Compare commits

...
107 Commits
Author SHA1 Message Date
sneak d886a9026f Merge branch 'main' into next
check / check (push) Successful in 4m8s
check / check (pull_request) Successful in 2m51s
2026-09-29 03:02:24 +02:00
clawbot 6e1f499048 Document that migrations are supported and none are added before 1.0 (closes #68)
check / check (push) Successful in 3m6s
check / check (pull_request) Successful in 3m8s
The docs now say vaultik supports migrations. The numbered files in `internal/database/schema/` are migrations: `schema_migrations` records which have run, and opening a database applies any that have not. None are added before 1.0 because nothing is installed anywhere yet, so a schema change edits `001.sql` directly. After 1.0 each change is a new numbered file, and an existing local database is migrated when vaultik is updated.

`docs/DATAMODEL.md` owns the explanation. The README caveat and roadmap entry and `AGENTS.md` policy 13 link to it. This replaces the wording from #146, which said there was no upgrade path.

Disclosure: `CLAUDE.md` line 33, the owner's file, changes from "do not need to support migrations" to "do not add migrations before 1.0".

Model: opus-5-5
2026-09-28 20:21:38 +02:00
sneak 15e6506e5d next: integrate accumulated work into main (#114)
check / check (push) Successful in 2m35s
Reviewed-on: #114
2026-09-23 13:03:02 +02:00
clawbot d24f5dc33c Adopt the canonical golangci-lint config (closes #90)
check / check (push) Successful in 3m9s
check / check (pull_request) Successful in 1m29s
The lint config is now the canonical file from `prompts`, which replaces the deprecated `gomodguard` with `gomodguard_v2`, so lint prints no deprecation warnings. It also turns on the `depguard` `test-support` rule. The one difference from canonical is that the deny list names vaultik's own test-only package `internal/storage/faultstore`, so shipped code cannot import it. The new config found nothing to fix in the source.

Issues and PRs that pin the old `.golangci.yml` sha256 as an untouched-file check need the new one: `7122fcf0dd0ea57441374f98ebd98bb3da23decb67f9209fee5175170838fbd1`.

Model: opus-5-5
2026-09-23 02:14:40 +02:00
clawbot eed117fe25 Route direct-stdout command output through internal/ui (closes #149)
check / check (pull_request) Successful in 1m27s
check / check (push) Successful in 3m18s
version, info, remote info, config, and database delete wrote plain text straight to stdout, so they were unstyled and ignored --quiet. Output is now governed by internal/ui in two buckets. Status lines and confirmations (config init, config set, the database delete prompt) go through the ui message methods and are silenced by --quiet. The data a command exists to produce is written plain -- the version/info/remote-info reports, the snapshot list table, config get values, and the --json documents -- and is never suppressed, since a script depends on it and a marker would corrupt a table or document. The database delete confirmation prompt is always shown. Pure-cli commands reach ui through a small commandUI helper.

Model: opus-4-8
2026-09-22 20:28:50 +02:00
clawbot dd7a610c23 Mark a snapshot complete only after its metadata export succeeds (closes #177)
check / check (pull_request) Successful in 1m42s
check / check (push) Successful in 3m35s
finalizeSnapshotMetadata marked the snapshot complete and then exported its metadata. A crash after completion but before/during the export left the local index showing the snapshot complete while the destination had no manifest or database, and PruneDatabase (which drops only NULL completed_at rows) kept it: a silently unrestorable snapshot.

Reorder so completion is recorded last. CompleteSnapshot is split into PopulateSnapshotBlobs (before the export) and MarkSnapshotComplete (after it). An interrupted export now leaves the snapshot incomplete, so the next run PruneDatabase drops it and re-backs-up the data; the reverse tiny window leaves a restorable snapshot the index reports honestly as remote-only. Update REPOSTRUCTURE.md guarantee 4 and the ARCHITECTURE.md flow. Add a fault-injection test driving the full create path.

Model: opus-4-8
2026-09-22 20:00:41 +02:00
clawbot 1548c0f933 Correct the security claims in docs and comments, and record the accepted risks (closes #171)
check / check (pull_request) Successful in 1m58s
check / check (push) Successful in 2m49s
Docs and comments only; no behaviour change. Corrects ten overclaims the security review found: snapshot names are hashed but the hash uses no secret, so a guessed hostname and name can be confirmed; a blob is named by hex(SHA256(SHA256(uncompressed contents))), stated once in docs/REPOSTRUCTURE.md and referenced elsewhere; double hashing does not hide known content (blob packing does); age uses ChaCha20-Poly1305, not XChaCha20; encryption is required, not optional; a snapshot is marked complete before its metadata is uploaded; the export comment now matches its only caller; deep verify detects corruption, not authorship; adding a recipient does not reach existing data; restore examples target a user-owned directory.

Adds an Accepted Risks subsection under Security Considerations with the seven documented risks, cross-referenced from the README.

Model: opus-4-8
2026-09-22 19:28:31 +02:00
clawbot c3bec7d3aa Reject a metadata database truncated to the age header and nonce (closes #152)
check / check (push) Successful in 2m5s
check / check (pull_request) Successful in 2m28s
An object holding just the age header and its 16-byte nonce decrypts without error: the truncated read surfaces as io.ErrUnexpectedEOF at the age layer, which the zstd decoder maps to a clean EOF at frame start. blobgen then reported zero bytes and no error, so a truncated stream was indistinguishable from a valid empty one, and the metadata database export slipped through -- restore built a fresh schema on the empty file and reported success.

blobgen.Reader.Read now, on EOF, reads once more from the age reader and surfaces io.ErrUnexpectedEOF unless that read is (0, io.EOF), the state a genuine end leaves. downloadSnapshotDB additionally rejects a zero-length decrypted database before any schema is built.

Model: opus-4-8
2026-09-22 18:11:57 +02:00
clawbot 1244c9e48d Bound download expansion and escape control chars on the terminal (closes #164)
check / check (pull_request) Successful in 1m47s
check / check (push) Successful in 3m11s
Objects fetched from the store are untrusted; several decode paths let one expand or print without limit.

- blobgen.LimitReader errors past a byte cap (not io.LimitReader silent EOF). DecodeManifest reads through caps on both compressed input and decompressed output, far above any real manifest, so json.Decode cannot buffer a compressible bomb. FetchAndDecryptBlob bounds decompression to the blob recorded uncompressed_size (not the restoring host blob_size_limit).
- downloadSnapshotDB streams straight from storage to its temp file with io.Copy, replacing two ReadAll calls that held the whole database twice.
- FetchBlob drops the per-blob Stat round-trip, its expectedSize parameter and returned size, all of which only fed a debug log.
- TTYHandler and ui.Writer escape control characters in messages, attribute keys/values, and rendered identifiers/paths before colour codes are applied, so a crafted value cannot drive the terminal.

Model: opus-4-8
2026-09-22 17:00:35 +02:00
clawbot 82c51a5337 Validate blob hashes, offsets and lengths from the destination (closes #155)
check / check (pull_request) Successful in 1m22s
check / check (push) Successful in 3m30s
A blob hash read back from the downloaded snapshot database or the store listing was trusted unchecked. A hostile remote could set a hash such as "aa/../../etc" and have a decrypted blob written outside the cache directory, or feed a short or negative value that panicked a command.

blobDiskCache.path now refuses any key with a path separator, and ReadAt rejects a negative offset or length, bounding so a sum cannot overflow past the check. A new isBlobHash helper gates FetchBlob, shallow and deep verify, and restore: buildBlobIndexes rejects every hash from the snapshot database before any fetch. The blobs/ and metadata/ listings skip a non-conforming name, and short-hash prefixes in log and error text go through a panic-safe shortHash helper.

Model: opus-4-8
2026-09-22 16:11:28 +02:00
clawbot d88ed64489 Parse the age identity key once and accept every identity in it (closes #165)
check / check (pull_request) Successful in 2m41s
check / check (push) Successful in 2m56s
Restore and verify --deep now parse the configured age secret key a single time through a new helper that uses age.ParseIdentities and hands every identity to age.Decrypt. A key file with several identities (a whole age-keygen file) is fully accepted, so a blob encrypted to any of its recipients decrypts, not just the first.

The helper is the first step of both commands, so a missing or unparseable key fails before anything is downloaded. Its error names the config source and never echoes the key value. config.extractAgeSecretKey and its silent fallback are removed; the key is stored raw and parsed only where decryption happens. README, the restore help, and the missing-key error now read the key from a file with \$(cat ...) rather than typed literally, keeping it out of shell history.

Model: opus-4-8
2026-09-22 15:45:27 +02:00
clawbot bd9656dbd4 Reject a decrypted snapshot database that is not the requested one (closes #156)
check / check (push) Successful in 1m26s
check / check (pull_request) Successful in 2m51s
Restore and deep verify downloaded and decrypted metadata/<key>/db.zst.age by object name alone. age decryption proves the database is readable, not that it is the snapshot that was asked for: an attacker who swaps in another valid db.zst.age could redirect the operation, and deep verify with a swapped database plus an empty manifest reported success with zero blobs verified.

After the database is opened, both paths now confirm its identity: an exported per-snapshot database holds one snapshot row, and a snapshot remote key derives from that row ID, so the database is the requested one exactly when its sole snapshot hashes back to the remote key fetched. The shared check lives in verifySnapshotDBIdentity, backed by a new SnapshotRepository.GetOnlySnapshot.

Model: opus-4-8
2026-09-22 15:01:02 +02:00
clawbot 7e611b95db Check blob sizes and the database in shallow verify (closes #169)
check / check (push) Successful in 1m21s
check / check (pull_request) Successful in 2m37s
Shallow snapshot verify only checked that each blob object existed and then reported "All blobs verified", overstating what it did.

It now compares each blob stored size against the manifest compressed_size, using the same comparison as the deep path, and checks that the snapshot encrypted database (db.zst.age) is present. A blob of the wrong size no longer counts as verified. The final line reports only what was checked: presence and size, not contents.

The README verify description and the CLI short/long text are corrected to match. Removed the now-unused resolveAndDownloadManifest helper and errBlobsMissing sentinel.

Model: opus-4-8
2026-09-22 14:28:44 +02:00
clawbot 4f27608560 Add negative and boundary tests for blobgen and types (closes #170)
check / check (push) Successful in 1m22s
check / check (pull_request) Successful in 1m18s
Test-only. internal/blobgen and internal/types had no negative or boundary coverage. Adds, in package blobgen_test: Writer-to-Reader round trips at the 64 KiB age-segment edges for random and compressible data, checking plaintext, byte counts and the reader/writer hashes by decrypting; a wrong-identity open; truncation and single-byte corruption of a multi-segment blob at every region; trailing bytes, empty input and garbage; rejected and accepted compression levels; nil, empty and invalid recipients; and a failing destination. In package types_test: Value/Scan round trips, NULL, wrong-type and malformed Scan, Parse and IsZero for FileID and BlobID.

The "cut right after the age header and nonce" truncation is excluded: it reads as valid and empty today and belongs to #152.

Model: opus-4-8
2026-09-22 14:28:34 +02:00
clawbot ae6aaaa388 Wait for the interrupted operation to clean up before exit (closes #159)
check / check (pull_request) Successful in 1m22s
check / check (push) Successful in 3m13s
On SIGINT/SIGTERM the process could exit before the interrupted command cleanup defers ran, leaving decrypted data in the temp directory (the blob cache and the decrypted snapshot database).

RunApp now mirrors fx run sequence: start, block on app.Wait(), then app.Stop(), returning only after Stop completes. fx delivers both an OS interrupt and the finished operation Shutdowner.Shutdown() on one channel. Stop runs the OnStop hooks; the operation hook cancels the command and waits for its goroutine to return (bounded by shutdownTimeout) before exit. The old code returned as soon as app.Done fired, without Stop, so a real interrupt unwound to os.Exit while cleanup still ran. Restore loops check the context between chunks and blobs so the wait ends promptly. A cli test drives RunApp through the OnStop hook.

Model: opus-4-8
2026-09-22 14:00:49 +02:00
clawbot f788668287 Remove the unused crypto path and write the blob-ID hash step once (closes #151)
check / check (push) Successful in 1m21s
check / check (pull_request) Successful in 2m39s
Production encryption and decryption already run through blobgen; the crypto package (Encryptor, Decryptor, UpdateRecipients, the fx Module) and Vaultik.GetEncryptor/GetDecryptor had no production caller. Delete crypto and route verify --deep through the same blobgen reader restore uses, parsing the age key once.

The second blob-ID hash step is now one exported blobgen.DoubleSHA256; Writer.Sum256 (the double hash) becomes Writer.ContentID so it no longer collides with Reader.Sum256 (the single plaintext hash). Also delete the never-adopted internal/types newtypes and the uncalled CleanupIncompleteSnapshots and its now-dead deleteSnapshot caller, and correct ARCHITECTURE.md. No production behavior changes.

Model: opus-4-8
2026-09-22 13:46:02 +02:00
clawbot 238ce3985f Scrub example config of real credentials and internal hosts (closes #172)
check / check (push) Successful in 1m20s
check / check (pull_request) Successful in 2m41s
config.example.yml carried a real-looking 20-char S3 access key id and 40-char secret, a private-address http:// endpoint, and a storage_url naming an internal rclone remote and pool path. Replace them with the same neutral placeholders the config init template uses: YOUR_ACCESS_KEY / YOUR_SECRET_KEY, a https://s3.example.com endpoint, a mybucket bucket, and rclone://myremote/path/to/backups. No behavior or other keys change.

The credentials live in the commented-out s3 block, which the loader never parses, so the new test reads the file raw text to assert the placeholders are present and no http:// endpoint remains, and also loads it to confirm the active storage_url still parses.

Model: opus-4-8
2026-09-22 13:45:52 +02:00
clawbot 548a7ae156 Give the local index and its export copy an explicit 0600 mode (closes #168)
check / check (pull_request) Successful in 1m23s
check / check (push) Successful in 2m57s
The local index lists every backed-up path and chunk hash, but its file mode was left to the SQLite driver and the umask, so under a typical 022 umask a fresh index (and its -wal/-shm side files) landed world-readable. The snapshot export copied the index to snapshot.db with a permissive create as well.

provideDatabase now calls ensureIndexFileMode before opening the driver: it creates the index 0600 if missing and chmods an existing one to 0600. Doing this before the driver opens the file matters because SQLite gives its -wal and -shm files the mode of the main database file. The export copy is now created 0600. Tests under umask 022 cover a fresh index, an existing 0644 index, and the export copy.

Model: opus-4-8
2026-09-22 13:12:07 +02:00
clawbot 3a58377127 Parse age_recipients at config load and never echo the entry (closes #153)
check / check (push) Successful in 1m23s
check / check (pull_request) Successful in 1m18s
Config.Validate now parses every age_recipients entry with age.ParseX25519Recipient, so a bad recipient fails at config load instead of deep in a backup after the snapshot row and tree walk. On failure the error names the position (age_recipients[N]) and never the value: a recipient string can itself be a secret key an operator pasted by mistake, and age's own error quotes its input. An entry starting with AGE-SECRET-KEY- gets a specific message.

The remaining parse sites (blobgen.NewWriter, crypto NewEncryptor and UpdateRecipients), reachable by callers that skip config.Load, likewise drop the value and age's wrapped error, naming only the position.

Model: opus-4-8
2026-09-22 13:01:00 +02:00
clawbot a6434de57f Open the downloaded snapshot database read-only, on a private temp dir (closes #162)
check / check (pull_request) Successful in 1m21s
check / check (push) Successful in 3m4s
Restore and deep verify used to open the decrypted snapshot database read-write through the local-index constructor, which applied migrations against whatever the file carried, and left the decrypted file in the shared temp directory. A forged file could redefine what restore queries return, and an interrupted open left decrypted metadata on disk.

Add database.OpenReadOnly: opens the file read-only (mode=ro) with query_only and trusted_schema=OFF, never applies schema files, and refuses a file whose schema carries a trigger, view or virtual table or lacks an expected table. Restore and deep verify now both use it, each inside its own private (0700) temp directory removed on every return path. pickNextDownload returns (FileID, bool) so a genuine nil-UUID file is not mistaken for "nothing left".

Model: opus-4-8
2026-09-22 12:45:53 +02:00
clawbot b4654f8e52 Abort the run when packing fails, even under --skip-errors (closes #161)
check / check (push) Successful in 1m22s
check / check (pull_request) Successful in 3m2s
A chunk is registered as pending (known, scanner-pending, packer pending-row) before it is packed. Under --skip-errors the scanner skipped a file on any processing error, including a failure inside addChunkToPacker (packing, database, encryption, upload). The pending chunk then stayed queued and a later blob finalize inserted it into the chunks table with no blob_chunks row, so a snapshot could complete holding a file whose chunk is in no blob and cannot be restored.

Errors from addChunkToPacker are now marked and abort the run regardless of --skip-errors; only open and read errors are skipped. The bookkeeping order is unchanged. Flag help and comments now say only unreadable files are skipped.

Model: opus-4-8
2026-09-22 12:28:44 +02:00
clawbot 39aef1c47c Stop config set echoing secrets; reject credential-bearing storage URLs (closes #166)
check / check (push) Successful in 1m21s
check / check (pull_request) Successful in 1m18s
config set now prints only the key name after a write, never the value: a value may be a secret such as s3.secret_access_key, and echoing it leaks into captured stdout and pasted terminals. The set logic moves into writeConfigSet so this is testable.

config set also tightens a pre-existing group- or world-readable config to 0600 after writing; the previous stat-and-preserve-mode block had no effect (os.WriteFile does not change an existing file mode) and is removed.

ParseStorageURL now rejects s3:// and rclone:// URLs that carry credentials in the userinfo or an unknown query parameter, naming s3.access_key_id and s3.secret_access_key as where credentials belong. On a url.Parse failure only the inner cause is wrapped, so the raw URL is not echoed. file:// is unchanged.

Model: opus-4-8
2026-09-22 12:28:32 +02:00
clawbot 96ebcd40d7 Reconcile purge against remote by hashed key, not human ID (closes #160)
check / check (pull_request) Successful in 1m20s
check / check (push) Successful in 2m51s
syncWithRemote compared human snapshot IDs against the hashed metadata/<key>/ directory names, which never match, so it deleted every local snapshot record; the purge that followed then found nothing to remove remotely. Reconcile via listAllRemoteSnapshotKeys and RemoteSnapshotKey(id), matching CleanupLocalSnapshots, so a row still backed by remote metadata is kept.

The purge tests only passed because their stubs used the human-ID layout production never writes; they now write metadata under the hashed remote key. New tests prove remotely-backed local rows survive the reconcile and that a purge removes the local row and remote metadata together.

Model: opus-4-8
2026-09-22 12:11:49 +02:00
clawbot d9f0220f94 Restore files at 0600 and make the blob hash check unskippable (closes #163)
check / check (pull_request) Successful in 1m21s
check / check (push) Successful in 2m42s
Regular files are now created with O_EXCL at mode 0600 and given their stored mode only after the content is written and closed, so a file whose stored mode is restrictive is never briefly readable by other local users mid-restore. A file whose write or close fails is removed rather than left partial, and a chmod failure is a user-visible warning instead of a debug line.

hashVerifyReader.Close now errors when closed before EOF, so a short read or early close can never obtain a blob whose hash was not verified; downloadBlobToCache drops the cache entry on any such failure.

verifyFile (--verify) now rejects a restored file with bytes past its last chunk. Tests cover each behaviour under umask 022.

Model: opus-4-8
2026-09-22 11:45:52 +02:00
clawbot 4c83e82543 Reject a blob_size_limit below the largest possible chunk (closes #167)
check / check (push) Successful in 1m20s
check / check (pull_request) Successful in 1m16s
Validate only rejected blob_size_limit below chunk_size, but the chunker can emit chunks up to chunk_size times the FastCDC size spread (four times), and the packer puts a single chunk of any size into an otherwise empty blob. A limit between one and four times chunk_size therefore let a blob reach four times the configured maximum, with most blobs holding a single chunk and so exposing individual chunk lengths to anyone who can list the destination.

Validate now rejects blob_size_limit below chunk_size times the spread, reusing the chunker's one constant (now exported as ChunkSizeSpread) instead of a second literal. The rule is stated in the error text, the Validate comment, the README config table, config.example.yml, and the generated config template.

Model: opus-4-8
2026-09-22 11:45:41 +02:00
clawbot 86361c8b50 Fail closed on unreadable manifests instead of losing blobs (closes #157)
check / check (pull_request) Successful in 1m20s
check / check (push) Successful in 2m42s
Prune learned which blobs are in use by reading every snapshot's manifest, but merely logged and skipped one it could not download or decode. Blobs referenced only by that snapshot then looked unreferenced and were deleted, with a zero exit -- and snapshot create --prune runs this unattended. collectReferencedBlobs now errors, naming the remote key, so prune deletes nothing and exits non-zero.

Manifest generation likewise skipped a blob whose lookup failed or was missing, yielding a manifest short of what the snapshot needs; it now fails. Deep verify only warned when the manifest omitted a database blob; it now fails on any divergence. Docs corrected.

Model: opus-4-8
2026-09-22 11:45:30 +02:00
clawbot d77663d039 Default a scheme-less s3.* endpoint to TLS (closes #158)
check / check (pull_request) Successful in 1m19s
check / check (push) Successful in 3m15s
With the s3.* config form and an endpoint written without a scheme, use_ssl being omitted built an http:// endpoint, while config.example.yml documented use_ssl as defaulting to true. Over plain HTTP a network observer sees manifests, object names, sizes and the access key id, and can alter responses.

use_ssl is now *bool: omitted (nil) means the default, TLS; only an explicit use_ssl: false forces plain HTTP. This matches the s3:// URL form, which already defaults to TLS. The config init template dropped its misleading use_ssl line from the s3:// block (that key is never read for URLs; ?ssl=false controls TLS there) and points at ?ssl=false instead.

Model: opus-4-8
2026-09-22 11:11:34 +02:00
clawbot 3abe9cbd9e Scope the PID lock to mutating commands (closes #150)
check / check (push) Successful in 1m19s
check / check (pull_request) Successful in 1m16s
RunWithApp took the process-wide PID lock for every fx-backed command, so read-only commands (info, snapshot list, snapshot verify, remote info) failed with "already running" while a backup held it.

AppOptions now carries a lockMode declared at each call site: only mutating commands (snapshot create, snapshot purge, snapshot remove, prune, remote nuke) acquire the lock; read-only ones run without it. snapshot restore is classified read-only -- it writes only to its target directory, not the local index or remote store. The decision moves to a small acquireLockIfMutating helper, with a test that a read-only command runs while the lock is held and two mutators still exclude. The README locking section is rewritten to match.

Model: opus-4-8
2026-09-22 11:01:29 +02:00
clawbot 76a6917a35 Keep restore writes inside the target directory (closes #154)
check / check (pull_request) Successful in 1m21s
check / check (push) Successful in 2m51s
restoreFile and verifyRestoredFiles joined the stored path onto the target with no containment check, so a ".." segment or an absolute path escaped the target, and a restored symlink could redirect a later child write anywhere on disk. Since age decryption proves a snapshot is readable but not honest, and restore usually runs as root, a forged snapshot became an arbitrary file write.

Both call sites now go through containedRestorePath: it rejects a stored path unless filepath.IsLocal accepts it with the leading separator removed (barring "..", absolute, and empty paths), then Lstats each existing ancestor below the target and refuses to descend through a symlink. The target directory itself may be a symlink, and honest symlinks pointing outside the tree are still written verbatim.

Model: opus-4-8
2026-09-22 11:01:01 +02:00
clawbot 38ebfd843a Trust only uploaded blobs for deduplication (closes #148)
check / check (pull_request) Successful in 1m23s
check / check (push) Successful in 3m10s
An interrupted blob upload left the blob's chunks, blob_chunks, and blobs rows committed before the upload was attempted, so a later run deduplicated against data that never reached storage and produced a snapshot that reported success but could not be restored.

Fix (issue option b): a chunk counts as known only when a blob holding it has uploaded_ts set, and each run drops un-uploaded blob rows and the chunks they orphan at startup, so the affected data is re-chunked and re-uploaded. A blob recorded with no remote backend is marked uploaded so the invariant holds uniformly.

The reproduction is the interrupted-upload test from #72: its t.Skip is removed and it passes against this fix, and this branch's earlier duplicate copy is dropped. The interrupted metadata-export case is split to #177.

Model: opus-4-8
2026-09-22 10:29:21 +02:00
clawbot 6b7517a4dc Add fault-injection tests for interruption and corruption (closes #72)
check / check (push) Successful in 1m22s
check / check (pull_request) Successful in 2m42s
Adds internal/storage/faultstore, a storage.Storer wrapper that injects faults through the storage seam without patching production code: an upload that dies mid-stream, a backend reporting success while storing nothing, and reads returning corrupt or truncated bytes. Covers all six scenarios from the issue, each asserting the observable end state (index, destination, and what the user is told), not merely that an error returned. Scenario 1b (retry after an interrupted upload) exposed a real dedup defect and is skipped with a pointer to #148, which also owns the half-exported-state repair. Tests run serially because each calls log.Initialize on the global logger. No production behavior changes.

Model: opus-4-8
2026-09-22 09:46:46 +02:00
clawbot 994e5de613 Quiet only the stdout UI under --json, not the log level (closes #112)
check / check (push) Successful in 2m23s
check / check (pull_request) Successful in 1m24s
Per the decision on the issue (option 1), --json no longer implies Quiet. Folding --json into Quiet pinned the stderr log level to WARN, so prune --json gave a machine consumer no record of the local index rows it deleted. The two effects are now split: a JSON field on log.Options drives only the stdout UI-quiet in setupGlobals, keeping the JSON document clean, while the stderr log level follows --verbose/--debug again (diagnostics have gone to stderr since #82). The same coupling is removed for snapshot verify, snapshot remove and remote info; snapshot list was already decoupled.

Model: opus-4-8
2026-09-22 09:05:52 +02:00
clawbot 42f4e648d7 Reconcile docs with code across the accuracy-sweep items (closes #74)
check / check (pull_request) Successful in 2m38s
check / check (push) Successful in 4m22s
Docs-only sweep of the accuracy items. Corrected ARCHITECTURE.md chunk sizes and the fx config type; documented the ls/rm aliases, the CPU/MEM profile env vars, the age_secret_key threat-model caveat, the four zstd presets, and a new locking section for the process-wide PID lock. Narrowed the internal/ui output claim to what holds today (refactor deferred to #149); lock-scoping deferred to #150. Added the missing ARCHITECTURE.md and config.example.yml README links. Every claim re-verified against the tree.

Disclosure: a pre-existing gomodguard linter deprecation surfaced during the gate; unrelated.

Model: opus-4-8
2026-09-22 09:05:36 +02:00
clawbot 343129f891 Accept a remote key for restore and verify, and document it (closes #124)
check / check (push) Failing after 1s
check / check (pull_request) Failing after 1s
A machine restoring after the original is gone has no local index and cannot know a snapshot's human ID; snapshot list shows such snapshots only by their remote key, but restore and verify accepted only the human ID, so recovery could not be done as documented.

Restore and verify now also accept a remote key, or an unambiguous leading part of it as snapshot list prints it, resolved against the store's metadata listing. Human IDs are never pure hex, which tells the two forms apart. Deep verify reads the single snapshot in the downloaded per-snapshot database. A new README section walks the recovery end to end; a test backs up, then lists, restores and deep-verifies with an empty index, another hostname and no age_recipients.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-22 01:01:25 +02:00
clawbot a50e3fa038 Add tests for internal/storage: URL parsing, backends, shared conformance suite (closes #66)
check / check (push) Failing after 1s
check / check (pull_request) Successful in 3m25s
internal/storage, the package that parses store URLs and selects the backend, had no tests.

Adds table-driven tests for URL parsing (each scheme, query parameters, malformed input, unknown scheme, backend type chosen); one shared conformance suite for the Storer interface, run against the file backend in a temp directory and the s3 backend on the in-process harness internal/s3 already uses, so a new backend inherits it; and rclone construction and argument tests using its in-process local backend. A comment records that rclone data operations need a configured remote and are not unit-tested. No production code changed and no defect surfaced.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-22 00:58:27 +02:00
clawbot 6fcd8e1668 Stamp Docker image version from the host; flush profiles on error exit (closes #75)
check / check (push) Failing after 1s
check / check (pull_request) Failing after 1s
Docker images reported commit unknown because the build ran git inside the container while .dockerignore excludes .git, and VERSION was never overridden. script/docker and script/cibuild now compute version, commit and date on the host and pass them as build args; the Dockerfile runs no git and falls back to dev and unknown, never empty, on a bare docker build.

Profiling a failing command gave a truncated or missing profile: Entry and each command goroutine called os.Exit(1), skipping the deferred profile writers in main. Entry now returns a status that main exits with after its defers run, and command goroutines report failure through one RunOperation helper, which also restores PID-lock release and graceful shutdown on failure.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)

Co-authored-by: clawbot <clawbot@noreply.example.org>
2026-09-21 22:01:05 +02:00
clawbot aab6a87f8c Reconcile the schema/migration docs with the code (closes #68)
check / check (pull_request) Failing after 1s
check / check (push) Successful in 2m46s
Four documents told different stories about the database schema. docs/DATAMODEL.md now owns the explanation and separates two things: the policy, which is unchanged (no supported upgrade path between versions; delete the local index with vaultik database delete and back up again), and the schema bootstrap that does exist (numbered files in internal/database/schema applied to a fresh database and recorded in schema_migrations).

README.md and AGENTS.md are reworded to match and link there. AGENTS.md names the real file to edit, internal/database/schema/001.sql, and notes that the pre-1.0 disposability clause expires on tagging. No code changed.

Judgement call: REPO_POLICIES.md still names a different schema file; it is cross-project policy and was left alone.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 21:58:37 +02:00
clawbot c355ef4d25 Report a prune count that could not be read as unknown, not 0 (closes #96)
check / check (pull_request) Failing after 1s
check / check (push) Successful in 2m46s
Prune read table row counts before and after to report how many orphaned files, chunks and blobs it removed, and discarded the error from every read. A failed query therefore reported as a count of 0, and the summary showed plausible wrong numbers.

A count that cannot be read is now logged as a warning (on stderr, also under --json) and shown as "unknown"; a difference computed from an unknown count is itself unknown. 0 still means the table was empty. No --json document carries these counts, so none can show a false 0.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 21:41:57 +02:00
clawbot 5927e1aa3d Write file:// blobs atomically via temp file and rename (closes #130)
check / check (push) Failing after 0s
check / check (pull_request) Failing after 0s
The file:// backend streamed each object straight to its final key, so an upload cut off mid-stream left a truncated object there. The next backup saw that Stat succeeded, recorded the blob as complete, and produced a snapshot that reported success but could not be restored.

Writes now go to a temporary file with a .partial suffix in the destination directory, are synced, then renamed onto the key. List and ListStream skip .partial files, so a leftover is never trusted as a blob and is overwritten when the key is written again. S3 PutObject is already atomic.

Disclosure: the containing directory is not synced after the rename, so a host crash right after it could still lose the object on some filesystems.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 21:24:42 +02:00
clawbot 9ca962969a Map s3 not-found to storage.ErrNotFound in Get and Stat (closes #129)
check / check (push) Failing after 1s
check / check (pull_request) Successful in 3m50s
The Storer interface documents that Get and Stat return storage.ErrNotFound for a missing object. The file and rclone backends did; the s3 backend returned the raw SDK error, so callers testing for ErrNotFound behaved differently on s3.

S3Storer.Get and Stat now wrap ErrNotFound when the SDK reports a missing object and leave every other error untouched. The SDK reports a missing key two ways (NoSuchKey from Get, NotFound from Head); both are recognised in one helper, s3.IsNotFound, which HeadObject now also uses. The mapping lives in the storage package because internal/s3 cannot import it.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 21:07:35 +02:00
clawbot 89ebfc78e2 Use one duration parser and fix the --older-than months example (closes #123)
check / check (push) Failing after 1s
check / check (pull_request) Failing after 1s
Two parseDuration functions existed with different grammars; only the one in internal/vaultik/helpers.go was reachable from a flag. The unused copy in internal/cli/duration.go is deleted, so no flag accepts anything it did not before.

The README gave 6m as the six-months example for snapshot purge --older-than, but m is minutes: that command removed every snapshot older than six minutes. The example is now 6mo, and the help for --older-than and --keep-newer-than states that m is minutes and mo is months.

The parser now rejects negative durations, which it used to accept or silently make positive.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)

Co-authored-by: clawbot <clawbot@noreply.example.org>
2026-09-21 20:58:30 +02:00
clawbot 07ef3a1c78 Drop the lint-guard shell scanner, keep the Dockerfile.lint checks (closes #121)
check / check (push) Failing after 0s
check / check (pull_request) Successful in 2m37s
The guard test in cmd/vaultik/lintdocker_test.go tried to prove that no script runs the linter outside the container by parsing shell scripts with a hand-written scanner. Four reviews each found another spelling it missed; such a parser cannot be complete, and nobody could follow it in one reading.

The scanner, its helpers and their tests are deleted. The plain Dockerfile.lint assertions stay: the linter image is pinned by digest, config verify runs before run, and the per-run value reaches both steps. TODO.md no longer claims a test proves the property; script/lint is the only lint entry point, and keeping it so is a review matter.

Judgement call: this drops a guard two reviewers asked to harden.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (decision, merge)
2026-09-21 20:41:33 +02:00
clawbot c423d13191 Hash the plaintext, not the encrypted bytes, in verify --deep (closes #131)
check / check (push) Failing after 0s
check / check (pull_request) Failing after 0s
The last step of verify --deep hashed the encrypted bytes it downloaded once with SHA256 and compared the result to the blob name. The name is the double SHA256 of the blob plaintext, so the two could never match and deep verification failed on every healthy blob with "blob hash mismatch".

It now hashes the decompressed plaintext as chunk verification streams it and compares the double SHA256 of that to the blob name, the same derivation the writer uses. A new test backs up a real snapshot, runs deep verify on it, then flips one byte in a stored blob and expects failure.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 20:24:37 +02:00
clawbot 75a10d3a22 Hash-verify the Go toolchain in the release workflow (closes #105)
check / check (push) Failing after 0s
check / check (pull_request) Successful in 4m13s
The release workflow installed Go with actions/setup-go, which pins the action but not the Go archive it downloads, so the compiler that builds the published binaries was verified against nothing in this repo.

New script/install-go, modelled on script/install-goreleaser, downloads the go.dev archive for the version in go.mod and refuses it unless its sha256 matches the value committed in the script. It fails if its version disagrees with go.mod, and on any OS or architecture other than the Linux release runners. GOTOOLCHAIN=local on the release step keeps the verified toolchain from switching itself.

Judgement call: release path only; script/bootstrap still uses the host Go.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 19:48:52 +02:00
clawbot 3d56dd7eb0 VACUUM snapshot metadata through the sqlite driver, not a CLI (closes #120)
check / check (push) Failing after 1s
check / check (pull_request) Successful in 2m57s
snapshot create compacted the metadata database by running a sqlite3 command-line binary, after every blob had already been uploaded. On a host without that binary, which includes anyone who installed with go install, the backup failed at the last step, and two tests failed the same way.

VACUUM now runs through the Go sqlite driver the program already uses, and its error is returned to the caller. The runtime Docker image no longer installs the sqlite package, since nothing in the binary calls it.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 19:41:28 +02:00
clawbot bdce350041 Delete dead code and stale fixtures, fix config set reindent (closes #70)
check / check (push) Failing after 1s
check / check (pull_request) Failing after 1s
Removes code and fixtures nothing uses: the internal/models package and its test, a second SnapshotInfo type in package cli that had no references (the live one is in internal/vaultik), and two config fixtures, test-config.yml and test/integration-config.yml, whose keys the config loader no longer accepts. test/config.yaml stays; a test uses it.

Also fixes config set, which rewrote the whole file with 4-space indentation on the first set despite the documented promise to preserve formatting. It now encodes with 2-space indent like the default template, and a test asserts comments and indentation survive a set.

Deviation: one commit, not one per deletion as the issue asked.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 19:25:01 +02:00
clawbot 753bc3ef60 Correct remote layout and privacy docs for hashed snapshot keys (closes #67)
check / check (pull_request) Failing after 1s
check / check (push) Successful in 3m11s
Three documents showed the remote layout with a plaintext snapshot ID as the metadata directory name, and docs/REPOSTRUCTURE.md blamed those IDs for the observable backup time. The store actually names each metadata directory with a one-way hash of the ID, so hostname and snapshot name are not visible; the backup time is, through the plaintext timestamp in the manifest, which is accepted behaviour.

README, ARCHITECTURE.md, docs/DATAMODEL.md and docs/REPOSTRUCTURE.md now show the hashed layout, the derivation is documented once, and the privacy section lists what the unencrypted manifest exposes. Two code comments that claimed the timestamp was hidden are corrected. No behaviour change.

Judgement call: docs/DATAMODEL.md was not named in the issue but had the same error.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 19:24:44 +02:00
clawbot d2a0510cb4 Trigger CI on next, not only main (closes #122)
check / check (pull_request) Failing after 0s
check / check (push) Successful in 3m33s
`check.yml` ran only on push to `main` and on pull requests against `main`. Every unit is a PR based on `next`, and `next` is pushed on each squash-merge, so no unit PR and no push to `next` ever ran CI; a broken `next` would first surface on the milestone PR. `next` is added to both branch lists; nothing else in the workflow changes. The README Entrypoints section now says where CI runs.

Disclosure: the CI run on the PR itself fired (the proof the trigger works) but was red because the runner had no disk space left before any check step ran; the local gate was green.

Model: opus-4-8 (implementation and review)
model: claude-fable-5
2026-09-21 14:55:59 +02:00
clawbot 583f65040a Correct --cron flag help to name warnings as unsuppressed (closes #87)
check / check (pull_request) Failing after 0s
`--cron` sets the UI quiet, but `Warningf` and `Errorf` are unconditional, and the snapshot summary is routed through `Warningf` on purpose so cron delivers something on a successful run. The help string said `silent unless error`, so a user could read normal cron output as a failure. It now says `silent unless warning or error`, matching the README. String only; no behavior change.

Model: opus-4-8 (implementation and review)
model: claude-fable-5
2026-09-21 14:55:45 +02:00
sneak d257f8f658 Lint in a container as a build step, via Dockerfile.lint (closes #113)
check / check (pull_request) Successful in 2m58s
Every lint run now happens inside its own container, invoked through
script/lint, and linting is a build step rather than a container
command: a successful build of the new root Dockerfile.lint IS a clean
lint. That shape also works where the docker daemon is remote and bind
mounts are impossible.

Its FROM line -- golangci/golangci-lint:v2.12.2, pinned by digest -- is
now the only pin of the linter version in this repo.

A container per run has its own lint cache and its own golangci-lint
lock, both discarded with it, so neither cross-worktree contamination
nor lock contention exists any more. The machinery that defended
against them is therefore gone: the per-worktree cache directories, the
lock-retry loop, and script/lint-audit, which existed to catch findings
replayed from a cache that no longer exists. So is the host lint path
in its entirety -- the native escape hatch, its version detection, and
VAULTIK_LINT_IN_CONTAINER in both script/lint and the Dockerfile.
Nothing lints on the host, at any version.

A cached build lints nothing, so the CHECK_EPOCH mechanism the product
Dockerfile already used is what makes a green mean something:
ARG CHECK_EPOCH with no default, placed below the module layers so
dependency caching survives, a `RUN [ -n "$CHECK_EPOCH" ] || exit 1`
guard so a build that withholds the arg fails instead of replaying, and
the value expanded into the lint command itself. script/lint computes
`epoch="$(date +%s%N)$$"` as a bare assignment on its own line, because
inline in the argument a failing substitution does not abort under
`set -eu` and yields a constant empty epoch -- which is exactly the
false green being prevented.

The product Dockerfile loses its lint stage rather than gaining a
second linter pin. That stage ran `make lint`, which is now
`docker build`: docker-in-docker inside a BuildKit step with no daemon.
Calling golangci-lint directly there instead would have meant two
independently bumpable digests for one tool. `make fmt-check` moves
beside `make test` in the builder stage, and script/cibuild now builds
Dockerfile.lint and then Dockerfile, each with its own fresh epoch,
failing on either. Consequence, stated in comments rather than left to
be discovered: script/docker builds the product image only and no
longer lints; script/check and script/cibuild are the gates.

`golangci-lint config verify` runs as its own epoch-keyed layer, above
the lint. It is not belt-and-braces: `golangci-lint run` rejects a
config it cannot PARSE but silently IGNORES an unknown top-level KEY.
Renaming .golangci.yml's `linters:` to `linterz:` -- one character --
discards `default: all`, the disable list and every threshold, leaves
only the small default linter set running, and exits 0 reporting
`0 issues.` on a tree the real config fails with an lll finding, in a
run whose lint layer demonstrably executed. That is a set-but-
ineffective config falling back to defaults instead of failing loudly,
sitting in the gate's own configuration. `config verify` catches it and
does so with the network genuinely off at this pin: under
`docker run --network none` against the pinned digest it exits 0 on
this repo's config and exits 3 on the `linterz:` variant. It is keyed
on CHECK_EPOCH like the lint itself, because a cached validation
validates nothing.

script/lint-fix is kept, reimplemented as a bind-mounted
docker run against the image parsed out of Dockerfile.lint -- a build
step cannot write fixes back to the worktree -- and its header states
outright that it is a developer convenience, never a gate, and needs a
local daemon.

cmd/vaultik/lintdocker_test.go parses both Dockerfiles and both scripts
and fails if any part of the mechanism is dropped: the digest pin, the
defaultless ARG below `go mod download`, the emptiness guard, the
expansion of the epoch into each check command, the bare per-invocation
epoch assignment in both scripts, cibuild building both files, the
config verification running before the lint, and -- structurally, not
by searching for one retired variable name -- that no script invokes
golangci-lint except through docker. Every one of those losses is
silent: the build still exits 0 and nothing is checked, which is why
they are asserted rather than trusted. The scanner behind the last of
those has its own test, because a structural check that goes blind
passes on every tree, including a broken one.

script/lint takes no arguments now, and says so instead of dropping
them: a build step has no command line to pass linter flags to.
2026-08-10 13:38:45 +00:00
clawbot 696ed9ab4d Gate prune's local-cleanup output on --json, and make make build build (closes #108)
check / check (push) Successful in 2m16s
Closes #110.

CleanupLocalSnapshots wrote three prose lines to stdout with no --json
awareness, covering every branch, so `vaultik prune --json | jq` failed
on any input. -q never helped either: printlnStdout and stdoutf write
straight to v.Stdout and never consult v.UI, which is what SetQuiet
affects. It now takes *PruneOptions, symmetric with its sibling phase
PruneBlobs, and gates all three writes.

Threading opts.JSON was chosen over moving the lines to log.Info,
because internal/log/log.go defaults the level to Warn: log.Info would
not have relocated them to stderr, it would have deleted them from a
plain `vaultik prune`, and "Removing stale local record" narrates the
deletion of local index rows. The stale-record count is deliberately not
added to PruneBlobsResult - every field there is blob-scoped and produced
by the phase that runs after this reconciliation, so adding it would
change a published --json schema as a side effect of a stream fix.

Note for anyone reading the --json contract: under --json the
stale-record removal now produces no signal in either stream. stdout is
correctly gated, stderr is level-pinned to Warn because --json sets
Quiet, and the count is not in the document. That is inherited behaviour
- PruneBlobs' own log.Info calls are equally invisible under --json - not
something this change introduced, and it is tracked separately.

make build exited 0 and produced nothing: .PHONY listed build with no
build: rule, and a phony target with no prerequisites and no recipe is
considered already satisfied, which turns what would be a hard error into
a silent success. In a repo where `make build` is the documented way to
build, a caller checking the exit code concluded the build worked. Now
`build: vaultik`, verified in both directions - a clean build produces
the binary, a deliberately broken one exits non-zero and produces none.

All 19 .PHONY names were audited; build was the only one lacking a rule.
TestPhonyTargetsAllHaveRules keeps that true for names added later, so
the class is closed rather than the instance.
2026-08-09 19:53:46 +02:00
clawbot f21e7c9e70 Suppress the startup banner for --json (closes #106)
check / check (push) Successful in 2m31s
The banner is printed to stdout before cobra parses, and
bannerSuppressedInArgs recognised only --quiet, -q and --cron. So every
--json document was preceded by two banner lines and a blank one, and
`vaultik snapshot list --json | jq` failed. Passing opts.JSON as
extraQuiet did not help: that calls UI.SetQuiet in an fx OnStart hook,
long after Entry has printed.

The raw-argv scan is extended rather than the banner moved after
parsing. root.go documents that the banner must survive cobra rejecting
its arguments and --help, and no single post-parse location covers those
paths. The subcommand-versus-persistent distinction does not decide it:
--cron is already in the suppression list and is itself subcommand-only,
existing on snapshot create alone, so this adds another instance of an
accepted imprecision rather than a new kind. The error directions are
asymmetric - a false positive loses a decorative banner, a false negative
corrupts a document - so the scan errs toward suppression, which is also
why --json=false suppresses, exactly as --quiet=false already does.

Four of the five --json commands now pipe into jq cleanly with no other
flags: snapshot list, snapshot verify, snapshot remove, remote info.
prune does not, because pruneLocalSnapshots writes three prose lines to
stdout with no --json awareness. That reproduces identically before this
change and -q never suppressed it either, since printlnStdout and
stdoutf bypass v.UI entirely. Tracked as #108.

Also fixed: TTYHandler's human-readable byte formatting did not survive
grouping, because the key check compared against the bare attribute name
and a grouped record presents it qualified. AGENTS.md policy 9 keyed the
log format on stdout's TTY-ness, which #82 made false by moving the
logger to stderr; it now names the log stream. Vaultik.Stderr keeps its
field with the comment amended to say outright that nothing writes to
it, and the dead listEnv.stderr is removed.
2026-08-09 19:18:36 +02:00
clawbot c16ef476a9 Log to stderr and stop discarding With attributes (closes #82)
check / check (push) Successful in 4m20s
Closes #97.

internal/log attached both handlers to os.Stdout, so any record that was
not suppressed landed in the middle of a --json document. WARN and ERROR
are never suppressed, so this was not hypothetical: a config file with
permissions looser than 0600 was enough to break
`vaultik snapshot list --json | jq`.

Both handlers now write to os.Stderr, and the TTY-vs-JSON format choice
tests os.Stderr rather than os.Stdout - the format has to follow the
stream the records land on, or a redirected stderr gets colorized
whenever stdout happens to be a terminal.

User-visible: --verbose and --debug output moves to stderr too, so
`vaultik snapshot list -v > out.txt` no longer captures diagnostics.
--quiet and --cron semantics are unchanged.

TTYHandler.WithAttrs and WithGroup discarded their arguments and returned
the receiver, while their doc comments claimed otherwise, so attributes
passed through the exported log.With vanished. The effect was
environment-dependent in the worst direction: handler choice is by
TTY-ness, so attributes disappeared on a terminal - where a developer is
debugging - and appeared correctly in CI. Both now return a new handler
with copied state rather than mutating the receiver, since slog permits a
handler to be shared and derived from concurrently. A test asserts the
TTY and JSON handlers emit the same attribute set, which is the test that
would have caught the original defect.

The local workaround in snapshot_list.go is removed now that the logger
no longer writes to stdout. The collect-then-emit machinery is kept, but
for a different reason than it was added: emitting from the fetch workers
would order warnings by network timing, whereas key-order emission after
group.Wait() is deterministic run to run.

Not yet complete: --json stdout still carries the startup banner, which
internal/cli/entry.go writes before cobra parses and which
bannerSuppressedInArgs does not recognise --json for. That is the
remaining stdout contamination path and is tracked in #106.
2026-08-09 18:43:55 +02:00
clawbot e3f407b440 Make the tagged-release path work on Gitea (closes #65)
check / check (push) Successful in 3m7s
No tag could be cut at all: .goreleaser.yaml had no gitea_urls block, so
goreleaser defaulted to the GitHub API, and the repo has zero tags.

.goreleaser.yaml now points at git.eeqj.de. Version derives from git via
a new script/version - exact tag with any leading v stripped, else
dev-<12-char sha>, with a -dirty suffix when tracked files are modified -
replacing the hardcoded 1.0.0-rc.1 that every local build was stamping
regardless of git state. A tag-triggered .gitea/workflows/release.yml
runs goreleaser with a scoped token (RELEASE_TOKEN); script/bootstrap
installs a sha256-verified goreleaser, and make release / release-snapshot
become script shims like every other target.

Two fabrications were removed rather than merely replaced. goreleaser's
snapshot.version_template was `{{ incpatch .Version }}-next`, which
invents a release number from the last tag - and with no tags, from
goreleaser's own fabricated v0.0.0. And internal/cli/version.go gated its
development-build notice on Version == "dev" exactly, so the moment
untagged builds carried a sha that notice would have gone silent and an
unreleased binary would have read as a release. Replaced with a tested
IsDevVersion predicate, and closed at both layers: the Makefile now
refuses to build when script/version yields nothing, and an empty version
counts as a development build - reachable today via
`docker build --build-arg VERSION=`.

The release workflow installs Go from a sha-pinned actions/setup-go
(v5.6.0) using go-version-file, so the compiler that produces released
binaries is pinned like every other external reference. Without it the
first tag push would either fail at goreleaser's before-hook or compile
the published artifacts with whatever unpinned Go the runner happened to
carry - the one unpinned thing in a release path that already refuses an
unpinned goreleaser.

Known gap: the Go tarball setup-go fetches is version-pinned but not
checksum-verified against a value in this repo, unlike the goreleaser
install and the Dockerfile digest.
2026-08-09 18:03:18 +02:00
clawbot b6e4a218a3 Isolate the lint cache and context-gate the native path (closes #99)
check / check (push) Successful in 2m23s
Closes #80.

script/lint pointed GOLANGCI_LINT_CACHE at a path shared by every
worktree of this repo. Two worktrees have identical Go file contents, so
their cache keys collided and one tree's stored findings replayed for
another, paths included - observed as 231 findings all citing another
session's worktree, with no parallel-runner message to signal it. The
failure is symmetric and only one direction is loud: a clean tree failed
by a dirty sibling gets investigated, a dirty tree passed by a clean
sibling does not.

The cache is now keyed per worktree on a digest of $ROOT, and remains
persistent. Independently of that, script/lint-audit inspects every run's
output and fails the run if any finding cites a path outside the tree
being linted. That guard is the load-bearing part: it converts a silent
unearned green into a hard error regardless of how the cache is keyed. It
is deliberately built so it can never certify a pass, only reject, so it
cannot itself become a gate that reports green.

The native path was gated on version equality alone, which admitted a
locally installed matching binary and bypassed the digest pin. It now
requires VAULTIK_LINT_IN_CONTAINER=1, set only by the Dockerfile lint
stage, in addition to version equality. /.dockerenv was rejected as the
signal because dockerd creates it for `docker run` but not reliably
during a BuildKit `docker build`, which is the case the exception exists
for. A version mismatch inside the container is now a hard error rather
than a fall-through.

This mattered more than the issue supposed: on this host a matching
golangci-lint exists on PATH, so script/lint was taking the native path
and linting against the global cache without ever running the pinned
image. That is the likely root of the observed contamination, and it is
closed here rather than mitigated.

The parallel-runner error is retried rather than reported. It is not a
lint result, and surfacing it as a non-zero exit is indistinguishable to
a caller from real findings; exhausted retries fail saying the tree was
never analysed. Note that a private cache alone does not remove lock
contention - measured with two concurrent runs using separate cache
directories.

script/bootstrap no longer reports success on a machine that cannot run
the gate: docker is now required by lint, check and precommit, so a
missing binary or unreachable daemon is a hard failure naming what will
not work.
2026-08-09 17:15:07 +02:00
clawbot c51f693527 Make the test gate unfakeable and stop test-integration lying (closes #93)
check / check (push) Successful in 3m42s
Closes #69.

script/test ran `go test` without -count=1, so Go's test cache satisfied
the gate without running anything: a repeat `make test` printed all 14 ok
lines in 0.42 seconds, every one marked (cached). Those lines count as ok
lines, so the evidence signal this repo relies on was forgeable. It sits
below the Docker layer cache - CHECK_EPOCH forces `RUN make test` to
re-execute, but a GOCACHE baked into an earlier image layer survives into
the re-executed step, so the step can run and still do no work.

-count=1 is applied unconditionally rather than only in the container,
because the pre-commit hook runs the same script and a gate honest only
in CI is dishonest where it is leaned on most. It costs about 11 seconds
on every repeat run, which is what it costs for a repeat run to mean
anything. test-coverage had the same omission and is fixed too; a
coverage profile assembled from cached results describes a run that did
not happen. Both invocations in script/test now share one run_tests
function so the quiet run and the verbose rerun cannot drift apart in
flags.

make test-integration passed -tags=integration while no file in the repo
carries any build tag, so it was an exact duplicate of make test. Removed
rather than given a tag scheme: the whole suite is 12s on the host, so
gating saves seconds in exchange for a mechanism whose failure mode is
"some tests silently stopped running" - a poor trade in a repo that has
found several ways for a gate to report an unearned green.

-timeout goes 30s to 120s. This DIVERGES from REPO_POLICIES.md:192, which
mandates 30s; the divergence is deliberate, recorded in script/test's
comment, and proposed upstream as #101. Measured worst case is 10.2s and
each fresh measurement has come in above the last, leaving 30s at 2.9x -
too thin for a loaded runner. A -timeout is a hang backstop, not a
performance budget.

Note for the record: cold-cache compilation is NOT charged against
-timeout. The flag reaches the test binary as -test.timeout and its clock
starts inside testing.M.Run, after compilation. Verified twice
independently - a run with an empty GOCACHE spent ~46s compiling and then
reported per-package durations within noise of warm. A shell
`timeout 30 go test ./...` does include compilation, but that is a
different mechanism.
2026-08-09 16:29:26 +02:00
clawbot 3f9c2e5033 Record the stale-branch triage and advance TODO.md (closes #71)
check / check (push) Successful in 2m5s
Twelve stale remote branches retired. Nine were ancestors of main with no
unique commits; golangci-v2.12.2 pointed at a tree byte-identical to the
one main's cc58583 already carries; fix/sync-snapshot-cleanup's one-line
change is present on main; and feature/restore-progress-bar was not only
superseded but regressive, since its diff deletes the #28 regression test
that landed separately.

Neither of the two branches that looked like unlanded correctness fixes
turned out to hold one, and in both cases main had moved past them by a
decision already recorded in the tracker.

fix/ctime-scanner-population would have restored a field that no longer
exists: ctime was removed outright by 1c72a37 (#54/#55). It never
participated in change detection either - the scanner compares size,
mtime, mode, uid and gid, exactly the five fields ARCHITECTURE.md
documents - so the feared dedup/data-loss failure was not reachable.

fix/sql-injection-whitelist would have reverted bfd7334, which replaced
a table-name allowlist with regex sanitization in response to review
feedback on PR #32, and would have broken main: its allowlist omits the
snapshots table, whose count main reads with the error discarded, so the
figure would silently have become zero. Exposure on main is nil -
getTableCount is unexported, every call site passes a literal, and the
sanitizing pattern admits no quote, space, semicolon or paren.

feature/daemon-mode is untouched pending the scope decision in #94.

The full disposition inventory, with the evidence for each branch, is
recorded on #71.
2026-08-09 10:23:30 +02:00
clawbot 50816b7415 Make a missing CHECK_EPOCH fail the build (closes #91)
check / check (push) Successful in 3m2s
PR #89 stopped script/cibuild replaying cached check layers, but left a
gap: a bare `docker build .` with no --build-arg still faked. An unset
ARG is an empty string, an empty string is a stable cache key, and the
check layers replay from it. That gap mattered because REPO_POLICIES.md
names `docker build .` verbatim as a command that must be green, so the
documented command was the one that lied.

Both check stages now carry `RUN [ -n "$CHECK_EPOCH" ] || exit 1`
immediately under their own ARG. Failed steps are never cached, so this
fails on every invocation rather than once - a bare build now stops with
a named error instead of reporting a green it did not earn. Each stage
needs its own guard because ARG scope is per-stage; a gate-carrying stage
without one is a silent hole if ordering ever changes.

The check RUNs now reference the value (`echo "check epoch: ${CHECK_EPOCH}"
&& make <target>`), so the cache miss is contractual rather than resting
on BuildKit's current treatment of unreferenced ARGs, and the epoch is
visible in the build log.

The epoch becomes "$(date +%s%N)$$" so concurrent invocations in the same
second cannot collide. busybox silently drops %N and exits 0, so $$ is
what makes it correct there. The bare-assignment form is retained
deliberately: inlining the substitution into --build-arg would, under
set -eu, yield an empty and therefore constant epoch without aborting.

script/docker gets the same treatment - it is not the gate, but two
entrypoints disagreeing about whether the tree is green is its own
hazard, and local builds are almost always warm.

Verified by negative control rather than inspection: a bare build fails
twice consecutively here and succeeds twice on the parent commit, so the
change is demonstrably not a no-op. The builder-stage guard was fired
directly with a targeted probe build, since the lint stage otherwise
fails first and would leave it unexercised.
2026-08-09 10:09:27 +02:00
clawbot c3bb3b5580 Make script/cibuild unable to report an unearned green (closes #85)
check / check (push) Successful in 3m13s
script/cibuild was a bare `docker build .`. On an unchanged tree Docker
served the check RUN layers from cache, so make fmt-check, make lint and
make test never executed - and the build still exited 0. Measured at
221ms with zero ok lines and every check layer CACHED, against 162s for a
real run. CI showed the same signature: 6 second "successes" on main.

An ARG CHECK_EPOCH now sits immediately above the check RUNs in both
stages - each stage declares its own, since ARG scope is per-stage - and
script/cibuild passes a fresh value per invocation. Dependency and module
layers sit above the ARG and still cache, so this does not make every
build cold.

The epoch is assigned before the build rather than inlined into the
--build-arg. Under `set -eu` a command substitution that fails inside an
argument does not abort the script: CHECK_EPOCH would become an empty
string, an empty string is a constant, a constant CHECK_EPOCH restores
the cached false green, and the guard would silently disarm itself while
still exiting 0. As a bare assignment, set -e catches a failing date and
no build starts.

The README and Dockerfile state the guarantee conditionally. It holds per
build context and CHECK_EPOCH value, and depends on script/cibuild
passing a fresh one - a bare `docker build .` with no --build-arg still
replays the check layers from the second consecutive run onward. That
residual gap is tracked in #91 along with the remaining upstream
hardening.

Verification is recorded once, in the PR's verification comment, rather
than restated with differing numbers in three places.
2026-08-09 09:37:55 +02:00
clawbot 3bcdbcfd83 Correct what --cron actually suppresses (closes #84)
check / check (push) Successful in 6s
The Vaultik.UI doc comment claimed the cli layer replaces the writer with
a discarding writer in --cron mode. It does not. UI is built once as
ui.New(os.Stdout) and never reassigned; internal/cli/app.go calls
UI.SetQuiet(true) when --cron or --quiet is set, which drops Begin,
Complete, Info, Notice, Detail, Progress and Banner - but Warningf and
Errorf have no quiet check and are still emitted.

That distinction matters: the end-of-run summary is deliberately routed
through UI.Warningf so cron delivers something, so a reader who believed
the comment would have concluded the opposite of how the code is meant to
work.

The README's --cron description carried the same imprecision ("Silent
unless error") and is corrected alongside it.

Comment and documentation only - the Go diff contains no non-comment
lines, so there is no behavior change.
2026-08-09 07:43:45 +02:00
clawbot 50e20b460e List remote snapshots without requiring the private key (closes #64)
check / check (push) Successful in 6s
ListSnapshots built its table entirely from the local SQLite index. The
only remote access, reportRemoteDrift, was gated on AgeSecretKey != "",
so on a correctly configured host - which by design holds no private key
- snapshot list never contacted the destination store at all. A user who
lost their local index could not see their own backups, and the
"<remote only>" cell the README documents was unreachable dead code.

The listing is now the union of the local index and the destination
store, with no age_secret_key gate. Remote-only snapshots cannot have
their hostname or name recovered - RemoteSnapshotKey is one-way and the
manifest stores the hash - so they are listed by abbreviated remote key
with the real timestamp and compressed size from the manifest, and
"<remote only>" in the two columns that require the local index. Nothing
new is written to remote storage and the human ID is never fabricated.

An unreachable destination degrades to local-only with a warning and a
zero exit code. remote_present is null rather than false in that case,
so "absent" and "unknown" stay distinguishable and no drift is claimed
from a listing that never happened.

Also:

- Snapshot timestamps are normalised to UTC in scanSnapshotRows, the
  single point where they enter the domain. Previously one of three
  scanners omitted .UTC(), so on a non-UTC host the same snapshot
  rendered a different time depending on whether it was locally tracked.
- The 1000-row cap and the unreadable-manifest count are reported in
  --json mode as well as table mode, so machine consumers cannot be
  silently truncated. The JSON shape is unchanged.
- Warnings raised while listing are routed to stderr rather than the
  logger, which writes to stdout and would corrupt the JSON document.
  This is a local workaround for the logger bug tracked in #82 and
  should be removed when that lands.
- downloadManifestByKey is now the only remote manifest reader, so the
  manifest privacy question in #81 has a single call site to change.
- The orphaned "vaultik snapshot cleanup" hint now names vaultik prune;
  that command was folded into prune by the 2026-07-02 consolidation.
2026-08-09 07:34:15 +02:00
clawbot af607e3597 Run the linter at the pinned version locally too (closes #78)
check / check (push) Successful in 6s
script/lint ran bare golangci-lint from PATH while CI and the Dockerfile
pinned v2.12.2 by digest, so make lint and CI could disagree about
findings. That drift ran both directions: it produced two false green
claims during the lint remediation, and on an ambient 2.10.1 it also
reported four gosec findings on a tree CI linted clean.

script/lint now extracts the image reference - tag and digest - from the
Dockerfile lint stage FROM line and runs that exact image under docker.
The Dockerfile FROM line is the single source of truth for the linter
version; the duplicate pins in the Makefile deps target and in
script/bootstrap are removed rather than kept in sync.

A golangci-lint on PATH is used only when its version exactly equals the
pin, which is what makes the in-container lint stage work (the Dockerfile
runs make lint inside the pinned image, where there is no docker daemon).
Any other version, or none, goes through docker. When docker is
unavailable the script fails with an actionable message and never falls
back to a different linter version.

script/lint-fix delegates to script/lint --fix so autofixes come from the
pinned linter too. The container mounts persistent build and module
caches and runs as the invoking uid/gid.

Verified by reinstating the four historical nolint directives that 2.10.1
requires and 2.12.2 reports as unused: the old script passed on that tree
and the new one fails with four nolintlint findings.
2026-08-09 04:52:22 +02:00
clawbot e496aa334b Finish the lint remediation: script/cibuild exits 0 (closes #61)
check / check (push) Successful in 5s
Clears the final 80 golangci-lint findings under the canonical
.golangci.yml (sha256 021cc83f4e6fc7c31b95b34b846723dfcf20b66b7baeea1dc40406e643346bcb),
taking the repo from red to green: script/cibuild exits 0.

- wsl_v5 (60): blank line above defer/go statements sharing no variable
  with the line above; blank-line-only diff.
- sqlclosecheck (10): the package-local CloseRows helper hid the close
  from the analyzer. Helper removed; all 18 call sites now defer an
  inline rows.Close(), preserving the fatal-on-close-error path. No
  resource leak existed - the rows were always being closed.
- prealloc (3): append targets given a starting capacity.
- revive (3): package-name findings suppressed with per-site directives
  pending the naming decision tracked in #76.

No gosec suppressions are needed under the pinned linter. .golangci.yml,
Dockerfile, Makefile, .gitea/ and script/ are byte-identical to main.

Verified with script/cibuild (digest-pinned golangci-lint v2.12.2), not
make check - the latter resolves the linter from PATH and is not a
trustworthy gate here; see #78.

Closes #59.
2026-08-09 04:25:11 +02:00
clawbotandsneak cc58583130 Update golangci-lint to v2.12.2 with canonical config (#62)
check / check (push) Successful in 5s
Updates golangci-lint to v2.12.2 everywhere it is pinned and installs the canonical `.golangci.yml`, then remediates every finding the new linter/config surfaces so `make check` is green.

## Version bump

- `Dockerfile` lint stage: `golangci/golangci-lint:v2.11.3-alpine` -> `v2.12.2-alpine` (digest-pinned, date comment updated)
- `Makefile` `deps` target: `go install` moved from the old v1 module path at `@latest` to the pinned `github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2`
- `.golangci.yml` replaced with the canonical config (v2 schema; settings under `linters.settings` so the thresholds actually apply; `default: all` with the standard six disables)
- `script/bootstrap` installs golangci-lint via the system package manager and carries no version pin, so it is unchanged
- CI (`.gitea/workflows/check.yml`) only runs `script/cibuild`, so it needed no change

## Lint remediation

The canonical config surfaced ~3,300 findings across 56k lines. All are fixed, behavior-preserving; incorporates and supersedes the per-package mechanical passes already merged to `main` (refs #61). Highlights:

- `err113`: dynamic errors replaced with package sentinels + `%w` wrapping; comparisons via `errors.Is`
- `goprintffuncname`: printf-style helpers renamed with an `f` suffix (`ui.Writer` message methods, `cli.ReportErrorf`, `database.Fatalf`) and all call sites updated
- `revive` stutter renames: `blob.Handler`, `blob.WithReader`, `blob.ChunkPosition`, `storage.URL`, `storage.Info`; missing doc comments added
- `contextcheck`/`noctx`: `context.Context` threaded through `blob.Packer` and the scanner call sites; context-aware `exec`/`sql` variants
- `funlen`/`cyclop`/`gocognit`/`dupl`: oversized and duplicated functions split into focused helpers (production and test code)
- tests: `t.Parallel()` added where safe (global logger init kept in the serial phase for `-race`), `t.TempDir()`/`t.Helper()` adopted, several suites converted to external test packages
- `gosec`: bounded integer conversions, `ReadHeaderTimeout` on the test HTTP server; remaining warnings suppressed per-site with justifications
- remaining `nolint` directives are rare, targeted, and each carries a reason (e.g. `nilnil` not-found contract in the repository layer, fx module globals, on-disk snake_case struct tags)
- removed the deprecated `log.LogOptions` alias (callers migrated to `log.Options`)

`make check` (tests with `-race`, lint, fmt-check) passes.

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #62
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-08-07 23:22:48 +02:00
sneak b87b72d4b9 Update TODO.md: lint remediation chunk 1 complete (refs #61)
check / check (push) Failing after 52s
2026-08-07 17:00:13 +00:00
sneak e7b49d58ab Fix noinlineerr findings: internal/vaultik (refs #61) 2026-08-07 16:59:56 +00:00
sneak 919229f224 Fix noinlineerr findings: internal/storage (refs #61) 2026-08-07 16:59:56 +00:00
sneak 68cffba35d Fix noinlineerr findings: internal/snapshot (refs #61) 2026-08-07 16:59:56 +00:00
sneak 26cbb63749 Fix noinlineerr findings: internal/pidlock (refs #61) 2026-08-07 16:59:56 +00:00
sneak bf1d3c6bad Fix noinlineerr findings: internal/database (refs #61) 2026-08-07 16:59:56 +00:00
sneak dca3c50cd2 Fix noinlineerr findings: internal/crypto (refs #61) 2026-08-07 16:59:56 +00:00
sneak 1ee0d291ca Fix noinlineerr findings: internal/config (refs #61) 2026-08-07 16:59:56 +00:00
sneak 2f7e37153c Fix noinlineerr findings: internal/cli (refs #61) 2026-08-07 16:59:56 +00:00
sneak 26909b058a Fix noinlineerr findings: internal/chunker (refs #61) 2026-08-07 16:59:56 +00:00
sneak 2dfdc5f095 Fix noinlineerr findings: internal/blobgen (refs #61) 2026-08-07 16:59:56 +00:00
sneak 217d60eeaa Fix noinlineerr findings: internal/blob (refs #61) 2026-08-07 16:59:56 +00:00
sneak f7ba056814 Fix noinlineerr findings: cmd/vaultik (refs #61) 2026-08-07 16:59:56 +00:00
sneak 82eb352eb5 Apply linter autofixes: internal/vaultik (refs #61) 2026-08-07 16:53:23 +00:00
sneak 0296e26210 Apply linter autofixes: internal/storage, types, ui (refs #61) 2026-08-07 16:53:23 +00:00
sneak 1e05fa0dd7 Apply linter autofixes: internal/snapshot (refs #61) 2026-08-07 16:53:23 +00:00
sneak bec964fc20 Apply linter autofixes: internal/globals, log, models, pidlock, s3 (refs #61) 2026-08-07 16:53:23 +00:00
sneak b1451bb17e Apply linter autofixes: internal/database (refs #61) 2026-08-07 16:53:19 +00:00
sneak ee83f50281 Apply linter autofixes: internal/crypto (refs #61) 2026-08-07 16:53:19 +00:00
sneak 76f79af733 Apply linter autofixes: internal/config (refs #61) 2026-08-07 16:53:19 +00:00
sneak 070a8a5447 Apply linter autofixes: internal/cli (refs #61) 2026-08-07 16:53:19 +00:00
sneak 40516d1263 Apply linter autofixes: internal/chunker (refs #61) 2026-08-07 16:53:19 +00:00
sneak a66e1f9844 Apply linter autofixes: internal/blobgen (refs #61) 2026-08-07 16:53:19 +00:00
sneak 5e4df7d04f Apply linter autofixes: internal/blob (refs #61) 2026-08-07 16:53:19 +00:00
sneak d34868a9c4 Apply linter autofixes: cmd/vaultik (refs #61) 2026-08-07 16:53:19 +00:00
sneak 04fce150bc Add script/lint-fix entrypoint and make lint-fix shim (refs #61) 2026-08-07 16:40:59 +00:00
sneak 23d22a0f19 Add standard .golangci.yml (refs #59)
check / check (push) Failing after 49s
Copied byte-for-byte from the vendored policy set in sneak/prompts.
The new config surfaces 2,990 lint findings; remediation is tracked
in issue #61 rather than being bundled here, so #59 stays open until
make check is green under this config.
2026-08-07 16:29:49 +00:00
sneak c9c72ef29d script/bootstrap: install sqlite3, which the test suite shells out to 2026-08-07 16:29:45 +00:00
clawbotandsneak 928c389a5a Add .editorconfig (refs #59) (#60)
check / check (push) Successful in 4s
Adds `.editorconfig`, copied byte-for-byte from `sneak/dnswatcher` (blob `2fe0ce0`). This is the small, safe half of #59; the `.golangci.yml` half is deferred — adopting the org-standard config surfaces ~2,990 lint findings on vaultik and needs a separate lint-cleanup decision (see #59). Hence `refs #59`, not `closes`.

`make check` and `docker build .` are green (a static config file does not affect them). Left open for review (not merged).

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #60
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-08-07 17:46:18 +02:00
sneak 2aaeeb4966 Refresh vendored REPO_POLICIES.md
check / check (push) Successful in 4s
2026-07-07 01:53:18 +02:00
sneak 43346e62db Adopt scripts-to-rule-them-all: script/ entrypoints, Makefile shims 2026-07-07 01:53:18 +02:00
sneak df975bb8f0 Add standard Workflow section to TODO.md
check / check (push) Successful in 6s
2026-07-06 21:06:38 +02:00
sneak fc56b0cb30 Add TODO.md
check / check (push) Successful in 6s
2026-07-06 20:35:44 +02:00
sneak 1f32820607 Merge branch 'refactor/cli-verb-consolidation'
check / check (push) Successful in 1m59s
2026-07-02 16:42:24 +02:00
sneak 34a4d163f2 Consolidate CLI verbs; retire overlapping commands
The verb surface accumulated overlapping cleanup commands. Consolidate
so each cleanup verb has one meaning:

- Rename 'database purge' -> 'database delete'. The command removes the
  SQLite file entirely; "purge" wrongly suggested purging contents.
- Fold 'snapshot cleanup' into 'prune'. Prune now runs three passes:
  reconcile local snapshots against the remote (previously the
  standalone cleanup command), drop orphaned local rows, then delete
  unreferenced remote blobs. One command, one mental model.
- Delete 'store info'. Its output was a strict subset of 'remote info',
  which already prints storage type + location. Any user reaching for
  either should reach for 'remote info'.
- Drop 'snapshot remove --all'. It duplicated 'remote nuke --force'.
  'remote nuke' is the single supported entry point for wiping the
  destination store.

Also update the storage-binding error message to reference the new
'vaultik database delete' name.
2026-07-02 16:42:20 +02:00
sneak 9497a31d0f Merge branch 'fix/bind-local-index-to-storage-url'
check / check (push) Successful in 2m18s
2026-07-02 16:35:44 +02:00
sneak d330f9f031 Bind the local index to its backup destination
The local index tracks which chunks and blobs already exist at the
backup destination. Nothing was recording *which* destination, so
changing storage_url and running a backup left the scanner treating
every already-seen chunk as still-present at the new (empty) location.
Uploads were skipped silently and the resulting snapshots pointed at
blobs that don't exist at the new destination.

Fix: record storage_url in a new local_meta key-value table on first
mutating command, and refuse to proceed when the configured URL later
differs from the stored one. The error explains the two recovery
paths (revert the config, or run 'vaultik database purge' to discard
the index and rebuild from a fresh full backup).

Wired into snapshot create / prune / snapshot remove / snapshot purge
/ snapshot cleanup. Read-only inspection commands (snapshot list,
remote info, store info) are exempt.
2026-07-02 16:35:41 +02:00
sneak fda6d7a7eb Merge branch 'fix/restore-skip-chown-non-root'
check / check (push) Successful in 2m0s
2026-06-28 07:48:32 +02:00
sneak 1a97a80a81 Skip chown when restore runs as non-root; warn at end
chown(2) requires root on every Unix-ish kernel. Restoring 39k files
as a non-root user produces 39k EPERM syscalls plus 39k matching debug
log lines, all for an operation that can't possibly succeed. Skip the
syscall entirely when euid != 0, and emit one warning at the end of the
restore so the user knows the on-disk UID/GID will reflect the running
user rather than the original owner.
2026-06-28 07:48:28 +02:00
sneak d492f34e8d Merge branch 'fix/snapshot-rm-no-auto-prune'
check / check (push) Successful in 1m54s
2026-06-28 06:20:13 +02:00
sneak c17426b556 snapshot rm: remove metadata only, print prune command for blobs
The previous change had snapshot rm auto-prune unreferenced blobs. The
correct division of labor is: rm removes a snapshot (local DB + remote
metadata), prune cleans up blobs. Reverting the auto-prune means rm
stays a cheap, deterministic operation: it touches one snapshot's worth
of state and emits the exact 'vaultik prune' command the user should
run next to delete blobs no longer referenced by any remote manifest.

This is correct because prune must consult every remote manifest
(including snapshots this host doesn't know about) to determine which
blobs are still referenced, and folding that work into rm would
silently turn rm into an expensive O(remote snapshots) operation that
also assumes the remote is fully reachable.
2026-06-28 06:20:09 +02:00
sneak c06d53ad45 Merge branch 'feature/snapshot-rm-removes-unique-blobs'
check / check (push) Successful in 2m6s
2026-06-28 06:10:29 +02:00
sneak b39d765374 Make snapshot rm clean up the remote by default
snapshot rm <id> now does the full cleanup: removes the local index
entry, strips the snapshot's metadata from the destination store, and
prunes any blobs that were only referenced by the just-removed manifest.
The --remote flag is retired; --local-only opts out for the rare case
where the user wants to forget a snapshot locally without touching the
remote.

If the destination store is unreachable, the local-DB removal still
completes and a warning is emitted; the user can rerun 'vaultik prune'
to retry the remote half later.

RemoveAllSnapshots gets the same treatment: after deleting every
snapshot's metadata (local + remote + orphan keys), an automatic blob
prune sweep removes the now-unreferenced blob set.
2026-06-28 06:10:26 +02:00
228 changed files with 26458 additions and 8037 deletions
+2
View File
@@ -3,6 +3,8 @@
*.md
LICENSE
vaultik
dist
.tool
coverage.out
coverage.html
.DS_Store
+12
View File
@@ -0,0 +1,12 @@
root = true
[*]
indent_style = space
indent_size = 4
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
[Makefile]
indent_style = tab
+3 -3
View File
@@ -1,9 +1,9 @@
name: check
on:
push:
branches: [main]
branches: [main, next]
pull_request:
branches: [main]
branches: [main, next]
jobs:
check:
runs-on: ubuntu-latest
@@ -11,4 +11,4 @@ jobs:
# actions/checkout v4, 2024-09-16
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Build and check
run: docker build .
run: script/cibuild
+53
View File
@@ -0,0 +1,53 @@
name: release
on:
push:
tags: ["v*"]
jobs:
release:
runs-on: ubuntu-latest
steps:
# actions/checkout v4, 2024-09-16
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
# goreleaser needs the tags and the full history: the version
# it stamps comes from the tag, and the changelog comes from
# the commits since the previous one. A shallow checkout
# silently produces a mislabelled release.
fetch-depth: 0
# goreleaser is not a compiler: it shells out to `go` for the
# `before:` hook and for every one of the four cross-compiles.
# Nothing else in this repo puts a Go toolchain on the runner --
# check.yml runs script/cibuild, which does all of its work inside
# the digest-pinned Dockerfile images -- so without this step the
# release either fails at the before-hook or, worse, ships binaries
# built by whatever Go the runner happens to carry.
#
# actions/setup-go would pin the action by commit sha, but the Go
# tarball it downloads at runtime is verified against no value in
# this repo, and the action exposes no checksum input.
# REPO_POLICIES.md requires every external reference to be pinned
# by hash with no exceptions, and this is the compiler that
# produces the published binaries -- the input where a substituted
# artifact matters most. So Go is installed the way goreleaser is:
# script/install-go downloads the exact archive for go.mod's `go`
# directive and refuses it unless its sha256 matches the value
# committed in the script, then puts .tool/go/bin on PATH for the
# steps below.
- name: Install Go
run: script/install-go
- name: Install goreleaser
run: script/install-goreleaser
- name: Release
run: script/release
env:
# RELEASE_TOKEN is a repository Actions secret: a Gitea access
# token with write access to this repository's releases (scope
# write:repository), owned by an account that can publish here.
# It is deliberately not the runner's automatic token, which is
# not guaranteed to carry that scope.
GITEA_TOKEN: ${{ secrets.RELEASE_TOKEN }}
# Build with the toolchain install-go just verified, never a
# different one auto-downloaded from a `toolchain` directive:
# the point of the hash pin is that this exact compiler makes
# the release.
GOTOOLCHAIN: local
+6
View File
@@ -1,6 +1,12 @@
# Binary
/vaultik
# goreleaser output
/dist/
# Locally installed pinned tools (script/install-goreleaser)
/.tool/
# Test artifacts
*.out
*.test
+102
View File
@@ -0,0 +1,102 @@
version: "2"
# Config schema uses the golangci-lint v2 layout (settings live under
# linters.settings, not top-level linters-settings) so that the
# thresholds below are actually applied by golangci-lint >= v2.
run:
timeout: 5m
modules-download-mode: readonly
linters:
default: all
enable:
# Successor to the deprecated gomodguard. Named explicitly, rather than
# left to `default: all`, because it carries the module policy below.
- gomodguard_v2
disable:
# Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields
- godot # Requires comments to end with periods
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
# Deprecated: the warning is attached to the old name, so it is
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
settings:
lll:
line-length: 88
funlen:
lines: 80
statements: 50
cyclop:
max-complexity: 15
dupl:
threshold: 100
depguard:
# Test-support code must not be compiled into the shipped binary. A
# test-support package exists to hand a test privileges the program
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name
# ends in `test`, are where that code belongs, and are exempt.
#
# The deny list below is the one part of this file a repository is
# expected to extend, and the only part it may. depguard matches an
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support
# packages have to be named here one at a time, by full import path,
# under a module path that differs from repository to repository. Add
# them; change nothing else.
rules:
test-support:
list-mode: lax
files:
- "$all"
- "!$test"
- "!**/*test/**"
deny:
- pkg: net/http/httptest
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
- pkg: sneak.berlin/go/vaultik/internal/storage/faultstore
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
# Only decisions already recorded in the Go package defaults are
# listed here. Every entry matches the module path exactly.
gomodguard_v2:
blocked:
- module: github.com/rs/zerolog
recommendations:
- log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues:
max-issues-per-linter: 0
max-same-issues: 0
+14 -1
View File
@@ -2,6 +2,13 @@ version: 2
project_name: vaultik
# This repo lives on Gitea, not GitHub. Without this block goreleaser
# talks to the GitHub API by default and a `goreleaser release` either
# fails outright or publishes somewhere nobody is looking.
gitea_urls:
api: https://git.eeqj.de/api/v1
download: https://git.eeqj.de
before:
hooks:
- go mod tidy
@@ -37,8 +44,14 @@ checksum:
name_template: "checksums.txt"
algorithm: sha256
# A snapshot is not a release and must not name itself like one. The
# previous `{{ incpatch .Version }}-next` derived a plausible-looking
# release number from the last tag -- and with no tags in the repo at
# all, from goreleaser's fabricated v0.0.0. This produces the same
# string script/version produces for an untagged build, so a snapshot
# binary and a `make vaultik` binary of the same clean commit agree.
snapshot:
version_template: "{{ incpatch .Version }}-next"
version_template: "dev-{{ slice .FullCommit 0 12 }}"
changelog:
sort: asc
+10 -7
View File
@@ -83,8 +83,8 @@ Version: 2025-06-08
possible to mock or stub these side-effects in tests.
9. Always use structured logging. Log any relevant state/context with the
messages (but do not log secrets). If stdout is not a terminal, output
the structured logs in jsonl format.
messages (but do not log secrets). If the log stream is not a terminal,
output the structured logs in jsonl format.
10. Avoid using bare strings or numbers in code, especially if they appear
anywhere more than once. Always define a constant (usually at the top
@@ -102,9 +102,12 @@ Version: 2025-06-08
build files are acceptable in the root, but source code and other files
should be organized in appropriate subdirectories.
13. Pre-1.0: NEVER write database migrations. There are no live databases
anywhere — every user's local index can be rebuilt from a fresh full
backup. When the schema changes, just change `schema.sql` (and any code
that touches the affected tables). The local index is disposable until
1.0 ships and is tagged.
13. Pre-1.0: NEVER add a database migration. Migrations are supported, but
nothing is installed anywhere yet, so there is nothing to migrate. To
change the schema, edit `internal/database/schema/001.sql` (and any
code that touches the affected tables) directly. After 1.0, each schema
change is a new numbered file in that directory and a released file is
never edited; an existing local database is then migrated when vaultik
is updated. See
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
+28 -19
View File
@@ -63,7 +63,7 @@ A content-addressed unit of data. Files are split into variable-size chunks usin
- `ChunkHash`: SHA256 hash of chunk content (primary key)
- `Size`: Chunk size in bytes
Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (typically 16KB-256KB for 64KB average).
Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for the 10MB default average).
#### FileChunk (`database.FileChunk`)
Maps files to their constituent chunks:
@@ -74,16 +74,16 @@ Maps files to their constituent chunks:
#### Blob (`database.Blob`)
The final storage unit uploaded to S3. Contains many compressed and encrypted chunks:
- `ID`: UUID assigned at creation
- `Hash`: SHA256 of final compressed+encrypted content
- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs))
- `UncompressedSize`: Total raw chunk data before compression
- `CompressedSize`: Size after zstd compression and age encryption
- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
Blob creation process:
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. Compressed with zstd
3. Encrypted with age (recipients configured in config)
4. SHA256 hash computed → becomes filename in S3
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and streamed to storage
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}`
#### BlobChunk (`database.BlobChunk`)
@@ -120,7 +120,7 @@ The CLI uses fx for dependency injection. Here's the instantiation order:
```go
// cli/app.go: NewApp()
fx.New(
fx.Supply(config.ConfigPath(opts.ConfigPath)), // 1. Config path
fx.Supply(config.Path(opts.ConfigPath)), // 1. Config path
fx.Supply(opts.LogOptions), // 2. Log options
fx.Provide(globals.New), // 3. Globals
fx.Provide(log.New), // 4. Logger config
@@ -193,7 +193,7 @@ scanner := v.ScannerFactory(snapshot.ScannerParams{
- **Created by**: `chunker.NewChunker(avgChunkSize)`
- **When**: Inside `snapshot.NewScanner()`
- **Configuration**:
- `avgChunkSize`: From config (typically 64KB)
- `avgChunkSize`: From config (default 10MB)
- `minChunkSize`: avgChunkSize / 4
- `maxChunkSize`: avgChunkSize * 4
@@ -284,9 +284,10 @@ Manages snapshot lifecycle and metadata export.
Key methods:
- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record
- `CompleteSnapshot(ctx, snapshotID)` → Mark snapshot complete
- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot references
- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3
- `CleanupIncompleteSnapshots(ctx, hostname)` → Remove failed snapshots
- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a successful export
- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark complete (no export between)
### `internal/database`
SQLite database for local index. Single-writer mode for thread safety.
@@ -307,7 +308,7 @@ Repository interfaces:
```
CreateSnapshot(opts)
│
├─► CleanupIncompleteSnapshots() // Critical: avoid dedup errors
├─► PruneDatabase() // Critical: avoid dedup errors
│
├─► SnapshotManager.CreateSnapshot() // Create DB record
│
@@ -336,16 +337,18 @@ CreateSnapshot(opts)
│
├─► SnapshotManager.UpdateSnapshotStatsExtended()
│
├─► SnapshotManager.CompleteSnapshot()
├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs
│
└─► SnapshotManager.ExportSnapshotMetadata()
├─► SnapshotManager.ExportSnapshotMetadata()
│ │
│ ├─► Copy database to temp file
│ ├─► Clean to only current snapshot data (VACUUM)
│ ├─► Compress binary SQLite with zstd
│ ├─► Encrypt with age
│ ├─► Upload db.zst.age to storage
│ └─► Upload manifest.json.zst to storage
│
├─► Copy database to temp file
├─► Clean to only current snapshot data (VACUUM)
├─► Compress binary SQLite with zstd
├─► Encrypt with age
├─► Upload db.zst.age to storage
└─► Upload manifest.json.zst to storage
└─► SnapshotManager.MarkSnapshotComplete() // only after the export succeeds
```
## Deduplication Strategy
@@ -366,11 +369,17 @@ bucket/
│ └── {full-hash} # Compressed+encrypted blob
│
└── metadata/
└── {snapshot-id}/
└── {remote-key}/
├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Blob list (for pruning/verification)
```
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
written to the store as a directory name. See
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation and a worked example.
## Thread Safety
- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
+1 -1
View File
@@ -30,7 +30,7 @@ Read the rules in AGENTS.md and follow them.
done provided to you in the initial instruction. Don't do part or most of
the work, do all of the work until the criteria for done are met.
* We do not need to support migrations; schema upgrades can be handled by
* We do not add migrations before 1.0; schema upgrades can be handled by
deleting the local state file and doing a full backup to re-create it.
* When testing on a 2.5Gbit/s ethernet to an s3 server backed by 2000MB/sec SSD,
+76 -30
View File
@@ -1,7 +1,28 @@
# Lint stage
# golangci/golangci-lint:v2.11.3-alpine, 2026-03-17
FROM golangci/golangci-lint:v2.11.3-alpine@sha256:b1c3de5862ad0a95b4e45a993b0f00415835d687e4f12c845c7493b86c13414e AS lint
# This file has no lint stage, deliberately.
#
# Linting lives in Dockerfile.lint, built by script/lint, and
# script/cibuild builds both. A lint stage here would have to either
# shell out to `make lint` -- which is now `docker build`, so
# docker-in-docker inside a BuildKit step with no daemon -- or call
# golangci-lint directly, which would mean a second, independently
# bumpable digest pin for the linter alongside the one in
# Dockerfile.lint. Two pins for one tool is the drift that
# https://git.eeqj.de/sneak/vaultik/issues/78 was filed over. See
# https://git.eeqj.de/sneak/vaultik/issues/113 for the ruling.
#
# Consequence, stated rather than left to be discovered: script/docker
# builds this file only and therefore does not lint. `make fmt-check`
# and `make test` still run here, so what a green build of this file
# means is "formatted, tested, and it compiles" -- the lint verdict
# comes from script/lint or script/cibuild.
# Build stage
# golang:1.26.1-alpine, 2026-03-17
FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f6c5bd6be49ed82039 AS builder
# Build tooling: make, plus a C toolchain because `go test -race` needs cgo.
# The sqlite driver is pure Go (modernc.org/sqlite), so no sqlite library or
# CLI is required.
RUN apk add --no-cache make build-base
WORKDIR /src
@@ -13,42 +34,67 @@ RUN go mod download
# Copy source code
COPY . .
# Run formatting check and linter
RUN make fmt-check
RUN make lint
# Build stage
# golang:1.26.1-alpine, 2026-03-17
FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f6c5bd6be49ed82039 AS builder
# Depend on lint stage passing
COPY --from=lint /src/go.sum /dev/null
# Run the format check and the tests.
#
# CHECK_EPOCH must stay immediately above these RUNs. These layers are
# keyed on its value, so they are cache-eligible only for a value
# already built against this same tree. script/cibuild and script/docker
# each pass a fresh value on every invocation, which is what makes their
# green mean the checks really executed.
#
# The value is expanded into each check command rather than left to a
# bare declaration, so the cache miss does not depend on BuildKit's
# unreferenced-ARG handling staying as it is. It also puts the epoch in
# the build log, where a reader can see the layer was keyed fresh.
#
# The guard is what makes a build that omits --build-arg fail instead of
# lie. An unset ARG is an empty string, and an empty string is a
# perfectly stable cache key: without the guard the first such build
# runs the checks and every one after it on an unchanged tree replays
# these layers from cache, executes nothing, and still exits 0. Failed
# steps are never cached, so the guard fails on EVERY invocation rather
# than once -- a bare `docker build .` is a loud error, not a quiet
# green. Do not give CHECK_EPOCH a default value; a default would
# satisfy the guard with a constant and restore the hole.
#
# Everything above this line (apk, go.mod, `go mod download`) is
# deliberately outside the busted range and keeps caching.
ARG CHECK_EPOCH
RUN [ -n "$CHECK_EPOCH" ] || exit 1
RUN echo "check epoch: ${CHECK_EPOCH}" && make fmt-check
RUN echo "check epoch: ${CHECK_EPOCH}" && make test
# Version, commit and build date are computed on the host by
# script/docker and script/cibuild (where .git exists) and passed in as
# build args. The build context excludes .git (see .dockerignore), so
# the build cannot derive them itself: it used to try, with `git
# rev-parse` inside this stage, and always got "unknown". VERSION comes
# from script/version, the source of truth shared with the Makefile, so
# it carries the same tag / dev-<sha> / -dirty rules and a Docker image
# reports the same string a local build of the same tree would.
#
# The defaults are the fallback for a bare `docker build .` that passes
# none of them: an unset arg would otherwise stamp an empty string and
# produce an image that cannot report its own version, commit or date.
# They match what an out-of-git build reports elsewhere.
#
# These ARGs sit here, after the checks, rather than at the top of the
# stage: every commit changes their values, and a value change
# invalidates all layers below the ARG. Declared up top they would bust
# `go mod download`; here they only rekey this build layer, which the
# COPY of the sources above already rebuilds on any change anyway.
ARG VERSION=dev
# Install build dependencies for CGO (mattn/go-sqlite3) and sqlite3 CLI (tests)
RUN apk add --no-cache make build-base sqlite
WORKDIR /src
# Copy go mod files first for better layer caching
COPY go.mod go.sum ./
RUN go mod download
# Copy source code
COPY . .
# Run tests
RUN make test
ARG COMMIT=unknown
ARG COMMIT_DATE=unknown
# Build (pure Go, no CGO required since we use modernc.org/sqlite)
RUN CGO_ENABLED=0 go build -ldflags "-X 'sneak.berlin/go/vaultik/internal/globals.Version=${VERSION}' -X 'sneak.berlin/go/vaultik/internal/globals.Commit=$(git rev-parse HEAD 2>/dev/null || echo unknown)' -X 'sneak.berlin/go/vaultik/internal/globals.CommitDate=$(git show -s --format=%cs HEAD 2>/dev/null || echo unknown)'" -o /vaultik ./cmd/vaultik
RUN CGO_ENABLED=0 go build -ldflags "-X 'sneak.berlin/go/vaultik/internal/globals.Version=${VERSION}' -X 'sneak.berlin/go/vaultik/internal/globals.Commit=${COMMIT}' -X 'sneak.berlin/go/vaultik/internal/globals.CommitDate=${COMMIT_DATE}'" -o /vaultik ./cmd/vaultik
# Runtime stage
# alpine:3.21, 2026-02-25
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
RUN apk add --no-cache ca-certificates sqlite
RUN apk add --no-cache ca-certificates
# Copy binary from builder
COPY --from=builder /vaultik /usr/local/bin/vaultik
+104
View File
@@ -0,0 +1,104 @@
# Lint image.
#
# Every lint run in this repo happens inside this image, invoked through
# script/lint, and linting is a BUILD STEP rather than a container
# command: a successful build of this file IS a clean lint. That shape
# also works where the docker daemon is remote and bind mounts are
# impossible, which `docker run` against a mounted worktree does not.
#
# This FROM line is the single source of truth for the linter version in
# this repo. Nothing else pins golangci-lint: the product Dockerfile has
# no lint stage, deliberately, so there is no second digest to bump and
# no pair of pins that can drift apart. Bump the tag AND the digest here
# and nowhere else.
#
# Note for readers coming from REPO_POLICIES.md: that document still
# describes the older pattern, a lint stage inside the product
# Dockerfile wired up with `COPY --from=lint /src/go.sum /dev/null`.
# That pattern is superseded here by the owner's ruling recorded in
# https://git.eeqj.de/sneak/vaultik/issues/113 -- lint runs in its own
# image, per run, with its own cache and its own lock, which is what
# makes concurrent runs on one host safe. The policy text is org-wide
# and is being amended separately; this file is what this repo does.
#
# golangci/golangci-lint:v2.12.2, 2026-08-10
FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240
WORKDIR /src
# Copy the dependency manifests first so the module download layer stays
# cached until they change. Everything above the ARG below is cacheable
# on purpose; a cold module download on every lint would make the inner
# loop unusable and buys nothing, because it is not what the gate is
# asserting.
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Force the check layers to execute on every invocation.
#
# CHECK_EPOCH must stay immediately above the RUNs below. Those layers
# are keyed on its value, so they are cache-eligible only for a value
# already built against this same tree; script/lint and script/cibuild
# each pass a fresh value on every invocation, which is what makes their
# green mean the linter really ran. Without it, `docker build -f
# Dockerfile.lint .` on an unchanged tree exits 0 in well under a second
# having linted nothing.
#
# The value is expanded into each check command itself rather than left
# to a bare declaration, so the cache miss does not depend on BuildKit's
# unreferenced-ARG handling staying as it is. It also puts the epoch in
# the build log, where a reader can see the layer was keyed fresh.
#
# The guard is what makes a build that omits --build-arg fail instead of
# lie. An unset ARG is an empty string, and an empty string is a
# perfectly stable cache key: without the guard the first such build
# lints and every one after it on an unchanged tree replays this layer,
# executes nothing, and still exits 0. Failed steps are never cached, so
# the guard fails on EVERY invocation rather than once. Do not give
# CHECK_EPOCH a default value; a default would satisfy the guard with a
# constant and restore the hole.
ARG CHECK_EPOCH
RUN [ -n "$CHECK_EPOCH" ] || exit 1
# Validate .golangci.yml before linting with it.
#
# This is not belt-and-braces; it closes a hole that `golangci-lint run`
# leaves wide open. `run` rejects YAML it cannot PARSE, but it silently
# IGNORES an unknown top-level KEY. Renaming `linters:` to `linterz:` --
# one character -- discards `default: all`, the whole disable list and
# every threshold, leaves only golangci-lint's small default linter set
# running, and exits 0 reporting `0 issues.` on a tree the real config
# fails. Demonstrated on this repo at this pin, recorded on
# https://git.eeqj.de/sneak/vaultik/pulls/114: with a planted
# over-length line, `script/lint` exits 1 naming the `revive` finding
# with `linters:` and exits 0 with `linterz:`. A set-but-ineffective
# config quietly falling back to defaults is precisely the false-green
# class this gate exists to eliminate, so it must not sit in the gate's
# own configuration.
#
# `config verify` catches it, and it does so OFFLINE at this pinned
# version -- verified, not assumed. Under `docker run --network none`
# against the pinned digest it exits 0 on this repo's config and exits 3
# on the `linterz:` variant with `additional properties 'linterz' not
# allowed`. An earlier revision of this file asserted the opposite, that
# the schema is fetched over live HTTPS from an unpinned URL, and used
# that to justify omitting this line. That claim was false at v2.12.2;
# the schema is embedded. If a future bump reintroduces a network fetch
# the failure is loud and this comment is where to record it.
#
# It is keyed on CHECK_EPOCH, like the lint run below, so it executes on
# every invocation. Content-addressing alone would arguably be enough --
# .golangci.yml arrives through `COPY . .`, so a cache hit here implies
# a byte-identical config was validated when the layer really ran. That
# argument is exactly the one that would also excuse caching the lint
# layer, and this repo has ruled it insufficient: a cached check layer
# checks nothing, and the cost of being wrong is silent. Forcing it costs
# milliseconds and puts the epoch in the log, where a reader can see that
# this validation ran rather than being replayed.
RUN echo "check epoch: ${CHECK_EPOCH}" && \
golangci-lint config verify --config .golangci.yml
RUN echo "check epoch: ${CHECK_EPOCH}" && \
golangci-lint run --config .golangci.yml ./...
+68 -26
View File
@@ -1,7 +1,20 @@
.PHONY: all check test lint fmt fmt-check build clean deps test-coverage test-integration local install release release-snapshot docker hooks
.PHONY: all bootstrap setup check test lint lint-fix fmt fmt-check build clean deps test-coverage local install release release-snapshot docker hooks
# Version number
VERSION := 1.0.0-rc.1
# Version number, derived from git by script/version -- the tag when
# HEAD is on one, otherwise dev-<sha>. This used to be a hardcoded
# constant, which meant every local build claimed to be a release that
# had never been tagged.
VERSION := $(shell script/version)
# $(shell) discards exit status, so a script/version that is missing,
# non-executable or broken would otherwise leave VERSION empty and every
# binary built here would print "vaultik " with no version at all. A
# build that cannot determine what it is must not produce an artifact.
ifeq ($(strip $(VERSION)),)
$(error script/version produced no version string; a build that cannot \
determine its version will not be made. Check that script/version exists \
and is executable)
endif
# Build variables
GIT_REVISION := $(shell git rev-parse HEAD 2>/dev/null || echo "unknown")
@@ -15,26 +28,56 @@ LDFLAGS := -X 'sneak.berlin/go/vaultik/internal/globals.Version=$(VERSION)' \
# Default target
all: vaultik
# Combined pre-commit/CI gate: lint, format check, then tests.
check: lint fmt-check test
# Install all development dependencies.
bootstrap:
@script/bootstrap
# Run tests only.
# Prepare a fresh clone: bootstrap plus pre-commit hook.
setup:
@script/setup
# Combined pre-commit/CI gate: tests, lint, format check.
check:
@script/check
# Run tests only. This runs the ENTIRE suite -- there is no separate
# integration target and no build-tagged subset held back. In
# particular internal/vaultik/integration_test.go, which does full
# chunk -> pack -> encrypt -> upload -> restore round-trips, runs here.
# A `test-integration` target used to exist and was removed: no file in
# the repo carried a build tag, so `-tags=integration` selected nothing
# extra and the target was an exact duplicate of this one.
test:
go test -race -timeout 30s ./...
@script/test
# Check if code is formatted (read-only).
fmt-check:
@test -z "$$(gofmt -l .)" || (echo "Files not formatted:" && gofmt -l . && exit 1)
@script/fmt-check
# Format code.
fmt:
go fmt ./...
@script/fmt
# Run linter only.
lint:
golangci-lint run ./...
@script/lint
# Apply the linter's autofixes (rewrites files).
lint-fix:
@script/lint-fix
# Build binary. `build` is the name the org convention reaches for and
# the one a caller checks the exit code of; `vaultik` is the file rule
# that does the work, so an unchanged tree still short-circuits.
#
# This alias is not decorative. `build` was listed in .PHONY with no
# rule, and a phony target with no prerequisites and no recipe is
# already satisfied: `make build` printed "Nothing to be done" and
# exited 0 without producing a binary (issue #110). Every name in
# .PHONY needs a rule for that reason; TestPhonyTargetsAllHaveRules in
# cmd/vaultik keeps it that way.
build: vaultik
# Build binary.
vaultik: internal/*/*.go cmd/vaultik/*.go
go build -ldflags "$(LDFLAGS)" -o $@ ./cmd/vaultik
@@ -43,20 +86,22 @@ clean:
rm -f vaultik
go clean
# Install dependencies.
# Install dependencies. The linter is deliberately not installed here:
# script/lint lints by building Dockerfile.lint, whose FROM line is the
# single source of truth for the linter version. A second, separately
# pinned copy on PATH could drift from it and make a local `make lint`
# disagree with CI.
deps:
go mod download
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
# Run tests with coverage.
# Run tests with coverage. -count=1 for the same reason script/test
# uses it: without it an unchanged package is served from Go's test
# result cache, and a coverage profile assembled from cached results
# describes a run that did not happen.
test-coverage:
go test -v -coverprofile=coverage.out ./...
go test -v -count=1 -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
# Run integration tests.
test-integration:
go test -v -tags=integration ./...
local:
VAULTIK_CONFIG=$(HOME)/etc/vaultik/config.yml ./vaultik snapshot --debug list 2>&1
VAULTIK_CONFIG=$(HOME)/etc/vaultik/config.yml ./vaultik snapshot --debug create 2>&1
@@ -66,19 +111,16 @@ install: vaultik
# Build and publish release artifacts (linux/darwin × amd64/arm64) via goreleaser.
release:
goreleaser release --clean
@script/release
# Dry-run a release build without publishing or tagging.
release-snapshot:
goreleaser release --clean --snapshot
@script/release-snapshot
# Build Docker image.
docker:
docker build -t vaultik .
@script/docker
# Install pre-commit hook.
hooks:
@printf '#!/bin/sh\nset -e\n' > .git/hooks/pre-commit
@printf 'go mod tidy\ngo fmt ./...\ngit diff --exit-code -- go.mod go.sum || { echo "go mod tidy changed files; please stage and retry"; exit 1; }\n' >> .git/hooks/pre-commit
@printf 'make check\n' >> .git/hooks/pre-commit
@chmod +x .git/hooks/pre-commit
@script/install-precommit
+474 -70
View File
@@ -38,7 +38,7 @@ vaultik snapshot list
Features:
* modern encryption ([age](https://age-encryption.org/), X25519 + XChaCha20-Poly1305)
* modern encryption ([age](https://age-encryption.org/), X25519 + ChaCha20-Poly1305)
* content-defined chunking with deduplication (FastCDC)
* incremental backups (only changed files are re-chunked)
* multithreaded zstd compression at configurable levels
@@ -71,19 +71,80 @@ Requirements that no existing tool meets:
## daily use
```sh
# verify a snapshot (shallow: checks all blobs exist)
# verify a snapshot (shallow: checks all blobs are present with the listed size)
vaultik snapshot verify <snapshot-id>
# deep verify (downloads and cryptographically verifies every blob)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot verify --deep <snapshot-id>
# put the private key file in the environment (reading it from the file
# keeps the key out of your shell history); the whole age-keygen file,
# with one or more identities, is accepted
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
# restore (requires the private key)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot restore <snapshot-id> /tmp/restored
# deep verify (downloads every blob, decrypts it, and re-hashes it to
# detect corruption — this checks integrity, not who wrote the blob)
vaultik snapshot verify --deep <snapshot-id>
# restore (requires the private key). Restore into a new directory you own,
# writable only by you — not a shared location like /tmp
vaultik snapshot restore <snapshot-id> ~/vaultik-restore
# daily cron job: back up, keep a 4-week rolling window of snapshots
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
```
## restoring on another machine
Restoring on a host that never ran the backup — a replacement machine
after the original is gone — is the case vaultik is built for. That host
needs only three things: the `vaultik` binary, the age **private** key,
and the storage credentials for the destination. It does **not** need the
local index, the original config file, or the original hostname.
```sh
# install
go install sneak.berlin/go/vaultik/cmd/vaultik@latest
# create a config and point it at the ORIGINAL backup destination
vaultik config init
vaultik config set storage_url "s3://bucket/prefix?endpoint=https://s3.example.com"
vaultik config set s3.access_key_id "..."
vaultik config set s3.secret_access_key "..."
# see what is on the destination store
vaultik snapshot list
```
`snapshot list` reads the destination store without the private key. A
snapshot that is not in this host's (empty) local index is shown as
remote-only: its row is identified by `<remote only:...>` rather than by
a `hostname_name_timestamp` name, because the name lives only in the
local index and the encrypted database and cannot be recovered from the
store. Its timestamp and compressed size are real. (See the `snapshot
list` description under [command details](#command-details) for the full
explanation.)
Use that remote key — the hex printed inside `<remote only:...>`, or the
full `remote_key` from `snapshot list --json` — to restore and verify:
```sh
# put the private key file in the environment (reading it from the file
# keeps the key out of your shell history)
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
# restore everything to a new directory you own (writable only by you, not a
# shared location like /tmp), then check every restored file's chunk hashes
vaultik snapshot restore --verify <remote-key> ~/vaultik-restore
# optionally, deep-verify the snapshot against the store (downloads every
# blob, decrypts it, and re-hashes it to detect corruption — this checks
# integrity, not who wrote the blob)
vaultik snapshot verify --deep <remote-key>
```
`age_recipients` (the public key) is not needed to restore — only the
private key in `VAULTIK_AGE_SECRET_KEY`. Both the abbreviated key printed
in the table and the full 64-character key from `--json` are accepted; a
leading part of the key is enough as long as it is unambiguous.
---
## cli
@@ -96,18 +157,16 @@ vaultik [--config <path>] config edit
vaultik [--config <path>] config get <key>
vaultik [--config <path>] config set <key> <value>
vaultik [--config <path>] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>]
vaultik [--config <path>] snapshot list [--json]
vaultik [--config <path>] snapshot list [--json] # alias: ls
vaultik [--config <path>] snapshot verify <snapshot-id> [--deep] [--json]
vaultik [--config <path>] snapshot purge [--keep-latest | --older-than <duration>] [--snapshot <name>...] [--force]
vaultik [--config <path>] snapshot remove <snapshot-id|--all> [--dry-run] [--force] [--remote] [--json]
vaultik [--config <path>] snapshot cleanup
vaultik [--config <path>] snapshot remove <snapshot-id> [--dry-run] [--force] [--local-only] [--json] # alias: rm
vaultik [--config <path>] snapshot restore <snapshot-id> <target-dir> [paths...] [--verify]
vaultik [--config <path>] prune [--force] [--json]
vaultik [--config <path>] info
vaultik [--config <path>] remote info [--json]
vaultik [--config <path>] remote nuke --force
vaultik [--config <path>] store info
vaultik [--config <path>] database purge [--force]
vaultik [--config <path>] database delete [--force]
vaultik completion <bash|zsh|fish|powershell>
vaultik version
```
@@ -115,16 +174,64 @@ vaultik version
### global flags
* `--config <path>`: Path to config file (default: `$VAULTIK_CONFIG`, then platform config dir, then `/etc/vaultik/config.yml`)
* `--verbose`, `-v`: Enable verbose output
* `--debug`: Enable debug output
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
* `--debug`: Enable debug output (on stderr — see below)
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
* `--skip-errors`: Continue past per-file errors instead of aborting (applies to `snapshot create` and `restore`)
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors (which would leave a chunk recorded but not stored) still abort the run.
### locking
Commands that write persistent state — `snapshot create`, `snapshot
purge`, `snapshot remove`, `prune`, and `remote nuke` — take a
process-wide lock at `$XDG_DATA_HOME/vaultik/vaultik.pid`
(`~/.local/share/vaultik/vaultik.pid` on Linux) for the whole run. Only
one of them runs at a time: a second one exits immediately with an
"already running" error rather than waiting, so two writers can never
corrupt the local index or the destination store.
Read-only commands — `info`, `snapshot list`, `snapshot verify`, and
`remote info` — do not take the lock and are never blocked, so they run
even while a backup is in progress. `snapshot restore` does not take the
lock either: it writes only to the target directory you name, not the
local index or the destination store. `config`, `database delete`,
`completion`, and `version` do not take the lock.
### stdout and stderr
Log output — everything from `--verbose` and `--debug`, and every
warning and error the logger emits — goes to **stderr**. stdout carries
the output you asked for: tables, and the documents produced by `--json`.
This means `vaultik snapshot list --verbose > out.txt` captures the
listing and leaves the diagnostics on your terminal. To capture both,
redirect stderr as well (`> out.txt 2> log.txt`, or `> out.txt 2>&1` to
interleave them).
The split is what makes `--json` usable from a script. Warnings and
errors are never suppressed — not by `--quiet`, not by `--cron` — so a
logger on stdout would eventually land a log line inside a JSON
document and break the parse. A config file with group- or
world-readable permissions is enough to trigger it.
Format follows the stream: when stderr is a terminal the records are
colorized one-liners, and when it is redirected or piped they are
JSON, one object per line.
Under `--json`, stdout holds the document and nothing else. The startup
banner is suppressed, as `--quiet` and `--cron` suppress it, and the
progress narration a command would otherwise print — such as the stale
local records `prune` reconciles away — is suppressed too, so it cannot
land ahead of the document. Every `--json` command therefore pipes on
its own, with no additional flag: `vaultik snapshot list --json | jq .`
and `vaultik prune --json | jq .` both work as written.
### environment variables
* `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `snapshot restore` and `snapshot verify --deep`)
* `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `snapshot restore` and `snapshot verify --deep`). May hold the whole `age-keygen` file — comments and every identity in it are accepted. Set it from the file, e.g. `export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"`, so the key is not typed into your shell history.
* `VAULTIK_CONFIG`: Path to config file (overridden by `--config`)
* `VAULTIK_INDEX_PATH`: Override local SQLite index path
* `VAULTIK_CPUPROFILE`: Write a CPU profile to this path for the duration of the run (development/debugging)
* `VAULTIK_MEMPROFILE`: Write a heap profile to this path when the run exits (development/debugging)
### shell completion
@@ -170,80 +277,129 @@ needed.
(System Settings → Privacy & Security → Full Disk Access) to read
TCC-protected directories; without it the backup aborts with a permission
error that explains how to fix it
* `--cron`: Silent unless error (for crontab)
* `--cron`: Silent on total success; warnings and errors are still printed
(for crontab)
* `--prune`: After backup, drop older snapshots of each backed-up name and
remove orphaned blobs from remote storage. By default keeps only the latest
snapshot per name; use `--keep-newer-than` for a rolling window.
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`)
**`snapshot list`**: Show every snapshot known to the destination
store with timestamps and three sizes per snapshot (compressed
remote size; total uncompressed chunk size; size of chunks newly
referenced by that snapshot). The uncompressed and "new chunk"
columns show `<remote only>` for snapshots not in the local index.
* `--json`: Output in JSON format
**`snapshot list`**: Show every snapshot known to this host — the union
of the local index and the backup destination store — with timestamps
and three sizes per snapshot (compressed remote size; total
uncompressed chunk size; size of chunks newly referenced by that
snapshot).
Listing the destination store does **not** require the age secret key,
so it works in vaultik's intended configuration, where the backed-up
host holds only the public key. A host that has lost its local index
can still see what it has backed up.
What that host cannot see is a remote-only snapshot's name. The
snapshot ID is hashed at the storage boundary and the manifest records
only the hash, so hostname and snapshot name exist solely in the local
index and in the encrypted per-snapshot database. Snapshots found only
on the destination store are therefore listed as
`<remote only:<abbreviated remote key>>` and show `<remote only>` in
the uncompressed and "new chunk" columns, which can only be computed
from the local index. Their timestamp and compressed size are real,
read from the manifest.
Snapshots in the local index with no counterpart on the destination
store are reported below the table as drift, with the `vaultik prune`
invocation that reconciles them.
If the destination store cannot be listed (unmounted volume,
permission denied, network down), the command warns, falls back to the
local index alone, and still exits zero.
* `--json`: Output in JSON format. Each entry carries `locally_tracked`
(whether the snapshot is in the local index), `remote_key` (the full
64-character storage key), and `remote_present` (whether it was seen
on the destination store, or `null` if the destination could not be
listed). Warnings about an unlistable destination, unreadable
manifests, and a truncated listing all go to stderr through the
logger, so stdout stays a single parseable document.
**`snapshot verify`**: Verify snapshot integrity.
* Default (shallow): checks that all blobs referenced in the manifest exist in storage
* Default (shallow): checks that every blob the manifest lists is present in
storage with the size the manifest records, and that the encrypted database is
present. It does not read blob contents.
* `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the
encrypted metadata database
* Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a
remote-only snapshot's remote key (or an unambiguous leading part of it)
* `--json`: Output results as JSON
**`snapshot purge`**: Remove old snapshots based on criteria. Retention is
per-snapshot-name (`--keep-latest` keeps the latest of each name, not the
latest globally).
* `--keep-latest`: Keep only the most recent snapshot of each name
* `--older-than <duration>`: Remove snapshots older than duration (e.g. `30d`, `6m`, `1y`)
* `--older-than <duration>`: Remove snapshots older than duration (e.g. `30d`,
`4w`, `6mo`, `1y`; `m` is minutes, `mo` is months)
* `--snapshot <name>`: Restrict to specific snapshot names (repeat for multiple)
* `--force`: Skip confirmation prompt
**`snapshot remove`**: Remove a specific snapshot from the local database.
Automatically cleans up local rows (files, chunks, blobs) that the removed
snapshot was the last referrer for — you don't need a separate prune step
after removal.
* `--remote`: Also remove snapshot metadata from remote storage
* `--all`: Remove all snapshots (requires `--force`)
**`snapshot remove`**: Remove one snapshot. By default this removes the
snapshot from the local index and strips the snapshot's metadata from
the backup destination store. Blobs are NOT touched — deleting blobs
requires reading every remaining remote manifest (the destination store
may hold snapshots this host doesn't know about), which is what
`vaultik prune` does. On success the command prints the exact `vaultik
prune` invocation to run as a follow-up. Local row cleanup (files,
chunks, blobs the snapshot was the last referrer for) runs
automatically. If the destination store is unreachable, the local-DB
removal still completes and a warning is emitted; rerun `vaultik prune`
once the store is reachable to finish remote cleanup. To wipe everything
on the destination in one go, use `vaultik remote nuke --force`.
* `--local-only`: Skip remote cleanup; only touch the local index
* `--dry-run`: Show what would be deleted without deleting
* `--force`: Skip confirmation prompt
* `--json`: Output result as JSON
**`snapshot cleanup`**: Remove stale local snapshot records that have no
corresponding metadata in remote storage. These are typically left behind
by incomplete or interrupted backups. Does not touch remote storage.
**`snapshot restore`**: Restore files from a backup snapshot.
* Requires `VAULTIK_AGE_SECRET_KEY` environment variable
* Accepts a snapshot ID, or — for a snapshot only on the destination
store — its remote key (or an unambiguous leading part of it) as shown
by `snapshot list`. See
[restoring on another machine](#restoring-on-another-machine).
* Optional path arguments to restore specific files/directories (default: all)
* Preserves file permissions, timestamps, ownership (ownership requires root),
symlinks, and empty directories
* `--verify`: After restoring, verify every file's chunk hashes match
**`prune`**: Tidy up everything that isn't needed. Removes orphaned local
database rows (files, chunks, blobs no longer referenced by any completed
snapshot) AND deletes unreferenced blobs from remote storage. `snapshot
create --prune`, `snapshot remove`, and `snapshot purge` run the same
cleanup automatically; this is the manual entry point for the same work.
**`prune`**: Tidy up everything that isn't needed. Runs three passes:
(1) reconcile the local index against the destination store — any
local snapshot whose remote metadata is missing is dropped from the
local index; (2) delete orphaned local rows (files, chunks, blobs no
longer referenced by any completed snapshot); (3) list every remote
manifest on the destination store to compute the still-referenced blob
set and delete any blob not in that set. Step (3) reads all remote
manifests — network cost scales with the number of snapshots. `snapshot
create --prune` runs the same cleanup automatically; this is the
manual entry point for the same work.
* `--force`: Skip confirmation prompt
* `--json`: Output stats as JSON
**`info`**: Display system configuration, storage settings, encryption
recipients, and local database statistics.
**`remote info`**: Show detailed remote storage information including per-snapshot
metadata sizes, blob counts, and orphaned blob detection.
**`remote info`**: Show storage backend type and location plus detailed
remote storage inventory: per-snapshot metadata sizes, blob counts, and
orphaned blob detection.
* `--json`: Output as JSON
**`remote nuke`**: Delete every snapshot's metadata and every blob from the
backup destination store, leaving the bucket prefix empty. Destructive and
irreversible.
irreversible. This is the single supported way to wipe the entire
destination store.
* `--force`: Required to confirm destruction.
**`store info`**: Display storage backend type and statistics.
**`database purge`**: Delete the local SQLite state database entirely. Remote
storage is unaffected; the next backup will do a full scan and re-deduplicate
against existing remote blobs.
**`database delete`**: Delete the local SQLite state database file
entirely. Remote storage is unaffected; the next backup will do a full
scan and re-deduplicate against existing remote blobs, and the local
index will re-bind to the currently configured storage destination.
Use this after changing `storage_url` to a different destination.
* `--force`: Skip confirmation prompt
---
@@ -270,6 +426,10 @@ both are set.
## architecture
For an implementation-level view of the internals — the data model, the
`fx` dependency-injection wiring, and the scanner — see
[`ARCHITECTURE.md`](ARCHITECTURE.md).
### remote storage layout
```
@@ -277,7 +437,7 @@ both are set.
├── blobs/
│ └── <aa>/<bb>/<full_blob_hash>
└── metadata/
└── <snapshot_id>/
└── <remote-key>/
├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Unencrypted blob list (for pruning)
```
@@ -288,8 +448,23 @@ both are set.
* `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling
pruning without the private key
Snapshot IDs follow the format `<hostname>_<snapshot-name>_<RFC3339-timestamp>`
(e.g. `server1_home_2025-06-01T12:00:00Z`).
Snapshot IDs follow the human-readable format
`<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g.
`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the
destination store in plaintext. Each snapshot's metadata directory is named
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a plain
listing of the store shows no hostname or snapshot name. The hash uses no
secret, though, so an observer who guesses a candidate hostname and snapshot
name can hash it and confirm the snapshot is present; the remote key keeps
names out of a listing but does not hide them from a guess. The backup time is
not hidden either: manifest.json.zst carries a plaintext timestamp, and object
modification times are visible at the storage layer regardless. For example,
`server1_home_2025-06-01T12:00:00Z` is stored under
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`.
See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation, and [Security Considerations](docs/REPOSTRUCTURE.md#security-considerations)
(including [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks)) for what the
format does and does not protect.
### data flow
@@ -306,7 +481,7 @@ Snapshot IDs follow the format `<hostname>_<snapshot-name>_<RFC3339-timestamp>`
**restore:**
1. Download and decrypt `metadata/<snapshot_id>/db.zst.age`
1. Download and decrypt `metadata/<remote-key>/db.zst.age`
2. Open the binary SQLite database
3. Query files (optionally filtered by paths)
4. Download and decrypt required blobs
@@ -330,32 +505,37 @@ Snapshot IDs follow the format `<hostname>_<snapshot-name>_<RFC3339-timestamp>`
### encryption
* Asymmetric encryption using age (X25519 + XChaCha20-Poly1305)
* Asymmetric encryption using age (X25519 + ChaCha20-Poly1305)
* Only the public key is needed on the source host
* Each blob and each metadata database is encrypted independently
* Multiple recipients supported (encrypt to multiple keys)
### compression
* zstd compression at configurable level (1-19, default 3)
* zstd compression at configurable level (1-19, default 3). The level is
accepted as 1-19 but maps onto zstd's four internal speed presets:
1-2 fastest, 3-5 default, 6-9 better, 10-19 best. Levels within the
same band compress identically.
* Applied before encryption at the blob level
---
## configuration reference
Run `vaultik config init` to generate a fully commented config file.
Key fields:
Run `vaultik config init` to generate a fully commented config file; a
complete annotated example also lives in
[`config.example.yml`](config.example.yml). Key fields:
| Field | Default | Description |
|-------|---------|-------------|
| `age_recipients` | (required) | Age public keys for encryption |
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
| `exclude` | | Global exclude patterns (applied to all snapshots) |
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit |
| `compression_level` | `3` | zstd compression level (1-19) |
| `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path |
@@ -379,9 +559,13 @@ Key fields:
sequentially. Restore speed is bound by single-stream throughput.
* **Device nodes, named pipes, and sockets are silently skipped.** Only
regular files, directories, and symlinks are backed up.
* **No database migrations.** If the local SQLite schema changes between
versions, delete the local database (`vaultik database purge`) and run
a full backup. Remote storage is unaffected.
* **Before 1.0, an update can make the local index unusable.** Vaultik
supports schema migrations, but none are added before 1.0 because
there is no installed base yet. If an update leaves your local index
unusable, run `vaultik database delete` and then a full backup; remote
storage is unaffected. After 1.0, the local index is migrated when
vaultik is updated. See
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
* **Files that change during backup may be inconsistent.** There is no
filesystem snapshot or freeze. If a file is modified between the scan
and chunk phases, the backed-up copy may reflect a partial write.
@@ -447,14 +631,11 @@ priority.
### infrastructure
* **Cross-machine restore documentation.** The "restore from
another host" workflow works but isn't documented as a
first-class operation in this README. Worth a dedicated section
once it's settled.
* **Schema migrations.** Currently nonexistent — pre-1.0 schema
changes are handled by `vaultik database purge` plus a full
re-scan. Post-1.0 we'll need a migration story to keep existing
index databases usable across upgrades.
* **Schema migrations after 1.0.** Migrations are supported, but none
are added before 1.0 because there is no installed base yet. After
1.0, each schema change is a new migration, so an existing local
index is migrated when vaultik is updated (see
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations)).
* **Storage backend coverage tests.** S3, file://, and rclone://
all share the Storer interface but the rclone path is the least
exercised in CI.
@@ -463,9 +644,30 @@ priority.
## output style
All user-facing output goes through helpers in `internal/ui` and conforms
to a uniform style. Color is enabled when stdout is a TTY and the
`NO_COLOR` environment variable is unset (https://no-color.org/).
Every command's user-facing output is governed by `internal/ui`, in one
of two ways. Color is enabled when stdout is a TTY and the `NO_COLOR`
environment variable is unset (https://no-color.org/).
* **Status, progress, warnings, and errors** go through the `internal/ui`
message methods below: marker-prefixed, colored on a TTY, and — except
warnings and errors — silenced by `--quiet`. This is the operational
narration of the long-running commands (`snapshot create`, `prune`,
`snapshot restore`, and the like) and the confirmations of
`config init`, `config set`, and `database delete`.
* **The data a command exists to produce** is written plain, with no
marker and no color, because a marker would corrupt a table or a
parsed document. This covers the `version`, `info`, and `remote info`
reports, the `snapshot list` table, `config get` values, and every
`--json` document. `--quiet` silences the human reports and tables
(`version`, `info`, `remote info`, `snapshot list`) but never the
machine-consumed `config get` value or the `--json` documents, which a
script depends on. The `database delete` confirmation prompt is also
written this way and always shown: it is an interactive exchange the
operator must see.
`internal/ui` writes to stdout; it is the output the user asked for.
Structured log records are a different thing and go through
`internal/log`, which writes to stderr (see "stdout and stderr" above).
Message classes:
@@ -526,6 +728,12 @@ regardless of color setting (emoji are not color).
## requirements
* Go 1.26 or later
* Docker, with a reachable daemon, to lint, check, or commit:
`script/lint` lints by building `Dockerfile.lint`, which runs the
digest-pinned `golangci-lint` image as a build step, and `make check`
and the pre-commit hook both run it. A `golangci-lint` installed on
`PATH` is not a substitute and is never used on a host, whatever its
version.
* S3-compatible object storage (or local filesystem, or rclone remote)
## development workflow
@@ -548,6 +756,202 @@ Do not commit directly to `main`. Do not skip steps.
Repository policies for AI agents are in [`AGENTS.md`](AGENTS.md).
## Entrypoints
This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call
them. We provide:
* `script/bootstrap` — install all development dependencies (go, Go
module download). It deliberately does not install `golangci-lint`;
see `script/lint` below.
* `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`
* `script/projectname` — print the project name (used for the Docker
image tag)
* `script/version` — print the version string to bake into the binary.
The `Makefile`'s `LDFLAGS` call this; it is the single source of truth
for the version. See [releasing](#releasing) for the rules.
* `script/install-goreleaser` — install the pinned `goreleaser` into
`.tool/bin` from a sha256-verified release archive. Idempotent, and
called by `script/bootstrap`; the release workflow calls it directly
because it needs `goreleaser` but not the Docker daemon
`script/bootstrap` insists on.
* `script/install-go` — install the Go toolchain named by `go.mod`'s
`go` directive into `.tool/go` from a sha256-verified `go.dev`
archive, and put it on `PATH`. Idempotent. Called only by the release
workflow, which needs a host Go for `goreleaser` to shell out to;
nothing else on the release runner does. `actions/setup-go` is not
used because it verifies the downloaded toolchain against no value in
this repo. Bumping Go edits `go.mod`, the checksum in this script, and
the `Dockerfile` `golang` digest together.
* `script/release` — cross-compile and publish the release artifacts
with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose
version is not the pinned one, on the same reasoning as `script/lint`.
* `script/release-snapshot` — the same build with no publishing and no
tagging, into `./dist`
* `script/test` — run the test suite (verbose rerun on failure). This
runs *everything*: there is no separate integration target and no
build-tagged subset held back, so the full round-trip tests in
`internal/vaultik/integration_test.go` run on every invocation. It
passes `-count=1`, which disables Go's test result cache. That is
deliberate and it is not free: on this repo's suite it costs about 11
seconds on every repeat run (measured, back to back: 0.4s cached
versus 11.6s with `-count=1`). That is the price of the run meaning
anything, because without it an unchanged package prints
`ok <pkg> (cached)`, which is indistinguishable from a package that
really ran, so the whole suite can report a full set of `ok` lines in
under half a second having executed nothing. The `-timeout` is a hang
backstop rather than a performance budget — it applies per test binary
to test execution only, not to compilation — and is set well above the
slowest package's measured runtime. Its 120s value deliberately
diverges from the 30s `REPO_POLICIES.md` mandates; the reasoning is in
the comment in the script, and issue #101 proposes amending the policy
text.
* `script/lint` — lint by building `Dockerfile.lint`, which runs
`golangci-lint run --config .golangci.yml ./...` as a build step
inside the digest-pinned `golangci-lint` image, so a successful build
*is* a clean lint. Nothing lints on the host, at any version, ever;
the script requires Docker and fails loudly rather than falling back
to a `golangci-lint` on `PATH`. That `FROM` line is the single source
of truth for the linter version — bump it there and nowhere else.
It takes no arguments, because a build step has no command line to
pass flags to, and it passes a fresh `--build-arg CHECK_EPOCH` on
every invocation so the lint layer cannot be replayed from cache (see
`script/cibuild` below for what that mechanism defends against). To
watch the linter execute, run it as
`BUILDKIT_PROGRESS=plain script/lint` and check that the lint layer
says `RUN … golangci-lint` rather than `CACHED`.
One container per run means one lint cache and one `golangci-lint`
lock per run, both private to it and discarded with it, so concurrent
runs on one host cannot contaminate or block each other.
* `script/lint-fix` — apply the linter's autofixes (rewrites files),
using the same pinned image, parsed out of `Dockerfile.lint`. It
cannot be a build step, because fixes have to land in the worktree, so
it bind-mounts the tree into a `docker run` and therefore needs a
*local* daemon. It is a developer convenience and never a gate: no
gate reads its exit status. Run `make lint` afterwards to find out
whether the tree is clean.
* `script/fmt` — format all code (writes)
* `script/fmt-check` — check formatting (read-only)
* `script/check` — run `script/test`, `script/lint`, and
`script/fmt-check`. This is authoritative *because* `script/lint`
builds `Dockerfile.lint`: a local `make check` and CI cannot disagree
about lint findings.
* `script/docker` — build the Docker image tagged via
`script/projectname`. Passes a fresh `--build-arg CHECK_EPOCH` for the
same reason `script/cibuild` does, so a local image build cannot be
green on checks it replayed from cache. It builds the *product* image
only, and the product `Dockerfile` has no lint stage, so it does not
lint: a green here means formatted, tested, and it compiles.
* `script/cibuild` — CI entrypoint, and the full gate. Two builds, in
order: `Dockerfile.lint` (the linter, as a build step) and then
`Dockerfile` (`make fmt-check` and `make test` in its builder stage,
then the product image). Either failing fails the script. It runs the
checks in the same containers CI does, from a clean copy of the tree,
so it also catches anything that depends on host state.
`.gitea/workflows/check.yml` runs it on every push to `main` and
`next` and on every pull request against either.
It passes a fresh `--build-arg CHECK_EPOCH` to each build, unique per
invocation, which both files declare immediately above their check
`RUN`s and expand into each check command. Those layers are keyed on
that value, so a new value re-runs them even on a byte-identical tree,
and a green from this script means the checks executed. Dependency and
module layers sit above the `ARG` and still cache, so a build is not
cold.
A build that supplies no `CHECK_EPOCH` — a bare `docker build .` or
`docker build -f Dockerfile.lint .` — fails rather than lying. An
unset `ARG` is an empty string and an empty string is a stable cache
key, so without a guard such a build would serve every check layer
from cache, execute nothing, and still exit 0. Each file therefore
asserts the value is non-empty before running anything, and because
failed steps are never cached that assertion fires on every
invocation rather than once. Use `script/lint`, `script/docker` or
`script/cibuild`, which pass the arg; a bare `docker build` is a loud
error.
* `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
not change files), then `script/check`
* `script/install-precommit` — install the git pre-commit hook that
runs `script/precommit`
## releasing
### version numbers
The version a binary reports comes from git, not from a constant in a
file. `script/version` decides it, and everything that stamps a binary
agrees with it:
* `HEAD` is exactly on a tag → that tag with a leading `v` stripped, so
the tag `v1.0.0` produces `vaultik 1.0.0`, matching the archive name
`vaultik_1.0.0_linux_amd64.tar.gz`. `goreleaser` strips the prefix the
same way.
* anything else → `dev-<12 chars of the commit sha>`.
* either, with uncommitted changes to tracked files → a `-dirty`
suffix, because a modified checkout of a tag is not that tag.
A build that is not a release never names itself like one. `vaultik
version` says so in as many words on a development build, and
`goreleaser --snapshot` stamps the same `dev-<sha>` string rather than
inventing the next patch number. If `script/version` cannot be run at
all, `make` stops with an error instead of building an unversioned
binary, and a binary that somehow carries an empty version string still
reports itself as a development build.
### cutting a release
Releases are cut by CI from a tag, not from a workstation:
```
git tag -a v1.2.3 -m 'v1.2.3'
git push origin v1.2.3
```
`.gitea/workflows/release.yml` triggers on `v*` tags, installs a Go
toolchain and the pinned `goreleaser`, and runs `script/release`, which
builds
`linux,darwin × amd64,arm64` archives plus `checksums.txt` and publishes
them to this repository's Gitea releases as a draft. `.goreleaser.yaml`
has a `gitea_urls:` block pointing at `https://git.eeqj.de/api/v1`;
without it `goreleaser` would talk to the GitHub API.
The workflow needs one repository Actions secret:
| Secret | What it is |
| --------------- | ------------------------------------------------------------------------------------------------------- |
| `RELEASE_TOKEN` | A Gitea access token with `write:repository` scope, owned by an account that can publish releases here. |
It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic
token is deliberately not used: it is not guaranteed to carry release
write access.
The Go toolchain that compiles the released binaries comes from an
`actions/setup-go` step pinned by commit sha, reading its version from
`go.mod` (currently `1.26.1`, the same version the `Dockerfile` builder
stage pins by digest). `goreleaser` shells out to `go` for every
cross-compile, so without that step the release would either fail
outright or ship binaries built by whatever unpinned toolchain the
runner happened to carry — the one unpinned thing in an otherwise
hash-pinned release path.
To rehearse the whole build without publishing or tagging anything:
```
make release-snapshot
```
Artifacts land in `./dist`, which is gitignored.
Release artifacts are not signed, carry no SBOM, and are not built
reproducibly; the archives contain the binary, `LICENSE`, and
`README.md` only (no shell completions or man page).
## license
[MIT](https://opensource.org/license/mit/)
+408
View File
@@ -0,0 +1,408 @@
---
title: Repository Policies
last_modified: 2026-07-06
---
This document covers repository structure, tooling, and workflow standards. Code
style conventions are in separate documents:
- [Code Styleguide](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE.md)
(general, bash, Docker)
- [Go](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_GO.md)
- [JavaScript](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_JS.md)
- [Python](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_PYTHON.md)
- [Go HTTP Server Conventions](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md)
---
- Cross-project documentation (such as this file) must include
`last_modified: YYYY-MM-DD` in the YAML front matter so it can be kept in sync
with the authoritative source as policies evolve.
- **ALL external references must be pinned by cryptographic hash.** This
includes Docker base images, Go modules, npm packages, GitHub Actions, and
anything else fetched from a remote source. Version tags (`@v4`, `@latest`,
`:3.21`, etc.) are server-mutable and therefore remote code execution
vulnerabilities. The ONLY acceptable way to reference an external dependency
is by its content hash (Docker `@sha256:...`, Go module hash in `go.sum`, npm
integrity hash in lockfile, GitHub Actions `@<commit-sha>`). No exceptions.
This also means never `curl | bash` to install tools like pyenv, nvm, rustup,
etc. Instead, download a specific release archive from GitHub, verify its hash
(hardcoded in the Dockerfile or script), and only then install. Unverified
install scripts are arbitrary remote code execution. This is the single most
important rule in this document. Double-check every external reference in
every file before committing. There are zero exceptions to this rule.
- Every repo with software must have a root `Makefile` with these targets:
`make bootstrap`, `make setup`, `make test`, `make lint`, `make fmt` (writes),
`make fmt-check` (read-only), `make check` (runs `test`, `lint`, `fmt-check`),
`make docker`, and `make hooks` (installs pre-commit hook). A model Makefile
is at `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`.
- Repos follow the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
pattern: the implementation of each Makefile target lives in an executable
script in `script/` (`script/bootstrap`, `script/setup`, `script/test`,
`script/lint`, `script/fmt`, `script/fmt-check`, `script/check`,
`script/docker`), and the Makefile targets are thin shims that call them. The
scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they run in
minimal containers (e.g. alpine images have no bash); locate the repo root
with `$(cd "$(dirname "$0")/.." && pwd -P)` and `cd` there before acting. From
the standard's canonical set we use `bootstrap`, `setup` (make the repo ready
for development after a fresh clone: runs `bootstrap`, then
`install-precommit`, plus any repo-specific initialization), `test`, and
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
assumes nothing is present: base tools come from nix, apt, brew, or apk
(detected in that order; apt runs noninteractive). For node it uses the
installed node if present; otherwise it installs a PINNED node version via
nvm, first installing nvm itself if missing — from a hash-verified GitHub
release archive (never `curl | sh`), with bash installed as an explicit
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root and runs `docker build .`; the Gitea workflow calls it. Four further
scripts are our own extensions to the standard: `script/check` runs
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
what the git pre-commit hook runs, and it calls `script/check`;
`script/install-precommit` installs the git pre-commit hook (the `make hooks`
target shims to it); and `script/projectname` (literally that filename) simply
outputs the project's name. Scripts that need the name call
`script/projectname` — e.g. `script/docker` assembles its image tag from it —
so those scripts stay byte-identical across all repos. Repo-type-specific
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/precommit`, not in the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
- Always use Makefile targets (`make fmt`, `make test`, `make lint`, etc.)
instead of invoking the underlying tools directly. The Makefile is the single
source of truth for how these operations are run.
- The Makefile is authoritative documentation for how the repo is used. Beyond
the required targets above, it should have targets for every common operation:
running a local development server (`make run`, `make dev`), re-initializing
or migrating the database (`make db-reset`, `make migrate`), building
artifacts (`make build`), generating code, seeding data, or anything else a
developer would do regularly. If someone checks out the repo and types
`make<tab>`, they should see every meaningful operation available. A new
contributor should be able to understand the entire development workflow by
reading the Makefile.
- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check`
as a build step so the build fails if the branch is not green. For non-server
repos, the Dockerfile should bring up a development environment and run
`make check`. For server repos, `make check` should run as an early build
stage before the final image is assembled. Dockerfiles install development
prerequisites by running `script/bootstrap` rather than duplicating installs
inline; COPY `script/` and the dependency manifests (`package.json` +
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
layer stays cached until dependencies change.
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
repos use a multistage build where linting runs in an independent stage based
on the `golangci/golangci-lint` image (pinned by hash). This stage runs
`make fmt-check` and `make lint` before the full build begins. The build stage
then declares an explicit dependency on the lint stage via
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
linting before proceeding to compilation and tests. This ensures lint failures
surface in seconds rather than minutes, without blocking on dependency
download or compilation in the build stage.
The standard pattern for a Go repo Dockerfile is:
```dockerfile
# Lint stage — fast feedback on formatting and lint issues
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make fmt-check
RUN make lint
# Build stage
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
WORKDIR /src
# Force BuildKit to run the lint stage before proceeding
COPY --from=lint /src/go.sum /dev/null
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make test
ARG VERSION=dev
RUN CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage
FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
Key points:
- The lint stage uses the `golangci/golangci-lint` image directly (it
includes both Go and the linter), so there is no need to install the
linter separately.
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
a stage dependency. BuildKit runs stages in parallel by default; without
this line, the build stage would not wait for lint to finish and a lint
failure might not fail the overall build.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint stage must
create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
The lint stage should not depend on the actual build output — it exists to
fail fast.
- If the project requires CGO or system libraries for linting (e.g.
`vips-dev`), install them in the lint stage with `apk add`.
- The build stage runs `make test` after compilation setup. Tests run in the
build stage, not the lint stage, because they may require compiled
artifacts or heavier dependencies.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` (which runs `docker build .`) on push. Since the
Dockerfile already runs `make check`, a successful build implies all checks
pass.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
two exceptions: four-space indents (except Go), and `proseWrap: always` for
Markdown (hard-wrap at 80 columns). Documentation and writing repos (Markdown,
HTML, CSS) should also have `.prettierrc` and `.prettierignore`.
- Pre-commit hook: runs `script/precommit`, which calls `script/check`. If local
testing is not possible in the repo, `script/precommit` may skip `script/test`
and run only `script/lint` and `script/fmt-check`. The hook is installed by
`script/install-precommit`; the Makefile must provide a `make hooks` target
that shims to it.
- All repos with software must have tests that run via the platform-standard
test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful
tests exist yet, add the most minimal test possible — e.g. importing the
module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op.
- `make test` must complete in under 20 seconds. Add a 30-second timeout in the
Makefile.
- **`make test` should use the conditional verbose rerun pattern.** Run tests
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
show full output. This keeps CI logs and `docker build` output clean on
success (just package/suite summaries) while providing full diagnostic detail
on failure (every test case, every assertion). The general shell pattern:
```makefile
test:
@<test-command> || \
{ echo "--- Rerunning with -v for details ---"; \
<test-command-with-v>; exit 1; }
```
Go example:
```makefile
test:
@go test -timeout 30s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 30s -race -v ./...; exit 1; }
```
Python example:
```makefile
test:
@python -m pytest || \
{ echo "--- Rerunning with -v for details ---"; \
python -m pytest -v; exit 1; }
```
The `exit 1` ensures the target always fails after a rerun — the first run
already proved the tests are broken, so the build must not pass even if a
flaky test happens to succeed on the second attempt. The rerun exists solely
for diagnostic output.
- Docker builds must complete in under 5 minutes.
- `make check` must not modify any files in the repo. Tests may use temporary
directories.
- `main` must always pass `make check`, no exceptions.
- Never commit secrets. `.env` files, credentials, API keys, and private keys
must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`.
Fetch the standard `.gitignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo.
- **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the
repository if it can be avoided. The build process (e.g. Dockerfile, Makefile)
should generate these at build time. Notable exception: Go protobuf generated
files (`.pb.go`) ARE committed because repos need to work with `go get`, which
downloads code but does not execute code generation.
- Never use `git add -A` or `git add .`. Always stage files explicitly by name.
- Never force-push to `main`.
- Make all changes on a feature branch. You can do whatever you want on a
feature branch.
- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only
manually by the user. Fetch from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`.
- When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD).
- Use `yarn`, not `npm`.
- Write all dates as YYYY-MM-DD (ISO 8601).
- Simple projects should be configured with environment variables.
- Dockerized web services listen on port 8080 by default, overridable with
`PORT`.
- **HTTP/web services must be hardened for production internet exposure before
tagging 1.0.** This means full compliance with security best practices
including, without limitation, all of the following:
- **Security headers** on every response:
- `Strict-Transport-Security` (HSTS) with `max-age` of at least one year
and `includeSubDomains`.
- `Content-Security-Policy` (CSP) with a restrictive default policy
(`default-src 'self'` as a baseline, tightened per-resource as
needed). Never use `unsafe-inline` or `unsafe-eval` unless
unavoidable, and document the reason.
- `X-Frame-Options: DENY` (or `SAMEORIGIN` if framing is required).
Prefer the `frame-ancestors` CSP directive as the primary control.
- `X-Content-Type-Options: nosniff`.
- `Referrer-Policy: strict-origin-when-cross-origin` (or stricter).
- `Permissions-Policy` restricting access to browser features the
application does not use (camera, microphone, geolocation, etc.).
- **Request and response limits:**
- Maximum request body size enforced on all endpoints (e.g. Go
`http.MaxBytesReader`). Choose a sane default per-route; never accept
unbounded input.
- Maximum response body size where applicable (e.g. paginated APIs).
- `ReadTimeout` and `ReadHeaderTimeout` on the `http.Server` to defend
against slowloris attacks.
- `WriteTimeout` on the `http.Server`.
- `IdleTimeout` on the `http.Server`.
- Per-handler execution time limits via `context.WithTimeout` or
chi/stdlib `middleware.Timeout`.
- **Authentication and session security:**
- Rate limiting on password-based authentication endpoints. API keys are
high-entropy and not susceptible to brute force, so they are exempt.
- CSRF tokens on all state-mutating HTML forms. API endpoints
authenticated via `Authorization` header (Bearer token, API key) are
exempt because the browser does not attach these automatically.
- Passwords stored using bcrypt, scrypt, or argon2 — never plain-text,
MD5, or SHA.
- Session cookies set with `HttpOnly`, `Secure`, and `SameSite=Lax` (or
`Strict`) attributes.
- **Reverse proxy awareness:**
- True client IP detection when behind a reverse proxy
(`X-Forwarded-For`, `X-Real-IP`). The application must accept
forwarded headers only from a configured set of trusted proxy
addresses — never trust `X-Forwarded-For` unconditionally.
- **CORS:**
- Authenticated endpoints must restrict `Access-Control-Allow-Origin` to
an explicit allowlist of known origins. Wildcard (`*`) is acceptable
only for public, unauthenticated read-only APIs.
- **Error handling:**
- Internal errors must never leak stack traces, SQL queries, file paths,
or other implementation details to the client. Return generic error
messages in production; detailed errors only when `DEBUG` is enabled.
- **TLS:**
- Services never terminate TLS directly. They are always deployed behind
a TLS-terminating reverse proxy. The service itself listens on plain
HTTP. However, HSTS headers and `Secure` cookie flags must still be
set by the application so that the browser enforces HTTPS end-to-end.
This list is non-exhaustive. Apply defense-in-depth: if a standard security
hardening measure exists for HTTP services and is not listed here, it is
still expected. When in doubt, harden.
- `README.md` is the primary documentation. Required sections:
- **Description**: First line must include the project name, purpose,
category (web server, SPA, CLI tool, etc.), license, and author. Example:
"µPaaS is an MIT-licensed Go web application by @sneak that receives
git-frontend webhooks and deploys applications via Docker in realtime."
- **Getting Started**: Copy-pasteable install/usage code block.
- **Entrypoints**: Opens by stating that the repo adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard (with that link), then documents each provided `script/`
entrypoint and its purpose.
- **Rationale**: Why does this exist?
- **Design**: How is the program structured?
- **TODO**: Update meticulously, even between commits. When planning, put
the todo list in the README so a new agent can pick up where the last one
left off.
- **License**: MIT, GPL, or WTFPL. Ask the user for new projects. Include a
`LICENSE` file in the repo root and a License section in the README.
- **Author**: [@sneak](https://sneak.berlin).
- First commit of a new repo should contain only `README.md`.
- Go module root: `sneak.berlin/go/<name>`. Always run `go mod tidy` before
committing.
- Use SemVer.
- Database migrations live in `internal/db/migrations/` and must be embedded in
the binary.
- `000_migration.sql` — contains ONLY the creation of the migrations
tracking table itself. Nothing else.
- `001_schema.sql` — the full application schema.
- **Pre-1.0.0:** never add additional migration files (002, 003, etc.).
There is no installed base to migrate. Edit `001_schema.sql` directly.
- **Post-1.0.0:** add new numbered migration files for each schema change.
Never edit existing migrations after release.
- All repos should have an `.editorconfig` enforcing the project's indentation
settings.
- Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
language-specific config). Everything else goes in a subdirectory. Canonical
subdirectory names:
- `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints
- `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root)
- `internal/` — Go internal packages
- `internal/db/migrations/` — database migrations
- `pkg/` — Go library packages
- `share/` — systemd units, data files
- `static/` — static assets (images, fonts, etc.)
- `web/` — web frontend source
- When setting up a new repo, files from the `prompts` repo may be used as
templates. Fetch them from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/<path>`.
- New repos must contain at minimum:
- `README.md`, `.git`, `.gitignore`, `.editorconfig`
- `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo)
- `Makefile`
- `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`,
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`install-precommit`)
- `Dockerfile`, `.dockerignore`
- `.gitea/workflows/check.yml`
- Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml`
+696
View File
@@ -0,0 +1,696 @@
# Workflow
* branch (from `main`)
* do the work in Next Step
* move Next Step to the top of Completed Steps
* move the top item of Future Steps into Next Step
* commit (`TODO.md` changes in the same commit as the work)
* merge to `main` if the branch is not protected, otherwise open a PR
* push
# Status
pre-1.0
# Next Step
Define the remaining scope for the first tagged release under the 1.0.0
milestone, then cut that tag. The mechanism to cut it now exists and is
exercised; what is left is the scope decision, which is the owner's.
This step deliberately names one version number: it previously said
"cut v0.1.0" while the `Makefile` baked in `1.0.0-rc.1` and the issue
milestone said 1.0.0, and three different answers to "what is the next
release" is exactly the contradiction
[issue #65](https://git.eeqj.de/sneak/vaultik/issues/65) was filed over.
# Completed Steps
- 2026-09-22: Routed the last direct-to-stdout command output through
`internal/ui`
([issue #149](https://git.eeqj.de/sneak/vaultik/issues/149)). The
`version`, `info`, `remote info`, `config`, and `database delete`
commands wrote plain text straight to stdout, so they were unstyled and
ignored `--quiet`. Output now falls in two buckets, both governed by
`internal/ui`: status lines and confirmations go through its message
methods (styled, and `--quiet` silences them), while the data a command
exists to produce — the reports, the `snapshot list` table, `config get`
values, and the `--json` documents — is written plain. `--quiet`
silences the human reports and tables but never the `config get` value
or the `--json` documents, which a script depends on, and the
`database delete` confirmation prompt is always shown. The README
output-style section now states this rule.
- 2026-09-22: Validated blob hashes, offsets and lengths read back from
the destination before using them
([issue #155](https://git.eeqj.de/sneak/vaultik/issues/155)). A blob
hash taken from the downloaded database or the store listing was
trusted unchecked, so a hostile remote could drive a decrypted blob to
be written outside the cache directory (a hash like `aa/../../etc`) or
crash a command with a short or negative value. `blobDiskCache.path`
now refuses any key containing a path separator, and `ReadAt` rejects a
negative offset or length and bounds with `length > size-offset` so a
sum cannot overflow past the check. A new `isBlobHash` helper (a plain
function, not a method — the packer stores `temp-placeholder-{uuid}` as
a hash) gates `FetchBlob`, shallow and deep verify; the `blobs/` and
`metadata/` listings skip a non-conforming name with a warning; and
short-hash prefixes in log and error text go through a `shortHash`
helper that cannot panic. `verify`'s chunk reader also rejects a
negative `blob_chunks` length and streams the chunk instead of
allocating a database-supplied size. `restore.go` and
`internal/database` were left untouched to avoid colliding with the
in-flight [issue #156](https://git.eeqj.de/sneak/vaultik/issues/156)
work; the cache-path and `FetchBlob` guards already stop the unsafe
write and fetch, so restore's own `buildBlobIndexes` early check is
deferred as fail-fast defense in depth.
- 2026-09-21: Stopped an interrupted blob upload from making a later
backup deduplicate against data that was never stored
([issue #148](https://git.eeqj.de/sneak/vaultik/issues/148)). The
packer commits a blob's `chunks`, `blob_chunks`, and `blobs` rows
before the upload is attempted, so a failed upload left chunk rows
behind and the next run skipped re-uploading them, producing a
snapshot that reported success but could not be restored. A run now
deduplicates only against chunks held by a blob whose `uploaded_ts` is
set, and at startup drops any un-uploaded blob rows (and the chunks
they orphan) so the affected data is re-chunked and re-uploaded. Blobs
recorded with no remote backend are marked uploaded so this invariant
holds uniformly.
- 2026-09-22: Made restore refuse any snapshot path that would write
outside the target directory
([issue #154](https://git.eeqj.de/sneak/vaultik/issues/154)).
`restoreFile` and `verifyRestoredFiles` joined the stored path onto the
target with no containment check, so a `..` segment or an absolute path
escaped the target and a restored symlink could redirect a later child
write anywhere on disk. Every stored path is now rejected unless
`filepath.IsLocal` accepts it with the leading separator removed, and
each existing ancestor directory below the target is `Lstat`ed to refuse
descending through a symlink; honest symlinks pointing outside the tree
are still written verbatim. age decryption proves a snapshot is
readable, not honest, and restore usually runs as root.
- 2026-09-21: Stopped `--json` from silencing stderr diagnostics
([issue #112](https://git.eeqj.de/sneak/vaultik/issues/112)). `--json`
used to be folded into `Quiet`, which pinned the log level to `WARN`,
so `prune --json` gave a machine consumer no record of the local index
rows it deleted even under `--verbose`. `--json` now quiets only the
stdout UI (the JSON document must stay clean, per
[issue #108](https://git.eeqj.de/sneak/vaultik/issues/108)); the stderr
log level follows `--verbose`/`--debug` again. The coupling was
removed the same way for `snapshot verify`, `snapshot remove`, and
`remote info`, which carried it for the same outdated reason.
- 2026-09-21: Stopped `prune` from reporting a failed row count as 0
([issue #96](https://git.eeqj.de/sneak/vaultik/issues/96)). The seven
`getTableCount` reads in `PruneDatabase` discarded their error, so a
query that could not run became a plausible `0` and the before/after
delta computed from it looked like real work. Each read now logs at
warn on failure and renders as `unknown`, never `0`, so an empty table
is distinguishable from one that could not be queried. The counts have
no `--json` representation — under `--json` the summary is suppressed
entirely — so nothing there can show a false `0`.
- 2026-09-21: Made the s3 storage backend report a missing object as
`storage.ErrNotFound`, like the `file` and `rclone` backends and as the
`Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw
AWS SDK error, so `errors.Is(err, storage.ErrNotFound)` was false on s3
and callers branched differently per backend. Added a small `s3.IsNotFound`
helper (reused by `HeadObject`) and a test that a missing key maps to
`ErrNotFound`
([issue #129](https://git.eeqj.de/sneak/vaultik/issues/129)).
- 2026-09-21: Fixed `verify --deep` reporting healthy snapshots as
corrupt. Its final blob-integrity check hashed the encrypted
downloaded bytes with a single SHA256 and compared that to the blob
ID, which is the double SHA256 of the plaintext, so the two could
never match. It now hashes the decompressed plaintext and compares the
double SHA256. Added a test that backs up a real snapshot, deep-verifies
it, then flips a byte in one stored blob and confirms deep verification
then fails
([issue #131](https://git.eeqj.de/sneak/vaultik/issues/131)).
- 2026-09-21: Made `snapshot create` VACUUM the per-snapshot metadata
database through the `modernc.org/sqlite` driver instead of shelling
out to the external `sqlite` command-line binary (issue #120). A
backup no longer needs that binary on `PATH`, so `make check` passes
on a stock `go install` host; `script/bootstrap` and the `Dockerfile`
(both the test-build and the shipped runtime stage) no longer install
it, and a new test asserts the uploaded database keeps no pages from
deleted rows. Dropped the now-false note on the 2026-08-07 entry below
that said bootstrap installs it.
- 2026-09-21: Made `.gitea/workflows/check.yml` run on pushes to `main`
and `next` and on pull requests against either, so unit PRs (whose
base is `next`) and `next` itself get a CI run instead of relying on a
local `make check`
([issue #122](https://git.eeqj.de/sneak/vaultik/issues/122)).
- 2026-09-21: Hash-verified the Go toolchain in the release workflow
([issue #105](https://git.eeqj.de/sneak/vaultik/issues/105)). New
`script/install-go` downloads the exact `go.dev` archive for `go.mod`'s
`go` directive and refuses it unless its sha256 matches a value
committed in the script; `.gitea/workflows/release.yml` calls it
instead of `actions/setup-go`, which verified the downloaded toolchain
against nothing in the repo. `GOTOOLCHAIN: local` on the release step
keeps that exact compiler from auto-switching. Bumping Go now touches
`go.mod`, the checksum, and the `Dockerfile` `golang` digest together.
- 2026-09-21: Collapsed the two duration parsers into one and fixed the
`--older-than` months example
([issue #123](https://git.eeqj.de/sneak/vaultik/issues/123)). Two
functions named `parseDuration` existed with different grammars;
`snapshot purge --older-than` and `--keep-newer-than` both already went
through the one in `internal/vaultik`, while the richer copy in
`internal/cli/duration.go` was reachable only from its own test. Kept
the live-path parser and deleted the unused one, so no flag's accepted
grammar changes. The trap the issue was filed over: `README.md`
documented `6m` as the months example for `--older-than`, but `m` is
minutes, so the documented command deleted every snapshot older than
six minutes on a destructive flag. Corrected the doc to `6mo` and put
both flags' help text on one example list that states `m` is minutes
and `mo` is months. The surviving parser now rejects negatives, which
it previously accepted (`-5h`) or silently made positive (`-5d`).
Table-driven tests cover every unit, `6m` as six minutes, `6mo` as 180
days, and rejection of a bare number, an unknown unit, and a negative.
- 2026-08-10: Moved every lint run into its own container, as a build
step ([issue #113](https://git.eeqj.de/sneak/vaultik/issues/113)).
New root `Dockerfile.lint`, built by `script/lint`, runs
`golangci-lint run --config .golangci.yml ./...` as a `RUN`
instruction in the digest-pinned `golangci/golangci-lint` image: a
successful build of that file *is* a clean lint, and it works even
where the daemon is remote and bind mounts are impossible. That
`FROM` line is now the only pin of the linter version in the repo.
This supersedes the per-worktree cache isolation landed for
[issue #99](https://git.eeqj.de/sneak/vaultik/issues/99). Isolation
fixed cross-worktree contamination but not lock contention — two
concurrent runs with entirely separate cache directories still
collided. A container per run has its own cache and its own lock, so
the whole class is gone, and with it the per-worktree cache
machinery, the lock-retry loop, and `script/lint-audit`, which
existed to catch replayed findings from a cache that no longer
exists. The host lint path went too: no escape hatch, no
`VAULTIK_LINT_IN_CONTAINER`, no version detection. Nothing lints on
the host at any version.
A cached build lints nothing, so the same `CHECK_EPOCH` mechanism the
product `Dockerfile` already used is what makes the green mean
something: `ARG CHECK_EPOCH` with no default below the module layers,
a `RUN [ -n "$CHECK_EPOCH" ] || exit 1` guard, the value expanded
into each check command, and a fresh `$(date +%s%N)$$` per invocation
computed as a bare assignment. `cmd/vaultik/lintdocker_test.go`
parses both Dockerfiles and both scripts and fails if any part of
that is dropped, because every way of losing it is silent. No test
asserts that no script runs the host linter: `script/lint` is the one
lint entry point and runs `golangci-lint` only inside the container,
and keeping it that way is a review matter, not something a test
proves.
The product `Dockerfile` lost its lint stage rather than gaining a
second linter pin: `make lint` is now `docker build`, so the stage
would have been docker-in-docker with no daemon, and calling
`golangci-lint` directly there would have restored the two-pins drift
of [issue #78](https://git.eeqj.de/sneak/vaultik/issues/78).
`make fmt-check` moved beside `make test` in the builder stage, and
`script/cibuild` now builds `Dockerfile.lint` and then `Dockerfile`,
each with its own fresh epoch. Consequence, stated rather than left
to be found: `script/docker` builds the product image only and no
longer lints; the gates are `script/check` and `script/cibuild`.
`golangci-lint config verify` runs as its own epoch-keyed layer,
above the lint. `golangci-lint run` rejects a config it cannot parse
but silently ignores an unknown top-level *key*: renaming `linters:`
to `linterz:` discarded `default: all` and every threshold and still
exited 0 on a tree the real config fails. `config verify` catches
that, and it does so with the network off at this pin — checked under
`docker run --network none`, not assumed. An earlier revision omitted
it on the claim that it fetches its schema over live HTTPS; that
claim was false at v2.12.2.
`script/lint-fix` is kept, reimplemented as a
bind-mounted `docker run` against the image parsed out of
`Dockerfile.lint` — it cannot be a build step, because fixes have to
land in the worktree — and marked in its header as a developer
convenience that no gate reads.
- 2026-08-09: Finished the `--json` stdout contract and gave `make build`
a rule ([issue #108](https://git.eeqj.de/sneak/vaultik/issues/108),
[issue #110](https://git.eeqj.de/sneak/vaultik/issues/110)). Two
unrelated defects of the same shape — a command reporting something it
did not do — landed together because both are small.
`CleanupLocalSnapshots` wrote three prose lines to stdout with no
`--json` awareness, covering every branch of the function, so no input
avoided them and `vaultik prune --json | jq` failed even after
[issue #106](https://git.eeqj.de/sneak/vaultik/issues/106) removed the
banner. `-q` never helped either: `printlnStdout` and `stdoutf` write
straight to `Vaultik.Stdout` and never consult `Vaultik.UI`, which is
what `SetQuiet` affects. The issue offered three fixes and asked for a
decision. Taken: thread `*PruneOptions` into the function and gate each
write on `!opts.JSON`, matching `PruneBlobs` (its sibling phase, which
already takes the same struct), `RemoveSnapshot` and `remote info`, so
the package has one pattern rather than two. Rejected: moving the lines
to `log.Info`, because the logger's default level is `slog.LevelWarn`,
so that would not relocate them to stderr — it would delete them from a
plain `vaultik prune`, and the removal of rows from the local index is
not something to narrate only under `--verbose`. Also rejected: putting
the stale-record count into `PruneBlobsResult`, whose every field is
blob-scoped and which is produced by the later phase; a prune document
covering both phases is a reasonable thing to want, but it is a schema
design question and not a stream-hygiene fix. The narration is
duplicated as `log.Info` records, which `PruneBlobs` already does
alongside its own prints, so the events survive on stderr for anyone
running `--verbose`.
`make build` printed "Nothing to be done for 'build'" and exited 0
without producing a binary: `build` was listed in `.PHONY` with no
`build:` rule anywhere, and declaring a name phony is exactly what
converts make's "No rule to make target" error into a silent success.
Fixed with `build: vaultik`, keeping `vaultik:` as the file rule. The
audit the issue asked for covers all 19 `.PHONY` names; `build` was the
only one without a rule, and `vaultik` is correctly absent from
`.PHONY`, being a real file target.
Tests, each verified to fail with the fix reverted rather than assumed
to: `CleanupLocalSnapshots` leaves stdout untouched under `--json` in
all three branches (stale records, none, empty index) and still emits
every line without it, so the guard cannot be satisfied by deleting the
output; `prune --json` run end to end through `Entry`, cobra and fx
over the process's real stdout descriptor against a `file://` store,
asserting exactly one JSON document, in both the stale and non-stale
branches; and a parse of the `Makefile` asserting every `.PHONY` name
has a rule and that `build` reaches the rule that produces the binary,
which keeps the audit true for names added later. That last one is a
parse rather than an invocation of `make`, since `make test` is what
runs it and shelling back into `make build` would nest a build inside
the test run. The property a parse cannot establish — that the recipe
still fails when the build fails — was verified by hand against a
deliberately broken tree: `make build` exits 2 and produces nothing.
`cmd/vaultik` gains its first test file, so `make test` now reports 16
packages `ok` where it reported 15.
- 2026-08-09: Stopped the startup banner from contaminating `--json`
documents ([issue #106](https://git.eeqj.de/sneak/vaultik/issues/106)).
`Entry` writes the banner to stdout before cobra parses anything, and
the flag scan that suppresses it knew `--quiet`, `-q` and `--cron` but
not `--json`, so every `--json` document arrived behind two lines of
prose and a blank line, and `vaultik snapshot list --json | jq` failed.
With the logger already on stderr from
[issue #82](https://git.eeqj.de/sneak/vaultik/issues/82), this was the
last writer that could put something on stdout that the caller did not
ask for. The design question the issue raised — extend the raw-argv
scan, or move the banner after parsing — is answered in favour of the
scan: the banner is printed first deliberately, so that it still
appears when cobra rejects the arguments and on `--help`, and after
parsing there is no single place that covers those paths. The stated
cost of the scan, that `--json` is a subcommand flag matched anywhere
in the vector, is a cost `--cron` already carries — it exists only on
`snapshot create` — so this adds an instance of an accepted
imprecision rather than a new kind, and the two error directions are
not symmetric: a false positive loses a decorative banner, a false
negative corrupts a document. Regression tests at the CLI layer, where
`internal/vaultik`'s existing guard cannot reach: one runs `Entry`
itself over the process's real stdout descriptor, through cobra and fx
to the document, made hermetic by `file://` storage; a second covers
the argument vectors of all five `--json` commands; a third asserts the
banner is still printed without a suppressing flag, so the first
cannot be satisfied by deleting the banner. Also corrected `AGENTS.md`
policy 9, which still keyed the structured-log format on stdout's
TTY-ness after #82 moved that decision to stderr — a rules file that
misdescribes the code misleads exactly the readers who trust it most.
Two smaller findings from the same review: `bytesAttrKey`'s
human-readable byte formatting silently stopped applying under an open
group, because the key reaching the comparison is group-qualified
(`transfer.bytes`), now matched on its final segment and tested both
ways; and `listEnv.stderr` in `snapshot_list_test.go`, assigned but
never read since those tests began capturing the process's stderr, is
removed. `Vaultik.Stderr` is kept — nothing writes to it today, which
its comment now says outright.
- 2026-08-09: Moved the logger to stderr and fixed `TTYHandler`'s
discarded attributes
([issue #82](https://git.eeqj.de/sneak/vaultik/issues/82),
[issue #97](https://git.eeqj.de/sneak/vaultik/issues/97)). Two defects
in `internal/log`, fixed together because both live in the handler
construction path. The first: both handlers were built over
`os.Stdout`, and `WARN`/`ERROR` are never suppressed, so a config file
with group- or world-readable permissions was enough to put a log
record inside a `--json` document and break `jq`. Diagnostics now go
to stderr, and the TTY/JSON format choice follows stderr rather than
stdout — testing the wrong stream would colorize records on a
redirected stderr whenever stdout happened to be a terminal. This is
user-visible: `--verbose` and `--debug` output moves to stderr too,
which is documented in `README.md` under "stdout and stderr". It also
let the local workaround in `internal/vaultik/snapshot_list.go` go:
`warnWhileListing` had been hand-rolling structured-log formatting to
reach a non-stdout writer, and the `jsonOutput` parameter threaded
through the remote-listing helpers existed only to choose between the
two writers. The collect-then-emit machinery around `listingWarning`
stays, but on its remaining merit — warnings emitted in key order
after `group.Wait()` are deterministic run to run, where emitting from
the fetch workers would order them by network timing. The second
defect: `TTYHandler.WithAttrs` and `WithGroup` discarded their
arguments and returned the receiver while their doc comments claimed
otherwise, so `log.With` attributes vanished on a terminal and
appeared correctly in CI — failing precisely when someone is debugging
interactively. Both now return a new handler (the receiver is never
written to, since `slog` permits concurrent derivation), attributes
persist across records, and grouping is implemented as dotted key
prefixes, which is the only honest rendering for a format with nowhere
to nest. New tests cover both, including one that feeds the same
derivation chain to the TTY and JSON handlers and compares the
attribute sets, so the two paths cannot drift apart again. Found and
filed while verifying: the startup banner is written to stdout and
`--json` does not suppress it
([issue #106](https://git.eeqj.de/sneak/vaultik/issues/106)), which is
a separate writer on a separate path and the remaining source of
stdout contamination.
- 2026-08-09: Made the tagged-release path actually work on Gitea
([issue #65](https://git.eeqj.de/sneak/vaultik/issues/65)). Three
independent blockers, one of which was the whole
release: `.goreleaser.yaml` had no `gitea_urls:` block, so goreleaser
defaulted to the GitHub API and a `goreleaser release` from this repo
would have failed or published where nobody is looking. It now points
at `https://git.eeqj.de/api/v1`. The version is the second: it was a
hardcoded `VERSION := 1.0.0-rc.1` in the `Makefile`, so every local
build claimed to be a release candidate that had never been tagged and
did not exist, while `git tag -l` was empty and `internal/globals`
defaulted to `dev`. Version now comes from git via the new
`script/version` — the exact tag with a leading `v` stripped (so a
`make` build and a goreleaser build of one commit report the same
string, and it matches the archive names), otherwise `dev-<12-char
sha>`, with `-dirty` appended in either case when tracked files are
modified. Untracked files are deliberately not counted, matching
`git describe --dirty`. The same honesty was owed by the snapshot
path: `snapshot.version_template` was `{{ incpatch .Version }}-next`,
which manufactures a release number from the last tag and, with no
tags at all, from goreleaser's fabricated `v0.0.0`; it now emits the
same `dev-<sha>`. The one non-obvious consequence is that
`internal/cli/version.go` gated its "this is a development build"
notice on the version being exactly `dev`, so the moment untagged
builds began carrying a commit sha that notice would have gone silent
and an unreleased binary would have read as a release — the gate is
now `globals.IsDevVersion`, which is a predicate over a string rather
than a comparison against a global precisely so it can be tested, and
it is tested at the boundary (`1.0.0-dev` is a release, `dev-<sha>`
is not). Release automation is the third blocker: a tag-triggered
`.gitea/workflows/release.yml` runs the build in CI rather than from
a laptop, with `fetch-depth: 0` because a shallow checkout has no
tags and would silently mislabel the release, and with the
`RELEASE_TOKEN` repository secret passed as `GITEA_TOKEN` (documented
in `README.md`; the runner's automatic token is not used because it
is not guaranteed to carry release write scope). `script/release`
unsets any `GITHUB_TOKEN`/`GITLAB_TOKEN` it finds, since goreleaser
chooses its forge from whichever token variable is set and refuses to
run when it sees more than one — a runner-provided token must not get
to decide where these artifacts are published. `make release` and
`make release-snapshot`, the last two Makefile targets that were not
shims, now call `script/release` and `script/release-snapshot`, which
resolve goreleaser exactly the way `script/lint` resolves the linter:
a `PATH` binary is used only at the pinned version, never as a silent
fallback. `script/bootstrap` installs it, from a sha256-verified
GitHub release archive per `REPO_POLICIES.md`, via a separate
`script/install-goreleaser` — separate because `script/bootstrap`
hard-fails without a usable Docker daemon by design, and the release
runner needs goreleaser without needing Docker. Verified by running
the thing rather than reading it: `make release-snapshot` produced
four archives and `checksums.txt`, and the linux/amd64 binary from
`dist/` reports `dev-<sha>` with the development-build notice. Tag
handling was exercised in a throwaway repository rather than by
tagging this one; no tag was created here, since that is the owner's
call. Signing, SBOM, reproducible builds, completions and a man page
are out of scope by the issue.
- 2026-08-09: Isolated the lint cache per worktree and context-gated the
native lint path (issues #99, #80). One defect seen twice:
`script/lint` decided whether it could skip the pinned image by asking
what version was on `PATH` rather than where it was running, and cache
isolation is part of that same question. The cache was one directory
per repo, shared by every worktree on the host, so two checkouts with
identical Go file contents collided and golangci-lint replayed the
stored analysis — paths and all. The loud direction of that failure
(a clean tree failed by a dirty sibling) is the harmless one; the
silent direction, a dirty tree **passed** by a clean sibling, is a
sixth way for a gate here to report a green it did not earn. The cache
is now keyed on a digest of the worktree path, and every run is
audited by the new `script/lint-audit`, which rejects output citing any
file that is not in the tree being linted — a backstop that runs on
clean output too, because that is the case nobody investigates. Caches
record the worktree they belong to and are collected when it
disappears, so throwaway worktrees do not accumulate them; the whole
tree lives under `XDG_CACHE_HOME` and is disposable. The
`parallel golangci-lint is running` refusal is now a bounded retry
rather than a verdict: it is not a lint result, and exiting non-zero
on it is indistinguishable to a caller from real findings (#88 showed
a private cache does not remove that contention). The native path now
requires `VAULTIK_LINT_IN_CONTAINER=1`, set only by the `Dockerfile`
lint stage, in addition to matching the pin, so a developer's locally
installed 2.12.2 no longer bypasses the digest pin; `/.dockerenv` was
rejected as the signal because `dockerd` creates it for `docker run`
and it is not reliably present during a BuildKit `docker build`, which
is the case the exception exists for. Version detection uses
`golangci-lint version --short` with the old banner scrape kept only
as a fallback. `script/bootstrap` no longer prints `bootstrap
complete` on a machine that cannot run the gate: a missing docker, or
one whose daemon is unreachable, is a hard failure naming exactly what
breaks. Verification was by reproduction rather than inspection — two
concurrent lints from two worktrees of differing cleanliness, a real
run made to report an outside path, a matching linter shimmed onto
`PATH`, and a `PATH` with docker removed — and is recorded on the pull
request.
- 2026-08-09: Closed the fifth false-green mechanism (issues #93, #69).
`script/test` omitted `-count=1`, so Go's test result cache could
satisfy the gate outright: a second back-to-back `make test` printed
the full set of 14 `ok` lines, every one marked `(cached)`, having
executed no test at all. Since `ok <pkg> (cached)` is an `ok` line,
the "14 `ok` lines means the suite ran" signal this repo leans on was
forgeable, one level below the Docker layer cache that #85 addressed.
Fixed with `-count=1` unconditionally rather than only in the
container, because the pre-commit hook runs the same script and a
gate honest only in CI is dishonest where people rely on it most;
`test-coverage` got the same flag, and `script/check` inherits it by
calling `script/test`. In the same area, `make test-integration` was
deleted rather than made real: no file in the repo carried a build
tag, so `-tags=integration` selected nothing and the target was an
exact duplicate of `make test`. Tagging a subset was rejected because
the entire suite runs in well under a minute, and a scheme whose
failure mode is "some tests silently stopped running" is a poor trade
for those seconds in a repo with this particular history. The
`-timeout` was raised from 30s after measuring rather than after
assuming: the standing claim that cold-cache compilation is charged
against `-timeout` is **false**, disproved by a containerised run
that spent 46s compiling and still reported per-package durations
within noise of a warm host run. `-timeout` reaches the test binary
as `-test.timeout` and its clock starts inside `testing.M.Run`, after
the build. The real exposure was margin, not compilation. The 120s
landed on is a **deliberate, documented divergence** from
`REPO_POLICIES.md:192`, which mandates 30s, and from that file's
canonical recipe at `:212-214`; the divergence is recorded in
`script/test`'s comment because `REPO_POLICIES.md` is org-canonical
and not editable here, and issue #101 proposes amending the policy
text upstream. Numbers and the full verification are recorded once,
on the pull request, and are deliberately not restated here.
- 2026-08-09: Triaged all fifteen stale remote branches (issue #71) and
deleted fourteen of them; the full per-branch disposition with
evidence is recorded on that issue. Method mattered more than the
outcome here: a three-dot `git diff main...branch` diffs from the
merge base, so it replays everything that landed on `main` after the
branch diverged and makes any old branch look like it holds unlanded
work. That artifact is what made `golangci-v2.12.2` appear to carry
126 files of unpushed changes when its tree was byte-identical to
`main`'s. Every containment claim here therefore rests on two-dot tip
diffs, tree-hash equality, `git cherry`, and `git branch -r --merged`.
Nine branches were plain ancestors of `main` with zero `git cherry`
`+` commits. `golangci-v2.12.2` had landed squashed as `cc58583`,
whose tree hash equals the branch tip's exactly; note the hash
recorded in the issue had gone stale because `main` advanced, so the
check had to be redone rather than repeated.
`fix/sync-snapshot-cleanup` was redundant, its one line already on
`main` in `syncWithRemote`. `feature/restore-progress-bar` was
superseded by `printRestoreProgress` and the disk-backed blob cache,
and had become actively regressive — it would have deleted
`internal/blobgen/compress_test.go`, the #28 regression test that
landed separately. The two branches this issue was filed for both
turned out to be closed questions that `main` had already moved past
by a recorded decision, so neither was landed and no regression test
was owed: `ctime` no longer exists anywhere in the codebase after
`1c72a37` removed the column, the `File.CTime` field and every use
(#54/#55), and change detection compares size, mtime, mode, uid and
gid only, exactly as `ARCHITECTURE.md` documents — so the
silently-skipped-file data-loss risk that made this a 1.0 item does
not exist. The SQL allow-list branch would have reverted `bfd7334`,
which replaced that very allow-list with regex sanitisation on review
feedback, and would have broken `getTableCount("snapshots")` because
its allow-list omits that table. `feature/daemon-mode` is untouched
and deferred to #94 pending an owner decision, so it is the one
branch besides `main` still on the remote. The stale `TODO.md` entry
named in the issue needed no fix: `e496aa3` had already removed it.
No product code changed.
- 2026-08-09: Adopted the remaining upstream `CHECK_EPOCH` hardening
(issue #91), closing the gap #85 knowingly left open. Four changes,
all four decided as adopt upstream in `sneak/prompts` #26. (1) Each
check stage now asserts `[ -n "$CHECK_EPOCH" ] || exit 1` before
running anything, so a build that supplies no `--build-arg` fails
instead of lying. This is the item that mattered: an unset `ARG` is
an empty string and an empty string is a stable cache key, so the
second and every later bare `docker build .` on an unchanged tree
replayed all three check layers and still exited 0 — and `docker
build .` is the command `REPO_POLICIES.md` names verbatim as a thing
that must be green, so the documented command was precisely the one
that lied. Failed steps are never cached, which is what makes the
guard fire on every invocation rather than once. (2) The epoch is now
expanded into each check command rather than left as a bare
declaration, so the cache miss no longer depends on BuildKit's
unreferenced-`ARG` handling staying as it is, and the value appears
in the build log. (3) `script/cibuild` uses
`epoch="$(date +%s%N)$$"`, unique per invocation rather than per
second; `%N` alone is insufficient because busybox drops it silently
and exits 0, and `$$` is what makes the guarantee hold regardless.
The bare-assignment form is kept deliberately — inlined in an
argument, a failing substitution does not abort under `set -eu` and
would yield an empty constant epoch, restoring the exact false green
being fixed. (4) `script/docker` passes the same fresh arg, so the
two entrypoints cannot disagree about whether the tree is green;
local builds are almost always warm, which made it the likelier
fooling in practice. The `ARG` placement from #85 is unchanged, below
`apk add`, `COPY go.mod go.sum` and `go mod download`, so dependency
layers still cache and the build is not cold. Verified by negative
control rather than inspection — a bare `docker build .` run twice
back to back, plus back-to-back pairs of both scripts and a host-side
`make check`; the measurements are recorded once, in the PR
verification comment, rather than restated here. `.golangci.yml`, the
lint-stage `FROM` line and its digest, `script/lint`,
`REPO_POLICIES.md` and `.gitea/workflows/check.yml` are all
untouched.
- 2026-08-09: Stopped `script/cibuild` from reporting a green it did
not earn (issue #85). A bare `docker build .` let Docker serve the
check layers from the layer cache whenever the tree had not changed:
the checks never executed and the build still exited 0. The fix is an
`ARG CHECK_EPOCH` declared immediately above the check `RUN`s in both
the lint stage and the builder stage (`ARG` scope is per-stage, so
each declares its own), with `script/cibuild` assigning
`epoch="$(date +%s)"` and passing `--build-arg CHECK_EPOCH="$epoch"`.
The assignment is separate on purpose: under `set -eu` a command
substitution that fails inside an argument does not abort the script,
which would leave an empty constant `CHECK_EPOCH` and restore the
very false green being fixed. Placement is the rest of the point —
the `ARG` sits below the `apk add`, `COPY go.mod go.sum`, and `go mod
download` layers, so only the checks are invalidated and the
dependency layers still cache. The guarantee is conditional on a
fresh value rather than absolute: a bare `docker build .` gets an
empty `CHECK_EPOCH` and can still serve the check layers from cache,
which `README.md` and the `Dockerfile` now say plainly, with issue
#91 tracking the upstream hardening (expanded `ARG` form, unset
guard, per-invocation epoch, `script/docker`) that would close it.
Verified by re-running the reproduction plus the withheld-`--build-arg`
counterfactual; the measurements are recorded once, in the PR #89
verification comment, rather than restated here. `.golangci.yml`, the
lint-stage `FROM` line and its digest, `script/lint`, and
`.gitea/workflows/check.yml` are all untouched.
- 2026-08-09: Corrected the `Vaultik.UI` doc comment (issue #84). It
claimed the cli layer replaces the writer with a discarding one in
`--cron` mode; the actual mechanism is `UI.SetQuiet(true)` in
`setupGlobals`, which drops Begin/Complete/Info/Notice/Detail/
Progress/Banner but still emits Warning and Error. The `--cron` line
in `README.md` said "Silent unless error", which understated what
survives, and now names warnings too. The other `--cron` comments
(`internal/log/log.go`, `internal/cli/snapshot.go`,
`internal/vaultik/snapshot.go`) were audited and already accurate.
Comments and docs only, no behavior change.
- 2026-08-09: Made `snapshot list` list the destination store without
the private key (issue #64). The listing is now the union of the
local index and a single streamed listing of the `metadata/` prefix,
with no `age_secret_key` gate — the manifest is unencrypted, so a
host holding only the public key can enumerate its own backups and a
host that lost its local index can still see them. A remote-only
snapshot's hostname and name are deliberately not recovered (they are
not recoverable without the private key, and making them so would
undo the privacy property tracked in issue #81); such rows are
labelled by an abbreviation of their remote key and carry the real
timestamp and compressed size from the manifest, with `<remote only>`
in the two columns that require the local index. Local-only snapshots
are reported as drift, and the hint now names `vaultik prune`, which
exists, instead of `vaultik snapshot cleanup`, which does not.
`reportRemoteDrift` collapsed into the merged view. Every remote
manifest read in the codebase now goes through
`downloadManifestByKey`, so issue #81 has one call site to change.
Review rework: snapshot timestamps now normalize to UTC in
`scanSnapshotRows`, the one place they enter the domain, so the merged
TIMESTAMP column cannot show local time for a locally tracked row and
UTC for a remote-only row on a non-UTC host; `GetIncompleteByHostname`
was folded onto that same scanner. `--json` now reports the
unreadable-manifest count and the 1000-row truncation on stderr
instead of returning a silently short document (the document's shape
is unchanged). The two per-snapshot `log.Warn` calls on the listing
path now route through the same JSON-aware writer as the existing
workaround, so one corrupt manifest can no longer put a log line on
stdout ahead of the document and break `| jq` — still a local
workaround pending issue #82. Verified with `script/cibuild` and with
an uncached `make check` (`0 issues.`, no cached test packages), plus
end to end against a `file://` destination with no secret key present.
- 2026-08-09: Closed the gap between `make lint` and CI (issue #78).
`script/lint` now runs the digest-pinned `golangci-lint` image taken
from the `Dockerfile` lint stage, which is the single source of truth
for the linter version; the duplicate pin in the `Makefile` `deps`
target and the unpinned `golangci-lint` install in `script/bootstrap`
are gone. A `golangci-lint` on `PATH` is used only when its version is
exactly the pinned one (which is how the lint stage runs it inside the
container); anything else goes through Docker, and a missing or
unreachable Docker daemon is a hard error rather than a silent
fallback. Only the **lint** leg of `make check` became equivalent to
`script/cibuild`; its tests and `gofmt` still run on the host against
the host toolchain, as `README.md` states. An earlier version of this
entry claimed `make check` was "as trustworthy as `script/cibuild`"
outright, which overstated it; corrected under issue #80.
- 2026-08-09: Finished the lint remediation under the canonical
`.golangci.yml` (issue #61, which also unblocks issue #59). The
remaining findings were fixed behavior-preservingly: `wsl_v5`
whitespace, `sqlclosecheck`, and `prealloc`. The `sqlclosecheck` sites
now close `sql.Rows` in a deferred closure instead of via the
`CloseRows` helper, which the linter could not see through. Only the
`revive` package-name findings remain suppressed, with per-site
`//nolint` directives; the package-rename question behind them is
tracked in issue #76. Verified with `script/cibuild`, which exits 0 —
that is the only trustworthy gate, because `script/lint` runs whatever
`golangci-lint` happens to be on `PATH` rather than the pinned
v2.12.2 that CI and the `Dockerfile` use, so `make check` can report
green on findings CI still fails. That tooling gap is tracked in issue
#78.
- 2026-08-09: The earlier next step "reconcile the uncommitted
`ARCHITECTURE.md` edits on `main`" needed no work: the working tree is
clean and `ARCHITECTURE.md` is committed on `main`.
- 2026-08-07: Updated golangci-lint to v2.12.2 everywhere it is pinned
(`Dockerfile` lint stage, `Makefile` deps target), replaced
`.golangci.yml` with the canonical config (v2 schema, `default: all`),
and remediated the bulk of the lint findings it surfaced (issue #61):
behavior-preserving fixes across every package, 2,990 findings down to
80. `make test` and `make fmt-check` were green at that point but
`make lint` was still red; the commit message claiming `make check`
was green was wrong.
- 2026-08-07: Added the standard `.golangci.yml` and `.editorconfig`
(issue #59); lint findings under the new config are tracked in issue
#61.
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section
- 2026-07-02: Consolidated CLI verbs, retired overlapping commands; bound
the local index to its backup destination URL.
- 2026-06-28: snapshot rm now removes metadata only and prints the prune
command; restore skips chown when running as non-root.
- 2026-06-26: Snapshot IDs hashed at the storage boundary; snapshot list
made resilient to bad remote entries.
- 2026-06-24: Collapsed snapshot prune into vaultik prune; restore streams
blobs to disk and restores files in blob-locality order; cron output
fixes.
- 2026-06-17: Restore overhaul: ReadAt chunk reads from cached blobs,
reference-counted blob sweeper, integration tests; new internal/ui
output layer, banner, and progress lines.
- 2025-12-18: Added ARCHITECTURE.md and godoc coverage for exported API.
- 2025-07-26: End-to-end integration tests; manifest format refactor;
renamed backup to snapshot; afero filesystem abstraction.
- 2025-07-20: Initial design and implementation: cobra + fx CLI skeleton,
SQLite index database, UUID blob storage with streaming chunking.
# Future Steps
None queued; the release-scoping item is now the Next Step.
+102
View File
@@ -0,0 +1,102 @@
package main_test
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// This file guards the version stamping of the product image (issue
// #75). The failure it protects against is silent: the image still
// builds and runs, but `vaultik version` inside it reports "commit:
// unknown", so an operator cannot tell which source produced a given
// backup. .dockerignore excludes .git, so the build cannot derive the
// commit itself; the values must be computed on the host and passed in.
//
// These are parses of the committed files, for the same reason the lint
// guards next door are: shelling out to docker would nest a build
// inside `make test`. That `vaultik version` in the built image really
// prints the host's version is verified by hand and recorded on the
// pull request.
// dockerScript is script/docker, relative to the repository root.
const dockerScript = "script/docker"
// versionArgs are the ldflag targets the build stamps and, matching
// them, the build args the host must supply. The names line up so the
// same list checks both files.
func versionArgs() []string {
return []string{"VERSION", "COMMIT", "COMMIT_DATE"}
}
// TestProductDockerfileTakesVersionAsBuildArgs fails unless the build
// declares each version arg and stamps it into the binary by ldflag
// reference, rather than computing it in the container.
func TestProductDockerfileTakesVersionAsBuildArgs(t *testing.T) {
t.Parallel()
found := instructions(t, productDockerfile)
for _, arg := range versionArgs() {
require.GreaterOrEqual(t, indexOf(found, "ARG "+arg), 0,
"%s must declare `ARG %s` so the host can pass it in",
productDockerfile, arg)
assertLdflagReferences(t, found, arg)
}
}
// TestProductDockerfileDoesNotDeriveVersionItself is the anti-regression
// for the original defect: the container ran `git rev-parse`, but .git
// is not in the build context, so it always resolved to "unknown". No
// git command may reach into a build that cannot see the history.
func TestProductDockerfileDoesNotDeriveVersionItself(t *testing.T) {
t.Parallel()
text := instructionText(readRepoFile(t, productDockerfile))
assert.NotContains(t, text, "git ",
"%s must not run git: .git is excluded from the build context, so"+
" any value it derives is wrong. Pass version, commit and date"+
" in as build args instead.", productDockerfile)
}
// TestDockerScriptComputesVersionOnTheHost fails unless script/docker
// derives each value where .git exists and passes it as a build arg,
// with VERSION coming from script/version so a Docker build reports the
// same string a local build of the same tree would.
func TestDockerScriptComputesVersionOnTheHost(t *testing.T) {
t.Parallel()
script := readRepoFile(t, dockerScript)
for _, arg := range versionArgs() {
assert.Contains(t, script, "--build-arg "+arg+"=",
"%s must pass --build-arg %s to the build", dockerScript, arg)
}
assert.Contains(t, script, "/version",
"%s must take VERSION from script/version, the source of truth"+
" shared with the Makefile", dockerScript)
}
// assertLdflagReferences fails unless some build instruction stamps the
// named variable from the ARG (a ${arg} reference), not from a value
// computed inside the container.
func assertLdflagReferences(t *testing.T, found []string, arg string) {
t.Helper()
for _, instruction := range found {
if strings.HasPrefix(instruction, "RUN ") &&
strings.Contains(instruction, "go build") &&
strings.Contains(instruction, "${"+arg+"}") {
return
}
}
assert.Fail(t, "version arg is declared but never stamped",
"the go build in %s must reference ${%s} in its ldflags, or the"+
" arg is passed and discarded", productDockerfile, arg)
}
+367
View File
@@ -0,0 +1,367 @@
package main_test
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// This file guards the shape of the lint gate. Every property asserted
// here is one whose loss is SILENT: the build still exits 0, the gate
// still looks green, and nothing was linted or tested.
//
// The gate is a build step. script/lint builds Dockerfile.lint, which
// runs golangci-lint as a RUN instruction, so a successful build is a
// clean lint. BuildKit will happily replay that RUN from cache on an
// unchanged tree in well under a second, which is why the check layers
// are keyed on a CHECK_EPOCH build arg that the calling script
// regenerates per invocation, and why an empty value is a hard error
// rather than a stable cache key.
//
// These are parses rather than invocations. Shelling out to docker from
// the test suite would nest a build inside `make test`, which itself
// runs inside a build in CI. The one property a parse cannot establish
// -- that a real finding actually fails the build -- is verified by
// hand against a deliberately broken tree, recorded on the pull
// request.
//
// One property is deliberately NOT tested here: that no script runs the
// linter on the host. script/lint is the only lint entry point, and it
// runs golangci-lint only inside the container; keeping it that way is a
// review matter, not something a test in this file establishes.
// The files under guard, relative to the repository root.
const (
lintDockerfile = "Dockerfile.lint"
productDockerfile = "Dockerfile"
lintScript = "script/lint"
cibuildScript = "script/cibuild"
)
// linterBinary is the linter's command name, used to locate the
// config-verify and lint steps in Dockerfile.lint.
const linterBinary = "golangci-lint"
// checkEpochARG is the declaration, with no default value. A default
// would satisfy the non-empty guard with a constant, and a constant is
// a stable cache key: the checks would be replayed from cache forever
// after the first build.
const checkEpochARG = "ARG CHECK_EPOCH"
// checkEpochGuard is what turns a build that omits --build-arg into a
// loud failure instead of a quiet green. Failed steps are never cached,
// so it fires on every such invocation rather than once.
const checkEpochGuard = `RUN [ -n "$CHECK_EPOCH" ] || exit 1`
// freshEpoch is the epoch computation the calling scripts must use, as
// a bare assignment on its own line. Inline in an argument, a failing
// `date` would not abort under `set -eu`; CHECK_EPOCH would become the
// empty string, and the guard above would be the only thing standing
// between that and a permanently cached green. `$$` is required because
// `date +%s` is second-granular and busybox silently drops `%N`, so
// without the pid two concurrent runs in one second can collide.
const freshEpoch = `epoch="$(date +%s%N)$$"`
// TestLintDockerfilePinsTheLinterByDigest fails if the lint image stops
// being pinned. An unpinned tag makes the gate's verdict depend on
// whatever the registry currently serves under that name.
func TestLintDockerfilePinsTheLinterByDigest(t *testing.T) {
t.Parallel()
from := ""
for _, instruction := range instructions(t, lintDockerfile) {
if strings.HasPrefix(instruction, "FROM ") {
from = instruction
break
}
}
require.NotEmpty(t, from, "%s declares no FROM", lintDockerfile)
assert.Contains(t, from, "golangci/golangci-lint",
"the lint image must be the golangci-lint image")
assert.Contains(t, from, "@sha256:",
"the lint image must be pinned by digest, not by tag alone")
}
// TestLintDockerfileCannotBeCachedGreen pins the whole cache-busting
// mechanism in the file that lints: the declaration with no default,
// the non-empty guard, and the value expanded into the lint command
// itself rather than merely declared.
func TestLintDockerfileCannotBeCachedGreen(t *testing.T) {
t.Parallel()
found := instructions(t, lintDockerfile)
argAt := indexOf(found, checkEpochARG)
require.GreaterOrEqual(t, argAt, 0,
"%s must declare `%s` with no default value",
lintDockerfile, checkEpochARG)
assert.GreaterOrEqual(t, indexOf(found, checkEpochGuard), argAt,
"%s must guard against an empty CHECK_EPOCH with `%s`",
lintDockerfile, checkEpochGuard)
assertEpochExpandedInto(t, found[argAt:], "golangci-lint run")
// Dependency layers must stay above the ARG, or every lint run
// re-downloads the module cache and the inner loop becomes
// unusable.
download := indexOf(found, "RUN go mod download")
require.GreaterOrEqual(t, download, 0,
"%s must download modules in their own layer", lintDockerfile)
assert.Less(t, download, argAt,
"`%s` must come after `go mod download` so dependency layers"+
" still cache", checkEpochARG)
}
// TestLintDockerfileVerifiesTheLinterConfig guards the validation of
// .golangci.yml itself. `golangci-lint run` rejects a config it cannot
// parse but silently IGNORES an unknown top-level key, so renaming
// `linters:` to `linterz:` discards `default: all` and every threshold
// and still exits 0 reporting no issues. `config verify` is what turns
// that into a failure, and it has to run BEFORE the lint, or the lint
// spends a minute reporting a verdict from a config already known to be
// wrong.
func TestLintDockerfileVerifiesTheLinterConfig(t *testing.T) {
t.Parallel()
found := instructions(t, lintDockerfile)
verify := linterBinary + " config verify"
verifyAt := indexContaining(found, verify)
require.GreaterOrEqual(t, verifyAt, 0,
"%s must run `%s --config .golangci.yml`: without it a typo'd"+
" top-level key in .golangci.yml is silently ignored and the"+
" gate passes with only the default linter set", lintDockerfile,
verify)
runAt := indexContaining(found, linterBinary+" run")
require.GreaterOrEqual(t, runAt, 0, "%s must lint", lintDockerfile)
assert.Less(t, verifyAt, runAt,
"%s must verify the config before linting with it", lintDockerfile)
// Keyed on the epoch like every other check layer, so it executes
// per invocation rather than being replayed. A cached validation
// validates nothing.
assertEpochExpandedInto(t, found, verify)
}
// TestProductDockerfileCannotBeCachedGreen holds the same line for the
// checks that remain in the product image build.
func TestProductDockerfileCannotBeCachedGreen(t *testing.T) {
t.Parallel()
found := instructions(t, productDockerfile)
argAt := indexOf(found, checkEpochARG)
require.GreaterOrEqual(t, argAt, 0,
"%s must declare `%s` with no default value",
productDockerfile, checkEpochARG)
assert.GreaterOrEqual(t, indexOf(found, checkEpochGuard), argAt,
"%s must guard against an empty CHECK_EPOCH", productDockerfile)
assertEpochExpandedInto(t, found[argAt:], "make fmt-check")
assertEpochExpandedInto(t, found[argAt:], "make test")
}
// TestProductDockerfileDoesNotLint records the split deliberately: the
// linter lives in Dockerfile.lint and nowhere else, so there is exactly
// one digest pinning it. A lint stage reintroduced here would either be
// docker-in-docker (`make lint` is now `docker build`) or a second,
// independently bumpable pin.
func TestProductDockerfileDoesNotLint(t *testing.T) {
t.Parallel()
contents := readRepoFile(t, productDockerfile)
for _, forbidden := range []string{"golangci", "make lint"} {
assert.NotContains(t, instructionText(contents), forbidden,
"%s must not lint: the linter is pinned once, in %s",
productDockerfile, lintDockerfile)
}
}
// TestLintScriptBuildsTheLintDockerfileWithAFreshEpoch is the other
// half of the mechanism. The Dockerfile's guard only rejects an EMPTY
// epoch; a constant non-empty one would satisfy it and still be served
// from cache forever.
func TestLintScriptBuildsTheLintDockerfileWithAFreshEpoch(t *testing.T) {
t.Parallel()
script := readRepoFile(t, lintScript)
assertBareEpochAssignment(t, script, lintScript)
assert.Contains(t, script, `--build-arg CHECK_EPOCH="$epoch"`,
"%s must pass the fresh epoch to the build", lintScript)
assert.Contains(t, script, lintDockerfile,
"%s must build %s", lintScript, lintDockerfile)
}
// TestCibuildBuildsBothDockerfilesWithFreshEpochs guards the CI gate:
// dropping either build silently removes a whole class of check from
// CI while leaving it green.
func TestCibuildBuildsBothDockerfilesWithFreshEpochs(t *testing.T) {
t.Parallel()
script := readRepoFile(t, cibuildScript)
assertBareEpochAssignment(t, script, cibuildScript)
assert.Equal(t, 2, strings.Count(script, freshEpoch),
"%s must compute a fresh epoch for each of its two builds",
cibuildScript)
assert.Equal(t, 2,
strings.Count(script, `--build-arg CHECK_EPOCH="$epoch"`),
"%s must pass a fresh epoch to both builds", cibuildScript)
assert.Contains(t, script, "-f Dockerfile.lint",
"%s must build %s", cibuildScript, lintDockerfile)
}
// assertEpochExpandedInto fails unless some instruction runs the named
// command with the epoch expanded into it. Expansion, not mere
// declaration: an ARG that no instruction references is not guaranteed
// to key the layer, and the expansion also puts the value in the build
// log where a reader can see the layer was keyed fresh.
func assertEpochExpandedInto(t *testing.T, found []string, command string) {
t.Helper()
for _, instruction := range found {
if !strings.HasPrefix(instruction, "RUN ") {
continue
}
if strings.Contains(instruction, command) &&
strings.Contains(instruction, "${CHECK_EPOCH}") {
return
}
}
assert.Fail(t, "no epoch-keyed layer runs the command",
"`%s` must run in a layer that expands ${CHECK_EPOCH}, or it"+
" will be replayed from cache without executing", command)
}
// assertBareEpochAssignment fails unless the script computes the epoch
// as a bare assignment on its own line.
func assertBareEpochAssignment(t *testing.T, script, name string) {
t.Helper()
for line := range strings.SplitSeq(script, "\n") {
if strings.TrimSpace(line) == freshEpoch {
return
}
}
assert.Fail(t, "no bare epoch assignment",
"%s must compute `%s` as a bare assignment on its own line, so"+
" `set -e` catches a failing date instead of quietly"+
" building with an empty epoch", name, freshEpoch)
}
// instructions returns the Dockerfile's instructions, one per element,
// with comments and blank lines dropped and continuation lines joined,
// so a multi-line RUN is one string.
func instructions(t *testing.T, name string) []string {
t.Helper()
return strings.Split(instructionText(readRepoFile(t, name)), "\n")
}
// instructionText is instructions' parse, before splitting: it is also
// what a "must not contain" assertion should look at, so that a word
// appearing only in a comment is not mistaken for behaviour.
func instructionText(contents string) string {
var (
out []string
continued string
isContinued bool
)
for line := range strings.SplitSeq(contents, "\n") {
trimmed := strings.TrimSpace(line)
if !isContinued && (trimmed == "" || strings.HasPrefix(trimmed, "#")) {
continue
}
isContinued = strings.HasSuffix(trimmed, `\`)
continued += strings.TrimSuffix(trimmed, `\`)
if isContinued {
continue
}
out = append(out, strings.Join(strings.Fields(continued), " "))
continued = ""
}
return strings.Join(out, "\n")
}
// indexOf returns the position of the first instruction equal to, or
// beginning with, want; -1 if there is none. An `ARG NAME=default`
// counts as beginning with `ARG NAME`, so a declared arg is found
// whether or not it carries a default.
func indexOf(found []string, want string) int {
for i, instruction := range found {
if instruction == want ||
strings.HasPrefix(instruction, want+" ") ||
strings.HasPrefix(instruction, want+"=") {
return i
}
}
return -1
}
// indexContaining returns the position of the first instruction
// containing want; -1 if there is none.
func indexContaining(found []string, want string) int {
for i, instruction := range found {
if strings.Contains(instruction, want) {
return i
}
}
return -1
}
// readRepoFile reads a file by its path relative to the repository
// root.
func readRepoFile(t *testing.T, name string) string {
t.Helper()
//nolint:gosec // G304: the path is a constant relative to this repo
contents, err := os.ReadFile(filepath.Join(repoRoot(t), name))
require.NoError(t, err)
return string(contents)
}
// repoRoot returns the repository root. The test binary runs with its
// package directory as the working directory, so the root is found by
// walking up until the module file appears.
func repoRoot(t *testing.T) string {
t.Helper()
dir, err := os.Getwd()
require.NoError(t, err)
for {
_, err = os.Stat(filepath.Join(dir, "go.mod"))
if err == nil {
return dir
}
parent := filepath.Dir(dir)
require.NotEqual(t, dir, parent,
"walked to the filesystem root without finding a go.mod")
dir = parent
}
}
+24 -5
View File
@@ -1,3 +1,4 @@
// Package main is the vaultik command-line entry point.
package main
import (
@@ -9,33 +10,51 @@ import (
)
func main() {
os.Exit(run())
}
// run sets up optional profiling, runs the CLI, and returns the process
// exit code. os.Exit lives in main so it fires only after run's deferred
// profile writers have flushed. cli.Entry returns a status code rather
// than calling os.Exit itself: an os.Exit from inside it would skip
// these defers and truncate the profile of a failing command -- exactly
// the command one most often wants to profile.
func run() int {
// CPU profiling: set VAULTIK_CPUPROFILE=/path/to/cpu.prof
if cpuProfile := os.Getenv("VAULTIK_CPUPROFILE"); cpuProfile != "" {
f, err := os.Create(cpuProfile)
f, err := os.Create(cpuProfile) //nolint:gosec // G304: operator-set path
if err != nil {
panic("could not create CPU profile: " + err.Error())
}
defer func() { _ = f.Close() }()
if err := pprof.StartCPUProfile(f); err != nil {
err = pprof.StartCPUProfile(f)
if err != nil {
panic("could not start CPU profile: " + err.Error())
}
defer pprof.StopCPUProfile()
}
// Memory profiling: set VAULTIK_MEMPROFILE=/path/to/mem.prof
if memProfile := os.Getenv("VAULTIK_MEMPROFILE"); memProfile != "" {
defer func() {
f, err := os.Create(memProfile)
f, err := os.Create(memProfile) //nolint:gosec // G304: operator-set path
if err != nil {
panic("could not create memory profile: " + err.Error())
}
defer func() { _ = f.Close() }()
runtime.GC() // get up-to-date statistics
if err := pprof.WriteHeapProfile(f); err != nil {
err = pprof.WriteHeapProfile(f)
if err != nil {
panic("could not write memory profile: " + err.Error())
}
}()
}
cli.CLIEntry()
return cli.Entry()
}
+151
View File
@@ -0,0 +1,151 @@
package main_test
import (
"regexp"
"slices"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// This file guards the Makefile that builds this program, which is why
// it lives beside it rather than in a package of its own.
//
// Issue #110: `build` was listed in .PHONY with no `build:` rule
// anywhere in the file. That combination is silently successful — make
// considers a phony target with no prerequisites and no recipe already
// satisfied, so `rm -f vaultik && make build` printed "Nothing to be
// done for 'build'" and exited 0 with no binary produced. Declaring the
// name phony is precisely what converts the "No rule to make target"
// error into a green.
//
// The guard is a parse of the Makefile rather than an invocation of
// make. `make test` is what runs these tests, so shelling back into
// `make build` here would nest a build inside the test run and drop a
// binary into the tree as a side effect of testing. The one property a
// parse cannot establish — that the recipe still fails when the build
// fails — is not testable from inside the build either; it is verified
// by hand against a deliberately broken tree.
// phonyDirective introduces the list of phony target names.
const phonyDirective = ".PHONY:"
// ruleLine matches a rule's target list: a target starts in column
// zero, so recipe lines (tab-indented) and the continuation lines of a
// variable assignment (space-indented) are excluded by construction.
//
// The trailing (?:[^=]|$) rejects `:=` assignments such as
// `VERSION := $(shell script/version)`, which are not rules. Directives
// and function calls (`.PHONY:`, `ifeq`, `$(error ...)`) do not match
// because a target here must begin with a letter, digit or underscore.
var ruleLine = regexp.MustCompile(`^([A-Za-z0-9_][A-Za-z0-9_./ -]*):(?:[^=]|$)`)
// TestPhonyTargetsAllHaveRules fails on any name in .PHONY that has no
// rule in the Makefile. Such a name is not a build target at all: it is
// a command that reports success without doing anything, which is worse
// than one that does not exist, because a caller checking the exit code
// cannot tell the difference.
func TestPhonyTargetsAllHaveRules(t *testing.T) {
t.Parallel()
makefile := readMakefile(t)
phony := phonyTargets(makefile)
require.NotEmpty(t, phony, "no .PHONY names found; the parser is broken")
rules := declaredRules(makefile)
// Sanity check on the rule parser before trusting its verdict: a
// parser that found nothing would pass this test by accident.
require.Contains(t, rules, "vaultik",
"the file rule that builds the binary must be recognized")
for _, target := range phony {
assert.Contains(t, rules, target,
"`.PHONY` lists %q but the Makefile declares no %q rule, so "+
"`make %s` exits 0 without doing anything", target, target, target)
}
}
// TestBuildTargetBuildsTheBinary pins the specific shape of issue #110:
// `make build` has to reach the rule that produces the binary. The test
// above would also pass if `build:` were given an empty recipe of its
// own, which would be the same silent success under a different
// spelling.
func TestBuildTargetBuildsTheBinary(t *testing.T) {
t.Parallel()
prerequisites := rulePrerequisites(readMakefile(t), "build")
require.NotNil(t, prerequisites, "the Makefile declares no `build` rule")
assert.Contains(t, prerequisites, "vaultik",
"`make build` must depend on the rule that builds the binary")
}
// readMakefile returns the contents of the repository's Makefile. The
// root is located by the shared walk in lintdocker_test.go.
func readMakefile(t *testing.T) string {
t.Helper()
return readRepoFile(t, "Makefile")
}
// phonyTargets returns every name declared phony, across all .PHONY
// lines.
func phonyTargets(makefile string) []string {
var targets []string
for line := range strings.SplitSeq(makefile, "\n") {
if !strings.HasPrefix(line, phonyDirective) {
continue
}
targets = append(targets,
strings.Fields(strings.TrimPrefix(line, phonyDirective))...)
}
return targets
}
// declaredRules returns the set of target names that have a rule.
func declaredRules(makefile string) map[string]bool {
rules := make(map[string]bool)
for line := range strings.SplitSeq(makefile, "\n") {
match := ruleLine.FindStringSubmatch(line)
if match == nil {
continue
}
// One rule may name several targets: `a b: prereq`.
for target := range strings.FieldsSeq(match[1]) {
rules[target] = true
}
}
return rules
}
// rulePrerequisites returns the prerequisites of the named rule, or nil
// if no such rule exists. A rule with none returns an empty slice, so
// "declared with nothing to do" is distinguishable from "not declared".
func rulePrerequisites(makefile, target string) []string {
for line := range strings.SplitSeq(makefile, "\n") {
match := ruleLine.FindStringSubmatch(line)
if match == nil {
continue
}
if !slices.Contains(strings.Fields(match[1]), target) {
continue
}
_, after, _ := strings.Cut(line, ":")
return append([]string{}, strings.Fields(after)...)
}
return nil
}
+10 -5
View File
@@ -257,16 +257,16 @@ exclude:
# Storage URL - use either this OR the s3 section below
# Supports: s3://bucket/prefix, file:///path, rclone://remote/path
storage_url: "rclone://las1stor1//srv/pool.2024.04/backups/heraklion"
storage_url: "rclone://myremote/path/to/backups"
# S3-compatible storage configuration
#s3:
# # S3-compatible endpoint URL
# # Examples: https://s3.amazonaws.com, https://storage.googleapis.com
# endpoint: http://10.100.205.122:8333
# endpoint: https://s3.example.com
#
# # Bucket name where backups will be stored
# bucket: testbucket
# bucket: mybucket
#
# # Prefix (folder) within the bucket for this host's backups
# # Useful for organizing backups from multiple hosts
@@ -274,8 +274,8 @@ storage_url: "rclone://las1stor1//srv/pool.2024.04/backups/heraklion"
# #prefix: "hosts/myserver/"
#
# # S3 access credentials
# access_key_id: Z9GT22M9YFU08WRMC5D4
# secret_access_key: Pi0tPKjFbN4rZlRhcA4zBtEkib04yy2WcIzI+AXk
# access_key_id: YOUR_ACCESS_KEY
# secret_access_key: YOUR_SECRET_KEY
#
# # S3 region
# # Default: us-east-1
@@ -304,6 +304,11 @@ storage_url: "rclone://las1stor1//srv/pool.2024.04/backups/heraklion"
# Maximum blob size
# Multiple chunks are packed into blobs up to this size
# Must be at least four times chunk_size (the largest chunk the chunker can
# emit); a smaller limit would let a single-chunk blob exceed it.
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
# not visible in its size; lowering the limit toward chunk_size exposes them.
# Supports: 1GB, 10G, 500MB, 1GiB, etc.
# Default: 10GB
#blob_size_limit: 10GB
+25 -9
View File
@@ -5,15 +5,29 @@
Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication.
**Important Notes:**
- **No Migration Support (pre-1.0)**: Vaultik does not support database schema
migrations. The local index is treated as disposable — if the schema changes,
delete the local SQLite database (`vaultik database purge`) and run a full
backup. The remote storage is unaffected; the new index will re-deduplicate
against existing remote blobs.
- **Version Compatibility**: In rare cases, you may need to use the same version
of Vaultik to restore a backup as was used to create it. This ensures
compatibility with the metadata format stored in S3.
## Schema Migrations
Vaultik supports schema migrations. They are the numbered SQL files in
`internal/database/schema/`, embedded in the binary: `000.sql` creates the
`schema_migrations` table, which records each migration that has run, and
`001.sql` creates the application tables. `database.New` opens a database and
applies, in order, every migration that database has not yet recorded.
**Before 1.0** no migrations are added, because nothing is installed anywhere
yet. A schema change edits `001.sql` (and the code that uses the affected
tables) directly. A local database created before the change has already
recorded `001.sql` as run, so it keeps the old schema and can become unusable;
`vaultik database delete` followed by a full backup rebuilds it.
**After 1.0** each schema change is a new numbered file, so an existing local
database is migrated the first time an updated vaultik opens it. A file that
has shipped in a release is never edited.
## Database Tables
### 1. `files`
@@ -71,7 +85,7 @@ Stores information about packed, compressed, and encrypted blob files.
**Columns:**
- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts
- `blob_hash` (TEXT UNIQUE) - SHA256 hash of final blob (NULL until finalized)
- `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (NULL until finalized); see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs)
- `created_ts` (INTEGER NOT NULL) - Creation timestamp
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
@@ -192,10 +206,12 @@ Tracks blob upload metrics.
After a snapshot is completed:
1. Copy database to temporary file
2. Clean temporary database to contain only current snapshot data
3. Export to SQL dump using sqlite3
3. VACUUM the trimmed database so deleted rows leave no pages behind
4. Compress with zstd and encrypt with age
5. Upload to S3 as `metadata/{snapshot-id}/db.zst.age`
6. Generate blob manifest and upload as `metadata/{snapshot-id}/manifest.json.zst`
5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext. The hash uses no secret, so a guessed hostname and snapshot name can still be confirmed against a listing; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its [Accepted Risks](REPOSTRUCTURE.md#accepted-risks).
### 4. Restore Process
+55 -22
View File
@@ -17,11 +17,13 @@ Vaultik stores all backup data in an S3-compatible object store. The repository
│ └── <hash[2:4]>/
│ └── <full-hash>
└── metadata/
└── <snapshot-id>/
└── <remote-key>/
├── db.zst.age
└── manifest.json.zst
```
The metadata subdirectory is named with the **remote key**, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See [Remote Key Derivation](#remote-key-derivation).
## Blobs Directory (`blobs/`)
### Structure
@@ -33,16 +35,18 @@ Vaultik stores all backup data in an S3-compatible object store. The repository
- **What it contains**: Packed collections of content-defined chunks from files
- **Format**: Zstandard compressed, then Age encrypted
- **Encryption**: Always encrypted with Age using the configured recipients
- **Naming**: Content-addressed using SHA256 hash of the encrypted blob
- **Naming**: Content-addressed. The blob's name is `hex(SHA256(SHA256(uncompressed blob contents)))` — the double SHA-256 of the concatenated chunk data, computed before compression and encryption, not a hash of the stored (compressed, encrypted) bytes. One consequence: only a holder of the age private key can check a stored blob's integrity, because matching a blob to its name means decrypting and decompressing it first — which is what `restore` and `verify --deep` do. Implemented in `internal/blobgen` (`DoubleSHA256`). This is the canonical description of blob naming; other documents and comments point here.
### Why Encrypted
Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy.
## Metadata Directory (`metadata/`)
Each snapshot has its own subdirectory named with the snapshot ID.
Each snapshot has its own subdirectory. The directory is **not** named with the human-readable snapshot ID; it is named with the remote key — a one-way hash of that ID. The human ID is never written to the destination store as a directory name (see [Remote Key Derivation](#remote-key-derivation)).
### Snapshot ID Format
The human-readable snapshot ID is used in CLI arguments, log lines, and the local database. It is not written to the destination store.
- **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>` if no
name was specified)
- **Example**: `laptop_home_2024-01-15T14:30:52Z`
@@ -51,6 +55,19 @@ Each snapshot has its own subdirectory named with the snapshot ID.
- Snapshot name from the configured `snapshots:` map (optional)
- RFC3339 UTC timestamp
This ID reveals the hostname, the configured snapshot name, and the backup time, so it is never used as the on-disk directory name — the remote key is used instead.
### Remote Key Derivation
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string. The hash is not reversible, but it uses no secret: an observer who guesses a candidate hostname and snapshot name can hash it the same way and confirm whether that snapshot is present. The remote key keeps names out of a plain listing; it does not hide them from a guess. Implemented in `internal/snapshot/remotekey.go`.
Worked example:
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
A plain listing of the destination store therefore shows only these hashes, not the hostname or snapshot name of any backup — but because the hash uses no secret, a guessed hostname and snapshot name can be hashed and confirmed against the listing. The same remote key is stored in the manifest's `snapshot_id` field.
### Files in Each Snapshot Directory
#### `db.zst.age` - Encrypted Database
@@ -68,16 +85,17 @@ Each snapshot has its own subdirectory named with the snapshot ID.
- **Structure**:
```json
{
"snapshot_id": "laptop_home_2024-01-15T14:30:52Z",
"timestamp": "2024-01-15T14:30:52Z",
"snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
"timestamp": "2025-06-01T12:00:00Z",
"blob_count": 42,
"total_compressed_size": 1048576,
"blobs": [
"cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
"deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678",
...
{ "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 24576 },
{ "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 32768 }
]
}
```
`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is written in the clear.
### Why Manifest is Unencrypted
The manifest must be readable without the private key to enable:
@@ -86,7 +104,7 @@ The manifest must be readable without the private key to enable:
3. **Verification** - Checking blob existence without decryption
4. **Cross-snapshot deduplication analysis** - Finding shared blobs between snapshots
The manifest only contains blob hashes, not file names or any other sensitive information.
The manifest contains the remote key, the backup timestamp, the blob count and total compressed size, and each blob's hash and compressed size. It contains no file names, paths, or other decrypted metadata.
## Security Considerations
@@ -96,39 +114,54 @@ The manifest only contains blob hashes, not file names or any other sensitive in
- **File-to-chunk mappings** (in db.zst.age)
### What's Not Encrypted
- **Blob hashes** (in manifest.json.zst)
- **Snapshot IDs** (directory names)
- **Blob count per snapshot** (in manifest.json.zst)
- **The remote key** — directory names and the manifest `snapshot_id`, a one-way hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation))
- **The backup timestamp** (in manifest.json.zst)
- **Blob hashes and their compressed sizes** (in manifest.json.zst)
- **Blob count and total compressed size per snapshot** (in manifest.json.zst)
### Privacy Implications
From the unencrypted data, an observer can determine:
- When backups were taken (from snapshot IDs)
- Which hostname created backups (from snapshot IDs)
- How many blobs each snapshot references
- Which blobs are shared between snapshots (deduplication patterns)
- The size of each encrypted blob
From the unencrypted data, an observer of the destination store can determine:
- **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear
- How many blobs each snapshot references, and the total compressed size
- The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)
- **Whether a guessed hostname and snapshot name are present** — the remote key is an unkeyed hash, so an observer holding candidate names can hash each one and match it against the directory listing. The human ID is never published, so it cannot be read off directly, but it can be confirmed by guessing.
Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless.
An observer cannot determine:
- The hostname or snapshot name of any backup by reading it off the store — the directory name and the manifest `snapshot_id` are unkeyed hashes of the human ID, so the text is never published (though a guessed name can be confirmed, as above)
- File names or paths
- File contents
- File permissions or ownership
- Directory structure
- Which chunks belong to which files
### Accepted Risks
These are known, deliberate properties of the format and the tooling, recorded so an operator can weigh them rather than discover them.
1. **No proof of authorship.** Restore and `verify --deep` prove that data decrypts with the age private key and matches its unkeyed content hashes. They do not prove who wrote it: anyone who knows a recipient public key and can replace objects on the destination can substitute a snapshot they built. The recipient string is not stored at the destination, but a compromised backed-up host has it. Defences live on the destination side — bucket versioning or object lock, credentials for the source host that cannot delete or overwrite existing versions, and pruning from a trusted host. Note that S3 `PutObject` overwrites an existing key, so PUT permission alone is not append-only.
2. **Compression reveals sizes.** Blobs and `db.zst.age` are zstd-compressed then age-encrypted; the manifest is compressed only. age does not pad, so an object's size is the exact compressed length of its contents. All new chunks packed into one blob share a single zstd stream (8 MiB window, 4 MiB at compression levels 1-2), and a blob is closed at `blob_size_limit` and at the end of each configured path. Because a stored chunk is never packed again, someone who can write into a backed-up file and watch blob sizes learns something only when their controlled data and a secret land in the same chunk of a file that keeps changing. Advice: back up any outsider-writable directory as its own snapshot.
3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of 2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them.
4. **Decrypted data on local disk.** Several commands stage plaintext under `$TMPDIR`: `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` and removed on success, but a `kill -9` or a power loss leaves them behind — delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` should be trusted to the same degree as the restore target.
5. **Store permissions differ per command.** The backed-up host needs only PUT to run `snapshot create`: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — `snapshot verify`, `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`, `snapshot remove`, and `remote nuke` also delete. The recommended cron line uses `--prune`, which runs `prune` on the backed-up host, so granting that host `--prune` gives it credentials that can delete its own backups. To keep the source host to PUT only, prune from a separate trusted host instead.
6. **Changing recipients does not re-encrypt existing data.** Deduplicated chunks and same-named blobs already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run `vaultik database delete` and take a full backup to a fresh destination or prefix.
7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients. Long-lived ciphertext held by a third party (the destination operator) has no fallback if X25519 is ever broken: there is no second recipient type and no post-quantum option.
## Consistency Guarantees
1. **Blobs are immutable** - Once written, a blob is never modified
2. **Blobs are written before metadata** - A snapshot's metadata is only written after all its blobs are successfully uploaded
3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst are written as complete files
4. **Snapshots are marked complete in local DB only after metadata upload** - Ensures consistency between local and remote state
4. **A snapshot is marked complete in the local DB only after its metadata is uploaded** - `finalizeSnapshotMetadata` runs `ExportSnapshotMetadata` first and records completion (`MarkSnapshotComplete`) only once the export succeeds (see the backup data flow in [ARCHITECTURE.md](../ARCHITECTURE.md)). A crash during the export therefore leaves the snapshot incomplete, so the next backup's `PruneDatabase` drops it and re-backs-up its data, rather than leaving a completed-looking row in the local index with no matching metadata on the destination store. (A crash in the brief moment after the export succeeds but before completion is recorded leaves a fully-restorable snapshot on the destination that the local index drops as incomplete on the next run; `snapshot list` then reports it honestly as remote-only, which is the safe direction: the destination copy stays restorable.)
## Pruning Safety
The prune operation is safe because:
1. It only deletes blobs not referenced in any manifest
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references
2. Manifests are unencrypted and can be read without keys
3. The operation compares the latest local DB snapshot with the latest S3 snapshot to ensure consistency
4. Pruning will fail if these don't match, preventing accidental deletion of needed blobs
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced
4. Prune requires exclusive access to the destination: running it during a concurrent backup can race a snapshot whose manifest is not yet written, so do not prune while a backup is in progress
## Restoration Requirements
+14 -1
View File
@@ -2,5 +2,18 @@ package blob
import "errors"
// ErrBlobSizeLimitExceeded is returned when adding a chunk would exceed the blob size limit
// ErrBlobSizeLimitExceeded is returned when adding a chunk would exceed
// the blob size limit.
var ErrBlobSizeLimitExceeded = errors.New("adding chunk would exceed blob size limit")
// ErrNoRecipients is returned when a Packer is created without any age
// recipients; blobs must always be encrypted.
var ErrNoRecipients = errors.New("recipients are required - blobs must be encrypted")
// ErrInvalidMaxBlobSize is returned when the configured maximum blob size
// is zero or negative.
var ErrInvalidMaxBlobSize = errors.New("max blob size must be positive")
// ErrNoFilesystem is returned when a Packer is created without a filesystem
// for temporary files.
var ErrNoFilesystem = errors.New("filesystem is required")
+189 -95
View File
@@ -1,14 +1,16 @@
// Package blob handles the creation of blobs - the final storage units for Vaultik.
// A blob is a large file (up to 10GB) containing many compressed and encrypted chunks
// from multiple source files. Blobs are content-addressed, meaning their filename
// is derived from the SHA256 hash of their compressed and encrypted content.
// from multiple source files. Blobs are content-addressed: a blob's filename is
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the concatenated
// chunk data before compression and encryption, not from the stored bytes. See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
//
// The blob creation process:
// 1. Chunks are accumulated from multiple files
// 2. The collection is compressed using zstd
// 3. The compressed data is encrypted using age
// 4. The encrypted blob is hashed to create its content-addressed name
// 5. The blob is uploaded to S3 using the hash as the filename
// 2. Each chunk's uncompressed bytes are fed to a running SHA-256 and, in the same
// pass, compressed with zstd and encrypted with age into the temp file
// 3. On finalize, the name is the double SHA-256 of that uncompressed content
// 4. The blob is uploaded to S3 using the name as the filename
//
// This design optimizes storage efficiency by batching many small chunks into
// larger blobs, reducing the number of S3 operations and associated costs.
@@ -18,6 +20,7 @@ import (
"context"
"database/sql"
"encoding/hex"
"errors"
"fmt"
"io"
"sync"
@@ -31,21 +34,28 @@ import (
"sneak.berlin/go/vaultik/internal/types"
)
// BlobHandler is a callback function invoked when a blob is finalized and ready for upload.
// The handler receives a BlobWithReader containing the blob metadata and a reader for
// the compressed and encrypted blob content. The handler is responsible for uploading
// the blob to storage and cleaning up any temporary files.
type BlobHandler func(blob *BlobWithReader) error
// Handler is a callback function invoked when a blob is finalized and
// ready for upload. The handler receives a WithReader containing the
// blob metadata and a reader for the compressed and encrypted blob content.
// The handler is responsible for uploading the blob to storage and cleaning
// up any temporary files.
type Handler func(blob *WithReader) error
// PackerConfig holds configuration for creating a Packer.
// All fields except BlobHandler are required.
type PackerConfig struct {
MaxBlobSize int64 // Maximum size of a blob before forcing finalization
CompressionLevel int // Zstd compression level (1-19, higher = better compression)
Recipients []string // Age recipients for encryption
Repositories *database.Repositories // Database repositories for tracking blob metadata
BlobHandler BlobHandler // Optional callback when blob is ready for upload
Fs afero.Fs // Filesystem for temporary files
// MaxBlobSize is the maximum size of a blob before forcing finalization.
MaxBlobSize int64
// CompressionLevel is the zstd level (1-19, higher = better compression).
CompressionLevel int
// Recipients holds the age recipients for encryption.
Recipients []string
// Repositories provides database access for tracking blob metadata.
Repositories *database.Repositories
// BlobHandler is an optional callback when a blob is ready for upload.
BlobHandler Handler
// Fs is the filesystem used for temporary files.
Fs afero.Fs
}
// PendingChunk represents a chunk waiting to be inserted into the database.
@@ -61,7 +71,7 @@ type Packer struct {
maxBlobSize int64
compressionLevel int
recipients []string // Age recipients for encryption
blobHandler BlobHandler // Called when blob is ready
blobHandler Handler // Called when blob is ready
repos *database.Repositories // For creating blob records
fs afero.Fs // Filesystem for temporary files
@@ -108,22 +118,23 @@ type FinishedBlob struct {
ID string
Hash string
Data []byte // Compressed data
Chunks []*BlobChunkRef
Chunks []*ChunkPosition
CreatedTS time.Time
Uncompressed int64
Compressed int64
}
// BlobChunkRef represents a chunk's position within a blob
type BlobChunkRef struct {
// ChunkPosition represents a chunk's position within a blob
type ChunkPosition struct {
ChunkHash string
Offset int64
Length int64
}
// BlobWithReader wraps a FinishedBlob with its data reader
type BlobWithReader struct {
// WithReader wraps a FinishedBlob with its data reader
type WithReader struct {
*FinishedBlob
Reader io.ReadSeeker
TempFile afero.File // Optional, only set for disk-based blobs
InsertedChunkHashes []string // Chunk hashes that were inserted to DB with this blob
@@ -134,14 +145,17 @@ type BlobWithReader struct {
// Returns an error if required configuration fields are missing or invalid.
func NewPacker(cfg PackerConfig) (*Packer, error) {
if len(cfg.Recipients) == 0 {
return nil, fmt.Errorf("recipients are required - blobs must be encrypted")
return nil, ErrNoRecipients
}
if cfg.MaxBlobSize <= 0 {
return nil, fmt.Errorf("max blob size must be positive")
return nil, ErrInvalidMaxBlobSize
}
if cfg.Fs == nil {
return nil, fmt.Errorf("filesystem is required")
return nil, ErrNoFilesystem
}
return &Packer{
maxBlobSize: cfg.MaxBlobSize,
compressionLevel: cfg.CompressionLevel,
@@ -157,9 +171,10 @@ func NewPacker(cfg PackerConfig) (*Packer, error) {
// The handler is responsible for uploading the blob to storage.
// If no handler is set, finalized blobs are stored in memory and can be
// retrieved with GetFinishedBlobs().
func (p *Packer) SetBlobHandler(handler BlobHandler) {
func (p *Packer) SetBlobHandler(handler Handler) {
p.mu.Lock()
defer p.mu.Unlock()
p.blobHandler = handler
}
@@ -169,6 +184,7 @@ func (p *Packer) SetBlobHandler(handler BlobHandler) {
func (p *Packer) AddPendingChunk(hash string, size int64) {
p.mu.Lock()
defer p.mu.Unlock()
p.pendingChunks = append(p.pendingChunks, PendingChunk{Hash: hash, Size: size})
}
@@ -177,13 +193,14 @@ func (p *Packer) AddPendingChunk(hash string, size int64) {
// In this case, the caller should finalize the current blob and retry.
// The chunk data is written immediately and can be garbage collected after this call.
// Thread-safe.
func (p *Packer) AddChunk(chunk *ChunkRef) error {
func (p *Packer) AddChunk(ctx context.Context, chunk *ChunkRef) error {
p.mu.Lock()
defer p.mu.Unlock()
// Initialize new blob if needed
if p.currentBlob == nil {
if err := p.startNewBlob(); err != nil {
err := p.startNewBlob(ctx)
if err != nil {
return fmt.Errorf("starting new blob: %w", err)
}
}
@@ -202,7 +219,8 @@ func (p *Packer) AddChunk(chunk *ChunkRef) error {
}
// Add chunk to current blob
if err := p.addChunkToCurrentBlob(chunk); err != nil {
err := p.addChunkToCurrentBlob(chunk)
if err != nil {
return err
}
@@ -213,12 +231,13 @@ func (p *Packer) AddChunk(chunk *ChunkRef) error {
// This should be called after all chunks have been added to ensure no data is lost.
// If a BlobHandler is set, it will be called with the finalized blob.
// Thread-safe.
func (p *Packer) Flush() error {
func (p *Packer) Flush(ctx context.Context) error {
p.mu.Lock()
defer p.mu.Unlock()
if p.currentBlob != nil && len(p.currentBlob.chunks) > 0 {
if err := p.finalizeCurrentBlob(); err != nil {
err := p.finalizeCurrentBlob(ctx)
if err != nil {
return fmt.Errorf("finalizing blob: %w", err)
}
}
@@ -232,7 +251,7 @@ func (p *Packer) Flush() error {
// BlobHandler (if set) or stored internally.
// Caller must handle retrying any chunk that triggered size limit exceeded.
// Not thread-safe - caller must hold the lock.
func (p *Packer) FinalizeBlob() error {
func (p *Packer) FinalizeBlob(ctx context.Context) error {
p.mu.Lock()
defer p.mu.Unlock()
@@ -240,7 +259,7 @@ func (p *Packer) FinalizeBlob() error {
return nil
}
return p.finalizeCurrentBlob()
return p.finalizeCurrentBlob(ctx)
}
// GetFinishedBlobs returns all completed blobs and clears the internal list.
@@ -253,11 +272,37 @@ func (p *Packer) GetFinishedBlobs() []*FinishedBlob {
blobs := p.finishedBlobs
p.finishedBlobs = make([]*FinishedBlob, 0)
return blobs
}
// PackChunks is a convenience method to pack multiple chunks at once.
func (p *Packer) PackChunks(ctx context.Context, chunks []*ChunkRef) error {
for _, chunk := range chunks {
err := p.AddChunk(ctx, chunk)
if errors.Is(err, ErrBlobSizeLimitExceeded) {
// Finalize current blob and retry
err = p.FinalizeBlob(ctx)
if err != nil {
return fmt.Errorf("finalizing blob before retry: %w", err)
}
// Retry the chunk
err = p.AddChunk(ctx, chunk)
if err != nil {
return fmt.Errorf(
"adding chunk %s after finalize: %w", chunk.Hash, err)
}
} else if err != nil {
return fmt.Errorf("adding chunk %s: %w", chunk.Hash, err)
}
}
return p.Flush(ctx)
}
// startNewBlob initializes a new blob (must be called with lock held)
func (p *Packer) startNewBlob() error {
func (p *Packer) startNewBlob(ctx context.Context) error {
// Generate UUID for the blob
blobID := uuid.New().String()
@@ -267,18 +312,24 @@ func (p *Packer) startNewBlob() error {
if err != nil {
return fmt.Errorf("parsing blob ID: %w", err)
}
blob := &database.Blob{
ID: blobIDTyped,
Hash: types.BlobHash("temp-placeholder-" + blobID), // Temporary placeholder until finalized
// Temporary placeholder hash until finalized.
Hash: types.BlobHash("temp-placeholder-" + blobID),
CreatedTS: time.Now().UTC(),
FinishedTS: nil,
UncompressedSize: 0,
CompressedSize: 0,
UploadedTS: nil,
}
if err := p.repos.WithTx(context.Background(), func(ctx context.Context, tx *sql.Tx) error {
return p.repos.Blobs.Create(ctx, tx, blob)
}); err != nil {
err = p.repos.WithTx(
ctx,
func(txCtx context.Context, tx *sql.Tx) error {
return p.repos.Blobs.Create(txCtx, tx, blob)
})
if err != nil {
return fmt.Errorf("creating blob record: %w", err)
}
}
@@ -294,6 +345,7 @@ func (p *Packer) startNewBlob() error {
if err != nil {
_ = tempFile.Close()
_ = p.fs.Remove(tempFile.Name())
return fmt.Errorf("creating blobgen writer: %w", err)
}
@@ -307,15 +359,20 @@ func (p *Packer) startNewBlob() error {
size: 0,
}
log.Debug("Created new blob container", "blob_id", blobID, "temp_file", tempFile.Name())
log.Debug("Created new blob container",
"blob_id", blobID, "temp_file", tempFile.Name())
return nil
}
// addChunkToCurrentBlob adds a chunk to the current blob (must be called with lock held)
// addChunkToCurrentBlob adds a chunk to the current blob (must be called
// with lock held).
func (p *Packer) addChunkToCurrentBlob(chunk *ChunkRef) error {
// Skip if chunk already in current blob
if p.currentBlob.chunkSet[chunk.Hash] {
log.Debug("Skipping duplicate chunk already in current blob", "chunk_hash", chunk.Hash)
log.Debug("Skipping duplicate chunk already in current blob",
"chunk_hash", chunk.Hash)
return nil
}
@@ -323,7 +380,8 @@ func (p *Packer) addChunkToCurrentBlob(chunk *ChunkRef) error {
offset := p.currentBlob.size
// Write to the blobgen writer (compression -> encryption -> disk)
if _, err := p.currentBlob.writer.Write(chunk.Data); err != nil {
_, err := p.currentBlob.writer.Write(chunk.Data)
if err != nil {
return fmt.Errorf("writing to blob stream: %w", err)
}
@@ -356,7 +414,7 @@ func (p *Packer) addChunkToCurrentBlob(chunk *ChunkRef) error {
}
// finalizeCurrentBlob completes the current blob (must be called with lock held)
func (p *Packer) finalizeCurrentBlob() error {
func (p *Packer) finalizeCurrentBlob(ctx context.Context) error {
if p.currentBlob == nil {
return nil
}
@@ -371,7 +429,8 @@ func (p *Packer) finalizeCurrentBlob() error {
chunksToInsert := p.pendingChunks
p.pendingChunks = nil
if err := p.commitBlobToDatabase(blobHash, finalSize, chunksToInsert); err != nil {
err = p.commitBlobToDatabase(ctx, blobHash, finalSize, chunksToInsert)
if err != nil {
return err
}
@@ -399,44 +458,59 @@ func (p *Packer) finalizeCurrentBlob() error {
return p.deliverFinishedBlob(finished, insertedChunkHashes)
}
// closeBlobWriter closes the writer, syncs to disk, and returns the blob hash and final size
// closeBlobWriter closes the writer, syncs to disk, and returns the blob
// hash and final size.
func (p *Packer) closeBlobWriter() (string, int64, error) {
if err := p.currentBlob.writer.Close(); err != nil {
err := p.currentBlob.writer.Close()
if err != nil {
p.cleanupTempFile()
return "", 0, fmt.Errorf("closing blobgen writer: %w", err)
}
if err := p.currentBlob.tempFile.Sync(); err != nil {
err = p.currentBlob.tempFile.Sync()
if err != nil {
p.cleanupTempFile()
return "", 0, fmt.Errorf("syncing temp file: %w", err)
}
finalSize, err := p.currentBlob.tempFile.Seek(0, io.SeekCurrent)
if err != nil {
p.cleanupTempFile()
return "", 0, fmt.Errorf("getting file size: %w", err)
}
if _, err := p.currentBlob.tempFile.Seek(0, io.SeekStart); err != nil {
_, err = p.currentBlob.tempFile.Seek(0, io.SeekStart)
if err != nil {
p.cleanupTempFile()
return "", 0, fmt.Errorf("seeking to start: %w", err)
}
finalHash := p.currentBlob.writer.Sum256()
finalHash := p.currentBlob.writer.ContentID()
return hex.EncodeToString(finalHash), finalSize, nil
}
// buildChunkRefs creates BlobChunkRef entries from the current blob's chunks
func (p *Packer) buildChunkRefs() []*BlobChunkRef {
refs := make([]*BlobChunkRef, 0, len(p.currentBlob.chunks))
// buildChunkRefs creates ChunkPosition entries from the current blob's chunks
func (p *Packer) buildChunkRefs() []*ChunkPosition {
refs := make([]*ChunkPosition, 0, len(p.currentBlob.chunks))
for _, chunk := range p.currentBlob.chunks {
refs = append(refs, &BlobChunkRef{
refs = append(refs, &ChunkPosition{
ChunkHash: chunk.Hash, Offset: chunk.Offset, Length: chunk.Size,
})
}
return refs
}
// commitBlobToDatabase inserts pending chunks, blob_chunks, and updates the blob record
func (p *Packer) commitBlobToDatabase(blobHash string, finalSize int64, chunksToInsert []PendingChunk) error {
func (p *Packer) commitBlobToDatabase(
ctx context.Context,
blobHash string, finalSize int64, chunksToInsert []PendingChunk,
) error {
if p.repos == nil {
return nil
}
@@ -444,13 +518,46 @@ func (p *Packer) commitBlobToDatabase(blobHash string, finalSize int64, chunksTo
blobIDTyped, parseErr := types.ParseBlobID(p.currentBlob.id)
if parseErr != nil {
p.cleanupTempFile()
return fmt.Errorf("parsing blob ID: %w", parseErr)
}
err := p.repos.WithTx(context.Background(), func(ctx context.Context, tx *sql.Tx) error {
err := p.repos.WithTx(
ctx,
func(txCtx context.Context, tx *sql.Tx) error {
return p.insertBlobRecords(txCtx, tx, blobIDTyped, blobHash,
finalSize, chunksToInsert)
})
if err != nil {
p.cleanupTempFile()
return fmt.Errorf("finalizing blob transaction: %w", err)
}
log.Debug("Committed blob transaction",
"chunks_inserted", len(chunksToInsert),
"blob_chunks_inserted", len(p.currentBlob.chunks))
return nil
}
// insertBlobRecords inserts pending chunks and blob_chunk rows, then marks
// the blob finished, all within the supplied transaction.
func (p *Packer) insertBlobRecords(
ctx context.Context,
tx *sql.Tx,
blobIDTyped types.BlobID,
blobHash string,
finalSize int64,
chunksToInsert []PendingChunk,
) error {
for _, chunk := range chunksToInsert {
dbChunk := &database.Chunk{ChunkHash: types.ChunkHash(chunk.Hash), Size: chunk.Size}
if err := p.repos.Chunks.Create(ctx, tx, dbChunk); err != nil {
dbChunk := &database.Chunk{
ChunkHash: types.ChunkHash(chunk.Hash), Size: chunk.Size,
}
err := p.repos.Chunks.Create(ctx, tx, dbChunk)
if err != nil {
return fmt.Errorf("creating chunk: %w", err)
}
}
@@ -460,62 +567,70 @@ func (p *Packer) commitBlobToDatabase(blobHash string, finalSize int64, chunksTo
BlobID: blobIDTyped, ChunkHash: types.ChunkHash(chunk.Hash),
Offset: chunk.Offset, Length: chunk.Size,
}
if err := p.repos.BlobChunks.Create(ctx, tx, blobChunk); err != nil {
err := p.repos.BlobChunks.Create(ctx, tx, blobChunk)
if err != nil {
return fmt.Errorf("creating blob_chunk: %w", err)
}
}
return p.repos.Blobs.UpdateFinished(ctx, tx, p.currentBlob.id, blobHash, p.currentBlob.size, finalSize)
})
if err != nil {
p.cleanupTempFile()
return fmt.Errorf("finalizing blob transaction: %w", err)
}
log.Debug("Committed blob transaction",
"chunks_inserted", len(chunksToInsert), "blob_chunks_inserted", len(p.currentBlob.chunks))
return nil
return p.repos.Blobs.UpdateFinished(ctx, tx, p.currentBlob.id, blobHash,
p.currentBlob.size, finalSize)
}
// deliverFinishedBlob passes the blob to the handler or stores it internally
func (p *Packer) deliverFinishedBlob(finished *FinishedBlob, insertedChunkHashes []string) error {
func (p *Packer) deliverFinishedBlob(
finished *FinishedBlob, insertedChunkHashes []string,
) error {
if p.blobHandler != nil {
if _, err := p.currentBlob.tempFile.Seek(0, io.SeekStart); err != nil {
_, err := p.currentBlob.tempFile.Seek(0, io.SeekStart)
if err != nil {
p.cleanupTempFile()
return fmt.Errorf("seeking for handler: %w", err)
}
blobWithReader := &BlobWithReader{
blobWithReader := &WithReader{
FinishedBlob: finished,
Reader: p.currentBlob.tempFile,
TempFile: p.currentBlob.tempFile,
InsertedChunkHashes: insertedChunkHashes,
}
if err := p.blobHandler(blobWithReader); err != nil {
err = p.blobHandler(blobWithReader)
if err != nil {
p.cleanupTempFile()
return fmt.Errorf("blob handler failed: %w", err)
}
p.currentBlob = nil
return nil
}
// No handler - read data for legacy behavior
log.Debug("No blob handler callback configured", "blob_hash", finished.Hash[:8]+"...")
if _, err := p.currentBlob.tempFile.Seek(0, io.SeekStart); err != nil {
_, err := p.currentBlob.tempFile.Seek(0, io.SeekStart)
if err != nil {
p.cleanupTempFile()
return fmt.Errorf("seeking to read data: %w", err)
}
data, err := io.ReadAll(p.currentBlob.tempFile)
if err != nil {
p.cleanupTempFile()
return fmt.Errorf("reading blob data: %w", err)
}
finished.Data = data
p.finishedBlobs = append(p.finishedBlobs, finished)
p.cleanupTempFile()
p.currentBlob = nil
return nil
}
@@ -527,24 +642,3 @@ func (p *Packer) cleanupTempFile() {
_ = p.fs.Remove(name)
}
}
// PackChunks is a convenience method to pack multiple chunks at once
func (p *Packer) PackChunks(chunks []*ChunkRef) error {
for _, chunk := range chunks {
err := p.AddChunk(chunk)
if err == ErrBlobSizeLimitExceeded {
// Finalize current blob and retry
if err := p.FinalizeBlob(); err != nil {
return fmt.Errorf("finalizing blob before retry: %w", err)
}
// Retry the chunk
if err := p.AddChunk(chunk); err != nil {
return fmt.Errorf("adding chunk %s after finalize: %w", chunk.Hash, err)
}
} else if err != nil {
return fmt.Errorf("adding chunk %s: %w", chunk.Hash, err)
}
}
return p.Flush()
}
+164 -231
View File
@@ -1,4 +1,4 @@
package blob
package blob_test
import (
"bytes"
@@ -6,12 +6,14 @@ import (
"crypto/sha256"
"database/sql"
"encoding/hex"
"errors"
"io"
"testing"
"filippo.io/age"
"github.com/klauspost/compress/zstd"
"github.com/spf13/afero"
"sneak.berlin/go/vaultik/internal/blob"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/types"
@@ -19,94 +21,98 @@ import (
const (
// Test key from test/insecure-integration-test.key
testPrivateKey = "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5"
testPrivateKey = "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7A" +
"PHXA2QS2NJA5"
testPublicKey = "age1ezrjmfpwsc95svdg0y54mums3zevgzu0x0ecq2f7tp8a05gl0sjq9q9wjg"
defaultMaxBlobSize = 10 * 1024 * 1024 // 10MB
testChunkSize = 1000
testChunkCount = 10
)
func TestPacker(t *testing.T) {
// Initialize logger for tests
log.Initialize(log.Config{})
// parseTestIdentity parses the fixed test age identity.
func parseTestIdentity(t *testing.T) *age.X25519Identity {
t.Helper()
// Parse test identity
identity, err := age.ParseX25519Identity(testPrivateKey)
if err != nil {
t.Fatalf("failed to parse test identity: %v", err)
}
t.Run("single chunk creates single blob", func(t *testing.T) {
// Create test database
return identity
}
// newTestPacker creates a test database and a Packer backed by it.
func newTestPacker(
t *testing.T, maxBlobSize int64,
) (*database.Repositories, *blob.Packer) {
t.Helper()
db, err := database.NewTestDB()
if err != nil {
t.Fatalf("failed to create test db: %v", err)
}
defer func() { _ = db.Close() }()
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
cfg := PackerConfig{
MaxBlobSize: 10 * 1024 * 1024, // 10MB
packer, err := blob.NewPacker(blob.PackerConfig{
MaxBlobSize: maxBlobSize,
CompressionLevel: 3,
Recipients: []string{testPublicKey},
Repositories: repos,
Fs: afero.NewMemMapFs(),
}
packer, err := NewPacker(cfg)
})
if err != nil {
t.Fatalf("failed to create packer: %v", err)
}
// Create a chunk
data := []byte("Hello, World!")
return repos, packer
}
// makeChunk creates a ChunkRef for data and registers the chunk in the
// database.
func makeChunk(
t *testing.T, repos *database.Repositories, data []byte,
) *blob.ChunkRef {
t.Helper()
hash := sha256.Sum256(data)
hashStr := hex.EncodeToString(hash[:])
// Create chunk in database first
dbChunk := &database.Chunk{
ChunkHash: types.ChunkHash(hashStr),
Size: int64(len(data)),
}
err = repos.WithTx(context.Background(), func(ctx context.Context, tx *sql.Tx) error {
err := repos.WithTx(
context.Background(),
func(ctx context.Context, tx *sql.Tx) error {
return repos.Chunks.Create(ctx, tx, dbChunk)
})
if err != nil {
t.Fatalf("failed to create chunk in db: %v", err)
}
chunk := &ChunkRef{
return &blob.ChunkRef{
Hash: hashStr,
Data: data,
}
}
// Add chunk
if err := packer.AddChunk(chunk); err != nil {
t.Fatalf("failed to add chunk: %v", err)
}
// decryptAndDecompress reverses the blob pipeline: age decrypt, then zstd
// decompress.
func decryptAndDecompress(
t *testing.T, blobData []byte, identity *age.X25519Identity,
) []byte {
t.Helper()
// Flush
if err := packer.Flush(); err != nil {
t.Fatalf("failed to flush: %v", err)
}
// Get finished blobs
blobs := packer.GetFinishedBlobs()
if len(blobs) != 1 {
t.Fatalf("expected 1 blob, got %d", len(blobs))
}
blob := blobs[0]
if len(blob.Chunks) != 1 {
t.Errorf("expected 1 chunk in blob, got %d", len(blob.Chunks))
}
// Note: Very small data may not compress well
t.Logf("Compression: %d -> %d bytes", blob.Uncompressed, blob.Compressed)
// Decrypt the blob data
decrypted, err := age.Decrypt(bytes.NewReader(blob.Data), identity)
decrypted, err := age.Decrypt(bytes.NewReader(blobData), identity)
if err != nil {
t.Fatalf("failed to decrypt blob: %v", err)
}
// Decompress the decrypted data
reader, err := zstd.NewReader(decrypted)
if err != nil {
t.Fatalf("failed to create decompressor: %v", err)
@@ -114,157 +120,144 @@ func TestPacker(t *testing.T) {
defer reader.Close()
var decompressed bytes.Buffer
if _, err := io.Copy(&decompressed, reader); err != nil {
_, err = io.Copy(&decompressed, reader)
if err != nil {
t.Fatalf("failed to decompress: %v", err)
}
if !bytes.Equal(decompressed.Bytes(), data) {
t.Error("decompressed data doesn't match original")
}
})
return decompressed.Bytes()
}
t.Run("multiple chunks packed together", func(t *testing.T) {
// Create test database
db, err := database.NewTestDB()
if err != nil {
t.Fatalf("failed to create test db: %v", err)
}
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
func TestPackerSingleChunk(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
cfg := PackerConfig{
MaxBlobSize: 10 * 1024 * 1024, // 10MB
CompressionLevel: 3,
Recipients: []string{testPublicKey},
Repositories: repos,
Fs: afero.NewMemMapFs(),
}
packer, err := NewPacker(cfg)
if err != nil {
t.Fatalf("failed to create packer: %v", err)
}
identity := parseTestIdentity(t)
repos, packer := newTestPacker(t, defaultMaxBlobSize)
ctx := context.Background()
// Create multiple small chunks
chunks := make([]*ChunkRef, 10)
for i := 0; i < 10; i++ {
data := bytes.Repeat([]byte{byte(i)}, 1000)
hash := sha256.Sum256(data)
hashStr := hex.EncodeToString(hash[:])
data := []byte("Hello, World!")
chunk := makeChunk(t, repos, data)
// Create chunk in database first
dbChunk := &database.Chunk{
ChunkHash: types.ChunkHash(hashStr),
Size: int64(len(data)),
}
err = repos.WithTx(context.Background(), func(ctx context.Context, tx *sql.Tx) error {
return repos.Chunks.Create(ctx, tx, dbChunk)
})
if err != nil {
t.Fatalf("failed to create chunk in db: %v", err)
}
chunks[i] = &ChunkRef{
Hash: hashStr,
Data: data,
}
}
// Add all chunks
for _, chunk := range chunks {
err := packer.AddChunk(chunk)
err := packer.AddChunk(ctx, chunk)
if err != nil {
t.Fatalf("failed to add chunk: %v", err)
}
}
// Flush
if err := packer.Flush(); err != nil {
err = packer.Flush(ctx)
if err != nil {
t.Fatalf("failed to flush: %v", err)
}
// Should have one blob with all chunks
blobs := packer.GetFinishedBlobs()
if len(blobs) != 1 {
t.Fatalf("expected 1 blob, got %d", len(blobs))
}
if len(blobs[0].Chunks) != 10 {
t.Errorf("expected 10 chunks in blob, got %d", len(blobs[0].Chunks))
finished := blobs[0]
if len(finished.Chunks) != 1 {
t.Errorf("expected 1 chunk in blob, got %d", len(finished.Chunks))
}
// Note: Very small data may not compress well
t.Logf("Compression: %d -> %d bytes",
finished.Uncompressed, finished.Compressed)
decompressed := decryptAndDecompress(t, finished.Data, identity)
if !bytes.Equal(decompressed, data) {
t.Error("decompressed data doesn't match original")
}
}
func TestPackerMultipleChunks(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
repos, packer := newTestPacker(t, defaultMaxBlobSize)
ctx := context.Background()
chunks := make([]*blob.ChunkRef, testChunkCount)
for i := range testChunkCount {
data := bytes.Repeat([]byte{byte(i)}, testChunkSize)
chunks[i] = makeChunk(t, repos, data)
}
for _, chunk := range chunks {
err := packer.AddChunk(ctx, chunk)
if err != nil {
t.Fatalf("failed to add chunk: %v", err)
}
}
err := packer.Flush(ctx)
if err != nil {
t.Fatalf("failed to flush: %v", err)
}
blobs := packer.GetFinishedBlobs()
if len(blobs) != 1 {
t.Fatalf("expected 1 blob, got %d", len(blobs))
}
if len(blobs[0].Chunks) != testChunkCount {
t.Errorf("expected %d chunks in blob, got %d",
testChunkCount, len(blobs[0].Chunks))
}
// Verify offsets are correct
expectedOffset := int64(0)
for i, chunkRef := range blobs[0].Chunks {
if chunkRef.Offset != expectedOffset {
t.Errorf("chunk %d: expected offset %d, got %d", i, expectedOffset, chunkRef.Offset)
t.Errorf("chunk %d: expected offset %d, got %d",
i, expectedOffset, chunkRef.Offset)
}
if chunkRef.Length != 1000 {
t.Errorf("chunk %d: expected length 1000, got %d", i, chunkRef.Length)
if chunkRef.Length != testChunkSize {
t.Errorf("chunk %d: expected length %d, got %d",
i, testChunkSize, chunkRef.Length)
}
expectedOffset += chunkRef.Length
}
})
}
t.Run("blob size limit enforced", func(t *testing.T) {
// Create test database
db, err := database.NewTestDB()
if err != nil {
t.Fatalf("failed to create test db: %v", err)
}
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
func TestPackerSizeLimit(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
// Small blob size limit to force multiple blobs
cfg := PackerConfig{
MaxBlobSize: 5000, // 5KB max
CompressionLevel: 3,
Recipients: []string{testPublicKey},
Repositories: repos,
Fs: afero.NewMemMapFs(),
}
packer, err := NewPacker(cfg)
if err != nil {
t.Fatalf("failed to create packer: %v", err)
}
const (
maxBlobSize = 5000 // 5KB max, forces multiple blobs
maxBlobawoOverhead = 6000 // allow some overhead over the limit
)
// Create chunks that will exceed the limit
chunks := make([]*ChunkRef, 10)
for i := 0; i < 10; i++ {
data := bytes.Repeat([]byte{byte(i)}, 1000) // 1KB each
hash := sha256.Sum256(data)
hashStr := hex.EncodeToString(hash[:])
repos, packer := newTestPacker(t, maxBlobSize)
ctx := context.Background()
// Create chunk in database first
dbChunk := &database.Chunk{
ChunkHash: types.ChunkHash(hashStr),
Size: int64(len(data)),
}
err = repos.WithTx(context.Background(), func(ctx context.Context, tx *sql.Tx) error {
return repos.Chunks.Create(ctx, tx, dbChunk)
})
if err != nil {
t.Fatalf("failed to create chunk in db: %v", err)
}
chunks[i] = &ChunkRef{
Hash: hashStr,
Data: data,
}
chunks := make([]*blob.ChunkRef, testChunkCount)
for i := range testChunkCount {
data := bytes.Repeat([]byte{byte(i)}, testChunkSize) // 1KB each
chunks[i] = makeChunk(t, repos, data)
}
blobCount := 0
// Add chunks and handle size limit errors
for _, chunk := range chunks {
err := packer.AddChunk(chunk)
if err == ErrBlobSizeLimitExceeded {
err := packer.AddChunk(ctx, chunk)
if errors.Is(err, blob.ErrBlobSizeLimitExceeded) {
// Finalize current blob
if err := packer.FinalizeBlob(); err != nil {
err = packer.FinalizeBlob(ctx)
if err != nil {
t.Fatalf("failed to finalize blob: %v", err)
}
blobCount++
// Retry adding the chunk
if err := packer.AddChunk(chunk); err != nil {
err = packer.AddChunk(ctx, chunk)
if err != nil {
t.Fatalf("failed to add chunk after finalize: %v", err)
}
} else if err != nil {
@@ -272,114 +265,54 @@ func TestPacker(t *testing.T) {
}
}
// Flush remaining
if err := packer.Flush(); err != nil {
err := packer.Flush(ctx)
if err != nil {
t.Fatalf("failed to flush: %v", err)
}
// Get all blobs
blobs := packer.GetFinishedBlobs()
totalBlobs := blobCount + len(blobs)
// Should have multiple blobs due to size limit
totalBlobs := blobCount + len(blobs)
if totalBlobs < 2 {
t.Errorf("expected multiple blobs due to size limit, got %d", totalBlobs)
}
// Verify each blob respects size limit (approximately)
for _, blob := range blobs {
if blob.Compressed > 6000 { // Allow some overhead
t.Errorf("blob size %d exceeds limit", blob.Compressed)
for _, finished := range blobs {
if finished.Compressed > maxBlobawoOverhead {
t.Errorf("blob size %d exceeds limit", finished.Compressed)
}
}
})
}
t.Run("with encryption", func(t *testing.T) {
// Create test database
db, err := database.NewTestDB()
if err != nil {
t.Fatalf("failed to create test db: %v", err)
}
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
func TestPackerEncryption(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
// Generate test identity (using the one from parent test)
cfg := PackerConfig{
MaxBlobSize: 10 * 1024 * 1024, // 10MB
CompressionLevel: 3,
Recipients: []string{testPublicKey},
Repositories: repos,
Fs: afero.NewMemMapFs(),
}
packer, err := NewPacker(cfg)
if err != nil {
t.Fatalf("failed to create packer: %v", err)
}
identity := parseTestIdentity(t)
repos, packer := newTestPacker(t, defaultMaxBlobSize)
ctx := context.Background()
// Create test data
data := bytes.Repeat([]byte("Test data for encryption!"), 100)
hash := sha256.Sum256(data)
hashStr := hex.EncodeToString(hash[:])
chunk := makeChunk(t, repos, data)
// Create chunk in database first
dbChunk := &database.Chunk{
ChunkHash: types.ChunkHash(hashStr),
Size: int64(len(data)),
}
err = repos.WithTx(context.Background(), func(ctx context.Context, tx *sql.Tx) error {
return repos.Chunks.Create(ctx, tx, dbChunk)
})
err := packer.AddChunk(ctx, chunk)
if err != nil {
t.Fatalf("failed to create chunk in db: %v", err)
}
chunk := &ChunkRef{
Hash: hashStr,
Data: data,
}
// Add chunk and flush
if err := packer.AddChunk(chunk); err != nil {
t.Fatalf("failed to add chunk: %v", err)
}
if err := packer.Flush(); err != nil {
err = packer.Flush(ctx)
if err != nil {
t.Fatalf("failed to flush: %v", err)
}
// Get blob
blobs := packer.GetFinishedBlobs()
if len(blobs) != 1 {
t.Fatalf("expected 1 blob, got %d", len(blobs))
}
blob := blobs[0]
// Decrypt the blob
decrypted, err := age.Decrypt(bytes.NewReader(blob.Data), identity)
if err != nil {
t.Fatalf("failed to decrypt blob: %v", err)
}
var decryptedData bytes.Buffer
if _, err := decryptedData.ReadFrom(decrypted); err != nil {
t.Fatalf("failed to read decrypted data: %v", err)
}
// Decompress
reader, err := zstd.NewReader(&decryptedData)
if err != nil {
t.Fatalf("failed to create decompressor: %v", err)
}
defer reader.Close()
var decompressed bytes.Buffer
if _, err := decompressed.ReadFrom(reader); err != nil {
t.Fatalf("failed to decompress: %v", err)
}
// Verify data
if !bytes.Equal(decompressed.Bytes(), data) {
decompressed := decryptAndDecompress(t, blobs[0].Data, identity)
if !bytes.Equal(decompressed, data) {
t.Error("decrypted and decompressed data doesn't match original")
}
})
}
-74
View File
@@ -1,74 +0,0 @@
package blobgen
import (
"bytes"
"encoding/hex"
"fmt"
"io"
)
// CompressResult contains the results of compression
type CompressResult struct {
Data []byte
UncompressedSize int64
CompressedSize int64
SHA256 string
}
// CompressData compresses and encrypts data, returning the result with hash
func CompressData(data []byte, compressionLevel int, recipients []string) (*CompressResult, error) {
var buf bytes.Buffer
// Create writer
w, err := NewWriter(&buf, compressionLevel, recipients)
if err != nil {
return nil, fmt.Errorf("creating writer: %w", err)
}
// Write data
if _, err := w.Write(data); err != nil {
_ = w.Close()
return nil, fmt.Errorf("writing data: %w", err)
}
// Close to flush
if err := w.Close(); err != nil {
return nil, fmt.Errorf("closing writer: %w", err)
}
return &CompressResult{
Data: buf.Bytes(),
UncompressedSize: int64(len(data)),
CompressedSize: int64(buf.Len()),
SHA256: hex.EncodeToString(w.Sum256()),
}, nil
}
// CompressStream compresses and encrypts from reader to writer, returning hash
func CompressStream(dst io.Writer, src io.Reader, compressionLevel int, recipients []string) (written int64, hash string, err error) {
// Create writer
w, err := NewWriter(dst, compressionLevel, recipients)
if err != nil {
return 0, "", fmt.Errorf("creating writer: %w", err)
}
closed := false
defer func() {
if !closed {
_ = w.Close()
}
}()
// Copy data
if _, err := io.Copy(w, src); err != nil {
return 0, "", fmt.Errorf("copying data: %w", err)
}
// Close to flush
if err := w.Close(); err != nil {
return 0, "", fmt.Errorf("closing writer: %w", err)
}
closed = true
return w.BytesWritten(), hex.EncodeToString(w.Sum256()), nil
}
-64
View File
@@ -1,64 +0,0 @@
package blobgen
import (
"bytes"
"crypto/rand"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// testRecipient is a static age recipient for tests.
const testRecipient = "age1cplgrwj77ta54dnmydvvmzn64ltk83ankxl5sww04mrtmu62kv3s89gmvv"
// TestCompressStreamNoDoubleClose is a regression test for issue #28.
// It verifies that CompressStream does not panic or return an error due to
// double-closing the underlying blobgen.Writer. Before the fix in PR #33,
// the explicit Close() on the happy path combined with defer Close() would
// cause a double close.
func TestCompressStreamNoDoubleClose(t *testing.T) {
input := []byte("regression test data for issue #28 double-close fix")
var buf bytes.Buffer
written, hash, err := CompressStream(&buf, bytes.NewReader(input), 3, []string{testRecipient})
require.NoError(t, err, "CompressStream should not return an error")
assert.True(t, written > 0, "expected bytes written > 0")
assert.NotEmpty(t, hash, "expected non-empty hash")
assert.True(t, buf.Len() > 0, "expected non-empty output")
}
// TestCompressStreamLargeInput exercises CompressStream with a larger payload
// to ensure no double-close issues surface under heavier I/O.
func TestCompressStreamLargeInput(t *testing.T) {
data := make([]byte, 512*1024) // 512 KB
_, err := rand.Read(data)
require.NoError(t, err)
var buf bytes.Buffer
written, hash, err := CompressStream(&buf, bytes.NewReader(data), 3, []string{testRecipient})
require.NoError(t, err)
assert.True(t, written > 0)
assert.NotEmpty(t, hash)
}
// TestCompressStreamEmptyInput verifies CompressStream handles empty input
// without double-close issues.
func TestCompressStreamEmptyInput(t *testing.T) {
var buf bytes.Buffer
_, hash, err := CompressStream(&buf, strings.NewReader(""), 3, []string{testRecipient})
require.NoError(t, err)
assert.NotEmpty(t, hash)
}
// TestCompressDataNoDoubleClose mirrors the stream test for CompressData,
// ensuring the explicit Close + error-path Close pattern is also safe.
func TestCompressDataNoDoubleClose(t *testing.T) {
input := []byte("CompressData regression test for double-close")
result, err := CompressData(input, 3, []string{testRecipient})
require.NoError(t, err)
assert.True(t, result.CompressedSize > 0)
assert.True(t, result.UncompressedSize == int64(len(input)))
assert.NotEmpty(t, result.SHA256)
}
+119
View File
@@ -0,0 +1,119 @@
package blobgen_test
import (
"bytes"
"crypto/rand"
"errors"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// ageChunkSize is age's STREAM plaintext chunk size (64 KiB); each encrypted
// chunk adds a 16-byte ChaCha20-Poly1305 tag.
const (
ageChunkSize = 64 * 1024
ageChunkTagSize = 16
ageSegmentSize = ageChunkSize + ageChunkTagSize
ageNonceSize = 16
)
// makeIdentity returns a fresh X25519 identity and its recipient string.
func makeIdentity(t *testing.T) (*age.X25519Identity, string) {
t.Helper()
id, err := age.GenerateX25519Identity()
require.NoError(t, err)
return id, id.Recipient().String()
}
// randomBytes returns n cryptographically random bytes, which do not compress
// so the encrypted payload spans multiple age segments.
func randomBytes(t *testing.T, n int) []byte {
t.Helper()
b := make([]byte, n)
_, err := rand.Read(b)
require.NoError(t, err)
return b
}
// compressibleBytes returns n bytes of a repeating pattern, which zstd packs
// down to a small payload.
func compressibleBytes(n int) []byte {
pattern := bytes.Repeat([]byte("compressible-"), n/13+1)
return pattern[:n]
}
// encryptBlob compresses, encrypts and returns a blob for plaintext at
// compression level 1.
func encryptBlob(t *testing.T, plaintext []byte, recipients ...string) []byte {
t.Helper()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, recipients)
require.NoError(t, err)
_, err = w.Write(plaintext)
require.NoError(t, err)
require.NoError(t, w.Close())
return buf.Bytes()
}
// ageHeaderLen returns the byte length of blob's age header, i.e. the offset
// of the 16-byte payload nonce that follows it. The header ends with a MAC
// line "--- <mac>\n"; the nonce begins right after that newline.
func ageHeaderLen(t *testing.T, blob []byte) int {
t.Helper()
i := bytes.Index(blob, []byte("\n--- "))
require.GreaterOrEqual(t, i, 0, "age MAC footer line not found")
nl := bytes.IndexByte(blob[i+1:], '\n')
require.GreaterOrEqual(t, nl, 0, "newline ending MAC line not found")
return i + 1 + nl + 1
}
// requireBlobUnreadable asserts that data never decrypts to a plaintext with a
// nil error: either NewReader fails, or reading it does.
func requireBlobUnreadable(t *testing.T, data []byte, id age.Identity) {
t.Helper()
r, err := blobgen.NewReader(bytes.NewReader(data), id)
if err != nil {
return
}
_, err = io.ReadAll(r)
_ = r.Close()
require.Error(t, err, "reading a damaged blob must fail")
}
// errFailWriter is returned by failAfterWriter once its byte limit is passed.
var errFailWriter = errors.New("destination write failed")
// failAfterWriter accepts writes until more than limit bytes have been sent,
// then fails every write. It models a destination that dies mid-blob.
type failAfterWriter struct {
limit int
written int
}
func (f *failAfterWriter) Write(p []byte) (int, error) {
f.written += len(p)
if f.written > f.limit {
return 0, errFailWriter
}
return len(p), nil
}
+49
View File
@@ -0,0 +1,49 @@
package blobgen
import (
"errors"
"io"
)
// ErrOutputTooLarge is returned by a reader from LimitReader once it has
// been asked for more than its limit. It bounds how far an untrusted
// compressed stream may expand, so a small, highly compressible object
// from the store cannot decompress without limit.
var ErrOutputTooLarge = errors.New("output exceeds size limit")
// LimitReader returns a reader that yields at most limit bytes from r and
// then fails with ErrOutputTooLarge. Unlike io.LimitReader, which reports
// a silent io.EOF at the limit (indistinguishable from a stream that
// simply ended), this fails, so a caller decoding or copying the stream
// sees an error rather than a truncated value. A stream of exactly limit
// bytes reads back cleanly to EOF; the first byte beyond it is the error.
func LimitReader(r io.Reader, limit int64) io.Reader {
// remaining counts down from limit+1: the extra byte is the one that,
// if it ever arrives, proves the stream is longer than the limit.
return &limitReader{r: r, remaining: limit + 1}
}
type limitReader struct {
r io.Reader
remaining int64
}
func (l *limitReader) Read(p []byte) (int, error) {
if l.remaining <= 0 {
return 0, ErrOutputTooLarge
}
if int64(len(p)) > l.remaining {
p = p[:l.remaining]
}
n, err := l.r.Read(p)
l.remaining -= int64(n)
if l.remaining <= 0 {
// The (limit+1)th byte was just read: the stream is too long.
return n, ErrOutputTooLarge
}
return n, err
}
+43
View File
@@ -0,0 +1,43 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestLimitReaderPassesExactSize checks that a stream of exactly the limit
// reads back cleanly to EOF: the bound must not reject a legitimate blob
// whose plaintext equals its recorded size.
func TestLimitReaderPassesExactSize(t *testing.T) {
t.Parallel()
const n = 1000
r := blobgen.LimitReader(bytes.NewReader(bytes.Repeat([]byte("a"), n)), n)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.Len(t, got, n)
}
// TestLimitReaderFailsPastLimit feeds a large, highly compressible run of
// zeros — the decompressed output a zip bomb would produce — through a
// small limit and checks it fails within the bound rather than passing
// the whole stream through.
func TestLimitReaderFailsPastLimit(t *testing.T) {
t.Parallel()
const limit = 1000
r := blobgen.LimitReader(
bytes.NewReader(bytes.Repeat([]byte{0}, limit*1000)), limit)
n, err := io.Copy(io.Discard, r)
require.ErrorIs(t, err, blobgen.ErrOutputTooLarge)
require.LessOrEqual(t, n, int64(limit)+1,
"reader must stop within one byte of the limit")
}
+190
View File
@@ -0,0 +1,190 @@
package blobgen_test
import (
"bytes"
"fmt"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestNewReaderWrongIdentity covers issue case 4: opening a blob with an
// identity other than the recipient reports no matching identity.
func TestNewReaderWrongIdentity(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
other, _ := makeIdentity(t)
blob := encryptBlob(t, []byte("secret payload"), recipient)
_, err := blobgen.NewReader(bytes.NewReader(blob), other)
require.Error(t, err)
var noMatch *age.NoIdentityMatchError
assert.ErrorAs(t, err, &noMatch)
}
// TestNewReaderTruncated covers issue case 6: a multi-segment blob cut at
// several points must never read back as valid data. The point immediately
// after the header and nonce is intentionally excluded: it reads as a valid
// empty blob today and is the regression case for
// https://git.eeqj.de/sneak/vaultik/issues/152.
func TestNewReaderTruncated(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
blob := encryptBlob(t, randomBytes(t, 4*65536+123), recipient)
h := ageHeaderLen(t, blob)
require.Greater(t, len(blob), h+ageNonceSize+ageSegmentSize,
"test needs a blob of at least two age segments")
cases := []struct {
name string
size int
}{
{"inside header", h / 2},
{"inside nonce", h + 8},
{"inside first segment", h + ageNonceSize + 100},
{"end of first full segment", h + ageNonceSize + ageSegmentSize},
{"last byte removed", len(blob) - 1},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, blob[:tc.size], id)
})
}
}
// TestNewReaderCorrupted covers issue case 7: one flipped byte in each region
// of a multi-segment blob makes it unreadable.
func TestNewReaderCorrupted(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
blob := encryptBlob(t, randomBytes(t, 4*65536+123), recipient)
h := ageHeaderLen(t, blob)
firstNL := bytes.IndexByte(blob, '\n')
require.Positive(t, firstNL, "header must have a version line")
cases := []struct {
name string
pos int
}{
{"header stanza", firstNL + 5},
{"header MAC line", h - 2},
{"nonce", h + 4},
{"body segment", h + ageNonceSize + 50},
{"final tag", len(blob) - 1},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
corrupt := append([]byte(nil), blob...)
corrupt[tc.pos] ^= 0xff
requireBlobUnreadable(t, corrupt, id)
})
}
}
// TestNewReaderTrailingAndGarbage covers issue case 8: bytes appended after a
// valid blob, empty input, and random garbage each fail to read.
func TestNewReaderTrailingAndGarbage(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
valid := encryptBlob(t, []byte("small payload"), recipient)
appended := append(append([]byte(nil), valid...), []byte("trailing junk")...)
t.Run("appended bytes", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, appended, id)
})
t.Run("empty input", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, []byte{}, id)
})
t.Run("random garbage", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, randomBytes(t, 512), id)
})
}
// TestNewWriterInvalidLevel covers the rejected end of issue case 9: an
// out-of-range compression level errors and writes nothing to the destination.
func TestNewWriterInvalidLevel(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
for _, level := range []int{0, -1, 20} {
t.Run(fmt.Sprintf("level%d", level), func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, level, []string{recipient})
require.ErrorIs(t, err, blobgen.ErrInvalidCompressionLevel)
assert.Nil(t, w)
assert.Zero(t, buf.Len(), "nothing written on an invalid level")
})
}
}
// TestNewWriterInvalidRecipients covers issue case 10: nil and empty recipient
// lists and an unparsable recipient string each error.
func TestNewWriterInvalidRecipients(t *testing.T) {
t.Parallel()
cases := []struct {
name string
recipients []string
}{
{"nil list", nil},
{"empty list", []string{}},
{"invalid recipient string", []string{"not-a-recipient"}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, tc.recipients)
require.Error(t, err)
assert.Nil(t, w)
})
}
}
// TestNewWriterFailingDestination covers issue case 11: a destination that
// fails mid-blob surfaces its error from Write or Close.
func TestNewWriterFailingDestination(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
// The limit clears the age header and nonce so NewWriter succeeds, then
// trips once the compressed body starts flowing.
dst := &failAfterWriter{limit: 512}
w, err := blobgen.NewWriter(dst, 1, []string{recipient})
require.NoError(t, err)
_, writeErr := w.Write(randomBytes(t, 256*1024))
closeErr := w.Close()
assert.True(t, writeErr != nil || closeErr != nil,
"destination failure must surface from Write or Close")
}
+29 -6
View File
@@ -2,6 +2,7 @@ package blobgen
import (
"crypto/sha256"
"errors"
"fmt"
"hash"
"io"
@@ -20,10 +21,12 @@ type Reader struct {
bytesRead int64
}
// NewReader creates a new Reader that decrypts, decompresses, and verifies data
func NewReader(r io.Reader, identity age.Identity) (*Reader, error) {
// NewReader creates a new Reader that decrypts, decompresses, and verifies
// data. Every supplied identity is offered to age.Decrypt, so a blob
// encrypted to any one of them can be read.
func NewReader(r io.Reader, identities ...age.Identity) (*Reader, error) {
// Create decryption reader
decReader, err := age.Decrypt(r, identity)
decReader, err := age.Decrypt(r, identities...)
if err != nil {
return nil, fmt.Errorf("creating decryption reader: %w", err)
}
@@ -50,19 +53,39 @@ func NewReader(r io.Reader, identity age.Identity) (*Reader, error) {
}
// Read implements io.Reader
func (r *Reader) Read(p []byte) (n int, err error) {
n, err = r.teeReader.Read(p)
func (r *Reader) Read(p []byte) (int, error) {
n, err := r.teeReader.Read(p)
r.bytesRead += int64(n)
// When the ciphertext is cut right after the age header plus its
// 16-byte nonce, the age reader's first read fails with
// io.ErrUnexpectedEOF, and the zstd decoder maps that to a clean
// io.EOF at frame start. That makes a truncated stream look like a
// valid empty one. Distinguish the two: on EOF, read once more from
// the age reader. A genuine end leaves it at (0, io.EOF); a truncated
// stream leaves its stored io.ErrUnexpectedEOF, which we surface.
if errors.Is(err, io.EOF) {
var probe [1]byte
m, ageErr := r.decryptor.Read(probe[:])
if m != 0 || !errors.Is(ageErr, io.EOF) {
return n, io.ErrUnexpectedEOF
}
}
return n, err
}
// Close closes the decompressor
func (r *Reader) Close() error {
r.decompressor.Close()
return nil
}
// Sum256 returns the SHA256 hash of all data read
// Sum256 returns the single SHA-256 of the plaintext read so far. This is the
// first hash only; the stored object name is its double hash, which callers
// obtain by passing this digest to DoubleSHA256.
func (r *Reader) Sum256() []byte {
return r.hasher.Sum(nil)
}
+54
View File
@@ -0,0 +1,54 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestMultipleRecipients verifies that data written for several recipients can
// be read back by each recipient's identity. Moved from internal/crypto, which
// held the only multi-recipient test; blobgen is now the sole encryption path.
func TestMultipleRecipients(t *testing.T) {
t.Parallel()
identities := make([]*age.X25519Identity, 3)
recipients := make([]string, 3)
for i := range identities {
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
identities[i] = identity
recipients[i] = identity.Recipient().String()
}
plaintext := []byte("Secret message for multiple recipients")
var encrypted bytes.Buffer
writer, err := blobgen.NewWriter(&encrypted, 3, recipients)
require.NoError(t, err)
_, err = writer.Write(plaintext)
require.NoError(t, err)
require.NoError(t, writer.Close())
// Every recipient's identity must recover the original plaintext.
for i, identity := range identities {
reader, err := blobgen.NewReader(
bytes.NewReader(encrypted.Bytes()), identity)
require.NoError(t, err, "recipient %d should open the reader", i+1)
got, err := io.ReadAll(reader)
require.NoError(t, err, "recipient %d should read the plaintext", i+1)
require.NoError(t, reader.Close())
assert.Equal(t, plaintext, got,
"recipient %d should recover the original plaintext", i+1)
}
}
+132
View File
@@ -0,0 +1,132 @@
package blobgen_test
import (
"bytes"
"crypto/sha256"
"fmt"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// checkRoundTrip writes input through a Writer, reads it back through a Reader,
// and verifies the plaintext, the byte counts, and the content hashes.
func checkRoundTrip(
t *testing.T, id *age.X25519Identity, recipient string,
level int, input []byte,
) {
t.Helper()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, level, []string{recipient})
require.NoError(t, err)
n, err := w.Write(input)
require.NoError(t, err)
assert.Equal(t, len(input), n)
require.NoError(t, w.Close())
require.Equal(t, int64(len(input)), w.BytesWritten())
r, err := blobgen.NewReader(bytes.NewReader(buf.Bytes()), id)
require.NoError(t, err)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.NoError(t, r.Close())
assert.Equal(t, input, got, "decrypted output must equal input")
require.Equal(t, int64(len(input)), r.BytesRead())
// The hash values are checked by decrypting: the reader's single SHA-256
// is the hash of the plaintext, and hashing it once more (DoubleSHA256)
// gives the writer's ContentID.
single := sha256.Sum256(got)
assert.Equal(t, single[:], r.Sum256())
assert.Equal(t, blobgen.DoubleSHA256(r.Sum256()), w.ContentID())
}
// TestWriterReaderRoundTrip covers issue cases 1 and 2: every size round trips
// for both random and compressible data, and the reader hash, its double hash
// and the byte counts all agree.
func TestWriterReaderRoundTrip(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
// Sizes exercise the age segment boundary (64 KiB) from just below to a
// few segments above it, plus the empty and single-byte edges.
sizes := []int{0, 1, 65535, 65536, 65537, 4*65536 + 123}
kinds := []struct {
name string
fill func(*testing.T, int) []byte
}{
{"random", randomBytes},
{"compressible", func(_ *testing.T, n int) []byte {
return compressibleBytes(n)
}},
}
for _, k := range kinds {
for _, size := range sizes {
name := fmt.Sprintf("%s/%d", k.name, size)
t.Run(name, func(t *testing.T) {
t.Parallel()
checkRoundTrip(t, id, recipient, 1, k.fill(t, size))
})
}
}
}
// TestZeroLengthNoWrite covers issue case 3: a Writer closed with no Write at
// all produces the double hash of the empty input, and the blob reads back as
// empty with no error.
func TestZeroLengthNoWrite(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, []string{recipient})
require.NoError(t, err)
require.NoError(t, w.Close())
assert.Equal(t, int64(0), w.BytesWritten())
empty := sha256.Sum256(nil)
doubled := sha256.Sum256(empty[:])
assert.Equal(t, doubled[:], w.ContentID(),
"ContentID of empty input is SHA256(SHA256(\"\"))")
r, err := blobgen.NewReader(bytes.NewReader(buf.Bytes()), id)
require.NoError(t, err)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.NoError(t, r.Close())
assert.Empty(t, got, "empty blob decrypts to empty output")
assert.Equal(t, int64(0), r.BytesRead())
assert.Equal(t, empty[:], r.Sum256())
}
// TestNewWriterValidLevelsRoundTrip covers the accepted end of issue case 9:
// the boundary compression levels 1 and 19 both round trip.
func TestNewWriterValidLevelsRoundTrip(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
input := randomBytes(t, 4096)
for _, level := range []int{1, 19} {
t.Run(fmt.Sprintf("level%d", level), func(t *testing.T) {
t.Parallel()
checkRoundTrip(t, id, recipient, level, input)
})
}
}
+70
View File
@@ -0,0 +1,70 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestReaderRejectsHeaderNonceTruncation guards against a stream cut right
// after the age header plus its 16-byte nonce. age.Decrypt still succeeds on
// such an object, and the zstd decoder maps the age reader's
// io.ErrUnexpectedEOF to a clean io.EOF at frame start, so without the extra
// check the truncated stream would read as a valid empty one. Reading it must
// now fail.
func TestReaderRejectsHeaderNonceTruncation(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
// Encrypting empty plaintext yields header + nonce(16) + a single
// 16-byte final chunk tag. Dropping the trailing tag leaves exactly the
// age header plus its nonce — the truncation point that triggers the bug.
var full bytes.Buffer
w, err := age.Encrypt(&full, identity.Recipient())
require.NoError(t, err)
require.NoError(t, w.Close())
truncated := full.Bytes()[:full.Len()-16]
reader, err := blobgen.NewReader(bytes.NewReader(truncated), identity)
require.NoError(t, err)
defer func() { _ = reader.Close() }()
_, err = io.ReadAll(reader)
require.Error(t, err)
require.ErrorIs(t, err, io.ErrUnexpectedEOF)
}
// TestReaderReadsGenuinelyEmptyBlob confirms the truncation check does not
// reject a legitimately empty payload: a blob written with no data must round
// trip back to zero bytes with no error.
func TestReaderReadsGenuinelyEmptyBlob(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
var encrypted bytes.Buffer
writer, err := blobgen.NewWriter(
&encrypted, 3, []string{identity.Recipient().String()})
require.NoError(t, err)
require.NoError(t, writer.Close())
reader, err := blobgen.NewReader(bytes.NewReader(encrypted.Bytes()), identity)
require.NoError(t, err)
defer func() { _ = reader.Close() }()
data, err := io.ReadAll(reader)
require.NoError(t, err)
require.Empty(t, data)
}
+76 -27
View File
@@ -1,7 +1,11 @@
// Package blobgen implements the blob data pipeline: streaming zstd
// compression, age encryption, and SHA256 content hashing for blob
// creation, plus the matching decrypt/decompress/verify reader.
package blobgen
import (
"crypto/sha256"
"errors"
"fmt"
"hash"
"io"
@@ -11,6 +15,44 @@ import (
"github.com/klauspost/compress/zstd"
)
// DoubleSHA256 returns the double SHA-256 of content whose single SHA-256
// digest is sum: it hashes that digest once more. Stored objects — a blob, and
// the metadata database export — are named by this second hash.
//
// The second hash does not hide whether known content is stored: an attacker
// who can reproduce an object's entire plaintext computes the same name simply
// by hashing twice, exactly as this code does. What limits that is blob
// packing, not the double hash — a blob's name covers all of its concatenated
// chunk plaintext, so a name can be confirmed only by someone who can
// reproduce the whole blob (a snapshot made entirely of known content, or a
// known file large enough to fill blobs on its own). An ordinary file that
// shares a blob with other, unknown data cannot be confirmed this way.
func DoubleSHA256(sum []byte) []byte {
h := sha256.Sum256(sum)
return h[:]
}
// Zstd compression level bounds accepted by NewWriter.
const (
minCompressionLevel = 1
maxCompressionLevel = 19
)
// reservedCompressionCPUs is how many CPUs are left free of zstd
// compression work for I/O and hashing.
const reservedCompressionCPUs = 2
// ErrInvalidCompressionLevel is returned when the zstd compression level
// is outside the accepted 1-19 range.
var ErrInvalidCompressionLevel = errors.New(
"invalid compression level: must be between 1 and 19")
// errInvalidRecipient is returned when a recipient string does not parse as
// an X25519 age1... public key. It omits the value, which can be sensitive.
var errInvalidRecipient = errors.New(
"not a valid X25519 age1... recipient")
// Writer wraps compression and encryption with SHA256 hashing.
// Data flows: input -> tee(hasher, compressor -> encryptor -> destination)
// The hash is computed on the uncompressed input for deterministic content-addressing.
@@ -23,11 +65,15 @@ type Writer struct {
bytesWritten int64
}
// NewWriter creates a new Writer that compresses, encrypts, and hashes data.
// The hash is computed on the uncompressed input for deterministic content-addressing.
func NewWriter(w io.Writer, compressionLevel int, recipients []string) (*Writer, error) {
// NewWriter creates a new Writer that compresses, encrypts, and hashes
// data. The hash is computed on the uncompressed input for deterministic
// content-addressing.
func NewWriter(
w io.Writer, compressionLevel int, recipients []string,
) (*Writer, error) {
// Validate compression level
if err := validateCompressionLevel(compressionLevel); err != nil {
err := validateCompressionLevel(compressionLevel)
if err != nil {
return nil, err
}
@@ -36,11 +82,15 @@ func NewWriter(w io.Writer, compressionLevel int, recipients []string) (*Writer,
// Parse recipients
var ageRecipients []age.Recipient
for _, recipient := range recipients {
for i, recipient := range recipients {
// The recipient string can be sensitive (e.g. a secret key pasted by
// mistake), so the error names its position, never its value.
r, err := age.ParseX25519Recipient(recipient)
if err != nil {
return nil, fmt.Errorf("parsing recipient %s: %w", recipient, err)
return nil, fmt.Errorf("%w: recipient %d", errInvalidRecipient, i)
}
ageRecipients = append(ageRecipients, r)
}
@@ -51,10 +101,7 @@ func NewWriter(w io.Writer, compressionLevel int, recipients []string) (*Writer,
}
// Calculate compression concurrency: CPUs - 2, minimum 1
concurrency := runtime.NumCPU() - 2
if concurrency < 1 {
concurrency = 1
}
concurrency := max(runtime.NumCPU()-reservedCompressionCPUs, 1)
// Create compression writer with encryption as destination
compressor, err := zstd.NewWriter(encWriter,
@@ -63,6 +110,7 @@ func NewWriter(w io.Writer, compressionLevel int, recipients []string) (*Writer,
)
if err != nil {
_ = encWriter.Close()
return nil, fmt.Errorf("creating compression writer: %w", err)
}
@@ -79,37 +127,36 @@ func NewWriter(w io.Writer, compressionLevel int, recipients []string) (*Writer,
}
// Write implements io.Writer
func (w *Writer) Write(p []byte) (n int, err error) {
n, err = w.teeWriter.Write(p)
func (w *Writer) Write(p []byte) (int, error) {
n, err := w.teeWriter.Write(p)
w.bytesWritten += int64(n)
return n, err
}
// Close closes all layers and returns any errors
func (w *Writer) Close() error {
// Close compressor first
if err := w.compressor.Close(); err != nil {
err := w.compressor.Close()
if err != nil {
return fmt.Errorf("closing compressor: %w", err)
}
// Then close encryptor
if err := w.encryptor.Close(); err != nil {
err = w.encryptor.Close()
if err != nil {
return fmt.Errorf("closing encryptor: %w", err)
}
return nil
}
// Sum256 returns the double SHA256 hash of the uncompressed input data.
// Double hashing (SHA256(SHA256(data))) prevents information leakage about
// the plaintext - an attacker cannot confirm existence of known content
// by computing its hash and checking for a matching blob filename.
func (w *Writer) Sum256() []byte {
// First hash: SHA256(plaintext)
firstHash := w.hasher.Sum(nil)
// Second hash: SHA256(firstHash) - this is the blob ID
secondHash := sha256.Sum256(firstHash)
return secondHash[:]
// ContentID returns the double SHA-256 of the uncompressed input data: the
// name under which this content is stored. It is the second hash of the
// running SHA-256, via DoubleSHA256; see that function for what naming content
// this way does and does not hide.
func (w *Writer) ContentID() []byte {
return DoubleSHA256(w.hasher.Sum(nil))
}
// BytesWritten returns the number of uncompressed bytes written
@@ -119,9 +166,11 @@ func (w *Writer) BytesWritten() int64 {
func validateCompressionLevel(level int) error {
// Zstd compression levels: 1-19 (default is 3)
// SpeedFastest = 1, SpeedDefault = 3, SpeedBetterCompression = 7, SpeedBestCompression = 11
if level < 1 || level > 19 {
return fmt.Errorf("invalid compression level %d: must be between 1 and 19", level)
// SpeedFastest = 1, SpeedDefault = 3, SpeedBetterCompression = 7,
// SpeedBestCompression = 11
if level < minCompressionLevel || level > maxCompressionLevel {
return fmt.Errorf("%w: got %d", ErrInvalidCompressionLevel, level)
}
return nil
}
+36 -13
View File
@@ -1,4 +1,4 @@
package blobgen
package blobgen_test
import (
"bytes"
@@ -9,12 +9,16 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestWriterHashIsDoubleHash verifies that Writer.Sum256() returns
// the double hash SHA256(SHA256(plaintext)) for security.
// Double hashing prevents attackers from confirming existence of known content.
// TestWriterHashIsDoubleHash verifies that Writer.ContentID() returns
// SHA256(SHA256(plaintext)). Stored objects are named by this second hash so a
// name is not the plaintext's own SHA-256; this does not stop someone who
// already holds the plaintext from confirming it.
func TestWriterHashIsDoubleHash(t *testing.T) {
t.Parallel()
// Test data - random data that doesn't compress well
testData := make([]byte, 1024*1024) // 1MB
_, err := rand.Read(testData)
@@ -27,7 +31,7 @@ func TestWriterHashIsDoubleHash(t *testing.T) {
var encryptedBuf bytes.Buffer
// Create blobgen writer
writer, err := NewWriter(&encryptedBuf, 3, []string{testRecipient})
writer, err := blobgen.NewWriter(&encryptedBuf, 3, []string{testRecipient})
require.NoError(t, err)
// Write test data
@@ -40,7 +44,7 @@ func TestWriterHashIsDoubleHash(t *testing.T) {
require.NoError(t, err)
// Get the hash from the writer
writerHash := hex.EncodeToString(writer.Sum256())
writerHash := hex.EncodeToString(writer.ContentID())
// Calculate the expected double hash: SHA256(SHA256(plaintext))
firstHash := sha256.Sum256(testData)
@@ -57,16 +61,18 @@ func TestWriterHashIsDoubleHash(t *testing.T) {
// The writer hash should match the double hash
assert.Equal(t, expectedDoubleHash, writerHash,
"Writer.Sum256() should return SHA256(SHA256(plaintext)) for security")
"Writer.ContentID() must be SHA256(SHA256(plaintext))")
// Verify it's NOT the single hash (would leak information)
// It must be the second hash, not the plaintext's own SHA-256.
assert.NotEqual(t, singleHashStr, writerHash,
"Writer hash should not be single hash (would allow content confirmation attacks)")
"Writer hash must be the double hash, not the single SHA-256")
}
// TestWriterDeterministicHash verifies that the same input always produces
// the same hash, even with non-deterministic encryption.
func TestWriterDeterministicHash(t *testing.T) {
t.Parallel()
// Test data
testData := []byte("Hello, World! This is test data for deterministic hashing.")
@@ -76,20 +82,20 @@ func TestWriterDeterministicHash(t *testing.T) {
// Create two writers and verify they produce the same hash
var buf1, buf2 bytes.Buffer
writer1, err := NewWriter(&buf1, 3, []string{testRecipient})
writer1, err := blobgen.NewWriter(&buf1, 3, []string{testRecipient})
require.NoError(t, err)
_, err = writer1.Write(testData)
require.NoError(t, err)
require.NoError(t, writer1.Close())
writer2, err := NewWriter(&buf2, 3, []string{testRecipient})
writer2, err := blobgen.NewWriter(&buf2, 3, []string{testRecipient})
require.NoError(t, err)
_, err = writer2.Write(testData)
require.NoError(t, err)
require.NoError(t, writer2.Close())
hash1 := hex.EncodeToString(writer1.Sum256())
hash2 := hex.EncodeToString(writer2.Sum256())
hash1 := hex.EncodeToString(writer1.ContentID())
hash2 := hex.EncodeToString(writer2.ContentID())
// Hashes should be identical (deterministic)
assert.Equal(t, hash1, hash2, "Same input should produce same hash")
@@ -103,3 +109,20 @@ func TestWriterDeterministicHash(t *testing.T) {
t.Logf("Encrypted size 1: %d bytes", buf1.Len())
t.Logf("Encrypted size 2: %d bytes", buf2.Len())
}
// TestNewWriterSecretKeyNotEchoed verifies that a secret key mistakenly passed
// as a recipient does not appear in the returned error. A recipient string can
// be sensitive, so the error must name only the position, not the value.
func TestNewWriterSecretKeyNotEchoed(t *testing.T) {
t.Parallel()
secretKey := "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GX" +
"VEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5"
var buf bytes.Buffer
_, err := blobgen.NewWriter(&buf, 3, []string{secretKey})
require.Error(t, err)
assert.NotContains(t, err.Error(), secretKey,
"error must not echo the recipient value")
}
+51 -27
View File
@@ -1,16 +1,21 @@
// Package chunker splits input data into content-defined chunks using the
// FastCDC algorithm so that identical data sequences produce identical
// chunks regardless of their position in the file.
package chunker
import (
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"os"
)
// Chunk represents a single chunk of data produced by the content-defined chunking algorithm.
// Each chunk is identified by its SHA256 hash and contains the raw data along with
// its position and size information from the original file.
// Chunk represents a single chunk of data produced by the content-defined
// chunking algorithm. Each chunk is identified by its SHA256 hash and
// contains the raw data along with its position and size information from
// the original file.
type Chunk struct {
Hash string // Content hash of the chunk
Data []byte // Chunk data
@@ -28,6 +33,11 @@ type Chunker struct {
maxChunkSize int
}
// ChunkSizeSpread is the FastCDC-recommended factor between the average
// chunk size and the minimum (avg/spread) and maximum (avg*spread) sizes.
// The largest chunk the chunker can emit is therefore avg*ChunkSizeSpread.
const ChunkSizeSpread = 4
// NewChunker creates a new chunker with the specified average chunk size.
// The actual chunk sizes will vary between avgChunkSize/4 and avgChunkSize*4
// as recommended by the FastCDC algorithm. Typical values for avgChunkSize
@@ -36,27 +46,31 @@ func NewChunker(avgChunkSize int64) *Chunker {
// FastCDC recommends min = avg/4 and max = avg*4
return &Chunker{
avgChunkSize: int(avgChunkSize),
minChunkSize: int(avgChunkSize / 4),
maxChunkSize: int(avgChunkSize * 4),
minChunkSize: int(avgChunkSize / ChunkSizeSpread),
maxChunkSize: int(avgChunkSize * ChunkSizeSpread),
}
}
// ChunkReader splits the reader into content-defined chunks and returns all chunks at once.
// This method loads all chunk data into memory, so it should only be used for
// reasonably sized inputs. For large files or streams, use ChunkReaderStreaming instead.
// ChunkReader splits the reader into content-defined chunks and returns all
// chunks at once. This method loads all chunk data into memory, so it should
// only be used for reasonably sized inputs. For large files or streams, use
// ChunkReaderStreaming instead.
// Returns an error if chunking fails or if reading from the input fails.
func (c *Chunker) ChunkReader(r io.Reader) ([]Chunk, error) {
chunker := AcquireReusableChunker(r, c.minChunkSize, c.avgChunkSize, c.maxChunkSize)
chunker := AcquireReusableChunker(
r, c.minChunkSize, c.avgChunkSize, c.maxChunkSize)
defer chunker.Release()
var chunks []Chunk
offset := int64(0)
for {
chunk, err := chunker.Next()
if err == io.EOF {
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return nil, fmt.Errorf("reading chunk: %w", err)
}
@@ -83,30 +97,36 @@ func (c *Chunker) ChunkReader(r io.Reader) ([]Chunk, error) {
// ChunkCallback is a function called for each chunk as it's processed.
// The callback receives a Chunk containing the hash, data, offset, and size.
// If the callback returns an error, chunk processing stops and the error is propagated.
// If the callback returns an error, chunk processing stops and the error is
// propagated.
type ChunkCallback func(chunk Chunk) error
// ChunkReaderStreaming splits the reader into chunks and calls the callback for each chunk.
// This is the preferred method for processing large files or streams as it doesn't
// accumulate all chunks in memory. The callback is invoked for each chunk as it's
// produced, allowing for streaming processing and immediate storage or transmission.
// Returns the SHA256 hash of the entire file content and an error if chunking fails,
// reading fails, or if the callback returns an error.
func (c *Chunker) ChunkReaderStreaming(r io.Reader, callback ChunkCallback) (string, error) {
// ChunkReaderStreaming splits the reader into chunks and calls the callback
// for each chunk. This is the preferred method for processing large files or
// streams as it doesn't accumulate all chunks in memory. The callback is
// invoked for each chunk as it's produced, allowing for streaming processing
// and immediate storage or transmission.
// Returns the SHA256 hash of the entire file content and an error if
// chunking fails, reading fails, or if the callback returns an error.
func (c *Chunker) ChunkReaderStreaming(
r io.Reader, callback ChunkCallback,
) (string, error) {
// Create a tee reader to calculate full file hash while chunking
fileHasher := sha256.New()
teeReader := io.TeeReader(r, fileHasher)
chunker := AcquireReusableChunker(teeReader, c.minChunkSize, c.avgChunkSize, c.maxChunkSize)
chunker := AcquireReusableChunker(
teeReader, c.minChunkSize, c.avgChunkSize, c.maxChunkSize)
defer chunker.Release()
offset := int64(0)
for {
chunk, err := chunker.Next()
if err == io.EOF {
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return "", fmt.Errorf("reading chunk: %w", err)
}
@@ -114,15 +134,17 @@ func (c *Chunker) ChunkReaderStreaming(r io.Reader, callback ChunkCallback) (str
// Calculate chunk hash
hash := sha256.Sum256(chunk.Data)
// Pass the data directly - caller must process it before we call Next() again
// (chunker reuses its internal buffer, but since we process synchronously
// and completely before continuing, no copy is needed)
if err := callback(Chunk{
// Pass the data directly - caller must process it before we call
// Next() again (chunker reuses its internal buffer, but since we
// process synchronously and completely before continuing, no copy
// is needed)
err = callback(Chunk{
Hash: hex.EncodeToString(hash[:]),
Data: chunk.Data,
Offset: offset,
Size: int64(len(chunk.Data)),
}); err != nil {
})
if err != nil {
return "", fmt.Errorf("callback error: %w", err)
}
@@ -138,12 +160,14 @@ func (c *Chunker) ChunkReaderStreaming(r io.Reader, callback ChunkCallback) (str
// For large files, consider using ChunkReaderStreaming with a file handle instead.
// Returns an error if the file cannot be opened or if chunking fails.
func (c *Chunker) ChunkFile(path string) ([]Chunk, error) {
file, err := os.Open(path)
file, err := os.Open(path) //nolint:gosec // G304: path is caller-supplied by design
if err != nil {
return nil, fmt.Errorf("opening file: %w", err)
}
defer func() {
if err := file.Close(); err != nil && err.Error() != "invalid argument" {
err := file.Close()
if err != nil && err.Error() != "invalid argument" {
// Log error or handle as needed
_ = err
}
+13 -4
View File
@@ -1,11 +1,15 @@
package chunker
package chunker_test
import (
"bytes"
"testing"
"sneak.berlin/go/vaultik/internal/chunker"
)
func TestChunkerExpectedChunkCount(t *testing.T) {
t.Parallel()
tests := []struct {
name string
fileSize int
@@ -38,16 +42,19 @@ func TestChunkerExpectedChunkCount(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
chunker := NewChunker(tt.avgChunkSize)
t.Parallel()
c := chunker.NewChunker(tt.avgChunkSize)
// Create data with some variation to trigger chunk boundaries
data := make([]byte, tt.fileSize)
for i := 0; i < len(data); i++ {
for i := range data {
// Use a pattern that should create boundaries
//nolint:gosec // G115: intentional byte truncation
data[i] = byte((i * 17) ^ (i >> 5))
}
chunks, err := chunker.ChunkReader(bytes.NewReader(data))
chunks, err := c.ChunkReader(bytes.NewReader(data))
if err != nil {
t.Fatalf("chunking failed: %v", err)
}
@@ -59,6 +66,7 @@ func TestChunkerExpectedChunkCount(t *testing.T) {
t.Errorf("too few chunks: got %d, expected at least %d",
len(chunks), tt.minExpected)
}
if len(chunks) > tt.maxExpected {
t.Errorf("too many chunks: got %d, expected at most %d",
len(chunks), tt.maxExpected)
@@ -69,6 +77,7 @@ func TestChunkerExpectedChunkCount(t *testing.T) {
for _, chunk := range chunks {
reconstructed = append(reconstructed, chunk.Data...)
}
if !bytes.Equal(data, reconstructed) {
t.Error("reconstructed data doesn't match original")
}
+45 -24
View File
@@ -1,17 +1,20 @@
package chunker
package chunker_test
import (
"bytes"
"crypto/rand"
"testing"
"sneak.berlin/go/vaultik/internal/chunker"
)
func TestChunker(t *testing.T) {
t.Run("small file produces single chunk", func(t *testing.T) {
chunker := NewChunker(1024 * 1024) // 1MB average
func TestChunkerSmallFileSingleChunk(t *testing.T) {
t.Parallel()
c := chunker.NewChunker(1024 * 1024) // 1MB average
data := bytes.Repeat([]byte("hello"), 100) // 500 bytes
chunks, err := chunker.ChunkReader(bytes.NewReader(data))
chunks, err := c.ChunkReader(bytes.NewReader(data))
if err != nil {
t.Fatalf("chunking failed: %v", err)
}
@@ -23,29 +26,34 @@ func TestChunker(t *testing.T) {
if chunks[0].Size != int64(len(data)) {
t.Errorf("expected chunk size %d, got %d", len(data), chunks[0].Size)
}
})
}
t.Run("large file produces multiple chunks", func(t *testing.T) {
chunker := NewChunker(256 * 1024) // 256KB average chunk size
func TestChunkerLargeFileMultipleChunks(t *testing.T) {
t.Parallel()
c := chunker.NewChunker(256 * 1024) // 256KB average chunk size
// Generate 2MB of random data
data := make([]byte, 2*1024*1024)
if _, err := rand.Read(data); err != nil {
_, err := rand.Read(data)
if err != nil {
t.Fatalf("failed to generate random data: %v", err)
}
chunks, err := chunker.ChunkReader(bytes.NewReader(data))
chunks, err := c.ChunkReader(bytes.NewReader(data))
if err != nil {
t.Fatalf("chunking failed: %v", err)
}
// Should produce multiple chunks - with FastCDC we expect around 8 chunks for 2MB with 256KB average
// Should produce multiple chunks - with FastCDC we expect around 8
// chunks for 2MB with 256KB average
if len(chunks) < 4 || len(chunks) > 16 {
t.Errorf("expected 4-16 chunks, got %d", len(chunks))
}
// Verify chunks reconstruct original data
var reconstructed []byte
reconstructed := make([]byte, 0, len(data))
for _, chunk := range chunks {
reconstructed = append(reconstructed, chunk.Data...)
}
@@ -56,17 +64,22 @@ func TestChunker(t *testing.T) {
// Verify offsets
var expectedOffset int64
for i, chunk := range chunks {
if chunk.Offset != expectedOffset {
t.Errorf("chunk %d: expected offset %d, got %d", i, expectedOffset, chunk.Offset)
t.Errorf("chunk %d: expected offset %d, got %d",
i, expectedOffset, chunk.Offset)
}
expectedOffset += chunk.Size
}
})
}
t.Run("deterministic chunking", func(t *testing.T) {
chunker1 := NewChunker(256 * 1024)
chunker2 := NewChunker(256 * 1024)
func TestChunkerDeterministic(t *testing.T) {
t.Parallel()
chunker1 := chunker.NewChunker(256 * 1024)
chunker2 := chunker.NewChunker(256 * 1024)
// Use deterministic data
data := bytes.Repeat([]byte("abcdefghijklmnopqrstuvwxyz"), 20000) // ~520KB
@@ -83,22 +96,25 @@ func TestChunker(t *testing.T) {
// Should produce same chunks
if len(chunks1) != len(chunks2) {
t.Fatalf("different number of chunks: %d vs %d", len(chunks1), len(chunks2))
t.Fatalf("different number of chunks: %d vs %d",
len(chunks1), len(chunks2))
}
for i := range chunks1 {
if chunks1[i].Hash != chunks2[i].Hash {
t.Errorf("chunk %d: different hashes", i)
}
if chunks1[i].Size != chunks2[i].Size {
t.Errorf("chunk %d: different sizes", i)
}
}
})
}
func TestChunkBoundaries(t *testing.T) {
chunker := NewChunker(256 * 1024) // 256KB average
t.Parallel()
c := chunker.NewChunker(256 * 1024) // 256KB average
// FastCDC uses avg/4 for min and avg*4 for max
avgSize := int64(256 * 1024)
@@ -107,11 +123,13 @@ func TestChunkBoundaries(t *testing.T) {
// Test that minimum chunk size is respected
data := make([]byte, minSize+1024)
if _, err := rand.Read(data); err != nil {
_, err := rand.Read(data)
if err != nil {
t.Fatalf("failed to generate random data: %v", err)
}
chunks, err := chunker.ChunkReader(bytes.NewReader(data))
chunks, err := c.ChunkReader(bytes.NewReader(data))
if err != nil {
t.Fatalf("chunking failed: %v", err)
}
@@ -119,10 +137,13 @@ func TestChunkBoundaries(t *testing.T) {
for i, chunk := range chunks {
// Last chunk can be smaller than minimum
if i < len(chunks)-1 && chunk.Size < minSize {
t.Errorf("chunk %d size %d is below minimum %d", i, chunk.Size, minSize)
t.Errorf("chunk %d size %d is below minimum %d",
i, chunk.Size, minSize)
}
if chunk.Size > maxSize {
t.Errorf("chunk %d size %d exceeds maximum %d", i, chunk.Size, maxSize)
t.Errorf("chunk %d size %d exceeds maximum %d",
i, chunk.Size, maxSize)
}
}
}
+77 -46
View File
@@ -1,6 +1,7 @@
package chunker
import (
"errors"
"io"
"math"
"sync"
@@ -27,32 +28,52 @@ type ReusableChunker struct {
}
// reusableChunkerPool pools ReusableChunker instances to avoid allocations.
//
//nolint:gochecknoglobals // process-wide object pool by design
var reusableChunkerPool = sync.Pool{
New: func() interface{} {
New: func() any {
return &ReusableChunker{}
},
}
// bufferPools contains pools for different buffer sizes.
// Key is the buffer size.
//
//nolint:gochecknoglobals // process-wide buffer pools by design
var bufferPools = sync.Map{}
func getBuffer(size int) []byte {
poolI, _ := bufferPools.LoadOrStore(size, &sync.Pool{
New: func() interface{} {
New: func() any {
buf := make([]byte, size)
return &buf
},
})
pool := poolI.(*sync.Pool)
return *pool.Get().(*[]byte)
pool, ok := poolI.(*sync.Pool)
if !ok {
panic("bufferPools holds a non-pool value")
}
buf, ok := pool.Get().(*[]byte)
if !ok {
panic("buffer pool holds a non-buffer value")
}
return *buf
}
func putBuffer(buf []byte) {
size := cap(buf)
poolI, ok := bufferPools.Load(size)
if ok {
pool := poolI.(*sync.Pool)
pool, isPool := poolI.(*sync.Pool)
if !isPool {
panic("bufferPools holds a non-pool value")
}
b := buf[:size]
pool.Put(&b)
}
@@ -66,17 +87,28 @@ type FastCDCChunk struct {
Fingerprint uint64
}
// AcquireReusableChunker gets a chunker from the pool and initializes it for the given reader.
func AcquireReusableChunker(rd io.Reader, minSize, avgSize, maxSize int) *ReusableChunker {
c := reusableChunkerPool.Get().(*ReusableChunker)
// bufSizeFactor sizes the internal read buffer relative to the maximum
// chunk size so a full chunk plus read-ahead always fits.
const bufSizeFactor = 2
bufSize := maxSize * 2
// AcquireReusableChunker gets a chunker from the pool and initializes it
// for the given reader.
func AcquireReusableChunker(
rd io.Reader, minSize, avgSize, maxSize int,
) *ReusableChunker {
c, ok := reusableChunkerPool.Get().(*ReusableChunker)
if !ok {
panic("reusableChunkerPool holds a non-chunker value")
}
bufSize := maxSize * bufSizeFactor
// Reuse buffer if it's the right size, otherwise get a new one
if c.buf == nil || cap(c.buf) != bufSize {
if c.buf != nil {
putBuffer(c.buf)
}
c.buf = getBuffer(bufSize)
} else {
// Restore buffer to full capacity (may have been truncated by previous EOF)
@@ -108,41 +140,14 @@ func (c *ReusableChunker) Release() {
reusableChunkerPool.Put(c)
}
func (c *ReusableChunker) fillBuffer() error {
n := len(c.buf) - c.cursor
if n >= c.maxSize {
return nil
}
// Move all data after the cursor to the start of the buffer
copy(c.buf[:n], c.buf[c.cursor:])
c.cursor = 0
if c.eof {
c.buf = c.buf[:n]
return nil
}
// Restore buffer to full capacity for reading
c.buf = c.buf[:c.bufSize]
// Fill the rest of the buffer
m, err := io.ReadFull(c.rd, c.buf[n:])
if err == io.EOF || err == io.ErrUnexpectedEOF {
c.buf = c.buf[:n+m]
c.eof = true
} else if err != nil {
return err
}
return nil
}
// Next returns the next chunk or io.EOF when done.
// The returned Data slice is only valid until the next call to Next.
func (c *ReusableChunker) Next() (FastCDCChunk, error) {
if err := c.fillBuffer(); err != nil {
err := c.fillBuffer()
if err != nil {
return FastCDCChunk{}, err
}
if len(c.buf) == 0 {
return FastCDCChunk{}, io.EOF
}
@@ -162,6 +167,37 @@ func (c *ReusableChunker) Next() (FastCDCChunk, error) {
return chunk, nil
}
func (c *ReusableChunker) fillBuffer() error {
n := len(c.buf) - c.cursor
if n >= c.maxSize {
return nil
}
// Move all data after the cursor to the start of the buffer
copy(c.buf[:n], c.buf[c.cursor:])
c.cursor = 0
if c.eof {
c.buf = c.buf[:n]
return nil
}
// Restore buffer to full capacity for reading
c.buf = c.buf[:c.bufSize]
// Fill the rest of the buffer
m, err := io.ReadFull(c.rd, c.buf[n:])
if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) {
c.buf = c.buf[:n+m]
c.eof = true
} else if err != nil {
return err
}
return nil
}
func (c *ReusableChunker) nextChunk(data []byte) (int, uint64) {
fp := uint64(0)
i := c.minSize
@@ -189,14 +225,9 @@ func (c *ReusableChunker) nextChunk(data []byte) (int, uint64) {
return i, fp
}
func min(a, b int) int {
if a < b {
return a
}
return b
}
// 256 random uint64s for the rolling hash function (from FastCDC paper)
//
//nolint:gochecknoglobals // immutable FastCDC gear lookup table
var table = [256]uint64{
0xe80e8d55032474b3, 0x11b25b61f5924e15, 0x03aa5bd82a9eb669, 0xc45a153ef107a38c,
0xeac874b86f0f57b9, 0xa5ccedec95ec79c7, 0xe15a3320ad42ac0a, 0x5ed3583fa63cec15,
+256 -64
View File
@@ -1,17 +1,19 @@
// Package cli implements the vaultik command-line interface: cobra
// commands, fx application wiring, and process-level concerns such as
// signal handling and the PID lock.
package cli
import (
"context"
"errors"
"fmt"
"os"
"os/signal"
"path/filepath"
"strings"
"syscall"
"sync"
"time"
"github.com/adrg/xdg"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
@@ -24,29 +26,61 @@ import (
"sneak.berlin/go/vaultik/internal/vaultik"
)
// AppOptions contains common options for creating the fx application.
// It includes the configuration file path, logging options, and additional
// fx modules and invocations that should be included in the application.
// shutdownTimeout bounds how long a signal-triggered graceful shutdown
// may take before we give up.
const shutdownTimeout = 30 * time.Second
// lockMode says whether a command mutates persistent state — the local
// index database or the remote store — and so must hold the process-wide
// PID lock, or only reads that state and may run alongside a mutator.
type lockMode int
const (
// mutating commands (snapshot create, snapshot purge, snapshot remove,
// prune, remote nuke) write the local index or the remote store. They
// hold the PID lock so that at most one runs at a time.
mutating lockMode = iota
// readOnly commands (info, snapshot list, snapshot verify, remote info,
// snapshot restore) do not write the local index or the remote store,
// so they run without the lock and are never blocked by a running
// mutator. restore writes only to the target directory it is given.
readOnly
)
// AppOptions contains common options for creating and running the fx
// application: the configuration file path, logging options, additional fx
// modules and invocations, and whether the command mutates persistent
// state (which decides whether it takes the PID lock).
type AppOptions struct {
ConfigPath string
LogOptions log.LogOptions
LogOptions log.Options
Modules []fx.Option
Invokes []fx.Option
Mode lockMode
}
// setupGlobals records the startup time and, when an output-suppression
// flag is active, marks the UI writer quiet so that Begin/Complete/
// Info/Notice/Detail/Progress are silenced. Warning and Error are NOT
// silenced — per the documented convention that --quiet suppresses
// non-error output only. The startup banner is printed by CLIEntry
// non-error output only. The startup banner is printed by Entry
// before cobra parses arguments, gated by the same arg-level check.
func setupGlobals(lc fx.Lifecycle, g *globals.Globals, v *vaultik.Vaultik, opts log.LogOptions) {
//
// --json quiets the UI here too, because stdout then carries a JSON
// document and human narration would corrupt it. Unlike Quiet it does
// not lower the stderr log level (issue #112), so --verbose/--debug
// still surface diagnostics alongside the document.
func setupGlobals(
lc fx.Lifecycle, g *globals.Globals, v *vaultik.Vaultik, opts log.Options,
) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
OnStart: func(_ context.Context) error {
g.StartTime = time.Now().UTC()
if opts.Cron || opts.Quiet {
if opts.Cron || opts.Quiet || opts.JSON {
v.UI.SetQuiet(true)
}
return nil
},
})
@@ -56,12 +90,12 @@ func setupGlobals(lc fx.Lifecycle, g *globals.Globals, v *vaultik.Vaultik, opts
// blank line. Used both from the fx hook (for subcommand invocations) and
// from the root cobra Run handler (for `vaultik` with no subcommand).
func writeStartupBanner(w *ui.Writer, startTime time.Time, shortCommit string) {
w.Banner("%s %s by %s (commit %s, built on %s) starting up at %s.",
w.Bannerf("%s %s by %s (commit %s, built on %s) starting up at %s.",
globals.Appname, globals.Version, globals.Author,
shortCommit, globals.CommitDate,
startTime.Format(time.RFC3339))
w.Banner("%s", globals.Homepage)
w.Banner("")
w.Bannerf("%s", globals.Homepage)
w.Bannerf("")
}
// NewApp creates a new fx application with common modules.
@@ -70,7 +104,7 @@ func writeStartupBanner(w *ui.Writer, startTime time.Time, shortCommit string) {
// The returned fx.App is ready to be started with RunApp.
func NewApp(opts AppOptions) *fx.App {
baseModules := []fx.Option{
fx.Supply(config.ConfigPath(opts.ConfigPath)),
fx.Supply(config.Path(opts.ConfigPath)),
fx.Supply(opts.LogOptions),
fx.Provide(globals.New),
fx.Provide(log.New),
@@ -84,12 +118,27 @@ func NewApp(opts AppOptions) *fx.App {
fx.NopLogger,
}
allOptions := append(baseModules, opts.Modules...)
capacity := len(baseModules) + len(opts.Modules) + len(opts.Invokes)
allOptions := make([]fx.Option, 0, capacity)
allOptions = append(allOptions, baseModules...)
allOptions = append(allOptions, opts.Modules...)
allOptions = append(allOptions, opts.Invokes...)
return fx.New(allOptions...)
}
// startupError carries a startup failure message that has been cleaned
// of fx dependency-injection noise. A distinct type (rather than
// errors.New) keeps the dynamic message out of err113's sight while
// preserving the exact user-facing text.
type startupError struct {
msg string
}
func (e *startupError) Error() string {
return e.msg
}
// cleanStartupError strips fx's dependency-injection call-chain noise from
// startup errors. fx wraps the underlying error with messages like
//
@@ -105,80 +154,223 @@ func cleanStartupError(err error) error {
if idx := strings.LastIndex(msg, "): "); idx >= 0 {
msg = msg[idx+3:]
}
return errors.New(msg)
return &startupError{msg: msg}
}
// RunApp starts and stops the fx application within the given context.
// It handles graceful shutdown on interrupt signals (SIGINT, SIGTERM) and
// ensures the application stops cleanly. The function blocks until the
// application completes or is interrupted. Returns an error if startup fails.
// RunApp starts the fx application, blocks until it is asked to stop, and
// then stops it. The app is asked to stop either by an OS interrupt
// (SIGINT/SIGTERM — fx installs its own handler when app.Wait is called) or,
// on normal completion, by the finished operation calling
// Shutdowner.Shutdown(); both arrive on the app.Wait channel.
//
// Stopping runs the fx OnStop hooks, and RunApp does not return until Stop
// returns. On an interrupt the operation's OnStop hook cancels the running
// command and waits for it to unwind — removing its decrypted scratch files —
// so the process cannot proceed to exit mid-cleanup (issue #159). Waiting for
// Stop before returning is what makes that hook effective: routing the
// interrupt through app.Stop and not returning until it completes is required,
// because fx also fires the app.Wait channel on the signal, and an earlier
// version returned on that alone — unwinding to os.Exit while the concurrent
// cleanup still ran. The stop is bounded by shutdownTimeout. Returns an error
// if startup fails.
func RunApp(ctx context.Context, app *fx.App) error {
// Set up signal handling for graceful shutdown
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
// Create a context that will be cancelled on signal
ctx, cancel := context.WithCancel(ctx)
defer cancel()
// Start the app
if err := app.Start(ctx); err != nil {
err := app.Start(ctx)
if err != nil {
return cleanStartupError(err)
}
// Handle shutdown
shutdownComplete := make(chan struct{})
go func() {
defer close(shutdownComplete)
<-sigChan
log.Notice("Received interrupt signal, shutting down gracefully...")
// Block until an interrupt or the finished operation's
// Shutdowner.Shutdown() arrives, then stop the app in this goroutine so we
// return only after its OnStop hooks — including the operation's cleanup
// wait — have run. Detach the stop from ctx's cancellation but keep its
// values, and bound it by shutdownTimeout.
<-app.Wait()
// Create a timeout context for shutdown
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer shutdownCancel()
shutdownCtx, cancel := context.WithTimeout(
context.WithoutCancel(ctx), shutdownTimeout)
defer cancel()
if err := app.Stop(shutdownCtx); err != nil {
err = app.Stop(shutdownCtx)
if err != nil {
log.Error("Error during shutdown", "error", err)
}
}()
// Wait for either the signal handler to complete shutdown or the app to request shutdown
select {
case <-shutdownComplete:
// Shutdown completed via signal
return nil
case <-ctx.Done():
// Context cancelled (shouldn't happen in normal operation)
if err := app.Stop(context.Background()); err != nil {
log.Error("Error stopping app", "error", err)
}
// errReported marks a failure the operation has already shown the user
// (and deliberately withheld under --json). Entry turns it into a
// non-zero exit status without printing anything further, so the error
// line is not doubled. It flows up from RunOperation through cobra to
// Entry.
var errReported = errors.New("operation failed")
// RunOperation runs op against the Vaultik instance inside the fx app
// and turns a failure into a returned error rather than an os.Exit from
// within the goroutine. An os.Exit there skipped main's deferred
// profile writers -- so profiling a failing command yielded a truncated
// profile (issue #75) -- and RunWithApp's PID-lock release, and denied
// the app any graceful shutdown; returning the error to the top runs
// all three.
//
// op runs in a goroutine so OnStart returns promptly and an interrupt
// can still cancel through OnStop; when it finishes, success or failure,
// it triggers shutdown, which is what lets RunWithApp return. On an
// interrupt OnStop cancels op and waits for the goroutine to return, so
// op's cleanup (removing decrypted scratch files) runs before the
// process exits; the wait is bounded by shutdownTimeout. report is
// called with a non-canceled failure so the caller can log it (and
// suppress it under --json) before it becomes errReported. A context
// cancellation is the interrupt path, not a failure: it is neither
// reported nor counted as one.
func RunOperation(
ctx context.Context, opts AppOptions,
op func(v *vaultik.Vaultik) error, report func(err error),
) error {
var (
mu sync.Mutex
failed bool
)
opts.Invokes = append(opts.Invokes,
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
var stop func(context.Context) bool
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
stop = v.StartOperation(func() {
err := op(v)
if err != nil && !errors.Is(err, context.Canceled) {
report(err)
mu.Lock()
failed = true
mu.Unlock()
}
return ctx.Err()
case <-app.Done():
// App finished running (e.g., backup completed)
stopErr := v.Shutdowner.Shutdown()
if stopErr != nil {
log.Error("Failed to shutdown", "error", stopErr)
}
})
return nil
},
// On an interrupt, cancel the operation and wait for it to
// unwind so its cleanup defers (which remove decrypted
// scratch files from the temp directory) run before the
// process exits. The wait is bounded by ctx, the existing
// shutdownTimeout.
OnStop: func(ctx context.Context) error {
if !stop(ctx) {
log.Warn("Shutdown timed out before the operation " +
"finished; decrypted temporary files may remain")
}
return nil
},
})
}))
err := RunWithApp(ctx, opts)
if err != nil {
return err
}
// The goroutine sets failed before triggering the shutdown that lets
// RunWithApp return, so the write is in place by the time we read it.
mu.Lock()
defer mu.Unlock()
if failed {
return errReported
}
return nil
}
// runVaultikApp runs the standard single-operation command lifecycle
// shared by the snapshot list/purge/remove and remote nuke subcommands:
// resolve the config, then run op against the Vaultik instance through
// RunOperation, reporting a failure prefixed with failMsg (suppressed
// while suppressErrors is true, e.g. under --json). mode says whether the
// command takes the PID lock. jsonOutput marks a command whose stdout is a
// JSON document: it quiets the UI but, unlike Quiet, leaves the stderr log
// level alone.
func runVaultikApp(
cmd *cobra.Command, mode lockMode, jsonOutput, suppressErrors bool,
failMsg string, op func(v *vaultik.Vaultik) error,
) error {
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
JSON: jsonOutput,
},
Mode: mode,
}, op, func(err error) {
if suppressErrors {
return
}
log.Error(failMsg, "error", err)
ReportErrorf("%s: %v", failMsg, err)
})
}
// RunWithApp is a helper that creates and runs an fx app with the given options.
// It combines NewApp and RunApp into a single convenient function. This is the
// preferred way to run CLI commands that need the full application context.
// It acquires a PID lock before starting to prevent concurrent instances.
// A mutating command takes the process-wide PID lock before starting so that
// only one runs at a time; a read-only command runs without it and is not
// blocked while a mutator holds the lock (opts.Mode).
func RunWithApp(ctx context.Context, opts AppOptions) error {
// Acquire PID lock to prevent concurrent instances
lockDir := filepath.Join(xdg.DataHome, "vaultik")
release, err := acquireLockIfMutating(opts.Mode,
filepath.Join(xdg.DataHome, "vaultik"))
if err != nil {
return err
}
defer release()
app := NewApp(opts)
return RunApp(ctx, app)
}
// acquireLockIfMutating takes the process-wide PID lock in lockDir for a
// mutating command and returns a function that releases it. A read-only
// command takes no lock, so it returns a no-op release and is never blocked
// while a mutator holds the lock. ErrAlreadyRunning (another mutator holds
// the lock) is surfaced as a "cannot start" error.
func acquireLockIfMutating(mode lockMode, lockDir string) (func(), error) {
if mode != mutating {
return func() {}, nil
}
lock, err := pidlock.Acquire(lockDir)
if err != nil {
if errors.Is(err, pidlock.ErrAlreadyRunning) {
return fmt.Errorf("cannot start: %w", err)
return nil, fmt.Errorf("cannot start: %w", err)
}
return fmt.Errorf("failed to acquire lock: %w", err)
return nil, fmt.Errorf("failed to acquire lock: %w", err)
}
defer func() {
if err := lock.Release(); err != nil {
return func() {
err := lock.Release()
if err != nil {
log.Warn("Failed to release PID lock", "error", err)
}
}()
app := NewApp(opts)
return RunApp(ctx, app)
}, nil
}
+60 -2
View File
@@ -1,11 +1,16 @@
package cli
package cli //nolint:testpackage // needs access to unexported cleanStartupError
import (
"errors"
"path/filepath"
"testing"
"sneak.berlin/go/vaultik/internal/pidlock"
)
func TestCleanStartupError(t *testing.T) {
t.Parallel()
tests := []struct {
name string
in string
@@ -13,7 +18,18 @@ func TestCleanStartupError(t *testing.T) {
}{
{
name: "real fx error chain",
in: `could not build arguments for function "sneak.berlin/go/vaultik/internal/cli".newSnapshotCreateCommand.func1.1 (/Users/user/dev/vaultik/internal/cli/snapshot.go:71): failed to build *vaultik.Vaultik: could not build arguments for function "sneak.berlin/go/vaultik/internal/vaultik".New (/Users/user/dev/vaultik/internal/vaultik/vaultik.go:59): failed to build storage.Storer: received non-nil error from function "sneak.berlin/go/vaultik/internal/storage".NewStorer (/Users/user/dev/vaultik/internal/storage/module.go:23): creating base path: mkdir /Volumes/BACKUPS: permission denied`,
in: `could not build arguments for function ` +
`"sneak.berlin/go/vaultik/internal/cli".newSnapshotCreateCommand.func1.1 ` +
`(/Users/user/dev/vaultik/internal/cli/snapshot.go:71): ` +
`failed to build *vaultik.Vaultik: ` +
`could not build arguments for function ` +
`"sneak.berlin/go/vaultik/internal/vaultik".New ` +
`(/Users/user/dev/vaultik/internal/vaultik/vaultik.go:59): ` +
`failed to build storage.Storer: ` +
`received non-nil error from function ` +
`"sneak.berlin/go/vaultik/internal/storage".NewStorer ` +
`(/Users/user/dev/vaultik/internal/storage/module.go:23): ` +
`creating base path: mkdir /Volumes/BACKUPS: permission denied`,
want: `creating base path: mkdir /Volumes/BACKUPS: permission denied`,
},
{
@@ -30,6 +46,9 @@ func TestCleanStartupError(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
//nolint:err113 // test constructs errors from table input
got := cleanStartupError(errors.New(tt.in)).Error()
if got != tt.want {
t.Errorf("got %q, want %q", got, tt.want)
@@ -37,3 +56,42 @@ func TestCleanStartupError(t *testing.T) {
})
}
}
// TestLockScopedToMutatingCommands proves the partition the PID lock now
// enforces: a read-only command runs while a mutator holds the lock, and
// two mutating commands still mutually exclude.
func TestLockScopedToMutatingCommands(t *testing.T) {
t.Parallel()
lockDir := filepath.Join(t.TempDir(), "vaultik")
// A mutating command takes the process-wide lock.
releaseMutator, err := acquireLockIfMutating(mutating, lockDir)
if err != nil {
t.Fatalf("mutating command could not acquire lock: %v", err)
}
// A read-only command runs to completion even while the lock is held.
releaseReader, err := acquireLockIfMutating(readOnly, lockDir)
if err != nil {
t.Fatalf("read-only command was blocked by held lock: %v", err)
}
releaseReader()
// A second mutating command is refused while the first holds the lock.
_, err = acquireLockIfMutating(mutating, lockDir)
if !errors.Is(err, pidlock.ErrAlreadyRunning) {
t.Fatalf("second mutating command was not excluded, got: %v", err)
}
// Once the first mutator releases, another mutating command may run.
releaseMutator()
release, err := acquireLockIfMutating(mutating, lockDir)
if err != nil {
t.Fatalf("mutating command could not acquire released lock: %v", err)
}
release()
}
+191 -45
View File
@@ -1,6 +1,8 @@
package cli
import (
"bytes"
"errors"
"fmt"
"os"
"os/exec"
@@ -10,6 +12,32 @@ import (
"github.com/spf13/cobra"
"gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/ui"
)
// configFileMode is the permission set for freshly written config files;
// configs may hold S3 credentials, so keep them owner-only.
const configFileMode = 0o600
// configSetArgs is the argument count of `config set <key> <value>`.
const configSetArgs = 2
// configDirMode is the permission set for created config directories;
// parent config dirs (e.g. ~/.config) are conventionally traversable.
const configDirMode = 0o755
// configYAMLIndent matches the 2-space indentation of defaultConfigTemplate,
// so `config set` writes the file back with the same indentation rather than
// yaml.Marshal's 4-space default.
const configYAMLIndent = 2
var (
errConfigExists = errors.New("config file already exists")
errEmptyConfig = errors.New("empty config file")
errKeyNotFound = errors.New("key not found")
errNeedNumericIndex = errors.New("key is a list; use a numeric index")
errIndexOutOfRange = errors.New("index out of range")
errNotMapOrList = errors.New("key is not a map or list")
)
const defaultConfigTemplate = `# vaultik configuration
@@ -18,8 +46,11 @@ const defaultConfigTemplate = `# vaultik configuration
# ─── REQUIRED ────────────────────────────────────────────────────────────────
# Age recipient public keys for encryption.
# Backups are encrypted to ALL listed recipients. Any one of the corresponding
# private keys can decrypt. Generate a keypair with:
# Backups are encrypted to ALL listed recipients; any one of the corresponding
# private keys can decrypt. Adding a recipient later does not re-encrypt data
# already stored: deduplicated chunks and existing blobs stay encrypted to the
# earlier recipients, so a newly added key cannot restore them on its own (see
# docs/REPOSTRUCTURE.md, Accepted Risks). Generate a keypair with:
# age-keygen -o vaultik_backup_private_key.txt
# grep 'public key' vaultik_backup_private_key.txt
age_recipients:
@@ -165,8 +196,8 @@ storage_url: ""
# access_key_id: YOUR_ACCESS_KEY
# secret_access_key: YOUR_SECRET_KEY
# # region: us-east-1 # Default: us-east-1
# # use_ssl: true # Default: true
# # part_size: 5MB # Multipart upload part size. Default: 5MB
# # For the s3:// form, disable TLS with ?ssl=false in the URL, not use_ssl.
# ─── OPTIONAL ────────────────────────────────────────────────────────────────
@@ -185,6 +216,8 @@ storage_url: ""
# chunk_size: 10MB
# Maximum blob size before splitting into a new blob.
# Must be at least four times chunk_size (the largest chunk the chunker can
# emit); a smaller limit would let a single-chunk blob exceed it.
# Accepts: 1GB, 10G, 500MB, etc.
# Default: 10GB
# blob_size_limit: 10GB
@@ -232,24 +265,33 @@ The config is written to the path from --config, $VAULTIK_CONFIG, or
the platform default config directory (e.g. ~/Library/Application Support/
on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
RunE: func(cmd *cobra.Command, _ []string) error {
path := configPathForInit()
if _, err := os.Stat(path); err == nil {
return fmt.Errorf("config file already exists: %s", path)
_, err := os.Stat(path)
if err == nil {
return fmt.Errorf("%w: %s", errConfigExists, path)
}
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, 0o755); err != nil {
err = os.MkdirAll(dir, configDirMode)
if err != nil {
return fmt.Errorf("creating config directory %s: %w", dir, err)
}
if err := os.WriteFile(path, []byte(defaultConfigTemplate), 0o600); err != nil {
err = os.WriteFile(path, []byte(defaultConfigTemplate), configFileMode)
if err != nil {
return fmt.Errorf("writing config file: %w", err)
}
fmt.Printf("Config written to %s\n", path)
fmt.Println("Edit it to set your age_recipients, snapshots, and storage_url.")
// A written-confirmation, not scriptable output: route it
// through the UI so it is styled and --quiet silences it.
out := commandUI(cmd)
out.Infof("Config written to %s.", path)
out.Infof(
"Edit it to set your age_recipients, snapshots, and storage_url.")
return nil
},
}
@@ -261,7 +303,7 @@ func newConfigEditCommand() *cobra.Command {
Use: "edit",
Short: "Open the config file in $EDITOR",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
RunE: func(cmd *cobra.Command, _ []string) error {
path, err := ResolveConfigPath()
if err != nil {
return err
@@ -272,10 +314,12 @@ func newConfigEditCommand() *cobra.Command {
editor = "vi"
}
ed := exec.Command(editor, path)
//nolint:gosec // G204: launching the operator's own $EDITOR is the point
ed := exec.CommandContext(cmd.Context(), editor, path)
ed.Stdin = os.Stdin
ed.Stdout = os.Stdout
ed.Stderr = os.Stderr
return ed.Run()
},
}
@@ -303,8 +347,14 @@ func newConfigGetCommand() *cobra.Command {
return err
}
// The value is scriptable output: it must stay machine-plain
// (no marker, no color) and is never silenced by --quiet, so it
// is written straight to stdout rather than through the UI.
w := cmd.OutOrStdout()
if node.Kind == yaml.ScalarNode {
fmt.Println(node.Value)
_, _ = fmt.Fprintln(w, node.Value)
return nil
}
@@ -312,7 +362,9 @@ func newConfigGetCommand() *cobra.Command {
if err != nil {
return fmt.Errorf("marshaling value: %w", err)
}
fmt.Print(string(out))
_, _ = fmt.Fprint(w, string(out))
return nil
},
}
@@ -332,52 +384,95 @@ Examples:
vaultik config set storage_url "s3://bucket/prefix?endpoint=host&region=us-east-1"
vaultik config set compression_level 9
vaultik config set s3.bucket mybucket # legacy S3 fields still supported`,
Args: cobra.ExactArgs(2),
Args: cobra.ExactArgs(configSetArgs),
RunE: func(cmd *cobra.Command, args []string) error {
path, err := ResolveConfigPath()
if err != nil {
return err
}
return writeConfigSet(commandUI(cmd), path, args[0], args[1])
},
}
}
// writeConfigSet applies key=value to the config at path, writes it back
// owner-only, and confirms the write by naming just the key through the
// UI writer (styled, and silenced by --quiet). The value is never
// echoed: it may be a secret such as s3.secret_access_key, and captured
// stdout or a pasted terminal would then leak it.
func writeConfigSet(out *ui.Writer, path, key, value string) error {
root, err := loadYAMLFile(path)
if err != nil {
return err
}
if err := yamlPathSet(root, strings.Split(args[0], "."), args[1]); err != nil {
err = yamlPathSet(root, strings.Split(key, "."), value)
if err != nil {
return err
}
out, err := yaml.Marshal(root)
data, err := marshalConfigYAML(root)
if err != nil {
return fmt.Errorf("marshaling config: %w", err)
}
mode := os.FileMode(0o600)
if info, err := os.Stat(path); err == nil {
mode = info.Mode().Perm()
}
if err := os.WriteFile(path, out, mode); err != nil {
err = os.WriteFile(path, data, configFileMode)
if err != nil {
return fmt.Errorf("writing config file: %w", err)
}
fmt.Printf("%s = %s\n", args[0], args[1])
return nil
},
// os.WriteFile does not change the mode of a file that already exists,
// so a config that was group- or world-readable stays that way. As it
// may hold S3 credentials, tighten it to owner-only after writing.
info, statErr := os.Stat(path)
if statErr == nil && info.Mode().Perm()&0o044 != 0 {
err = os.Chmod(path, configFileMode)
if err != nil {
return fmt.Errorf("tightening config file permissions: %w", err)
}
}
out.Infof("Set %s.", key)
return nil
}
// marshalConfigYAML renders a config document tree with 2-space indentation,
// matching defaultConfigTemplate. yaml.Marshal defaults to 4 spaces, which
// would reindent the whole file on the first `config set` despite the promise
// to preserve formatting.
func marshalConfigYAML(root *yaml.Node) ([]byte, error) {
var buf bytes.Buffer
enc := yaml.NewEncoder(&buf)
enc.SetIndent(configYAMLIndent)
err := enc.Encode(root)
if err != nil {
return nil, err
}
err = enc.Close()
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}
// loadYAMLFile parses a YAML file into a yaml.Node document tree,
// which preserves comments and ordering for round-tripping.
func loadYAMLFile(path string) (*yaml.Node, error) {
data, err := os.ReadFile(path)
data, err := os.ReadFile(path) //nolint:gosec // G304: config path is operator-supplied
if err != nil {
return nil, fmt.Errorf("reading config file: %w", err)
}
var root yaml.Node
if err := yaml.Unmarshal(data, &root); err != nil {
err = yaml.Unmarshal(data, &root)
if err != nil {
return nil, fmt.Errorf("parsing config file: %w", err)
}
@@ -399,8 +494,9 @@ func yamlPathGet(root *yaml.Node, keys []string) (*yaml.Node, error) {
node := root
if node.Kind == yaml.DocumentNode {
if len(node.Content) == 0 {
return nil, fmt.Errorf("empty config file")
return nil, errEmptyConfig
}
node = node.Content[0]
}
@@ -408,27 +504,40 @@ func yamlPathGet(root *yaml.Node, keys []string) (*yaml.Node, error) {
switch node.Kind {
case yaml.MappingNode:
found := false
for j := 0; j+1 < len(node.Content); j += 2 {
if node.Content[j].Value == key {
node = node.Content[j+1]
found = true
break
}
}
if !found {
return nil, fmt.Errorf("key not found: %s", strings.Join(keys[:i+1], "."))
return nil, fmt.Errorf("%w: %s",
errKeyNotFound, strings.Join(keys[:i+1], "."))
}
case yaml.SequenceNode:
idx, err := strconv.Atoi(key)
if err != nil {
return nil, fmt.Errorf("key %q is a list; use a numeric index", strings.Join(keys[:i], "."))
return nil, fmt.Errorf("%w: %s",
errNeedNumericIndex, strings.Join(keys[:i], "."))
}
if idx < 0 || idx >= len(node.Content) {
return nil, fmt.Errorf("index %d out of range for %s (len %d)", idx, strings.Join(keys[:i], "."), len(node.Content))
return nil, fmt.Errorf("%w: index %d for %s (len %d)",
errIndexOutOfRange, idx, strings.Join(keys[:i], "."),
len(node.Content))
}
node = node.Content[idx]
case yaml.DocumentNode, yaml.ScalarNode, yaml.AliasNode:
return nil, fmt.Errorf("%w: %s",
errNotMapOrList, strings.Join(keys[:i], "."))
default:
return nil, fmt.Errorf("key %q is not a map or list", strings.Join(keys[:i], "."))
return nil, fmt.Errorf("%w: %s",
errNotMapOrList, strings.Join(keys[:i], "."))
}
}
@@ -445,6 +554,7 @@ func yamlPathSet(root *yaml.Node, keys []string, value string) error {
if len(node.Content) == 0 {
node.Content = []*yaml.Node{{Kind: yaml.MappingNode}}
}
node = node.Content[0]
}
@@ -453,52 +563,86 @@ func yamlPathSet(root *yaml.Node, keys []string, value string) error {
switch node.Kind {
case yaml.MappingNode:
node = yamlSetInMapping(node, key, value, last)
case yaml.SequenceNode:
next, err := yamlSetInSequence(node, keys, i, value, last)
if err != nil {
return err
}
node = next
case yaml.DocumentNode, yaml.ScalarNode, yaml.AliasNode:
return fmt.Errorf("%w: %s",
errNotMapOrList, strings.Join(keys[:i], "."))
default:
return fmt.Errorf("%w: %s",
errNotMapOrList, strings.Join(keys[:i], "."))
}
}
return nil
}
// yamlSetInMapping resolves (creating if needed) the value node for key
// within a mapping node, setting it to value when it is the final path
// element, and returns the node to descend into.
func yamlSetInMapping(node *yaml.Node, key, value string, last bool) *yaml.Node {
var valueNode *yaml.Node
for j := 0; j+1 < len(node.Content); j += 2 {
if node.Content[j].Value == key {
valueNode = node.Content[j+1]
break
}
}
if valueNode == nil {
keyNode := &yaml.Node{Kind: yaml.ScalarNode, Value: key}
valueNode = &yaml.Node{Kind: yaml.MappingNode}
if last {
valueNode = &yaml.Node{Kind: yaml.ScalarNode, Value: value}
}
node.Content = append(node.Content, keyNode, valueNode)
} else if last {
setScalar(valueNode, value)
}
node = valueNode
return valueNode
}
case yaml.SequenceNode:
idx, err := strconv.Atoi(key)
// yamlSetInSequence indexes (or appends to) a sequence node using the
// numeric path element keys[i], setting the element to value when it is
// the final path element, and returns the node to descend into.
func yamlSetInSequence(
node *yaml.Node, keys []string, i int, value string, last bool,
) (*yaml.Node, error) {
idx, err := strconv.Atoi(keys[i])
if err != nil {
return fmt.Errorf("key %q is a list; use a numeric index", strings.Join(keys[:i], "."))
return nil, fmt.Errorf("%w: %s",
errNeedNumericIndex, strings.Join(keys[:i], "."))
}
if idx < 0 || idx > len(node.Content) {
return fmt.Errorf("index %d out of range for %s (len %d)", idx, strings.Join(keys[:i], "."), len(node.Content))
return nil, fmt.Errorf("%w: index %d for %s (len %d)",
errIndexOutOfRange, idx, strings.Join(keys[:i], "."),
len(node.Content))
}
if idx == len(node.Content) {
newNode := &yaml.Node{Kind: yaml.MappingNode}
if last {
newNode = &yaml.Node{Kind: yaml.ScalarNode, Value: value}
}
node.Content = append(node.Content, newNode)
} else if last {
setScalar(node.Content[idx], value)
}
node = node.Content[idx]
default:
return fmt.Errorf("key %q is not a map or list", strings.Join(keys[:i], "."))
}
}
return nil
return node.Content[idx], nil
}
// setScalar overwrites a node in place with a plain scalar value.
@@ -516,8 +660,10 @@ func configPathForInit() string {
if rootFlags.ConfigPath != "" {
return rootFlags.ConfigPath
}
if envPath := os.Getenv("VAULTIK_CONFIG"); envPath != "" {
return envPath
}
return DefaultConfigPath()
}
+155 -10
View File
@@ -1,18 +1,26 @@
package cli
package cli //nolint:testpackage // exercises unexported yamlPathGet/yamlPathSet
import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
"gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/ui"
)
// TestDefaultConfigTemplateParses ensures the init template is valid YAML
// that unmarshals into the Config struct with the expected snapshots.
func TestDefaultConfigTemplateParses(t *testing.T) {
t.Parallel()
var cfg config.Config
if err := yaml.Unmarshal([]byte(defaultConfigTemplate), &cfg); err != nil {
err := yaml.Unmarshal([]byte(defaultConfigTemplate), &cfg)
if err != nil {
t.Fatalf("default config template is not valid YAML: %v", err)
}
@@ -24,9 +32,11 @@ func TestDefaultConfigTemplateParses(t *testing.T) {
if !ok {
t.Fatal("expected 'home' snapshot in default config")
}
if len(home.Paths) == 0 {
t.Error("home snapshot should have at least one path")
}
if len(home.Exclude) == 0 {
t.Error("home snapshot should have exclude patterns")
}
@@ -35,9 +45,11 @@ func TestDefaultConfigTemplateParses(t *testing.T) {
if !ok {
t.Fatal("expected 'apps' snapshot in default config")
}
if len(apps.Paths) != 1 || apps.Paths[0] != "/Applications" {
t.Errorf("apps snapshot should back up /Applications, got %v", apps.Paths)
}
if len(apps.Exclude) == 0 {
t.Error("apps snapshot should have exclude patterns")
}
@@ -58,14 +70,20 @@ snapshots:
func parseTestYAML(t *testing.T) *yaml.Node {
t.Helper()
var root yaml.Node
if err := yaml.Unmarshal([]byte(testYAML), &root); err != nil {
err := yaml.Unmarshal([]byte(testYAML), &root)
if err != nil {
t.Fatalf("parsing test yaml: %v", err)
}
return &root
}
func TestYAMLPathGet(t *testing.T) {
t.Parallel()
root := parseTestYAML(t)
tests := []struct {
@@ -86,16 +104,21 @@ func TestYAMLPathGet(t *testing.T) {
for _, tt := range tests {
t.Run(tt.path, func(t *testing.T) {
t.Parallel()
node, err := yamlPathGet(root, splitPath(tt.path))
if tt.err {
if err == nil {
t.Fatalf("expected error for %q", tt.path)
}
return
}
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if node.Value != tt.want {
t.Errorf("get %q = %q, want %q", tt.path, node.Value, tt.want)
}
@@ -104,29 +127,40 @@ func TestYAMLPathGet(t *testing.T) {
}
func TestYAMLPathSet(t *testing.T) {
t.Parallel()
root := parseTestYAML(t)
// Overwrite existing nested value
if err := yamlPathSet(root, splitPath("s3.bucket"), "newbucket"); err != nil {
err := yamlPathSet(root, splitPath("s3.bucket"), "newbucket")
if err != nil {
t.Fatalf("set s3.bucket: %v", err)
}
// Create new nested key with intermediate map
if err := yamlPathSet(root, splitPath("s3.endpoint"), "s3.example.com"); err != nil {
err = yamlPathSet(root, splitPath("s3.endpoint"), "s3.example.com")
if err != nil {
t.Fatalf("set s3.endpoint: %v", err)
}
if err := yamlPathSet(root, splitPath("newmap.newkey"), "val"); err != nil {
err = yamlPathSet(root, splitPath("newmap.newkey"), "val")
if err != nil {
t.Fatalf("set newmap.newkey: %v", err)
}
// Overwrite a sequence element and append a new one
if err := yamlPathSet(root, splitPath("age_recipients.0"), "age1bbb"); err != nil {
err = yamlPathSet(root, splitPath("age_recipients.0"), "age1bbb")
if err != nil {
t.Fatalf("set age_recipients.0: %v", err)
}
if err := yamlPathSet(root, splitPath("age_recipients.1"), "age1ccc"); err != nil {
err = yamlPathSet(root, splitPath("age_recipients.1"), "age1ccc")
if err != nil {
t.Fatalf("append age_recipients.1: %v", err)
}
if err := yamlPathSet(root, splitPath("age_recipients.5"), "age1ddd"); err == nil {
err = yamlPathSet(root, splitPath("age_recipients.5"), "age1ddd")
if err == nil {
t.Error("expected out-of-range append to fail")
}
@@ -135,9 +169,14 @@ func TestYAMLPathSet(t *testing.T) {
if err != nil {
t.Fatalf("marshal: %v", err)
}
text := string(out)
for _, want := range []string{"newbucket", "s3.example.com", "newkey: val", "# top comment", "# inline comment", "age1bbb", "age1ccc"} {
wants := []string{
"newbucket", "s3.example.com", "newkey: val",
"# top comment", "# inline comment", "age1bbb", "age1ccc",
}
for _, want := range wants {
if !contains(text, want) {
t.Errorf("round-tripped YAML missing %q:\n%s", want, text)
}
@@ -147,11 +186,117 @@ func TestYAMLPathSet(t *testing.T) {
if err != nil {
t.Fatalf("get after set: %v", err)
}
if got.Value != "newbucket" {
t.Errorf("s3.bucket = %q after set, want newbucket", got.Value)
}
}
// TestConfigSetPreservesFormatting asserts the `config set` write path
// (marshalConfigYAML) round-trips a 2-space-indented file without reindenting
// it to yaml.Marshal's 4-space default, and keeps comments.
func TestConfigSetPreservesFormatting(t *testing.T) {
t.Parallel()
root := parseTestYAML(t)
err := yamlPathSet(root, splitPath("s3.bucket"), "newbucket")
if err != nil {
t.Fatalf("set s3.bucket: %v", err)
}
out, err := marshalConfigYAML(root)
if err != nil {
t.Fatalf("marshal: %v", err)
}
text := string(out)
for _, want := range []string{"# top comment", "# inline comment"} {
if !contains(text, want) {
t.Errorf("round-tripped YAML dropped comment %q:\n%s", want, text)
}
}
// Nested map keys stay at 2-space indent; the bug reindented them to 4.
if !contains(text, "\n bucket: newbucket") {
t.Errorf("expected 2-space indent for s3.bucket, got:\n%s", text)
}
if contains(text, "\n bucket:") {
t.Errorf("s3.bucket reindented to 4 spaces:\n%s", text)
}
// Sequence items under a key also stay at 2 spaces.
if !contains(text, "\n - age1aaa") {
t.Errorf("expected 2-space indent for sequence item, got:\n%s", text)
}
}
// TestWriteConfigSetHidesSecret checks that setting a secret key prints
// only the key name, never the value, to the confirmation output.
func TestWriteConfigSetHidesSecret(t *testing.T) {
t.Parallel()
const secret = "SUPERSECRETVALUE"
path := filepath.Join(t.TempDir(), "config.yaml")
err := os.WriteFile(path, []byte("version: 1\n"), 0o600)
if err != nil {
t.Fatalf("seed config: %v", err)
}
var buf bytes.Buffer
err = writeConfigSet(ui.NewWithColor(&buf, false), path,
"s3.secret_access_key", secret)
if err != nil {
t.Fatalf("writeConfigSet: %v", err)
}
if strings.Contains(buf.String(), secret) {
t.Errorf("output echoed the secret value: %q", buf.String())
}
if !strings.Contains(buf.String(), "s3.secret_access_key") {
t.Errorf("output did not confirm the key name: %q", buf.String())
}
}
// TestWriteConfigSetTightensMode checks that a pre-existing group- or
// world-readable config is tightened to owner-only after a set, since
// os.WriteFile leaves an existing file's mode untouched.
func TestWriteConfigSetTightensMode(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "config.yaml")
// Seed a world-readable config; the loose mode is the condition under
// test, so gosec's G306 is expected here.
err := os.WriteFile(path, []byte("version: 1\n"), 0o644) //nolint:gosec // G306
if err != nil {
t.Fatalf("seed config: %v", err)
}
var buf bytes.Buffer
err = writeConfigSet(ui.NewWithColor(&buf, false), path,
"compression_level", "9")
if err != nil {
t.Fatalf("writeConfigSet: %v", err)
}
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat config: %v", err)
}
if info.Mode().Perm() != 0o600 {
t.Errorf("config mode = %04o, want 0600", info.Mode().Perm())
}
}
func splitPath(s string) []string {
return strings.Split(s, ".")
}
+37 -21
View File
@@ -18,32 +18,37 @@ func NewDatabaseCommand() *cobra.Command {
}
cmd.AddCommand(
newDatabasePurgeCommand(),
newDatabaseDeleteCommand(),
)
return cmd
}
// newDatabasePurgeCommand creates the database purge command
func newDatabasePurgeCommand() *cobra.Command {
// newDatabaseDeleteCommand creates the database delete command.
// (Renamed from "purge"; the operation removes the SQLite file
// entirely, which is a delete, not a purge of content.)
func newDatabaseDeleteCommand() *cobra.Command {
var force bool
cmd := &cobra.Command{
Use: "purge",
Short: "Delete the local state database",
Use: "delete",
Short: "Delete the local state database file",
Long: `Completely removes the local SQLite state database.
This will erase all local tracking of:
- File metadata and change detection state
- Chunk and blob mappings
- Local snapshot records
- The storage-binding record
The remote storage is NOT affected. After purging, the next backup will
perform a full scan and re-deduplicate against existing remote blobs.
The remote storage is NOT affected. After deletion, the next backup
will perform a full scan and re-deduplicate against existing remote
blobs, and the local index will re-bind to the currently configured
storage destination on that run.
Use --force to skip the confirmation prompt.`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
RunE: func(cmd *cobra.Command, _ []string) error {
// Resolve config path
configPath, err := ResolveConfigPath()
if err != nil {
@@ -57,26 +62,40 @@ Use --force to skip the confirmation prompt.`,
}
dbPath := cfg.IndexPath
out := commandUI(cmd)
// Check if database exists
if _, err := os.Stat(dbPath); os.IsNotExist(err) {
fmt.Printf("Database does not exist: %s\n", dbPath)
_, err = os.Stat(dbPath)
if os.IsNotExist(err) {
out.Infof("Local state database does not exist: %s.", dbPath)
return nil
}
// Confirm unless --force
// Confirm unless --force. The prompt and its immediate result
// are an interactive exchange the operator must see, so they go
// straight to stdout rather than through the UI and --quiet does
// not silence them.
if !force {
fmt.Printf("This will delete the local state database at:\n %s\n\n", dbPath)
fmt.Print("Are you sure? Type 'yes' to confirm: ")
w := cmd.OutOrStdout()
_, _ = fmt.Fprintf(w,
"This will delete the local state database at:\n %s\n\n", dbPath)
_, _ = fmt.Fprint(w, "Are you sure? Type 'yes' to confirm: ")
var confirm string
if _, err := fmt.Scanln(&confirm); err != nil || confirm != "yes" {
fmt.Println("Aborted.")
_, err = fmt.Scanln(&confirm)
if err != nil || confirm != "yes" {
_, _ = fmt.Fprintln(w, "Aborted.")
//nolint:nilerr // a failed/aborted confirmation is a clean abort
return nil
}
}
// Delete the database file
if err := os.Remove(dbPath); err != nil {
err = os.Remove(dbPath)
if err != nil {
return fmt.Errorf("failed to delete database: %w", err)
}
@@ -86,12 +105,9 @@ Use --force to skip the confirmation prompt.`,
_ = os.Remove(walPath) // Ignore errors - files may not exist
_ = os.Remove(shmPath)
rootFlags := GetRootFlags()
if !rootFlags.Quiet {
fmt.Printf("Database purged: %s\n", dbPath)
}
out.Infof("Local state database deleted: %s.", dbPath)
log.Info("Local state database deleted", "path", dbPath)
log.Info("Local state database purged", "path", dbPath)
return nil
},
}
-94
View File
@@ -1,94 +0,0 @@
package cli
import (
"fmt"
"regexp"
"strconv"
"strings"
"time"
)
// parseDuration parses duration strings. Supports standard Go duration format
// (e.g., "3h30m", "1h45m30s") as well as extended units:
// - d: days (e.g., "30d", "7d")
// - w: weeks (e.g., "2w", "4w")
// - mo: months (30 days) (e.g., "6mo", "1mo")
// - y: years (365 days) (e.g., "1y", "2y")
//
// Can combine units: "1y6mo", "2w3d", "1d12h30m"
func parseDuration(s string) (time.Duration, error) {
// First try standard Go duration parsing
if d, err := time.ParseDuration(s); err == nil {
return d, nil
}
// Extended duration parsing
// Check for negative values
if strings.HasPrefix(strings.TrimSpace(s), "-") {
return 0, fmt.Errorf("negative durations are not supported")
}
// Pattern matches: number + unit, repeated
re := regexp.MustCompile(`(\d+(?:\.\d+)?)\s*([a-zA-Z]+)`)
matches := re.FindAllStringSubmatch(s, -1)
if len(matches) == 0 {
return 0, fmt.Errorf("invalid duration format: %q", s)
}
var total time.Duration
for _, match := range matches {
valueStr := match[1]
unit := strings.ToLower(match[2])
value, err := strconv.ParseFloat(valueStr, 64)
if err != nil {
return 0, fmt.Errorf("invalid number %q: %w", valueStr, err)
}
var d time.Duration
switch unit {
// Standard time units
case "ns", "nanosecond", "nanoseconds":
d = time.Duration(value)
case "us", "µs", "microsecond", "microseconds":
d = time.Duration(value * float64(time.Microsecond))
case "ms", "millisecond", "milliseconds":
d = time.Duration(value * float64(time.Millisecond))
case "s", "sec", "second", "seconds":
d = time.Duration(value * float64(time.Second))
case "m", "min", "minute", "minutes":
d = time.Duration(value * float64(time.Minute))
case "h", "hr", "hour", "hours":
d = time.Duration(value * float64(time.Hour))
// Extended units
case "d", "day", "days":
d = time.Duration(value * float64(24*time.Hour))
case "w", "week", "weeks":
d = time.Duration(value * float64(7*24*time.Hour))
case "mo", "month", "months":
// Using 30 days as approximation
d = time.Duration(value * float64(30*24*time.Hour))
case "y", "year", "years":
// Using 365 days as approximation
d = time.Duration(value * float64(365*24*time.Hour))
default:
// Try parsing as standard Go duration unit
testStr := fmt.Sprintf("1%s", unit)
if _, err := time.ParseDuration(testStr); err == nil {
// It's a valid Go duration unit, parse the full value
fullStr := fmt.Sprintf("%g%s", value, unit)
if d, err = time.ParseDuration(fullStr); err != nil {
return 0, fmt.Errorf("invalid duration %q: %w", fullStr, err)
}
} else {
return 0, fmt.Errorf("unknown time unit %q", unit)
}
}
total += d
}
return total, nil
}
-263
View File
@@ -1,263 +0,0 @@
package cli
import (
"testing"
"time"
"github.com/stretchr/testify/assert"
)
func TestParseDuration(t *testing.T) {
tests := []struct {
name string
input string
expected time.Duration
wantErr bool
}{
// Standard Go durations
{
name: "standard seconds",
input: "30s",
expected: 30 * time.Second,
},
{
name: "standard minutes",
input: "45m",
expected: 45 * time.Minute,
},
{
name: "standard hours",
input: "2h",
expected: 2 * time.Hour,
},
{
name: "standard combined",
input: "3h30m",
expected: 3*time.Hour + 30*time.Minute,
},
{
name: "standard complex",
input: "1h45m30s",
expected: 1*time.Hour + 45*time.Minute + 30*time.Second,
},
{
name: "standard with milliseconds",
input: "1s500ms",
expected: 1*time.Second + 500*time.Millisecond,
},
// Extended units - days
{
name: "single day",
input: "1d",
expected: 24 * time.Hour,
},
{
name: "multiple days",
input: "7d",
expected: 7 * 24 * time.Hour,
},
{
name: "fractional days",
input: "1.5d",
expected: 36 * time.Hour,
},
{
name: "days spelled out",
input: "3days",
expected: 3 * 24 * time.Hour,
},
// Extended units - weeks
{
name: "single week",
input: "1w",
expected: 7 * 24 * time.Hour,
},
{
name: "multiple weeks",
input: "4w",
expected: 4 * 7 * 24 * time.Hour,
},
{
name: "weeks spelled out",
input: "2weeks",
expected: 2 * 7 * 24 * time.Hour,
},
// Extended units - months
{
name: "single month",
input: "1mo",
expected: 30 * 24 * time.Hour,
},
{
name: "multiple months",
input: "6mo",
expected: 6 * 30 * 24 * time.Hour,
},
{
name: "months spelled out",
input: "3months",
expected: 3 * 30 * 24 * time.Hour,
},
// Extended units - years
{
name: "single year",
input: "1y",
expected: 365 * 24 * time.Hour,
},
{
name: "multiple years",
input: "2y",
expected: 2 * 365 * 24 * time.Hour,
},
{
name: "years spelled out",
input: "1year",
expected: 365 * 24 * time.Hour,
},
// Combined extended units
{
name: "weeks and days",
input: "2w3d",
expected: 2*7*24*time.Hour + 3*24*time.Hour,
},
{
name: "years and months",
input: "1y6mo",
expected: 365*24*time.Hour + 6*30*24*time.Hour,
},
{
name: "days and hours",
input: "1d12h",
expected: 24*time.Hour + 12*time.Hour,
},
{
name: "complex combination",
input: "1y2mo3w4d5h6m7s",
expected: 365*24*time.Hour + 2*30*24*time.Hour + 3*7*24*time.Hour + 4*24*time.Hour + 5*time.Hour + 6*time.Minute + 7*time.Second,
},
{
name: "with spaces",
input: "1d 12h 30m",
expected: 24*time.Hour + 12*time.Hour + 30*time.Minute,
},
// Edge cases
{
name: "zero duration",
input: "0s",
expected: 0,
},
{
name: "large duration",
input: "10y",
expected: 10 * 365 * 24 * time.Hour,
},
// Error cases
{
name: "empty string",
input: "",
wantErr: true,
},
{
name: "invalid format",
input: "abc",
wantErr: true,
},
{
name: "unknown unit",
input: "5x",
wantErr: true,
},
{
name: "invalid number",
input: "xyzd",
wantErr: true,
},
{
name: "negative not supported",
input: "-5d",
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := parseDuration(tt.input)
if tt.wantErr {
assert.Error(t, err, "expected error for input %q", tt.input)
return
}
assert.NoError(t, err, "unexpected error for input %q", tt.input)
assert.Equal(t, tt.expected, got, "duration mismatch for input %q", tt.input)
})
}
}
func TestParseDurationSpecialCases(t *testing.T) {
// Test that standard Go durations work exactly as expected
standardDurations := []string{
"300ms",
"1.5h",
"2h45m",
"72h",
"1us",
"1µs",
"1ns",
}
for _, d := range standardDurations {
expected, err := time.ParseDuration(d)
assert.NoError(t, err)
got, err := parseDuration(d)
assert.NoError(t, err)
assert.Equal(t, expected, got, "standard duration %q should parse identically", d)
}
}
func TestParseDurationRealWorldExamples(t *testing.T) {
// Test real-world snapshot purge scenarios
tests := []struct {
description string
input string
olderThan time.Duration
}{
{
description: "keep snapshots from last 30 days",
input: "30d",
olderThan: 30 * 24 * time.Hour,
},
{
description: "keep snapshots from last 6 months",
input: "6mo",
olderThan: 6 * 30 * 24 * time.Hour,
},
{
description: "keep snapshots from last year",
input: "1y",
olderThan: 365 * 24 * time.Hour,
},
{
description: "keep snapshots from last week and a half",
input: "1w3d",
olderThan: 10 * 24 * time.Hour,
},
{
description: "keep snapshots from last 90 days",
input: "90d",
olderThan: 90 * 24 * time.Hour,
},
}
for _, tt := range tests {
t.Run(tt.description, func(t *testing.T) {
got, err := parseDuration(tt.input)
assert.NoError(t, err)
assert.Equal(t, tt.olderThan, got)
// Verify the duration makes sense for snapshot purging
assert.Greater(t, got, time.Hour, "snapshot purge duration should be at least an hour")
})
}
}
+73 -23
View File
@@ -1,6 +1,8 @@
package cli
import (
"errors"
"io"
"os"
"strings"
"time"
@@ -9,50 +11,97 @@ import (
"sneak.berlin/go/vaultik/internal/ui"
)
// CLIEntry is the main entry point for the CLI application.
// It prints the startup banner (unless a quiet flag is present in os.Args),
// executes the root cobra command, and routes any returned error through
// the ui.Writer so the user sees a properly formatted "🛑 ERROR:" line.
func CLIEntry() {
if !bannerSuppressedInArgs(os.Args[1:]) {
short := globals.Commit
if len(short) > 12 {
short = short[:12]
}
writeStartupBanner(ui.New(os.Stdout), time.Now().UTC(), short)
}
// shortCommitLen is the number of git commit hash characters shown in
// the startup banner.
const shortCommitLen = 12
// Entry is the main entry point for the CLI application.
// It prints the startup banner to stdout (unless a banner-suppressing
// flag is present in os.Args — see bannerSuppressedInArgs), executes the
// root cobra command, and routes any returned error through the
// ui.Writer so the user sees a properly formatted "🛑 ERROR:" line.
//
// It returns the process exit code (0 on success, 1 on error) rather
// than calling os.Exit, so that main's deferred profile writers run
// before the process ends. See run in cmd/vaultik/main.go.
func Entry() int {
emitStartupBanner(os.Args[1:], os.Stdout)
rootCmd := NewRootCommand()
rootCmd.SilenceErrors = true
if err := rootCmd.Execute(); err != nil {
ReportError("%s", err.Error())
os.Exit(1)
err := rootCmd.Execute()
if err != nil {
// An operation that ran inside the fx app has already reported
// its own failure (and suppressed it under --json); errReported
// says so. Printing it again here would double the error line.
// Every other error — bad arguments, a config that would not
// load — reaches Entry unreported, so it is shown here.
if !errors.Is(err, errReported) {
ReportErrorf("%s", err.Error())
}
return 1
}
return 0
}
// ReportError emits a user-facing error to stderr in the standard
// emitStartupBanner writes the startup banner to w unless args (the
// argument vector with the program name already stripped) contains a
// flag that suppresses it. Split out of Entry so that the decision — the
// only thing standing between a --json invocation and a parseable
// stdout — is reachable from a test without running the whole CLI.
func emitStartupBanner(args []string, w io.Writer) {
if bannerSuppressedInArgs(args) {
return
}
short := globals.Commit
if len(short) > shortCommitLen {
short = short[:shortCommitLen]
}
writeStartupBanner(ui.New(w), time.Now().UTC(), short)
}
// ReportErrorf emits a user-facing error to stderr in the standard
// 🛑 ERROR: format. Use it from goroutine error paths (where returning
// an error to cobra isn't an option) and anywhere else a CLI command
// must surface a failure outside the normal RunE return path.
func ReportError(format string, args ...any) {
ui.New(os.Stderr).Error(format, args...)
func ReportErrorf(format string, args ...any) {
ui.New(os.Stderr).Errorf(format, args...)
}
// bannerSuppressedInArgs reports whether any of args is a flag that
// should suppress the startup banner (--quiet/-q/--cron). Stops at the
// "--" argument terminator. Recognizes both long forms and short -q,
// including combined short flags like "-qv".
// should suppress the startup banner (--quiet/-q/--cron/--json). Stops
// at the "--" argument terminator. Recognizes both long forms and short
// -q, including combined short flags like "-qv".
//
// This scans the raw argument vector because the banner is printed
// before cobra parses anything — deliberately, so that it still appears
// when cobra rejects the arguments and on --help. The consequence is
// that a flag is matched wherever it occurs in the vector, including
// positions where the command it belongs to would not accept it.
// --json is a subcommand flag rather than a persistent one, but so is
// --cron (it exists only on `snapshot create`), so this adds no new
// class of imprecision. The only cost of a false positive is a missing
// decorative banner; the cost of a false negative is a corrupt document
// on stdout, so the scan errs deliberately in that direction.
func bannerSuppressedInArgs(args []string) bool {
for _, a := range args {
if a == "--" {
return false
}
switch a {
case "--quiet", "-q", "--cron":
case "--quiet", "-q", "--cron", "--json":
return true
}
if strings.HasPrefix(a, "--quiet=") || strings.HasPrefix(a, "--cron=") {
if strings.HasPrefix(a, "--quiet=") ||
strings.HasPrefix(a, "--cron=") ||
strings.HasPrefix(a, "--json=") {
return true
}
// Combined short flags like -qv or -vq.
@@ -64,5 +113,6 @@ func bannerSuppressedInArgs(args []string) bool {
}
}
}
return false
}
+300
View File
@@ -0,0 +1,300 @@
package cli //nolint:testpackage // needs access to unexported emitStartupBanner
import (
"bytes"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"testing"
"github.com/adrg/xdg"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Command words and flags used to build argument vectors below. They are
// constants rather than repeated literals so that a rename shows up as a
// compile error in one place.
const (
cmdSnapshot = "snapshot"
cmdList = "list"
cmdCreate = "create"
cmdVerify = "verify"
cmdRemove = "remove"
cmdPrune = "prune"
cmdRemote = "remote"
cmdInfo = "info"
flagJSON = "--json"
flagQuiet = "--quiet"
flagConfig = "--config"
// programName is argv[0] as the real process receives it. Entry
// strips it before scanning, so it has to be present.
programName = "vaultik"
// someSnapshotID is any snapshot identifier: these tests never run
// the command, so it only has to occupy the positional argument.
someSnapshotID = "host_2026-01-01T00:00:00Z"
)
// placeholderJSONDocument stands in for whatever document a --json
// command writes to stdout. `snapshot list --json` with no snapshots
// prints exactly this; the other --json commands print an object rather
// than an array, but this test is not about their shape. It is about
// what is on stdout *before* them, which is the same for all of them
// because Entry prints the banner before cobra has parsed anything and
// therefore before it can know which command is running.
const placeholderJSONDocument = "[]\n"
// jsonArgumentVectors are the argument vectors of every --json
// invocation the CLI accepts, with the program name stripped exactly as
// Entry strips it. Each one must leave stdout untouched by the banner.
//
//nolint:gochecknoglobals // read-only test fixture shared by two tests
var jsonArgumentVectors = map[string][]string{
"snapshot list": {cmdSnapshot, cmdList, flagJSON},
"snapshot verify": {cmdSnapshot, cmdVerify, someSnapshotID, flagJSON},
"snapshot remove": {cmdSnapshot, cmdRemove, someSnapshotID, flagJSON},
"prune": {cmdPrune, flagJSON},
"remote info": {cmdRemote, cmdInfo, flagJSON},
// --json before the subcommand, and with an explicit value: the
// scan is positional, so both forms have to be recognized.
"json first": {flagJSON, cmdSnapshot, cmdList},
"json with value": {cmdSnapshot, cmdList, flagJSON + "=true"},
// A --json invocation that also carries a flag with a value, so the
// scan cannot be fooled by an argument that consumes the next one.
"json with config": {
flagConfig, "/nonexistent/vaultik.yml", cmdSnapshot, cmdList, flagJSON,
},
}
// TestJSONInvocationStdoutIsExactlyOneDocument is the CLI-layer
// regression guard for issue #106: `vaultik snapshot list --json | jq`
// must work with no other flags.
//
// internal/vaultik's TestListSnapshots_JSONStdoutIsOnlyTheDocument
// guards the same contract one layer down, but it calls the library
// function directly and so cannot see Entry, which is where the
// contamination was: the startup banner is written to stdout before
// cobra parses anything, and the suppression scan did not know about
// --json. The two banner lines and the blank line landed ahead of the
// document and `jq` refused the result.
//
// The document is a constant here because this test is about the
// argument vectors, one per --json command; the one that runs a real
// command end to end is TestEntryJSONStdoutIsExactlyOneDocument below.
func TestJSONInvocationStdoutIsExactlyOneDocument(t *testing.T) {
t.Parallel()
for name, argv := range jsonArgumentVectors {
t.Run(name, func(t *testing.T) {
t.Parallel()
var stdout bytes.Buffer
emitStartupBanner(argv, &stdout)
require.Empty(t, stdout.String(),
"nothing may reach stdout ahead of a --json document")
_, err := stdout.WriteString(placeholderJSONDocument)
require.NoError(t, err)
requireExactlyOneJSONDocument(t, stdout.String())
})
}
}
// TestBannerStillPrintedWithoutSuppressingFlag pins the other half of
// the contract. Without it, deleting the banner outright would satisfy
// the test above, and the banner is wanted on interactive invocations.
func TestBannerStillPrintedWithoutSuppressingFlag(t *testing.T) {
t.Parallel()
for name, argv := range map[string][]string{
"no flags": {cmdSnapshot, cmdList},
"verbose": {cmdSnapshot, cmdList, "--verbose"},
"after the terminator": {
cmdSnapshot, "restore", "--", flagJSON,
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
var stdout bytes.Buffer
emitStartupBanner(argv, &stdout)
assert.Contains(t, stdout.String(), "starting up at",
"the banner belongs on invocations that did not opt out")
})
}
}
// TestBannerSuppressedInArgs covers the suppression scan directly,
// including the flags that suppressed the banner before --json joined
// them, so that adding --json cannot regress them.
func TestBannerSuppressedInArgs(t *testing.T) {
t.Parallel()
for name, testCase := range map[string]struct {
args []string
suppressed bool
}{
"quiet long": {[]string{cmdSnapshot, cmdCreate, flagQuiet}, true},
"quiet short": {[]string{cmdSnapshot, cmdCreate, "-q"}, true},
"quiet combined": {[]string{cmdSnapshot, cmdCreate, "-qv"}, true},
"cron": {[]string{cmdSnapshot, cmdCreate, "--cron"}, true},
"json": {[]string{cmdSnapshot, cmdList, flagJSON}, true},
"nothing": {[]string{cmdSnapshot, cmdList}, false},
"empty": {nil, false},
"json after dashes": {
[]string{cmdSnapshot, cmdList, "--", flagJSON}, false,
},
"quiet after dashes": {
[]string{cmdSnapshot, cmdCreate, "--", "-q"}, false,
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
assert.Equal(t, testCase.suppressed,
bannerSuppressedInArgs(testCase.args))
})
}
}
// hermeticConfig is a complete, valid config that needs no network and
// no credentials: file:// storage is exempt from the S3 credential
// checks, and FileStorer over a directory that does not exist lists
// zero objects without erroring. Chunk, blob and compression settings
// are filled in by config.Load.
const hermeticConfig = `age_recipients:
- age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj
snapshots:
test:
paths:
- %s
storage_url: file://%s
index_path: %s
hostname: test-host
`
// TestEntryJSONStdoutIsExactlyOneDocument runs the real thing: Entry,
// with a real argument vector, over the process's real stdout file
// descriptor, all the way through cobra and the fx graph to the
// document. It is the assertion the issue asks for — `vaultik snapshot
// list --json | jq .` with no other flags — with the pipe replaced by a
// decoder.
//
// `snapshot list` is the command chosen because it is the only --json
// command that reaches its document without a populated destination
// store: it reads the local index, streams `metadata/` (empty here),
// and treats a barren destination as an empty list rather than a
// failure.
//
// Not parallel: it replaces os.Args, os.Stdout and the xdg globals.
func TestEntryJSONStdoutIsExactlyOneDocument(t *testing.T) {
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
filepath.Join(dir, "store"),
filepath.Join(dir, "index.sqlite"))
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
xdg.Reload()
t.Cleanup(xdg.Reload)
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
os.Args = []string{
programName, flagConfig, configPath, cmdSnapshot, cmdList, flagJSON,
}
stdout := captureProcessStdout(t, func() { _ = Entry() })
requireExactlyOneJSONDocument(t, stdout)
var snapshots []any
require.NoError(t, json.Unmarshal([]byte(stdout), &snapshots))
assert.Empty(t, snapshots,
"a destination store with no snapshots lists none")
}
// captureProcessStdout redirects the process's own stdout to a pipe for
// the duration of fn and returns what was written to it. The redirection
// has to be at the file-descriptor level rather than through an injected
// writer, because the banner and the JSON encoder reach os.Stdout
// independently and the point of the test is that both land in the same
// place.
//
// Not parallel-safe: os.Stdout is process-global.
func captureProcessStdout(t *testing.T, fn func()) string {
t.Helper()
reader, writer, err := os.Pipe()
require.NoError(t, err)
previous := os.Stdout
os.Stdout = writer
captured := make(chan string, 1)
go func() {
var buf bytes.Buffer
_, _ = io.Copy(&buf, reader)
captured <- buf.String()
}()
fn()
os.Stdout = previous
require.NoError(t, writer.Close())
out := <-captured
require.NoError(t, reader.Close())
return out
}
// requireExactlyOneJSONDocument fails unless stdout decodes as a single
// JSON value with nothing before or after it — the property that makes
// `| jq` work.
func requireExactlyOneJSONDocument(t *testing.T, stdout string) {
t.Helper()
decoder := json.NewDecoder(strings.NewReader(stdout))
var document any
err := decoder.Decode(&document)
require.NoError(t, err,
"stdout must parse as JSON, got:\n%s", stdout)
_, err = decoder.Token()
require.ErrorIs(t, err, io.EOF,
"stdout must hold exactly one JSON document, got:\n%s", stdout)
}
@@ -0,0 +1,140 @@
package cli //nolint:testpackage // shares the prune fixtures and capture helpers
import (
"bytes"
"io"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// staleRecordLogMessage is the local-cleanup audit line CleanupLocalSnapshots
// logs for each stale record. It is exactly the signal issue #112 says a
// machine consumer lost under --json: gated off stdout, and pinned below
// the log level on stderr because --json used to force Quiet.
const staleRecordLogMessage = "Removing stale local snapshot record"
// TestEntryPruneJSONStderrHonoursVerbosity is the end-to-end regression
// guard for issue #112. Under --json the log level must still follow
// --verbose/--debug rather than being pinned to WARN, so the
// local-cleanup records reach stderr under --verbose while stdout stays
// exactly one JSON document; without --verbose they stay below the
// level, as they do without --json.
//
// Both halves are asserted together on the same run, because the fix has
// to keep the document clean (issue #108) while freeing stderr.
//
// Not parallel: it replaces os.Args, os.Stdout, os.Stderr and the xdg
// globals.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntryPruneJSONStderrHonoursVerbosity(t *testing.T) {
for _, testCase := range []struct {
name string
verbose bool
wantOnStderr bool
}{
{
name: "verbose json surfaces the cleanup record on stderr",
verbose: true,
wantOnStderr: true,
},
{
name: "json alone keeps the cleanup record below the level",
verbose: false,
wantOnStderr: false,
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath := writeHermeticPruneConfig(t, true)
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
args := []string{
programName, flagConfig, configPath, cmdPrune, flagJSON,
}
if testCase.verbose {
args = append(args, "--verbose")
}
os.Args = args
stdout, stderr := captureProcessStdoutAndStderr(t,
func() { _ = Entry() })
// The document stays clean in both cases: freeing stderr must
// not regress issue #108.
requireExactlyOneJSONDocument(t, stdout)
if testCase.wantOnStderr {
assert.Contains(t, stderr, staleRecordLogMessage,
"--verbose --json must emit the cleanup record on stderr")
assert.Contains(t, stderr, stalePruneSnapshotID,
"the record must name the snapshot it removed")
} else {
assert.NotContains(t, stderr, staleRecordLogMessage,
"without --verbose the record stays below the log level")
}
})
}
}
// captureProcessStdoutAndStderr redirects both of the process's own
// standard streams to pipes for the duration of fn and returns what was
// written to each. The redirection is at the file-descriptor level
// because the logger binds os.Stderr when it initializes inside fn, and
// the JSON document reaches os.Stdout independently; the point is to see
// where each actually lands.
//
// Not parallel-safe: os.Stdout and os.Stderr are process-global.
func captureProcessStdoutAndStderr(t *testing.T, fn func()) (string, string) {
t.Helper()
outReader, outWriter, err := os.Pipe()
require.NoError(t, err)
errReader, errWriter, err := os.Pipe()
require.NoError(t, err)
previousOut, previousErr := os.Stdout, os.Stderr
os.Stdout, os.Stderr = outWriter, errWriter
capturedOut := drain(outReader)
capturedErr := drain(errReader)
fn()
os.Stdout, os.Stderr = previousOut, previousErr
require.NoError(t, outWriter.Close())
require.NoError(t, errWriter.Close())
out, errOut := <-capturedOut, <-capturedErr
require.NoError(t, outReader.Close())
require.NoError(t, errReader.Close())
return out, errOut
}
// drain copies a reader to a string on a goroutine and delivers the
// result once the writer end is closed.
func drain(reader io.Reader) <-chan string {
captured := make(chan string, 1)
go func() {
var buf bytes.Buffer
_, _ = io.Copy(&buf, reader)
captured <- buf.String()
}()
return captured
}
+165
View File
@@ -0,0 +1,165 @@
package cli //nolint:testpackage // shares hermeticConfig and the capture helpers
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"os"
"path/filepath"
"testing"
"time"
"github.com/adrg/xdg"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
// pruneJSONDocument is the shape `prune --json` writes: the
// PruneBlobsResult document, and nothing else.
//
//nolint:tagliatelle // snake_case is the established JSON output format
type pruneJSONDocument struct {
BlobsFound int `json:"blobs_found"`
BlobsDeleted int `json:"blobs_deleted"`
BytesFreed int64 `json:"bytes_freed"`
}
// stalePruneSnapshotID is seeded into the local index with no manifest
// on the destination store, which is exactly what makes it stale.
const stalePruneSnapshotID = "test-host_test_2026-04-01T09:00:00Z"
// TestEntryPruneJSONStdoutIsExactlyOneDocument is the end-to-end
// regression guard for issue #108: `vaultik prune --json | jq .` must
// work with no other flags.
//
// It runs Entry over the process's real stdout descriptor, through
// cobra and the fx graph, against a hermetic file:// destination store
// — the same construction TestEntryJSONStdoutIsExactlyOneDocument uses
// for `snapshot list`, with the pipe to jq replaced by a decoder.
//
// Both branches of the local-snapshot reconciliation are exercised
// because the three stdout writes that broke this covered all of them:
// one line per stale record and a summary when there were any, and a
// "No stale local snapshots found." line when there were none. No input
// avoided the contamination, so no single branch demonstrates the fix.
//
// Not parallel: it replaces os.Args, os.Stdout and the xdg globals.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and the xdg globals
func TestEntryPruneJSONStdoutIsExactlyOneDocument(t *testing.T) {
for _, testCase := range []struct {
name string
seedStale bool
description string
}{
{
name: "no stale local records",
seedStale: false,
description: "the empty-index branch used to print a 'No stale' line",
},
{
name: "stale local records present",
seedStale: true,
description: "the removal branch used to print a line per record " +
"plus a summary",
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath := writeHermeticPruneConfig(t, testCase.seedStale)
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
os.Args = []string{
programName, flagConfig, configPath, cmdPrune, flagJSON,
}
stdout := captureProcessStdout(t, func() { _ = Entry() })
requireExactlyOneJSONDocument(t, stdout)
var document pruneJSONDocument
require.NoError(t, json.Unmarshal([]byte(stdout), &document),
testCase.description)
// A destination store with no blobs has none to prune. The
// assertion that matters is the one above; this one keeps the
// test honest about which document it decoded.
assert.Equal(t, 0, document.BlobsFound)
})
}
}
// writeHermeticPruneConfig builds a config over a temp directory and, if
// seedStale is set, creates the index database up front with one
// snapshot record that has no counterpart on the destination store.
// Returns the config path.
func writeHermeticPruneConfig(t *testing.T, seedStale bool) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
indexPath := filepath.Join(dir, "index.sqlite")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
filepath.Join(dir, "store"),
indexPath)
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
xdg.Reload()
t.Cleanup(xdg.Reload)
if seedStale {
seedStaleSnapshotRecord(t, indexPath)
}
return configPath
}
// seedStaleSnapshotRecord creates the index database at path and
// inserts one completed snapshot into it. Nothing is written to the
// destination store, so `prune` finds the record stale and removes it —
// the branch that printed a line per record.
func seedStaleSnapshotRecord(t *testing.T, path string) {
t.Helper()
ctx := context.Background()
db, err := database.New(ctx, path)
require.NoError(t, err)
defer func() { require.NoError(t, db.Close()) }()
startedAt := time.Date(2026, 4, 1, 9, 0, 0, 0, time.UTC)
completedAt := startedAt.Add(time.Minute)
snap := &database.Snapshot{
ID: types.SnapshotID(stalePruneSnapshotID),
Hostname: "test-host",
VaultikVersion: "test",
StartedAt: startedAt,
CompletedAt: &completedAt,
}
repos := database.NewRepositories(db)
err = repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
return repos.Snapshots.Create(ctx, tx, snap)
})
require.NoError(t, err)
}
+58
View File
@@ -0,0 +1,58 @@
package cli //nolint:testpackage // shares programName and the capture helpers
import (
"os"
"testing"
"github.com/stretchr/testify/assert"
)
// TestEntryReturnsStatusCode pins the contract main() relies on for
// issue #75: Entry reports success or failure through its return value
// and never calls os.Exit. An os.Exit from inside Entry would skip
// main's deferred profile writers and truncate the profile of a failing
// command. main turns this code into os.Exit only after those defers
// run, so a failing command must come back with a non-zero code rather
// than ending the process here.
//
// Stdout is captured only to keep the banner and command output off the
// test log; the assertion is on the returned code.
//
//nolint:paralleltest // replaces os.Args and rootFlags
func TestEntryReturnsStatusCode(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
want int
}{
{
// version is self-contained: it needs no config and no
// destination store, so it exercises the success path.
name: "successful command returns zero",
args: []string{programName, "version"},
want: 0,
},
{
name: "unknown command returns one",
args: []string{programName, "no-such-command"},
want: 1,
},
} {
t.Run(testCase.name, func(t *testing.T) {
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
os.Args = testCase.args
var code int
_ = captureProcessStdout(t, func() { code = Entry() })
assert.Equal(t, testCase.want, code)
})
}
}
+18 -4
View File
@@ -1,14 +1,18 @@
package cli
package cli_test
import (
"testing"
"sneak.berlin/go/vaultik/internal/cli"
)
// TestCLIEntry ensures the CLI can be imported and basic initialization works
func TestCLIEntry(t *testing.T) {
t.Parallel()
// This test primarily serves as a compilation test
// to ensure all imports resolve correctly
cmd := NewRootCommand()
cmd := cli.NewRootCommand()
if cmd == nil {
t.Fatal("NewRootCommand() returned nil")
}
@@ -18,15 +22,20 @@ func TestCLIEntry(t *testing.T) {
}
// Verify all subcommands are registered
expectedCommands := []string{"config", "snapshot", "store", "prune", "info", "version", "remote", "database"}
expectedCommands := []string{
"config", "snapshot", "prune", "info", "version", "remote", "database",
}
for _, expected := range expectedCommands {
found := false
for _, cmd := range cmd.Commands() {
if cmd.Use == expected || cmd.Name() == expected {
found = true
break
}
}
if !found {
t.Errorf("Expected command '%s' not found", expected)
}
@@ -38,15 +47,20 @@ func TestCLIEntry(t *testing.T) {
t.Errorf("Failed to find snapshot command: %v", err)
} else {
// Check snapshot subcommands
expectedSubCommands := []string{"create", "list", "purge", "verify", "cleanup", "restore"}
expectedSubCommands := []string{
"create", "list", "purge", "verify", "remove", "restore",
}
for _, expected := range expectedSubCommands {
found := false
for _, subcmd := range snapshotCmd.Commands() {
if subcmd.Use == expected || subcmd.Name() == expected {
found = true
break
}
}
if !found {
t.Errorf("Expected snapshot subcommand '%s' not found", expected)
}
+9 -32
View File
@@ -1,11 +1,7 @@
package cli
import (
"context"
"os"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik"
)
@@ -22,7 +18,7 @@ func NewInfoCommand() *cobra.Command {
- Encryption configuration (recipients)
- Local database statistics`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
RunE: func(cmd *cobra.Command, _ []string) error {
// Use unified config resolution
configPath, err := ResolveConfigPath()
if err != nil {
@@ -31,39 +27,20 @@ func NewInfoCommand() *cobra.Command {
// Use the app framework
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
return RunOperation(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
go func() {
if err := v.ShowInfo(); err != nil {
if err != context.Canceled {
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.ShowInfo()
}, func(err error) {
log.Error("Failed to show info", "error", err)
ReportError("Failed to show info: %v", err)
os.Exit(1)
}
}
if err := v.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
v.Cancel()
return nil
},
})
}),
},
ReportErrorf("Failed to show info: %v", err)
})
},
}
+14 -39
View File
@@ -1,11 +1,7 @@
package cli
import (
"context"
"os"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik"
)
@@ -30,7 +26,7 @@ Snapshot create --prune and snapshot remove run the same cleanup
automatically; this command is the manual entry point for the same
work (e.g. after a crashed backup or to reclaim storage).`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
RunE: func(cmd *cobra.Command, _ []string) error {
// Use unified config resolution
configPath, err := ResolveConfigPath()
if err != nil {
@@ -39,46 +35,25 @@ work (e.g. after a crashed backup or to reclaim storage).`,
// Use the app framework like other commands
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
return RunOperation(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet || opts.JSON,
Quiet: rootFlags.Quiet,
JSON: opts.JSON,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
// Start the prune operation in a goroutine
go func() {
// Run the prune operation
if err := v.Prune(opts); err != nil {
if err != context.Canceled {
if !opts.JSON {
log.Error("Prune operation failed", "error", err)
ReportError("Prune failed: %v", err)
}
os.Exit(1)
}
Mode: mutating,
}, func(v *vaultik.Vaultik) error {
return v.Prune(opts)
}, func(err error) {
if opts.JSON {
return
}
// Shutdown the app when prune completes
if err := v.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
log.Debug("Stopping prune operation")
v.Cancel()
return nil
},
})
}),
},
log.Error("Prune operation failed", "error", err)
ReportErrorf("Prune failed: %v", err)
})
},
}
+206
View File
@@ -0,0 +1,206 @@
package cli //nolint:testpackage // sets the unexported rootFlags directly
import (
"bytes"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/spf13/cobra"
)
// setRootFlags overrides the global rootFlags for the duration of one
// test and restores it afterward. These tests must not run in parallel:
// the flags are process-global, so the whole struct is saved and put
// back rather than left mutated for the next test.
func setRootFlags(t *testing.T, f RootFlags) {
t.Helper()
old := rootFlags
rootFlags = f
t.Cleanup(func() { rootFlags = old })
}
// seedFile writes content to a fresh file and returns its path.
func seedFile(t *testing.T, dir, name, content string) string {
t.Helper()
path := filepath.Join(dir, name)
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("seeding %s: %v", name, err)
}
return path
}
// mustExecute runs a command with its output captured and fails the test
// if it errors, returning what the command printed.
func mustExecute(t *testing.T, cmd *cobra.Command, args ...string) string {
t.Helper()
var out bytes.Buffer
cmd.SetOut(&out)
cmd.SetArgs(args)
err := cmd.Execute()
if err != nil {
t.Fatalf("%s failed: %v", cmd.Name(), err)
}
return out.String()
}
// TestVersionQuietSuppressesReport checks that --quiet silences the whole
// version report: it is human-facing output, not a scriptable value.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestVersionQuietSuppressesReport(t *testing.T) {
setRootFlags(t, RootFlags{Quiet: true})
out := mustExecute(t, NewVersionCommand())
if out != "" {
t.Errorf("--quiet version printed %q, want nothing", out)
}
}
// TestConfigGetIgnoresQuiet checks that a config value is printed even
// under --quiet: it is scriptable output a caller depends on, so --quiet
// must not suppress it, and it stays machine-plain (no marker, no color).
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigGetIgnoresQuiet(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "storage_url: file:///mnt/x\n")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigGetCommand(), "storage_url")
if out != "file:///mnt/x\n" {
t.Errorf("config get --quiet = %q, want the plain value", out)
}
}
// TestConfigSetQuietSuppressesConfirmation checks that --quiet silences
// the confirmation line while still writing the value to the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigSetQuietSuppressesConfirmation(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "compression_level: 3\n")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigSetCommand(), "compression_level", "9")
if out != "" {
t.Errorf("--quiet config set printed %q, want nothing", out)
}
data, err := os.ReadFile(path) //nolint:gosec // G304: test-controlled path
if err != nil {
t.Fatalf("reading config back: %v", err)
}
if !strings.Contains(string(data), "compression_level: 9") {
t.Errorf("config set did not write the value under --quiet:\n%s", data)
}
}
// TestConfigSetConfirmsWhenNotQuiet checks that the confirmation names
// the key (styled) when --quiet is not set.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigSetConfirmsWhenNotQuiet(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "compression_level: 3\n")
setRootFlags(t, RootFlags{ConfigPath: path})
out := mustExecute(t, newConfigSetCommand(), "compression_level", "9")
if !strings.Contains(out, "compression_level") {
t.Errorf("config set did not confirm the key: %q", out)
}
}
// TestConfigInitQuietSuppressesConfirmation checks that --quiet silences
// the "config written" confirmation while still writing the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigInitQuietSuppressesConfirmation(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "new-config.yml")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigInitCommand())
if out != "" {
t.Errorf("--quiet config init printed %q, want nothing", out)
}
_, err := os.Stat(path)
if err != nil {
t.Errorf("config init did not write the file under --quiet: %v", err)
}
}
// seedDatabaseDeleteConfig writes a valid config whose index_path is a
// seeded database file, and returns both paths.
func seedDatabaseDeleteConfig(t *testing.T, dir string) (string, string) {
t.Helper()
dbPath := seedFile(t, dir, "index.sqlite", "not-a-real-db")
cfg := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"), filepath.Join(dir, "store"), dbPath)
cfgPath := seedFile(t, dir, "config.yml", cfg)
return dbPath, cfgPath
}
// TestDatabaseDeleteQuietSuppressesMessage checks that --quiet silences
// the "database deleted" line while still removing the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestDatabaseDeleteQuietSuppressesMessage(t *testing.T) {
dir := t.TempDir()
dbPath, cfgPath := seedDatabaseDeleteConfig(t, dir)
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: cfgPath})
out := mustExecute(t, newDatabaseDeleteCommand(), "--force")
if out != "" {
t.Errorf("--quiet database delete printed %q, want nothing", out)
}
_, err := os.Stat(dbPath)
if !os.IsNotExist(err) {
t.Errorf("database delete did not remove the file: stat err = %v", err)
}
}
// TestDatabaseDeleteReportsWhenNotQuiet checks that the deletion is
// reported when --quiet is not set.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestDatabaseDeleteReportsWhenNotQuiet(t *testing.T) {
dir := t.TempDir()
_, cfgPath := seedDatabaseDeleteConfig(t, dir)
setRootFlags(t, RootFlags{ConfigPath: cfgPath})
out := mustExecute(t, newDatabaseDeleteCommand(), "--force")
if !strings.Contains(out, "deleted") {
t.Errorf("database delete did not report the deletion: %q", out)
}
}
+27 -77
View File
@@ -1,16 +1,17 @@
package cli
import (
"context"
"fmt"
"os"
"errors"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// errNukeNeedsForce guards the destructive 'remote nuke' subcommand.
var errNukeNeedsForce = errors.New(
"remote nuke requires --force (this deletes ALL remote snapshots and blobs)")
// NewRemoteCommand creates the remote command and subcommands
func NewRemoteCommand() *cobra.Command {
cmd := &cobra.Command{
@@ -39,55 +40,20 @@ empty and the next backup starts from scratch.
This is destructive and irreversible. Requires --force.`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
RunE: func(cmd *cobra.Command, _ []string) error {
if !force {
return fmt.Errorf("remote nuke requires --force (this deletes ALL remote snapshots and blobs)")
return errNukeNeedsForce
}
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
go func() {
if err := v.NukeRemote(true); err != nil {
if err != context.Canceled {
log.Error("Remote nuke failed", "error", err)
ReportError("Remote nuke failed: %v", err)
os.Exit(1)
}
}
if err := v.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
v.Cancel()
return nil
},
})
}),
},
return runVaultikApp(cmd, mutating, false, false, "Remote nuke failed",
func(v *vaultik.Vaultik) error {
return v.NukeRemote(true)
})
},
}
cmd.Flags().BoolVar(&force, "force", false, "Required: confirm destruction of ALL remote data")
cmd.Flags().BoolVar(&force, "force", false,
"Required: confirm destruction of ALL remote data")
return cmd
}
@@ -105,7 +71,7 @@ func newRemoteInfoCommand() *cobra.Command {
- Count and size of referenced blobs (from all manifests)
- Count and size of orphaned blobs (not referenced by any manifest)`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
RunE: func(cmd *cobra.Command, _ []string) error {
// Use unified config resolution
configPath, err := ResolveConfigPath()
if err != nil {
@@ -113,41 +79,25 @@ func newRemoteInfoCommand() *cobra.Command {
}
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
return RunOperation(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet || jsonOutput,
Quiet: rootFlags.Quiet,
JSON: jsonOutput,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
go func() {
if err := v.RemoteInfo(jsonOutput); err != nil {
if err != context.Canceled {
if !jsonOutput {
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.RemoteInfo(jsonOutput)
}, func(err error) {
if jsonOutput {
return
}
log.Error("Failed to get remote info", "error", err)
ReportError("Failed to get remote info: %v", err)
}
os.Exit(1)
}
}
if err := v.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
v.Cancel()
return nil
},
})
}),
},
ReportErrorf("Failed to get remote info: %v", err)
})
},
}
+56 -15
View File
@@ -1,6 +1,7 @@
package cli
import (
"errors"
"fmt"
"os"
"path/filepath"
@@ -8,8 +9,12 @@ import (
"github.com/adrg/xdg"
"github.com/spf13/cobra"
"sneak.berlin/go/vaultik/internal/ui"
)
// errConfigNotFound is wrapped by all config-resolution failures.
var errConfigNotFound = errors.New("config file not found")
// RootFlags holds global flags that apply to all commands.
// These flags are defined on the root command and inherited by all subcommands.
type RootFlags struct {
@@ -20,6 +25,7 @@ type RootFlags struct {
SkipErrors bool
}
//nolint:gochecknoglobals // cobra persistent flags bind to package state
var rootFlags RootFlags
// NewRootCommand creates the root cobra command for the vaultik CLI.
@@ -34,26 +40,32 @@ public keys and uploads to S3-compatible storage. No private keys are needed
on the source system.`,
SilenceUsage: true,
// Bare 'vaultik' (no subcommand): print help. The banner is
// printed once at process startup by CLIEntry, before cobra
// printed once at process startup by Entry, before cobra
// parses arguments, so it appears even when cobra rejects
// args (e.g. "requires at least 2 arg(s)") and on --help.
Run: func(cmd *cobra.Command, args []string) {
Run: func(cmd *cobra.Command, _ []string) {
_ = cmd.Help()
},
}
// Add global flags
cmd.PersistentFlags().StringVar(&rootFlags.ConfigPath, "config", "", "Path to config file (default: $VAULTIK_CONFIG or platform config dir)")
cmd.PersistentFlags().BoolVarP(&rootFlags.Verbose, "verbose", "v", false, "Enable verbose output")
cmd.PersistentFlags().BoolVar(&rootFlags.Debug, "debug", false, "Enable debug output")
cmd.PersistentFlags().BoolVarP(&rootFlags.Quiet, "quiet", "q", false, "Suppress non-error output")
cmd.PersistentFlags().BoolVar(&rootFlags.SkipErrors, "skip-errors", false, "Continue past per-file errors instead of aborting (applies to snapshot create and restore)")
cmd.PersistentFlags().StringVar(&rootFlags.ConfigPath, "config", "",
"Path to config file (default: $VAULTIK_CONFIG or platform config dir)")
cmd.PersistentFlags().BoolVarP(&rootFlags.Verbose, "verbose", "v", false,
"Enable verbose output")
cmd.PersistentFlags().BoolVar(&rootFlags.Debug, "debug", false,
"Enable debug output")
cmd.PersistentFlags().BoolVarP(&rootFlags.Quiet, "quiet", "q", false,
"Suppress non-error output")
cmd.PersistentFlags().BoolVar(&rootFlags.SkipErrors, "skip-errors", false,
"Skip files that cannot be read when creating a snapshot, or "+
"that cannot be restored when restoring, instead of aborting "+
"(packing and storage errors still abort)")
// Add subcommands
cmd.AddCommand(
NewConfigCommand(),
NewPruneCommand(),
NewStoreCommand(),
NewSnapshotCommand(),
NewInfoCommand(),
NewVersionCommand(),
@@ -70,32 +82,60 @@ func GetRootFlags() RootFlags {
return rootFlags
}
// commandUI returns a UI writer for a command's stdout, in quiet mode
// when the global --quiet flag is set. This is how the pure-cli
// commands (version, config, database) reach internal/ui: color follows
// the writer (a TTY gets color, a captured test buffer does not), and
// --quiet silences the same message classes it silences everywhere else.
func commandUI(cmd *cobra.Command) *ui.Writer {
w := ui.New(cmd.OutOrStdout())
if GetRootFlags().Quiet {
w.SetQuiet(true)
}
return w
}
// ResolveConfigPath resolves the config file path from flags, environment, or default.
// Search order: --config flag, VAULTIK_CONFIG env, XDG config dir, /etc/vaultik/config.yml.
// Search order: --config flag, VAULTIK_CONFIG env, XDG config dir,
// /etc/vaultik/config.yml.
// Explicit paths from --config and $VAULTIK_CONFIG are checked for existence
// so the user gets a clear error instead of a downstream YAML parser failure.
func ResolveConfigPath() (string, error) {
if path := rootFlags.ConfigPath; path != "" {
if _, err := os.Stat(path); err != nil {
return "", fmt.Errorf("config file from --config not found: %s (run 'vaultik config init --config %s' to create it)", path, path)
_, err := os.Stat(path)
if err != nil {
return "", fmt.Errorf(
"%w: from --config: %s (run 'vaultik config init --config %s' to create it)",
errConfigNotFound, path, path)
}
return path, nil
}
if path := os.Getenv("VAULTIK_CONFIG"); path != "" {
if _, err := os.Stat(path); err != nil {
return "", fmt.Errorf("config file from $VAULTIK_CONFIG not found: %s (unset VAULTIK_CONFIG, point it at an existing file, or run 'vaultik config init')", path)
_, err := os.Stat(path) //nolint:gosec // G703: path is operator-supplied by design
if err != nil {
return "", fmt.Errorf(
"%w: from $VAULTIK_CONFIG: %s (unset VAULTIK_CONFIG, point it at "+
"an existing file, or run 'vaultik config init')",
errConfigNotFound, path)
}
return path, nil
}
for _, path := range defaultConfigPaths() {
if _, err := os.Stat(path); err == nil {
_, err := os.Stat(path)
if err == nil {
return path, nil
}
}
return "", fmt.Errorf("no config file found at %s (run 'vaultik config init' to create the default config, or pass --config <path>)", strings.Join(defaultConfigPaths(), " or "))
return "", fmt.Errorf(
"%w: searched %s (run 'vaultik config init' to create the default "+
"config, or pass --config <path>)",
errConfigNotFound, strings.Join(defaultConfigPaths(), " or "))
}
// defaultConfigPaths returns the ordered list of config paths to search.
@@ -115,5 +155,6 @@ func DefaultConfigPath() string {
if os.Getuid() == 0 {
return "/etc/vaultik/config.yml"
}
return filepath.Join(xdg.ConfigHome, "vaultik", "config.yml")
}
+97
View File
@@ -0,0 +1,97 @@
package cli_test
import (
"context"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/cli"
)
// TestRunAppWaitsForOperationCleanupOnShutdown drives RunApp with an fx app
// wired the way RunOperation wires a command: a single lifecycle hook whose
// OnStart launches the operation in its own goroutine and whose OnStop cancels
// it and blocks until that goroutine returns. The operation stands in for a
// restore blocked mid-download — it holds a decrypted "scratch" file and only
// removes it as it unwinds on cancellation.
//
// The app is asked to stop once the operation is running (standing in for an
// OS interrupt; fx delivers a real signal and Shutdowner.Shutdown() on the
// same app.Wait channel, so both drive the identical shutdown path). RunApp
// must not return until app.Stop has run the OnStop hook, so the scratch file
// must be gone by the time RunApp returns. Before the fix RunApp returned as
// soon as the app.Wait/Done channel fired, without running app.Stop, so the
// cleanup never ran and this file would still be on disk (issue #159).
func TestRunAppWaitsForOperationCleanupOnShutdown(t *testing.T) {
t.Parallel()
scratch := filepath.Join(t.TempDir(), "decrypted-scratch")
require.NoError(t, os.WriteFile(scratch, []byte("secret"), 0o600))
// Cancel and reap the operation even if RunApp returns without doing so
// (the buggy path), so the goroutine cannot leak past the test.
opCtx, opCancel := context.WithCancel(context.Background())
t.Cleanup(opCancel)
var stop func(context.Context) bool
app := fx.New(
fx.NopLogger,
fx.Invoke(func(lc fx.Lifecycle, sh fx.Shutdowner) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
done := make(chan struct{})
go func() {
defer close(done)
// Blocked mid-operation until cancelled, then run the
// cleanup an interrupted restore would run.
<-opCtx.Done()
_ = os.Remove(scratch)
}()
stop = func(ctx context.Context) bool {
opCancel()
select {
case <-done:
return true
case <-ctx.Done():
return false
}
}
// Ask the app to stop now that the operation is running.
go func() { _ = sh.Shutdown() }()
return nil
},
OnStop: func(ctx context.Context) error {
stop(ctx)
return nil
},
})
}),
)
done := make(chan error, 1)
go func() { done <- cli.RunApp(context.Background(), app) }()
select {
case err := <-done:
require.NoError(t, err)
case <-time.After(30 * time.Second):
t.Fatal("RunApp did not return after shutdown was requested")
}
_, err := os.Stat(scratch)
require.True(t, os.IsNotExist(err),
"RunApp returned before the operation removed its decrypted scratch file")
}
+116 -303
View File
@@ -1,16 +1,40 @@
package cli
import (
"context"
"errors"
"fmt"
"os"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik"
)
var (
errSnapshotIDRequired = errors.New("snapshot ID required")
errWrongArgCount = errors.New("wrong argument count")
errPurgeCriteriaNeeded = errors.New(
"must specify either --keep-latest or --older-than")
errPurgeCriteriaBoth = errors.New(
"cannot specify both --keep-latest and --older-than")
)
// requireSnapshotIDArg validates that exactly one positional argument
// (the snapshot ID) was supplied, printing help otherwise.
func requireSnapshotIDArg(cmd *cobra.Command, args []string) error {
if len(args) != 1 {
_ = cmd.Help()
if len(args) == 0 {
return errSnapshotIDRequired
}
return fmt.Errorf("%w: expected 1 argument, got %d",
errWrongArgCount, len(args))
}
return nil
}
// NewSnapshotCommand creates the snapshot command and subcommands
func NewSnapshotCommand() *cobra.Command {
cmd := &cobra.Command{
@@ -25,7 +49,6 @@ func NewSnapshotCommand() *cobra.Command {
cmd.AddCommand(newSnapshotPurgeCommand())
cmd.AddCommand(newSnapshotVerifyCommand())
cmd.AddCommand(newSnapshotRemoveCommand())
cmd.AddCommand(newSnapshotCleanupCommand())
cmd.AddCommand(newSnapshotRestoreCommand())
return cmd
@@ -59,53 +82,35 @@ specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
// Use the backup functionality from cli package
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
// --cron suppression is wired through v.UI by setupGlobals.
return RunOperation(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Cron: opts.Cron,
Quiet: rootFlags.Quiet,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
// Start the snapshot creation in a goroutine
go func() {
// --cron suppression is wired through v.UI by setupGlobals.
if err := v.CreateSnapshot(opts); err != nil {
if err != context.Canceled {
Mode: mutating,
}, func(v *vaultik.Vaultik) error {
return v.CreateSnapshot(opts)
}, func(err error) {
log.Error("Snapshot creation failed", "error", err)
ReportError("Snapshot creation failed: %v", err)
os.Exit(1)
}
}
// Shutdown the app when snapshot completes
if err := v.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
log.Debug("Stopping snapshot creation")
// Cancel the Vaultik context
v.Cancel()
return nil
},
})
}),
},
ReportErrorf("Snapshot creation failed: %v", err)
})
},
}
cmd.Flags().BoolVar(&opts.Cron, "cron", false, "Run in cron mode (silent unless error)")
cmd.Flags().BoolVar(&opts.Prune, "prune", false, "After backup, drop older snapshots of the same name and remove orphaned blobs")
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "", "With --prune: keep snapshots newer than this duration (e.g. 4w, 30d, 6mo) instead of only the latest")
cmd.Flags().BoolVar(&opts.Cron, "cron", false,
"Run in cron mode (silent unless warning or error)")
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
"After backup, drop older snapshots of the same name and remove "+
"orphaned blobs")
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
"With --prune: keep snapshots newer than this duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
"instead of only the latest")
return cmd
}
@@ -120,47 +125,11 @@ func newSnapshotListCommand() *cobra.Command {
Short: "List all snapshots",
Long: "Lists all snapshots with their ID, timestamp, and compressed size",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
// Use unified config resolution
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
go func() {
if err := v.ListSnapshots(jsonOutput); err != nil {
if err != context.Canceled {
log.Error("Failed to list snapshots", "error", err)
ReportError("Failed to list snapshots: %v", err)
os.Exit(1)
}
}
if err := v.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
v.Cancel()
return nil
},
})
}),
},
RunE: func(cmd *cobra.Command, _ []string) error {
return runVaultikApp(cmd, readOnly, false, false,
"Failed to list snapshots",
func(v *vaultik.Vaultik) error {
return v.ListSnapshots(jsonOutput)
})
},
}
@@ -183,63 +152,32 @@ Retention is per-snapshot-name: --keep-latest keeps the latest of each
configured snapshot name, not the latest globally. Use --snapshot to
restrict the operation to specific snapshot names.`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
RunE: func(cmd *cobra.Command, _ []string) error {
// Validate flags
if !opts.KeepLatest && opts.OlderThan == "" {
return fmt.Errorf("must specify either --keep-latest or --older-than")
return errPurgeCriteriaNeeded
}
if opts.KeepLatest && opts.OlderThan != "" {
return fmt.Errorf("cannot specify both --keep-latest and --older-than")
return errPurgeCriteriaBoth
}
// Use unified config resolution
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
go func() {
if err := v.PurgeSnapshotsWithOptions(opts); err != nil {
if err != context.Canceled {
log.Error("Failed to purge snapshots", "error", err)
ReportError("Failed to purge snapshots: %v", err)
os.Exit(1)
}
}
if err := v.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
v.Cancel()
return nil
},
})
}),
},
return runVaultikApp(cmd, mutating, false, false,
"Failed to purge snapshots",
func(v *vaultik.Vaultik) error {
return v.PurgeSnapshotsWithOptions(opts)
})
},
}
cmd.Flags().BoolVar(&opts.KeepLatest, "keep-latest", false, "Keep only the latest snapshot of each name")
cmd.Flags().StringVar(&opts.OlderThan, "older-than", "", "Remove snapshots older than duration (e.g., 30d, 6m, 1y)")
cmd.Flags().BoolVar(&opts.KeepLatest, "keep-latest", false,
"Keep only the latest snapshot of each name")
cmd.Flags().StringVar(&opts.OlderThan, "older-than", "",
"Remove snapshots older than duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months)")
cmd.Flags().BoolVar(&opts.Force, "force", false, "Skip confirmation prompt")
cmd.Flags().StringArrayVar(&opts.Names, "snapshot", nil, "Restrict to snapshots with these names (repeat for multiple)")
cmd.Flags().StringArrayVar(&opts.Names, "snapshot", nil,
"Restrict to snapshots with these names (repeat for multiple)")
return cmd
}
@@ -250,18 +188,16 @@ func newSnapshotVerifyCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "verify <snapshot-id>",
Short: "Verify snapshot integrity",
Long: "Verifies that all blobs referenced in a snapshot exist",
Args: func(cmd *cobra.Command, args []string) error {
if len(args) != 1 {
_ = cmd.Help()
if len(args) == 0 {
return fmt.Errorf("snapshot ID required")
}
return fmt.Errorf("expected 1 argument, got %d", len(args))
}
return nil
},
Short: "Check a snapshot's blobs are present with the listed size",
Long: "Checks that every blob the snapshot's manifest lists is present\n" +
"in storage with the size the manifest records, and that the\n" +
"snapshot's encrypted database is present. It does not read blob\n" +
"contents; use --deep to download, decrypt, and re-hash every blob\n" +
"to detect corruption -- integrity, not who wrote it.\n\n" +
"The snapshot may be named by its ID or, on a host with no local\n" +
"index, by the remote key that 'snapshot list' prints for a\n" +
"remote-only snapshot (an unambiguous leading part is enough).",
Args: requireSnapshotIDArg,
RunE: func(cmd *cobra.Command, args []string) error {
snapshotID := args[0]
@@ -272,41 +208,25 @@ func newSnapshotVerifyCommand() *cobra.Command {
}
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
return RunOperation(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet || opts.JSON,
Quiet: rootFlags.Quiet,
JSON: opts.JSON,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
go func() {
if err := v.VerifySnapshotWithOptions(snapshotID, opts); err != nil {
if err != context.Canceled {
if !opts.JSON {
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.VerifySnapshotWithOptions(snapshotID, opts)
}, func(err error) {
if opts.JSON {
return
}
log.Error("Verification failed", "error", err)
ReportError("Verification failed: %v", err)
}
os.Exit(1)
}
}
if err := v.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
v.Cancel()
return nil
},
})
}),
},
ReportErrorf("Verification failed: %v", err)
})
},
}
@@ -322,152 +242,45 @@ func newSnapshotRemoveCommand() *cobra.Command {
opts := &vaultik.RemoveOptions{}
cmd := &cobra.Command{
Use: "remove [snapshot-id]",
Use: "remove <snapshot-id>",
Aliases: []string{"rm"},
Short: "Remove a snapshot from the local database",
Long: `Removes a snapshot from the local database.
Short: "Remove a snapshot from local index and remote metadata",
Long: `Removes a snapshot.
By default, only removes from the local database. Use --remote to also remove
the snapshot metadata from remote storage.
By default, this removes the snapshot from the local index database and
strips the snapshot's metadata from the backup destination store. Blobs
are NOT touched: deleting them requires reading every remaining remote
manifest (the destination store may hold snapshots this host doesn't
know about), which is what 'vaultik prune' does. On success the command
prints the exact 'vaultik prune' invocation to run as a follow-up.
Note: This does NOT remove blobs. Use 'vaultik prune' to remove orphaned blobs
after removing snapshots.
Use --local-only to skip the remote half (e.g. when you want to forget a
snapshot locally without touching the destination store).
Use --all --force to remove all snapshots.`,
Args: func(cmd *cobra.Command, args []string) error {
all, _ := cmd.Flags().GetBool("all")
if all {
if len(args) > 0 {
_ = cmd.Help()
return fmt.Errorf("--all cannot be used with a snapshot ID")
}
return nil
}
if len(args) != 1 {
_ = cmd.Help()
if len(args) == 0 {
return fmt.Errorf("snapshot ID required (or use --all --force)")
}
return fmt.Errorf("expected 1 argument, got %d", len(args))
}
return nil
},
If the remote is unreachable, the local-database removal still completes
and a warning is emitted; rerun 'vaultik prune' once the destination store
is reachable to finish remote cleanup.
To wipe the entire destination store and start over, use 'vaultik remote
nuke --force' — it is the single supported entry point for that.`,
Args: requireSnapshotIDArg,
RunE: func(cmd *cobra.Command, args []string) error {
// Use unified config resolution
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
return runVaultikApp(cmd, mutating, opts.JSON, opts.JSON,
"Failed to remove snapshot",
func(v *vaultik.Vaultik) error {
_, err := v.RemoveSnapshot(args[0], opts)
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet || opts.JSON,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
go func() {
var err error
if opts.All {
_, err = v.RemoveAllSnapshots(opts)
} else {
_, err = v.RemoveSnapshot(args[0], opts)
}
if err != nil {
if err != context.Canceled {
if !opts.JSON {
log.Error("Failed to remove snapshot", "error", err)
ReportError("Failed to remove snapshot: %v", err)
}
os.Exit(1)
}
}
if err := v.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
v.Cancel()
return nil
},
})
}),
},
return err
})
},
}
cmd.Flags().BoolVarP(&opts.Force, "force", "f", false, "Skip confirmation prompt")
cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false, "Show what would be removed without removing")
cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false,
"Show what would be removed without removing")
cmd.Flags().BoolVar(&opts.JSON, "json", false, "Output result as JSON")
cmd.Flags().BoolVar(&opts.Remote, "remote", false, "Also remove snapshot metadata from remote storage")
cmd.Flags().BoolVar(&opts.All, "all", false, "Remove all snapshots (requires --force)")
return cmd
}
// newSnapshotCleanupCommand creates the 'snapshot cleanup' subcommand
func newSnapshotCleanupCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "cleanup",
Short: "Remove stale local snapshot records not found in remote storage",
Long: `Removes local database records for snapshots whose metadata no longer
exists in remote storage. These are typically left behind by incomplete
or interrupted backups.
This command does not delete anything from remote storage.`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
go func() {
if err := v.CleanupLocalSnapshots(); err != nil {
if err != context.Canceled {
log.Error("Cleanup failed", "error", err)
ReportError("Cleanup failed: %v", err)
os.Exit(1)
}
}
if err := v.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
v.Cancel()
return nil
},
})
}),
},
})
},
}
cmd.Flags().BoolVar(&opts.LocalOnly, "local-only", false,
"Skip remote cleanup; only touch the local index")
return cmd
}
+38 -85
View File
@@ -1,18 +1,15 @@
package cli
import (
"context"
"os"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// restoreMinArgs is the minimum positional argument count of
// `snapshot restore <snapshot-id> <target-dir> [paths...]`.
const restoreMinArgs = 2
// RestoreOptions contains options for the restore command
type RestoreOptions struct {
TargetDir string
@@ -20,15 +17,6 @@ type RestoreOptions struct {
Verify bool // Verify restored files after restore
}
// RestoreApp contains all dependencies needed for restore
type RestoreApp struct {
Globals *globals.Globals
Config *config.Config
Storage storage.Storer
Vaultik *vaultik.Vaultik
Shutdowner fx.Shutdowner
}
// newSnapshotRestoreCommand creates the 'snapshot restore' subcommand
func newSnapshotRestoreCommand() *cobra.Command {
opts := &RestoreOptions{}
@@ -38,118 +26,83 @@ func newSnapshotRestoreCommand() *cobra.Command {
Short: "Restore files from a snapshot",
Long: `Download and decrypt files from a backup snapshot.
This command will restore files from the specified snapshot to the target directory.
This command will restore files from the specified snapshot to the
target directory.
If no paths are specified, all files are restored.
If paths are specified, only matching files/directories are restored.
Requires the VAULTIK_AGE_SECRET_KEY environment variable to be set with the age private key.
The snapshot may be named by its ID or, when restoring on a host with no
local index, by the remote key that 'snapshot list' prints for a
remote-only snapshot (an unambiguous leading part is enough).
Requires the age private key in the VAULTIK_AGE_SECRET_KEY environment
variable. The variable may hold the whole age-keygen file (comments and
all of its identities are accepted); read it from the file rather than
typing the key, so it does not land in your shell history:
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
Examples:
# Restore entire snapshot
vaultik snapshot restore myhost_docs_2025-01-01T12:00:00Z /restore
# Restore specific file
vaultik snapshot restore myhost_docs_2025-01-01T12:00:00Z /restore /home/user/important.txt
vaultik snapshot restore myhost_docs_2025-01-01T12:00:00Z /restore \
/home/user/important.txt
# Restore specific directory
vaultik snapshot restore myhost_docs_2025-01-01T12:00:00Z /restore /home/user/documents/
vaultik snapshot restore myhost_docs_2025-01-01T12:00:00Z /restore \
/home/user/documents/
# Restore and verify all files
vaultik snapshot restore --verify myhost_docs_2025-01-01T12:00:00Z /restore`,
Args: cobra.MinimumNArgs(2),
Args: cobra.MinimumNArgs(restoreMinArgs),
RunE: func(cmd *cobra.Command, args []string) error {
return runRestore(cmd, args, opts)
},
}
cmd.Flags().BoolVar(&opts.Verify, "verify", false, "Verify restored files by checking chunk hashes")
cmd.Flags().BoolVar(&opts.Verify, "verify", false,
"Verify restored files by checking chunk hashes")
return cmd
}
// runRestore parses arguments and runs the restore operation through the app framework
// runRestore parses arguments and runs the restore operation through the
// app framework.
func runRestore(cmd *cobra.Command, args []string, opts *RestoreOptions) error {
snapshotID := args[0]
opts.TargetDir = args[1]
if len(args) > 2 {
opts.Paths = args[2:]
if len(args) > restoreMinArgs {
opts.Paths = args[restoreMinArgs:]
}
// Use unified config resolution
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
// Use the app framework like other commands
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
return RunOperation(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Modules: buildRestoreModules(),
Invokes: buildRestoreInvokes(snapshotID, opts),
})
}
// buildRestoreModules returns the fx.Options for dependency injection in restore
func buildRestoreModules() []fx.Option {
return []fx.Option{
fx.Provide(fx.Annotate(
func(g *globals.Globals, cfg *config.Config,
storer storage.Storer, v *vaultik.Vaultik, shutdowner fx.Shutdowner) *RestoreApp {
return &RestoreApp{
Globals: g,
Config: cfg,
Storage: storer,
Vaultik: v,
Shutdowner: shutdowner,
}
},
)),
}
}
// buildRestoreInvokes returns the fx.Options that wire up the restore lifecycle
func buildRestoreInvokes(snapshotID string, opts *RestoreOptions) []fx.Option {
return []fx.Option{
fx.Invoke(func(app *RestoreApp, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
// Start the restore operation in a goroutine
go func() {
// Run the restore operation
restoreOpts := &vaultik.RestoreOptions{
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.Restore(&vaultik.RestoreOptions{
SnapshotID: snapshotID,
TargetDir: opts.TargetDir,
Paths: opts.Paths,
Verify: opts.Verify,
SkipErrors: GetRootFlags().SkipErrors,
}
if err := app.Vaultik.Restore(restoreOpts); err != nil {
if err != context.Canceled {
SkipErrors: rootFlags.SkipErrors,
})
}, func(err error) {
log.Error("Restore operation failed", "error", err)
ReportError("Restore failed: %v", err)
os.Exit(1)
}
}
// Shutdown the app when restore completes
if err := app.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
log.Debug("Stopping restore operation")
app.Vaultik.Cancel()
return nil
},
ReportErrorf("Restore failed: %v", err)
})
}),
}
}
+37
View File
@@ -0,0 +1,37 @@
package cli //nolint:testpackage // exercises the unexported command constructor
import (
"strings"
"testing"
"github.com/spf13/pflag"
)
// TestRestoreCommandDoesNotTakeKeyAsArgument guards the fix for the age
// key being echoed on the command line: restore must take the key only
// from the environment, never as a flag value, and its help must show the
// file-based form rather than a literal key that would land in shell
// history.
func TestRestoreCommandDoesNotTakeKeyAsArgument(t *testing.T) {
t.Parallel()
cmd := newSnapshotRestoreCommand()
cmd.Flags().VisitAll(func(f *pflag.Flag) {
lower := strings.ToLower(f.Name)
for _, banned := range []string{"key", "secret", "age", "identity"} {
if strings.Contains(lower, banned) {
t.Errorf("restore must not accept the key as a flag; found --%s", f.Name)
}
}
})
help := cmd.Long
if strings.Contains(help, "AGE-SECRET-KEY-") {
t.Error("restore help must not show a literal age private key to type")
}
if !strings.Contains(help, "$(cat ") {
t.Error("restore help should read the key from a file, e.g. $(cat ...)")
}
}
-158
View File
@@ -1,158 +0,0 @@
package cli
import (
"context"
"fmt"
"strings"
"time"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
)
// StoreApp contains dependencies for store commands
type StoreApp struct {
Storage storage.Storer
Shutdowner fx.Shutdowner
}
// NewStoreCommand creates the store command and subcommands
func NewStoreCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "store",
Short: "Storage information commands",
Long: "Commands for viewing information about the storage backend",
}
// Add subcommands
cmd.AddCommand(newStoreInfoCommand())
return cmd
}
// newStoreInfoCommand creates the 'store info' subcommand
func newStoreInfoCommand() *cobra.Command {
return &cobra.Command{
Use: "info",
Short: "Display storage information",
Long: "Shows storage configuration and statistics including snapshots and blobs",
RunE: func(cmd *cobra.Command, args []string) error {
return runWithApp(cmd.Context(), func(app *StoreApp) error {
return app.Info(cmd.Context())
})
},
}
}
// Info displays storage information
func (app *StoreApp) Info(ctx context.Context) error {
// Get storage info
storageInfo := app.Storage.Info()
fmt.Printf("Storage Information\n")
fmt.Printf("==================\n\n")
fmt.Printf("Storage Configuration:\n")
fmt.Printf(" Type: %s\n", storageInfo.Type)
fmt.Printf(" Location: %s\n\n", storageInfo.Location)
// Count snapshots by listing metadata/ prefix
snapshotCount := 0
snapshotCh := app.Storage.ListStream(ctx, "metadata/")
snapshotDirs := make(map[string]bool)
for object := range snapshotCh {
if object.Err != nil {
return fmt.Errorf("listing snapshots: %w", object.Err)
}
// Extract snapshot ID from path like metadata/2024-01-15-143052-hostname/
parts := strings.Split(object.Key, "/")
if len(parts) >= 2 && parts[0] == "metadata" && parts[1] != "" {
snapshotDirs[parts[1]] = true
}
}
snapshotCount = len(snapshotDirs)
// Count blobs and calculate total size by listing blobs/ prefix
blobCount := 0
var totalSize int64
blobCh := app.Storage.ListStream(ctx, "blobs/")
for object := range blobCh {
if object.Err != nil {
return fmt.Errorf("listing blobs: %w", object.Err)
}
if !strings.HasSuffix(object.Key, "/") { // Skip directories
blobCount++
totalSize += object.Size
}
}
fmt.Printf("Storage Statistics:\n")
fmt.Printf(" Snapshots: %d\n", snapshotCount)
fmt.Printf(" Blobs: %d\n", blobCount)
fmt.Printf(" Total Size: %s\n", formatBytes(totalSize))
return nil
}
// formatBytes formats bytes into human-readable format
func formatBytes(bytes int64) string {
const unit = 1024
if bytes < unit {
return fmt.Sprintf("%d B", bytes)
}
div, exp := int64(unit), 0
for n := bytes / unit; n >= unit; n /= unit {
div *= unit
exp++
}
return fmt.Sprintf("%.1f %cB", float64(bytes)/float64(div), "KMGTPE"[exp])
}
// runWithApp creates the FX app and runs the given function
func runWithApp(ctx context.Context, fn func(*StoreApp) error) error {
var result error
rootFlags := GetRootFlags()
// Use unified config resolution
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
err = RunWithApp(ctx, AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Modules: []fx.Option{
fx.Provide(func(storer storage.Storer, shutdowner fx.Shutdowner) *StoreApp {
return &StoreApp{
Storage: storer,
Shutdowner: shutdowner,
}
}),
},
Invokes: []fx.Option{
fx.Invoke(func(app *StoreApp, shutdowner fx.Shutdowner) {
result = fn(app)
// Shutdown after command completes
go func() {
time.Sleep(100 * time.Millisecond) // Brief delay to ensure clean shutdown
if err := shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
}),
},
})
if err != nil {
return err
}
return result
}
-10
View File
@@ -1,10 +0,0 @@
package cli
import "time"
// SnapshotInfo represents snapshot information for listing
type SnapshotInfo struct {
ID string `json:"id"`
Timestamp time.Time `json:"timestamp"`
CompressedSize int64 `json:"compressed_size"`
}
+37 -15
View File
@@ -6,6 +6,7 @@ import (
"github.com/spf13/cobra"
"sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/ui"
)
// NewVersionCommand creates the version command
@@ -15,23 +16,44 @@ func NewVersionCommand() *cobra.Command {
Short: "Print version information",
Long: `Print version, git commit, and build information for vaultik.`,
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("vaultik %s\n", globals.Version)
fmt.Printf(" commit: %s\n", globals.Commit)
fmt.Printf(" build date: %s\n", globals.CommitDate)
fmt.Printf(" go: %s\n", runtime.Version())
fmt.Printf(" os/arch: %s/%s\n", runtime.GOOS, runtime.GOARCH)
fmt.Printf(" author: %s\n", globals.Author)
fmt.Printf(" homepage: %s\n", globals.Homepage)
fmt.Printf(" license: %s\n", globals.License)
if globals.Version == "dev" {
fmt.Println()
fmt.Println("This is a development build (no version information embedded).")
fmt.Println("Build a release binary with 'make vaultik' or download from")
fmt.Println("https://sneak.berlin/go/vaultik for embedded version metadata.")
}
Run: func(cmd *cobra.Command, _ []string) {
writeVersion(commandUI(cmd))
},
}
return cmd
}
// writeVersion prints the version report through the UI writer. The
// report is the output this command exists to produce, so it is written
// plain (markers would corrupt the aligned report) via the writer's
// underlying stdout; --quiet silences it like any other non-error
// output.
func writeVersion(out *ui.Writer) {
if out.Quiet() {
return
}
w := out.Out()
_, _ = fmt.Fprintf(w, "vaultik %s\n", globals.Version)
_, _ = fmt.Fprintf(w, " commit: %s\n", globals.Commit)
_, _ = fmt.Fprintf(w, " build date: %s\n", globals.CommitDate)
_, _ = fmt.Fprintf(w, " go: %s\n", runtime.Version())
_, _ = fmt.Fprintf(w, " os/arch: %s/%s\n", runtime.GOOS, runtime.GOARCH)
_, _ = fmt.Fprintf(w, " author: %s\n", globals.Author)
_, _ = fmt.Fprintf(w, " homepage: %s\n", globals.Homepage)
_, _ = fmt.Fprintf(w, " license: %s\n", globals.License)
if globals.IsDevVersion(globals.Version) {
_, _ = fmt.Fprintln(w)
_, _ = fmt.Fprintln(w,
"This is a development build: it was not built from a tagged")
_, _ = fmt.Fprintln(w,
"commit, so it carries no release version. Released binaries")
_, _ = fmt.Fprintf(w,
"are published at %s\n", globals.ReleasesURL)
_, _ = fmt.Fprintln(w,
"and report their tag on the first line above.")
}
}
+77
View File
@@ -0,0 +1,77 @@
package cli_test
import (
"bytes"
"strings"
"testing"
"sneak.berlin/go/vaultik/internal/cli"
"sneak.berlin/go/vaultik/internal/globals"
)
// runVersionCommand executes `vaultik version` with its output
// captured, and returns what it printed.
func runVersionCommand(t *testing.T) string {
t.Helper()
cmd := cli.NewVersionCommand()
var out bytes.Buffer
cmd.SetOut(&out)
cmd.SetErr(&out)
cmd.SetArgs([]string{})
err := cmd.Execute()
if err != nil {
t.Fatalf("version command failed: %v", err)
}
return out.String()
}
// TestVersionCommandReportsBuildVersion checks that the first line of
// the report is the version the binary was actually built with. The
// test binary carries no -ldflags, so that is the "dev" default -- the
// same string an untagged `make vaultik` build stamps a prefix of.
//
//nolint:paralleltest // executes a command that reads the global rootFlags
func TestVersionCommandReportsBuildVersion(t *testing.T) {
out := runVersionCommand(t)
wantFirst := "vaultik " + globals.Version
if first, _, _ := strings.Cut(out, "\n"); first != wantFirst {
t.Errorf("first line = %q, want %q", first, wantFirst)
}
if !strings.Contains(out, "commit:") {
t.Error("output does not report the commit")
}
}
// TestVersionCommandFlagsDevelopmentBuild is the regression test for
// the thing this command exists to prevent: a build that is not a
// release must say so. The notice used to be gated on the version
// being exactly "dev", so once untagged builds started carrying their
// commit sha it would have gone silent and an unreleased binary would
// have looked like a release.
//
//nolint:paralleltest // executes a command that reads the global rootFlags
func TestVersionCommandFlagsDevelopmentBuild(t *testing.T) {
if !globals.IsDevVersion(globals.Version) {
t.Skipf("test binary was stamped with release version %q",
globals.Version)
}
out := runVersionCommand(t)
if !strings.Contains(out, "development build") {
t.Errorf("dev build did not print the development-build notice:\n%s",
out)
}
if !strings.Contains(out, globals.ReleasesURL) {
t.Errorf("development-build notice does not point at %s:\n%s",
globals.ReleasesURL, out)
}
}
+207 -65
View File
@@ -1,6 +1,10 @@
// Package config loads, validates, and provides the vaultik YAML
// configuration, including snapshot definitions, encryption recipients,
// and storage settings.
package config
import (
"errors"
"fmt"
"os"
"path/filepath"
@@ -12,21 +16,71 @@ import (
"github.com/adrg/xdg"
"go.uber.org/fx"
"gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/log"
)
const appName = "vaultik"
// secretKeyPrefix marks an age secret (private) key. It is compared
// case-insensitively so a recipient entry that is actually a private key is
// caught and never passed to age or echoed back.
//
//nolint:gosec // G101: marker for detecting a pasted secret key, not a credential
const secretKeyPrefix = "AGE-SECRET-KEY-"
// Defaults and validation bounds for tunable settings.
const (
defaultBlobSizeLimit = Size(10 * 1024 * 1024 * 1024) // 10GB
defaultChunkSize = Size(10 * 1024 * 1024) // 10MB
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MB
defaultCompressionLevel = 3
minChunkSize = 1024 * 1024 // 1MB
minCompressionLevel = 1
maxCompressionLevel = 19
)
// Sentinel validation errors.
var (
errNoConfigPath = errors.New("config path not provided")
errNoAgeRecipients = errors.New(
"at least one age_recipient is required (generate with: age-keygen)")
errRecipientIsSecretKey = errors.New(
"an age secret key was given where a public key (age1...) belongs")
errRecipientNotX25519 = errors.New(
"not a valid recipient; only X25519 age1... public keys are supported")
errNoSnapshots = errors.New(
"at least one snapshot must be configured (see config.example.yml)")
errSnapshotNoPaths = errors.New("snapshot must have at least one path")
errChunkSizeTooSmall = errors.New("chunk_size must be at least 1MB")
errBlobSizeTooSmall = errors.New(
"blob_size_limit must be at least the largest chunk the chunker can " +
"emit (chunk_size times the FastCDC size spread)")
errBadCompression = errors.New("compression_level must be between 1 and 19")
errBadStorageScheme = errors.New(
"storage_url must start with s3://, file://, or rclone://")
errStorageNotConfigured = errors.New(
"storage not configured; set storage_url or provide s3.endpoint + " +
"s3.bucket + credentials")
errS3BucketRequired = errors.New("s3.bucket is required (or set storage_url)")
errS3KeyIDRequired = errors.New("s3.access_key_id is required")
errS3SecretRequired = errors.New("s3.secret_access_key is required")
)
// expandTilde expands ~ at the start of a path to the user's home directory.
func expandTilde(path string) string {
if path == "~" {
home, _ := os.UserHomeDir()
return home
}
if strings.HasPrefix(path, "~/") {
home, _ := os.UserHomeDir()
return filepath.Join(home, path[2:])
}
return path
}
@@ -34,8 +88,10 @@ func expandTilde(path string) string {
func expandTildeInURL(url string) string {
if strings.HasPrefix(url, "file://~/") {
home, _ := os.UserHomeDir()
return "file://" + filepath.Join(home, url[9:])
}
return url
}
@@ -63,6 +119,7 @@ func (c *Config) GetExcludes(snapshotName string) []string {
combined := make([]string, 0, len(c.Exclude)+len(snap.Exclude))
combined = append(combined, c.Exclude...)
combined = append(combined, snap.Exclude...)
return combined
}
@@ -74,19 +131,48 @@ func (c *Config) SnapshotNames() []string {
}
// Sort for deterministic order
sort.Strings(names)
return names
}
// Names of the two places the age secret key can be configured, used by
// AgeSecretKeySourceName for error messages that must not echo the value.
//
//nolint:gosec // G101: these are the names of the config sources, not a key
const (
ageSecretKeySourceEnv = "VAULTIK_AGE_SECRET_KEY"
ageSecretKeySourceConfig = "age_secret_key"
)
// AgeSecretKeySourceName returns the human name of where AgeSecretKey was
// configured. A Config built directly (as in tests) has no recorded
// source, so it reports the config-file field name.
func (c *Config) AgeSecretKeySourceName() string {
if c.AgeSecretKeySource != "" {
return c.AgeSecretKeySource
}
return ageSecretKeySourceConfig
}
// Config represents the application configuration for Vaultik.
// It defines all settings for backup operations, including source directories,
// encryption recipients, storage configuration, and performance tuning parameters.
// Configuration is typically loaded from a YAML file.
//
//nolint:tagliatelle // snake_case is the established config-file format
type Config struct {
AgeRecipients []string `yaml:"age_recipients"`
AgeSecretKey string `yaml:"age_secret_key"`
// AgeSecretKeySource names where AgeSecretKey was configured
// ("VAULTIK_AGE_SECRET_KEY" or "age_secret_key") so a later parse
// failure can name the source without echoing the secret value. It is
// set by Load and never read from or written to the config file.
AgeSecretKeySource string `yaml:"-"`
BlobSizeLimit Size `yaml:"blob_size_limit"`
ChunkSize Size `yaml:"chunk_size"`
Exclude []string `yaml:"exclude"` // Global excludes applied to all snapshots
// Exclude holds global excludes applied to all snapshots.
Exclude []string `yaml:"exclude"`
Hostname string `yaml:"hostname"`
IndexPath string `yaml:"index_path"`
S3 S3Config `yaml:"s3"`
@@ -98,13 +184,16 @@ type Config struct {
// Supported formats:
// - s3://bucket/prefix?endpoint=host&region=us-east-1
// - file:///path/to/backup
// For S3 URLs, credentials are still read from s3.access_key_id and s3.secret_access_key.
// For S3 URLs, credentials are still read from s3.access_key_id
// and s3.secret_access_key.
StorageURL string `yaml:"storage_url"`
}
// S3Config represents S3 storage configuration for backup storage.
// It supports both AWS S3 and S3-compatible storage services.
// All fields except UseSSL and PartSize are required.
//
//nolint:tagliatelle // snake_case is the established config-file format
type S3Config struct {
Endpoint string `yaml:"endpoint"`
Bucket string `yaml:"bucket"`
@@ -112,21 +201,23 @@ type S3Config struct {
AccessKeyID string `yaml:"access_key_id"`
SecretAccessKey string `yaml:"secret_access_key"`
Region string `yaml:"region"`
UseSSL bool `yaml:"use_ssl"`
// UseSSL selects HTTPS for a scheme-less endpoint. Omitted (nil) means
// the default, TLS; set it to false only to force plain HTTP.
UseSSL *bool `yaml:"use_ssl"`
PartSize Size `yaml:"part_size"`
}
// ConfigPath wraps the config file path for fx dependency injection.
// Path wraps the config file path for fx dependency injection.
// This type allows the config file path to be injected as a distinct type
// rather than a plain string, avoiding conflicts with other string dependencies.
type ConfigPath string
type Path string
// New creates a new Config instance by loading from the specified path.
// This function is used by the fx dependency injection framework.
// Returns an error if the path is empty or if loading fails.
func New(path ConfigPath) (*Config, error) {
func New(path Path) (*Config, error) {
if path == "" {
return nil, fmt.Errorf("config path not provided")
return nil, errNoConfigPath
}
cfg, err := Load(string(path))
@@ -151,20 +242,22 @@ func Load(path string) (*Config, error) {
cfg := &Config{
// Set defaults
BlobSizeLimit: Size(10 * 1024 * 1024 * 1024), // 10GB
ChunkSize: Size(10 * 1024 * 1024), // 10MB
BlobSizeLimit: defaultBlobSizeLimit,
ChunkSize: defaultChunkSize,
IndexPath: filepath.Join(xdg.DataHome, appName, "index.sqlite"),
CompressionLevel: 3,
CompressionLevel: defaultCompressionLevel,
}
// Convert smartconfig data to YAML then unmarshal
configData := sc.Data()
yamlBytes, err := yaml.Marshal(configData)
if err != nil {
return nil, fmt.Errorf("failed to marshal config data: %w", err)
}
if err := yaml.Unmarshal(yamlBytes, cfg); err != nil {
err = yaml.Unmarshal(yamlBytes, cfg)
if err != nil {
return nil, fmt.Errorf("failed to parse config: %w", err)
}
@@ -177,6 +270,7 @@ func Load(path string) (*Config, error) {
for i, path := range snap.Paths {
snap.Paths[i] = expandTilde(path)
}
cfg.Snapshots[name] = snap
}
@@ -185,10 +279,7 @@ func Load(path string) (*Config, error) {
cfg.IndexPath = expandTilde(envIndexPath)
}
// Check for environment variable override for AgeSecretKey
if envAgeSecretKey := os.Getenv("VAULTIK_AGE_SECRET_KEY"); envAgeSecretKey != "" {
cfg.AgeSecretKey = extractAgeSecretKey(envAgeSecretKey)
}
cfg.setAgeSecretKey()
// Get hostname if not set
if cfg.Hostname == "" {
@@ -196,6 +287,7 @@ func Load(path string) (*Config, error) {
if err != nil {
return nil, fmt.Errorf("failed to get hostname: %w", err)
}
cfg.Hostname = hostname
}
@@ -203,12 +295,15 @@ func Load(path string) (*Config, error) {
if cfg.S3.Region == "" {
cfg.S3.Region = "us-east-1"
}
if cfg.S3.PartSize == 0 {
cfg.S3.PartSize = Size(5 * 1024 * 1024) // 5MB
cfg.S3.PartSize = defaultS3PartSize
}
// Check config file permissions (warn if world or group readable)
if info, err := os.Stat(path); err == nil {
//nolint:gosec // G703: config path is operator-supplied by design
info, statErr := os.Stat(path)
if statErr == nil {
mode := info.Mode().Perm()
if mode&0044 != 0 { // group or world readable
log.Warn("Config file has insecure permissions (contains S3 credentials)",
@@ -218,7 +313,8 @@ func Load(path string) (*Config, error) {
}
}
if err := cfg.Validate(); err != nil {
err = cfg.Validate()
if err != nil {
return nil, fmt.Errorf("invalid config: %w", err)
}
@@ -227,113 +323,159 @@ func Load(path string) (*Config, error) {
// Validate checks if the configuration is valid and complete.
// It ensures all required fields are present and have valid values:
// - At least one age recipient must be specified
// - At least one age recipient must be specified, and every recipient must
// parse as an X25519 age1... public key (so a bad entry fails at load, not
// mid-backup); errors name the position, never the value
// - At least one snapshot must be configured with at least one path
// - Storage must be configured (either storage_url or s3.* fields)
// - Chunk size must be at least 1MB
// - Blob size limit must be at least the chunk size
// - Blob size limit must be at least the largest chunk the chunker can emit
// (chunk_size times chunker.ChunkSizeSpread), so a single-chunk blob never
// exceeds the configured limit
// - Compression level must be between 1 and 19
//
// Returns an error describing the first validation failure encountered.
func (c *Config) Validate() error {
if len(c.AgeRecipients) == 0 {
return fmt.Errorf("at least one age_recipient is required (generate with: age-keygen)")
return errNoAgeRecipients
}
for i, recipient := range c.AgeRecipients {
err := validateAgeRecipient(recipient)
if err != nil {
return fmt.Errorf("age_recipients[%d]: %w", i, err)
}
}
if len(c.Snapshots) == 0 {
return fmt.Errorf("at least one snapshot must be configured (see config.example.yml)")
return errNoSnapshots
}
for name, snap := range c.Snapshots {
if len(snap.Paths) == 0 {
return fmt.Errorf("snapshot %q must have at least one path", name)
return fmt.Errorf("%w: %q", errSnapshotNoPaths, name)
}
}
// Validate storage configuration
if err := c.validateStorage(); err != nil {
err := c.validateStorage()
if err != nil {
return err
}
if c.ChunkSize.Int64() < 1024*1024 { // 1MB minimum
return fmt.Errorf("chunk_size must be at least 1MB")
if c.ChunkSize.Int64() < minChunkSize {
return errChunkSizeTooSmall
}
if c.BlobSizeLimit.Int64() < c.ChunkSize.Int64() {
return fmt.Errorf("blob_size_limit must be at least chunk_size")
// The chunker can emit chunks up to chunk_size * ChunkSizeSpread, and the
// packer places a single such chunk into an otherwise empty blob. A limit
// below that bound would let a blob exceed it, so reject it.
largestChunk := c.ChunkSize.Int64() * chunker.ChunkSizeSpread
if c.BlobSizeLimit.Int64() < largestChunk {
return fmt.Errorf("%w: need at least %d bytes",
errBlobSizeTooSmall, largestChunk)
}
if c.CompressionLevel < 1 || c.CompressionLevel > 19 {
return fmt.Errorf("compression_level must be between 1 and 19")
if c.CompressionLevel < minCompressionLevel ||
c.CompressionLevel > maxCompressionLevel {
return errBadCompression
}
return nil
}
// validateAgeRecipient parses one age_recipients entry with the age library
// and returns a value-free error on failure. A recipient string can be
// sensitive (an operator may paste a secret key by mistake), so neither the
// entry nor age's own error (which quotes its input) is ever included.
func validateAgeRecipient(recipient string) error {
if strings.HasPrefix(strings.ToUpper(recipient), secretKeyPrefix) {
return errRecipientIsSecretKey
}
_, err := age.ParseX25519Recipient(recipient)
if err != nil {
return errRecipientNotX25519
}
return nil
}
// setAgeSecretKey records the age secret key and where it came from. The
// value is stored raw and parsed only where decryption happens
// (internal/vaultik), so backup, list and prune keep working whatever the
// field holds. The environment variable overrides the config-file field.
func (c *Config) setAgeSecretKey() {
if c.AgeSecretKey != "" {
c.AgeSecretKeySource = ageSecretKeySourceConfig
}
if env := os.Getenv("VAULTIK_AGE_SECRET_KEY"); env != "" {
c.AgeSecretKey = env
c.AgeSecretKeySource = ageSecretKeySourceEnv
}
}
// validateStorage validates storage configuration.
// If StorageURL is set, it takes precedence. S3 URLs require credentials.
// File URLs don't require any S3 configuration.
// If StorageURL is not set, legacy S3 configuration is required.
func (c *Config) validateStorage() error {
if c.StorageURL != "" {
// URL-based configuration
if strings.HasPrefix(c.StorageURL, "file://") {
// File storage doesn't need S3 credentials
return nil
}
if strings.HasPrefix(c.StorageURL, "s3://") {
// S3 storage needs credentials
if c.S3.AccessKeyID == "" {
return fmt.Errorf("s3.access_key_id is required for s3:// URLs")
}
if c.S3.SecretAccessKey == "" {
return fmt.Errorf("s3.secret_access_key is required for s3:// URLs")
}
return nil
}
if strings.HasPrefix(c.StorageURL, "rclone://") {
// Rclone storage uses rclone's own config
return nil
}
return fmt.Errorf("storage_url must start with s3://, file://, or rclone://")
return c.validateStorageURL()
}
// Legacy S3 configuration
if c.S3.Endpoint == "" {
return fmt.Errorf("storage not configured; set storage_url or provide s3.endpoint + s3.bucket + credentials")
return errStorageNotConfigured
}
if c.S3.Bucket == "" {
return fmt.Errorf("s3.bucket is required (or set storage_url)")
return errS3BucketRequired
}
if c.S3.AccessKeyID == "" {
return fmt.Errorf("s3.access_key_id is required")
return errS3KeyIDRequired
}
if c.S3.SecretAccessKey == "" {
return fmt.Errorf("s3.secret_access_key is required")
return errS3SecretRequired
}
return nil
}
// extractAgeSecretKey extracts the AGE-SECRET-KEY from the input using
// the age library's parser, which handles comments and whitespace.
func extractAgeSecretKey(input string) string {
identities, err := age.ParseIdentities(strings.NewReader(input))
if err != nil || len(identities) == 0 {
// Fall back to trimmed input if parsing fails
return strings.TrimSpace(input)
// validateStorageURL validates URL-based storage configuration. File and
// rclone URLs need no credentials; S3 URLs require the legacy s3.*
// credential fields.
func (c *Config) validateStorageURL() error {
switch {
case strings.HasPrefix(c.StorageURL, "file://"):
// File storage doesn't need S3 credentials
return nil
case strings.HasPrefix(c.StorageURL, "rclone://"):
// Rclone storage uses rclone's own config
return nil
case strings.HasPrefix(c.StorageURL, "s3://"):
// S3 storage needs credentials
if c.S3.AccessKeyID == "" {
return fmt.Errorf("%w for s3:// URLs", errS3KeyIDRequired)
}
// Return the string representation of the first identity
if id, ok := identities[0].(*age.X25519Identity); ok {
return id.String()
if c.S3.SecretAccessKey == "" {
return fmt.Errorf("%w for s3:// URLs", errS3SecretRequired)
}
return nil
default:
return errBadStorageScheme
}
return strings.TrimSpace(input)
}
// Module exports the config module for fx dependency injection.
// It provides the Config type to other modules in the application.
//
//nolint:gochecknoglobals // fx module definitions are package globals
var Module = fx.Module("config",
fx.Provide(New),
)
+244 -52
View File
@@ -1,21 +1,30 @@
package config
package config //nolint:testpackage // exercises unexported source constants
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"sneak.berlin/go/vaultik/internal/chunker"
)
const (
TEST_SNEAK_AGE_PUBLIC_KEY = "age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj"
TEST_INTEGRATION_AGE_PUBLIC_KEY = "age1ezrjmfpwsc95svdg0y54mums3zevgzu0x0ecq2f7tp8a05gl0sjq9q9wjg"
TEST_INTEGRATION_AGE_PRIVATE_KEY = "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5"
testSneakAgePublicKey = "age1278m9q7dp3chsh2dcy82qk27v047zywyvt" +
"xwnj4cvt0z65jw6a7q5dqhfj"
testIntegrationAgePublicKey = "age1ezrjmfpwsc95svdg0y54mums3zevgzu" +
"0x0ecq2f7tp8a05gl0sjq9q9wjg"
testIntegrationAgePrivateKey = "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GX" +
"VEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5"
)
func TestMain(m *testing.M) {
// Set up test environment
testConfigPath := filepath.Join("..", "..", "test", "config.yaml")
if absPath, err := filepath.Abs(testConfigPath); err == nil {
absPath, err := filepath.Abs(testConfigPath)
if err == nil {
_ = os.Setenv("VAULTIK_CONFIG", absPath)
}
@@ -23,8 +32,11 @@ func TestMain(m *testing.M) {
os.Exit(code)
}
// TestConfigLoad ensures the config package can be imported and basic functionality works
// TestConfigLoad ensures the config package can be imported and basic
// functionality works.
func TestConfigLoad(t *testing.T) {
t.Parallel()
// Use the test config file
configPath := os.Getenv("VAULTIK_CONFIG")
if configPath == "" {
@@ -41,8 +53,10 @@ func TestConfigLoad(t *testing.T) {
if len(cfg.AgeRecipients) != 2 {
t.Errorf("Expected 2 age recipients, got %d", len(cfg.AgeRecipients))
}
if cfg.AgeRecipients[0] != TEST_SNEAK_AGE_PUBLIC_KEY {
t.Errorf("Expected first age recipient to be %s, got '%s'", TEST_SNEAK_AGE_PUBLIC_KEY, cfg.AgeRecipients[0])
if cfg.AgeRecipients[0] != testSneakAgePublicKey {
t.Errorf("Expected first age recipient to be %s, got '%s'",
testSneakAgePublicKey, cfg.AgeRecipients[0])
}
if len(cfg.Snapshots) != 1 {
@@ -59,11 +73,13 @@ func TestConfigLoad(t *testing.T) {
}
if testSnap.Paths[0] != "/tmp/vaultik-test-source" {
t.Errorf("Expected first path to be '/tmp/vaultik-test-source', got '%s'", testSnap.Paths[0])
t.Errorf("Expected first path to be '/tmp/vaultik-test-source', got '%s'",
testSnap.Paths[0])
}
if cfg.S3.Bucket != "vaultik-test-bucket" {
t.Errorf("Expected S3 bucket to be 'vaultik-test-bucket', got '%s'", cfg.S3.Bucket)
t.Errorf("Expected S3 bucket to be 'vaultik-test-bucket', got '%s'",
cfg.S3.Bucket)
}
if cfg.Hostname != "test-host" {
@@ -71,76 +87,252 @@ func TestConfigLoad(t *testing.T) {
}
}
// TestExampleConfigIsScrubbedAndLoads checks that the shipped
// config.example.yml carries only neutral placeholders (no real credentials,
// private addresses, or internal host names) and still parses.
func TestExampleConfigIsScrubbedAndLoads(t *testing.T) {
t.Parallel()
examplePath := filepath.Join("..", "..", "config.example.yml")
cfg, err := Load(examplePath)
if err != nil {
t.Fatalf("Failed to load config.example.yml: %v", err)
}
if cfg.StorageURL != "rclone://myremote/path/to/backups" {
t.Errorf("Expected neutral storage_url, got '%s'", cfg.StorageURL)
}
//nolint:gosec // G304: examplePath is a fixed in-repo path, not user input
raw, err := os.ReadFile(examplePath)
if err != nil {
t.Fatalf("Failed to read config.example.yml: %v", err)
}
text := string(raw)
wantSubstrings := []string{
"YOUR_ACCESS_KEY",
"YOUR_SECRET_KEY",
"endpoint: https://",
}
for _, want := range wantSubstrings {
if !strings.Contains(text, want) {
t.Errorf("Expected config.example.yml to contain %q", want)
}
}
// A raw "http://" scheme would mean a plaintext, likely private endpoint.
if strings.Contains(text, "http://") {
t.Error("config.example.yml should not contain an http:// endpoint")
}
}
// TestConfigFromEnv tests loading config path from environment variable
func TestConfigFromEnv(t *testing.T) {
t.Parallel()
configPath := os.Getenv("VAULTIK_CONFIG")
if configPath == "" {
t.Skip("VAULTIK_CONFIG not set")
}
// Verify the file exists
if _, err := os.Stat(configPath); os.IsNotExist(err) {
t.Errorf("Config file does not exist at path from VAULTIK_CONFIG: %s", configPath)
//nolint:gosec // G703: test config path comes from the test environment
_, err := os.Stat(configPath)
if os.IsNotExist(err) {
t.Errorf("Config file does not exist at path from VAULTIK_CONFIG: %s",
configPath)
}
}
// TestExtractAgeSecretKey tests extraction of AGE-SECRET-KEY from various inputs
func TestExtractAgeSecretKey(t *testing.T) {
// TestValidateBlobSizeLimit checks the blob_size_limit boundary: it must be at
// least the largest chunk the chunker can emit (chunk_size times
// chunker.ChunkSizeSpread), because the packer places a single such chunk into
// an otherwise empty blob. A limit between chunk_size and that bound is rejected.
func TestValidateBlobSizeLimit(t *testing.T) {
t.Parallel()
const chunkSize = Size(10 * 1024 * 1024) // 10MB
largestChunk := chunkSize.Int64() * chunker.ChunkSizeSpread
newConfig := func(blobLimit Size) *Config {
return &Config{
AgeRecipients: []string{testSneakAgePublicKey},
Snapshots: map[string]SnapshotConfig{"test": {Paths: []string{"/tmp/src"}}},
StorageURL: "file:///tmp/vaultik-test-store",
ChunkSize: chunkSize,
BlobSizeLimit: blobLimit,
CompressionLevel: 3,
}
}
tests := []struct {
name string
input string
expected string
blobLimit Size
wantErr bool
}{
{
name: "plain key",
input: "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5",
expected: "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5",
name: "at chunk_size but below largest chunk is rejected",
blobLimit: chunkSize,
wantErr: true,
},
{
name: "key with trailing newline",
input: "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5\n",
expected: "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5",
name: "between chunk_size and largest chunk is rejected",
blobLimit: Size(chunkSize.Int64() * 2),
wantErr: true,
},
{
name: "full age-keygen output",
input: `# created: 2025-01-14T12:00:00Z
# public key: age1ezrjmfpwsc95svdg0y54mums3zevgzu0x0ecq2f7tp8a05gl0sjq9q9wjg
AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5
`,
expected: "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5",
name: "one byte below largest chunk is rejected",
blobLimit: Size(largestChunk - 1),
wantErr: true,
},
{
name: "age-keygen output with extra blank lines",
input: `# created: 2025-01-14T12:00:00Z
# public key: age1ezrjmfpwsc95svdg0y54mums3zevgzu0x0ecq2f7tp8a05gl0sjq9q9wjg
AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5
`,
expected: "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5",
name: "exactly at largest chunk is accepted",
blobLimit: Size(largestChunk),
wantErr: false,
},
{
name: "key with leading whitespace",
input: " AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5 ",
expected: "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GXVEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5",
},
{
name: "empty input",
input: "",
expected: "",
},
{
name: "only comments",
input: "# this is a comment\n# another comment",
expected: "# this is a comment\n# another comment",
name: "above largest chunk is accepted",
blobLimit: Size(largestChunk * 100),
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := extractAgeSecretKey(tt.input)
if result != tt.expected {
t.Errorf("extractAgeSecretKey(%q) = %q, want %q", tt.input, result, tt.expected)
t.Parallel()
err := newConfig(tt.blobLimit).Validate()
if tt.wantErr {
if !errors.Is(err, errBlobSizeTooSmall) {
t.Fatalf("Validate() error = %v, want errBlobSizeTooSmall", err)
}
return
}
if err != nil {
t.Fatalf("Validate() unexpected error: %v", err)
}
})
}
}
// TestValidateAgeRecipients checks that recipients are parsed at config load
// (a bad entry fails immediately, not mid-backup) and that no invalid entry —
// least of all a pasted secret key — is echoed in the error.
func TestValidateAgeRecipients(t *testing.T) {
t.Parallel()
baseConfig := func(recipients []string) *Config {
return &Config{
AgeRecipients: recipients,
Snapshots: map[string]SnapshotConfig{"test": {Paths: []string{"/tmp/src"}}},
StorageURL: "file:///tmp/vaultik-test-store",
ChunkSize: Size(10 * 1024 * 1024),
BlobSizeLimit: Size(10 * 1024 * 1024 * 1024),
CompressionLevel: 3,
}
}
tests := []struct {
name string
recipients []string
wantErr bool
}{
{
name: "config init placeholder is rejected",
recipients: []string{"age1REPLACE_WITH_YOUR_PUBLIC_KEY"},
wantErr: true,
},
{
name: "ssh-ed25519 recipient is rejected",
recipients: []string{"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIexamplekeydata"},
wantErr: true,
},
{
name: "truncated age1 string is rejected",
recipients: []string{"age1short"},
wantErr: true,
},
{
name: "secret key passed as recipient is rejected",
recipients: []string{testIntegrationAgePrivateKey},
wantErr: true,
},
{
name: "two valid recipients are accepted",
recipients: []string{testSneakAgePublicKey, testIntegrationAgePublicKey},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
err := baseConfig(tt.recipients).Validate()
if !tt.wantErr {
if err != nil {
t.Fatalf("Validate() unexpected error: %v", err)
}
return
}
if err == nil {
t.Fatal("Validate() returned nil, want error")
}
// The entry itself must never appear in the error, since a
// recipient string can be a secret key.
for _, recipient := range tt.recipients {
if strings.Contains(err.Error(), recipient) {
t.Fatalf("Validate() error echoed the recipient value: %v", err)
}
}
})
}
}
// TestAgeSecretKeySourceName checks the name reported for the configured
// age secret key: the recorded source when Load set one, and the
// config-file field name for a Config built directly (as in tests).
func TestAgeSecretKeySourceName(t *testing.T) {
t.Parallel()
tests := []struct {
name string
source string
want string
}{
{
name: "unset defaults to config field",
source: "",
want: ageSecretKeySourceConfig,
},
{
name: "environment source",
source: ageSecretKeySourceEnv,
want: ageSecretKeySourceEnv,
},
{
name: "config-file source",
source: ageSecretKeySourceConfig,
want: ageSecretKeySourceConfig,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
cfg := &Config{AgeSecretKeySource: tt.source}
if got := cfg.AgeSecretKeySourceName(); got != tt.want {
t.Errorf("AgeSecretKeySourceName() = %q, want %q", got, tt.want)
}
})
}
+29 -4
View File
@@ -1,31 +1,45 @@
package config
import (
"errors"
"fmt"
"math"
"github.com/dustin/go-humanize"
)
var (
errSizeType = errors.New("size must be a number or string")
errSizeTooLarge = errors.New("size exceeds maximum supported value")
)
// Size represents a byte size that can be specified in configuration files.
// It can unmarshal from both numeric values (interpreted as bytes) and
// human-readable strings like "10MB", "2.5GB", or "1TB".
//
//nolint:recvcheck // UnmarshalYAML requires a pointer; String/Int64 are value reads
type Size int64
// UnmarshalYAML implements yaml.Unmarshaler for Size, allowing it to be
// parsed from YAML configuration files. It accepts both numeric values
// (interpreted as bytes) and string values with units (e.g., "10MB").
func (s *Size) UnmarshalYAML(unmarshal func(interface{}) error) error {
func (s *Size) UnmarshalYAML(unmarshal func(any) error) error {
// Try to unmarshal as int64 first
var intVal int64
if err := unmarshal(&intVal); err == nil {
err := unmarshal(&intVal)
if err == nil {
*s = Size(intVal)
return nil
}
// Try to unmarshal as string
var strVal string
if err := unmarshal(&strVal); err != nil {
return fmt.Errorf("size must be a number or string")
err = unmarshal(&strVal)
if err != nil {
return errSizeType
}
// Parse the string using go-humanize
@@ -34,7 +48,12 @@ func (s *Size) UnmarshalYAML(unmarshal func(interface{}) error) error {
return fmt.Errorf("invalid size format: %w", err)
}
if bytes > math.MaxInt64 {
return fmt.Errorf("%w: %s", errSizeTooLarge, strVal)
}
*s = Size(bytes)
return nil
}
@@ -49,6 +68,7 @@ func (s Size) Int64() int64 {
// For example, 1048576 bytes would be formatted as "1.0 MB".
// This implements the fmt.Stringer interface.
func (s Size) String() string {
//nolint:gosec // G115: sizes are non-negative by construction
return humanize.Bytes(uint64(s))
}
@@ -58,5 +78,10 @@ func ParseSize(s string) (Size, error) {
if err != nil {
return 0, fmt.Errorf("invalid size format: %w", err)
}
if bytes > math.MaxInt64 {
return 0, fmt.Errorf("%w: %s", errSizeTooLarge, s)
}
return Size(bytes), nil
}
-209
View File
@@ -1,209 +0,0 @@
package crypto
import (
"bytes"
"fmt"
"io"
"sync"
"filippo.io/age"
"go.uber.org/fx"
)
// Encryptor provides thread-safe encryption using the age encryption library.
// It supports encrypting data for multiple recipients simultaneously, allowing
// any of the corresponding private keys to decrypt the data. This is useful
// for backup scenarios where multiple parties should be able to decrypt the data.
type Encryptor struct {
recipients []age.Recipient
mu sync.RWMutex
}
// NewEncryptor creates a new encryptor with the given age public keys.
// Each public key should be a valid age X25519 recipient string (e.g., "age1...")
// At least one recipient must be provided. Returns an error if any of the
// public keys are invalid or if no recipients are specified.
func NewEncryptor(publicKeys []string) (*Encryptor, error) {
if len(publicKeys) == 0 {
return nil, fmt.Errorf("at least one recipient is required")
}
recipients := make([]age.Recipient, 0, len(publicKeys))
for _, key := range publicKeys {
recipient, err := age.ParseX25519Recipient(key)
if err != nil {
return nil, fmt.Errorf("parsing age recipient %s: %w", key, err)
}
recipients = append(recipients, recipient)
}
return &Encryptor{
recipients: recipients,
}, nil
}
// Encrypt encrypts data using age encryption for all configured recipients.
// The encrypted data can be decrypted by any of the corresponding private keys.
// This method is suitable for small to medium amounts of data that fit in memory.
// For large data streams, use EncryptStream or EncryptWriter instead.
func (e *Encryptor) Encrypt(data []byte) ([]byte, error) {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
var buf bytes.Buffer
// Create encrypted writer for all recipients
w, err := age.Encrypt(&buf, recipients...)
if err != nil {
return nil, fmt.Errorf("creating encrypted writer: %w", err)
}
// Write data
if _, err := w.Write(data); err != nil {
return nil, fmt.Errorf("writing encrypted data: %w", err)
}
// Close to flush
if err := w.Close(); err != nil {
return nil, fmt.Errorf("closing encrypted writer: %w", err)
}
return buf.Bytes(), nil
}
// EncryptStream encrypts data from reader to writer using age encryption.
// This method is suitable for encrypting large files or streams as it processes
// data in a streaming fashion without loading everything into memory.
// The encrypted data is written directly to the destination writer.
func (e *Encryptor) EncryptStream(dst io.Writer, src io.Reader) error {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
// Create encrypted writer for all recipients
w, err := age.Encrypt(dst, recipients...)
if err != nil {
return fmt.Errorf("creating encrypted writer: %w", err)
}
// Copy data
if _, err := io.Copy(w, src); err != nil {
return fmt.Errorf("copying encrypted data: %w", err)
}
// Close to flush
if err := w.Close(); err != nil {
return fmt.Errorf("closing encrypted writer: %w", err)
}
return nil
}
// EncryptWriter creates a writer that encrypts data written to it.
// All data written to the returned WriteCloser will be encrypted and written
// to the destination writer. The caller must call Close() on the returned
// writer to ensure all encrypted data is properly flushed and finalized.
// This is useful for integrating encryption into existing writer-based pipelines.
func (e *Encryptor) EncryptWriter(dst io.Writer) (io.WriteCloser, error) {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
// Create encrypted writer for all recipients
w, err := age.Encrypt(dst, recipients...)
if err != nil {
return nil, fmt.Errorf("creating encrypted writer: %w", err)
}
return w, nil
}
// UpdateRecipients updates the recipients for future encryption operations.
// This method is thread-safe and can be called while other encryption operations
// are in progress. Existing encryption operations will continue with the old
// recipients. At least one recipient must be provided. Returns an error if any
// of the public keys are invalid or if no recipients are specified.
func (e *Encryptor) UpdateRecipients(publicKeys []string) error {
if len(publicKeys) == 0 {
return fmt.Errorf("at least one recipient is required")
}
recipients := make([]age.Recipient, 0, len(publicKeys))
for _, key := range publicKeys {
recipient, err := age.ParseX25519Recipient(key)
if err != nil {
return fmt.Errorf("parsing age recipient %s: %w", key, err)
}
recipients = append(recipients, recipient)
}
e.mu.Lock()
e.recipients = recipients
e.mu.Unlock()
return nil
}
// Decryptor provides thread-safe decryption using the age encryption library.
// It uses a private key to decrypt data that was encrypted for the corresponding
// public key.
type Decryptor struct {
identity age.Identity
mu sync.RWMutex
}
// NewDecryptor creates a new decryptor with the given age private key.
// The private key should be a valid age X25519 identity string.
// Returns an error if the private key is invalid.
func NewDecryptor(privateKey string) (*Decryptor, error) {
identity, err := age.ParseX25519Identity(privateKey)
if err != nil {
return nil, fmt.Errorf("parsing age identity: %w", err)
}
return &Decryptor{
identity: identity,
}, nil
}
// Decrypt decrypts data using age decryption.
// This method is suitable for small to medium amounts of data that fit in memory.
// For large data streams, use DecryptStream instead.
func (d *Decryptor) Decrypt(data []byte) ([]byte, error) {
d.mu.RLock()
identity := d.identity
d.mu.RUnlock()
r, err := age.Decrypt(bytes.NewReader(data), identity)
if err != nil {
return nil, fmt.Errorf("creating decrypted reader: %w", err)
}
decrypted, err := io.ReadAll(r)
if err != nil {
return nil, fmt.Errorf("reading decrypted data: %w", err)
}
return decrypted, nil
}
// DecryptStream returns a reader that decrypts data from the provided reader.
// This method is suitable for decrypting large files or streams as it processes
// data in a streaming fashion without loading everything into memory.
// The caller should close the input reader when done.
func (d *Decryptor) DecryptStream(src io.Reader) (io.Reader, error) {
d.mu.RLock()
identity := d.identity
d.mu.RUnlock()
r, err := age.Decrypt(src, identity)
if err != nil {
return nil, fmt.Errorf("creating decrypted reader: %w", err)
}
return r, nil
}
// Module exports the crypto module for fx dependency injection.
var Module = fx.Module("crypto")
-157
View File
@@ -1,157 +0,0 @@
package crypto
import (
"bytes"
"testing"
"filippo.io/age"
)
func TestEncryptor(t *testing.T) {
// Generate a test key pair
identity, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity: %v", err)
}
publicKey := identity.Recipient().String()
// Create encryptor
enc, err := NewEncryptor([]string{publicKey})
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Test data
plaintext := []byte("Hello, World! This is a test message.")
// Encrypt
ciphertext, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Verify it's actually encrypted (should be larger and different)
if bytes.Equal(plaintext, ciphertext) {
t.Error("ciphertext equals plaintext")
}
// Decrypt to verify
r, err := age.Decrypt(bytes.NewReader(ciphertext), identity)
if err != nil {
t.Fatalf("failed to decrypt: %v", err)
}
var decrypted bytes.Buffer
if _, err := decrypted.ReadFrom(r); err != nil {
t.Fatalf("failed to read decrypted data: %v", err)
}
if !bytes.Equal(plaintext, decrypted.Bytes()) {
t.Error("decrypted data doesn't match original")
}
}
func TestEncryptorMultipleRecipients(t *testing.T) {
// Generate three test key pairs
identity1, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity1: %v", err)
}
identity2, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity2: %v", err)
}
identity3, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity3: %v", err)
}
publicKeys := []string{
identity1.Recipient().String(),
identity2.Recipient().String(),
identity3.Recipient().String(),
}
// Create encryptor with multiple recipients
enc, err := NewEncryptor(publicKeys)
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Test data
plaintext := []byte("Secret message for multiple recipients")
// Encrypt
ciphertext, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Verify each recipient can decrypt
identities := []age.Identity{identity1, identity2, identity3}
for i, identity := range identities {
r, err := age.Decrypt(bytes.NewReader(ciphertext), identity)
if err != nil {
t.Fatalf("recipient %d failed to decrypt: %v", i+1, err)
}
var decrypted bytes.Buffer
if _, err := decrypted.ReadFrom(r); err != nil {
t.Fatalf("recipient %d failed to read decrypted data: %v", i+1, err)
}
if !bytes.Equal(plaintext, decrypted.Bytes()) {
t.Errorf("recipient %d: decrypted data doesn't match original", i+1)
}
}
}
func TestEncryptorUpdateRecipients(t *testing.T) {
// Generate two identities
identity1, _ := age.GenerateX25519Identity()
identity2, _ := age.GenerateX25519Identity()
publicKey1 := identity1.Recipient().String()
publicKey2 := identity2.Recipient().String()
// Create encryptor with first key
enc, err := NewEncryptor([]string{publicKey1})
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Encrypt with first key
plaintext := []byte("test data")
ciphertext1, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Update to second key
if err := enc.UpdateRecipients([]string{publicKey2}); err != nil {
t.Fatalf("failed to update recipients: %v", err)
}
// Encrypt with second key
ciphertext2, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// First ciphertext should only decrypt with first identity
if _, err := age.Decrypt(bytes.NewReader(ciphertext1), identity1); err != nil {
t.Error("failed to decrypt with identity1")
}
if _, err := age.Decrypt(bytes.NewReader(ciphertext1), identity2); err == nil {
t.Error("should not decrypt with identity2")
}
// Second ciphertext should only decrypt with second identity
if _, err := age.Decrypt(bytes.NewReader(ciphertext2), identity2); err != nil {
t.Error("failed to decrypt with identity2")
}
if _, err := age.Decrypt(bytes.NewReader(ciphertext2), identity1); err == nil {
t.Error("should not decrypt with identity1")
}
}
+55 -12
View File
@@ -3,18 +3,25 @@ package database
import (
"context"
"database/sql"
"errors"
"fmt"
)
// BlobChunkRepository provides access to the blob_chunks table, which maps
// blobs to the chunks they contain (with offset and length).
type BlobChunkRepository struct {
db *DB
}
// NewBlobChunkRepository creates a BlobChunkRepository backed by db.
func NewBlobChunkRepository(db *DB) *BlobChunkRepository {
return &BlobChunkRepository{db: db}
}
func (r *BlobChunkRepository) Create(ctx context.Context, tx *sql.Tx, bc *BlobChunk) error {
// Create inserts a blob_chunks row, using tx when non-nil.
func (r *BlobChunkRepository) Create(
ctx context.Context, tx *sql.Tx, bc *BlobChunk,
) error {
query := `
INSERT INTO blob_chunks (blob_id, chunk_hash, offset, length)
VALUES (?, ?, ?, ?)
@@ -34,7 +41,11 @@ func (r *BlobChunkRepository) Create(ctx context.Context, tx *sql.Tx, bc *BlobCh
return nil
}
func (r *BlobChunkRepository) GetByBlobID(ctx context.Context, blobID string) ([]*BlobChunk, error) {
// GetByBlobID returns all chunks contained in the given blob, ordered by
// their offset within the blob.
func (r *BlobChunkRepository) GetByBlobID(
ctx context.Context, blobID string,
) ([]*BlobChunk, error) {
query := `
SELECT blob_id, chunk_hash, offset, length
FROM blob_chunks
@@ -46,22 +57,35 @@ func (r *BlobChunkRepository) GetByBlobID(ctx context.Context, blobID string) ([
if err != nil {
return nil, fmt.Errorf("querying blob chunks: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var blobChunks []*BlobChunk
for rows.Next() {
var bc BlobChunk
err := rows.Scan(&bc.BlobID, &bc.ChunkHash, &bc.Offset, &bc.Length)
if err != nil {
return nil, fmt.Errorf("scanning blob chunk: %w", err)
}
blobChunks = append(blobChunks, &bc)
}
return blobChunks, rows.Err()
}
func (r *BlobChunkRepository) GetByChunkHash(ctx context.Context, chunkHash string) (*BlobChunk, error) {
// GetByChunkHash returns one blob_chunks row containing the given chunk,
// or nil if the chunk is not packed in any blob.
func (r *BlobChunkRepository) GetByChunkHash(
ctx context.Context, chunkHash string,
) (*BlobChunk, error) {
query := `
SELECT blob_id, chunk_hash, offset, length
FROM blob_chunks
@@ -70,7 +94,9 @@ func (r *BlobChunkRepository) GetByChunkHash(ctx context.Context, chunkHash stri
`
LogSQL("GetByChunkHash", query, chunkHash)
var bc BlobChunk
err := r.db.conn.QueryRowContext(ctx, query, chunkHash).Scan(
&bc.BlobID,
&bc.ChunkHash,
@@ -78,21 +104,27 @@ func (r *BlobChunkRepository) GetByChunkHash(ctx context.Context, chunkHash stri
&bc.Length,
)
if err == sql.ErrNoRows {
if errors.Is(err, sql.ErrNoRows) {
LogSQL("GetByChunkHash", "No rows found", chunkHash)
return nil, nil
return nil, nil //nolint:nilnil // nil,nil signals not-found; callers check nil
}
if err != nil {
LogSQL("GetByChunkHash", "Error", chunkHash, err)
return nil, fmt.Errorf("querying blob chunk: %w", err)
}
LogSQL("GetByChunkHash", "Found blob", chunkHash, "blob", bc.BlobID)
return &bc, nil
}
// GetByChunkHashTx retrieves a blob chunk within a transaction
func (r *BlobChunkRepository) GetByChunkHashTx(ctx context.Context, tx *sql.Tx, chunkHash string) (*BlobChunk, error) {
func (r *BlobChunkRepository) GetByChunkHashTx(
ctx context.Context, tx *sql.Tx, chunkHash string,
) (*BlobChunk, error) {
query := `
SELECT blob_id, chunk_hash, offset, length
FROM blob_chunks
@@ -101,7 +133,9 @@ func (r *BlobChunkRepository) GetByChunkHashTx(ctx context.Context, tx *sql.Tx,
`
LogSQL("GetByChunkHashTx", query, chunkHash)
var bc BlobChunk
err := tx.QueryRowContext(ctx, query, chunkHash).Scan(
&bc.BlobID,
&bc.ChunkHash,
@@ -109,20 +143,25 @@ func (r *BlobChunkRepository) GetByChunkHashTx(ctx context.Context, tx *sql.Tx,
&bc.Length,
)
if err == sql.ErrNoRows {
if errors.Is(err, sql.ErrNoRows) {
LogSQL("GetByChunkHashTx", "No rows found", chunkHash)
return nil, nil
return nil, nil //nolint:nilnil // nil,nil signals not-found; callers check nil
}
if err != nil {
LogSQL("GetByChunkHashTx", "Error", chunkHash, err)
return nil, fmt.Errorf("querying blob chunk: %w", err)
}
LogSQL("GetByChunkHashTx", "Found blob", chunkHash, "blob", bc.BlobID)
return &bc, nil
}
// DeleteOrphaned deletes blob_chunks entries where either the blob or chunk no longer exists
// DeleteOrphaned deletes blob_chunks entries where either the blob or the
// chunk no longer exists.
func (r *BlobChunkRepository) DeleteOrphaned(ctx context.Context) error {
// Delete blob_chunks where the blob doesn't exist
query1 := `
@@ -132,7 +171,9 @@ func (r *BlobChunkRepository) DeleteOrphaned(ctx context.Context) error {
WHERE blobs.id = blob_chunks.blob_id
)
`
if _, err := r.db.ExecWithLog(ctx, query1); err != nil {
_, err := r.db.ExecWithLog(ctx, query1)
if err != nil {
return fmt.Errorf("deleting blob_chunks with missing blobs: %w", err)
}
@@ -144,7 +185,9 @@ func (r *BlobChunkRepository) DeleteOrphaned(ctx context.Context) error {
WHERE chunks.chunk_hash = blob_chunks.chunk_hash
)
`
if _, err := r.db.ExecWithLog(ctx, query2); err != nil {
_, err = r.db.ExecWithLog(ctx, query2)
if err != nil {
return fmt.Errorf("deleting blob_chunks with missing chunks: %w", err)
}
+130 -85
View File
@@ -1,4 +1,4 @@
package database
package database_test
import (
"context"
@@ -6,71 +6,107 @@ import (
"testing"
"time"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
func TestBlobChunkRepository(t *testing.T) {
db, cleanup := setupTestDB(t)
defer cleanup()
// Chunk hashes used across the blob_chunks tests.
const (
chunk1Hash = "chunk1"
chunk2Hash = "chunk2"
chunk3Hash = "chunk3"
)
// mustCreateChunks registers the given chunk hashes (1024 bytes each).
func mustCreateChunks(
t *testing.T,
repos *database.Repositories,
hashes ...types.ChunkHash,
) {
t.Helper()
ctx := context.Background()
repos := NewRepositories(db)
// Create blob first
blob := &Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash("blob1-hash"),
CreatedTS: time.Now(),
}
err := repos.Blobs.Create(ctx, nil, blob)
if err != nil {
t.Fatalf("failed to create blob: %v", err)
}
// Create chunks
chunks := []types.ChunkHash{"chunk1", "chunk2", "chunk3"}
for _, chunkHash := range chunks {
chunk := &Chunk{
for _, chunkHash := range hashes {
chunk := &database.Chunk{
ChunkHash: chunkHash,
Size: 1024,
}
err = repos.Chunks.Create(ctx, nil, chunk)
err := repos.Chunks.Create(ctx, nil, chunk)
if err != nil {
t.Fatalf("failed to create chunk %s: %v", chunkHash, err)
}
}
}
// mustCreateBlob creates a blob row with the given hash.
func mustCreateBlob(
t *testing.T,
repos *database.Repositories,
hash types.BlobHash,
) *database.Blob {
t.Helper()
blob := &database.Blob{
ID: types.NewBlobID(),
Hash: hash,
CreatedTS: time.Now(),
}
err := repos.Blobs.Create(context.Background(), nil, blob)
if err != nil {
t.Fatalf("failed to create blob %s: %v", hash, err)
}
return blob
}
func TestBlobChunkRepository(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
blob := mustCreateBlob(t, repos, "blob1-hash")
mustCreateChunks(t, repos, chunk1Hash, chunk2Hash, chunk3Hash)
// Test Create
bc1 := &BlobChunk{
bc1 := &database.BlobChunk{
BlobID: blob.ID,
ChunkHash: types.ChunkHash("chunk1"),
ChunkHash: types.ChunkHash(chunk1Hash),
Offset: 0,
Length: 1024,
}
err = repos.BlobChunks.Create(ctx, nil, bc1)
err := repos.BlobChunks.Create(ctx, nil, bc1)
if err != nil {
t.Fatalf("failed to create blob chunk: %v", err)
}
// Add more chunks to the same blob
bc2 := &BlobChunk{
bc2 := &database.BlobChunk{
BlobID: blob.ID,
ChunkHash: types.ChunkHash("chunk2"),
ChunkHash: types.ChunkHash(chunk2Hash),
Offset: 1024,
Length: 2048,
}
err = repos.BlobChunks.Create(ctx, nil, bc2)
if err != nil {
t.Fatalf("failed to create second blob chunk: %v", err)
}
bc3 := &BlobChunk{
bc3 := &database.BlobChunk{
BlobID: blob.ID,
ChunkHash: types.ChunkHash("chunk3"),
ChunkHash: types.ChunkHash(chunk3Hash),
Offset: 3072,
Length: 512,
}
err = repos.BlobChunks.Create(ctx, nil, bc3)
if err != nil {
t.Fatalf("failed to create third blob chunk: %v", err)
@@ -81,6 +117,7 @@ func TestBlobChunkRepository(t *testing.T) {
if err != nil {
t.Fatalf("failed to get blob chunks: %v", err)
}
if len(blobChunks) != 3 {
t.Errorf("expected 3 chunks, got %d", len(blobChunks))
}
@@ -89,92 +126,97 @@ func TestBlobChunkRepository(t *testing.T) {
expectedOffsets := []int64{0, 1024, 3072}
for i, bc := range blobChunks {
if bc.Offset != expectedOffsets[i] {
t.Errorf("wrong chunk order: expected offset %d, got %d", expectedOffsets[i], bc.Offset)
t.Errorf("wrong chunk order: expected offset %d, got %d",
expectedOffsets[i], bc.Offset)
}
}
// Test GetByChunkHash
bc, err := repos.BlobChunks.GetByChunkHash(ctx, "chunk2")
if err != nil {
t.Fatalf("failed to get blob chunk by chunk hash: %v", err)
}
if bc == nil {
t.Fatal("expected blob chunk, got nil")
}
if bc.BlobID != blob.ID {
t.Errorf("wrong blob ID: expected %s, got %s", blob.ID, bc.BlobID)
}
if bc.Offset != 1024 {
t.Errorf("wrong offset: expected 1024, got %d", bc.Offset)
}
// Test duplicate insert (should fail due to primary key constraint)
err = repos.BlobChunks.Create(ctx, nil, bc1)
if err == nil {
t.Fatal("duplicate blob_chunk insert should fail due to primary key constraint")
}
if !strings.Contains(err.Error(), "UNIQUE") && !strings.Contains(err.Error(), "constraint") {
if !strings.Contains(err.Error(), "UNIQUE") &&
!strings.Contains(err.Error(), "constraint") {
t.Fatalf("expected constraint error, got: %v", err)
}
}
func TestBlobChunkRepositoryGetByChunkHash(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
blob := mustCreateBlob(t, repos, "blob-gbch-hash")
mustCreateChunks(t, repos, chunk2Hash)
bc2 := &database.BlobChunk{
BlobID: blob.ID,
ChunkHash: types.ChunkHash(chunk2Hash),
Offset: 1024,
Length: 2048,
}
err := repos.BlobChunks.Create(ctx, nil, bc2)
if err != nil {
t.Fatalf("failed to create blob chunk: %v", err)
}
// Test GetByChunkHash
bc, err := repos.BlobChunks.GetByChunkHash(ctx, chunk2Hash)
if err != nil {
t.Fatalf("failed to get blob chunk by chunk hash: %v", err)
}
if bc == nil {
t.Fatal("expected blob chunk, got nil")
}
if bc.BlobID != blob.ID {
t.Errorf("wrong blob ID: expected %s, got %s", blob.ID, bc.BlobID)
}
if bc.Offset != 1024 {
t.Errorf("wrong offset: expected 1024, got %d", bc.Offset)
}
// Test non-existent chunk
bc, err = repos.BlobChunks.GetByChunkHash(ctx, "nonexistent")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if bc != nil {
t.Error("expected nil for non-existent chunk")
}
}
func TestBlobChunkRepositoryMultipleBlobs(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := NewRepositories(db)
repos := database.NewRepositories(db)
// Create blobs
blob1 := &Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash("blob1-hash"),
CreatedTS: time.Now(),
}
blob2 := &Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash("blob2-hash"),
CreatedTS: time.Now(),
}
err := repos.Blobs.Create(ctx, nil, blob1)
if err != nil {
t.Fatalf("failed to create blob1: %v", err)
}
err = repos.Blobs.Create(ctx, nil, blob2)
if err != nil {
t.Fatalf("failed to create blob2: %v", err)
}
// Create chunks
chunkHashes := []types.ChunkHash{"chunk1", "chunk2", "chunk3"}
for _, chunkHash := range chunkHashes {
chunk := &Chunk{
ChunkHash: chunkHash,
Size: 1024,
}
err = repos.Chunks.Create(ctx, nil, chunk)
if err != nil {
t.Fatalf("failed to create chunk %s: %v", chunkHash, err)
}
}
blob1 := mustCreateBlob(t, repos, "blob1-hash")
blob2 := mustCreateBlob(t, repos, "blob2-hash")
mustCreateChunks(t, repos, chunk1Hash, chunk2Hash, chunk3Hash)
// Create chunks across multiple blobs
// Some chunks are shared between blobs (deduplication scenario)
blobChunks := []BlobChunk{
{BlobID: blob1.ID, ChunkHash: types.ChunkHash("chunk1"), Offset: 0, Length: 1024},
{BlobID: blob1.ID, ChunkHash: types.ChunkHash("chunk2"), Offset: 1024, Length: 1024},
{BlobID: blob2.ID, ChunkHash: types.ChunkHash("chunk2"), Offset: 0, Length: 1024}, // chunk2 is shared
{BlobID: blob2.ID, ChunkHash: types.ChunkHash("chunk3"), Offset: 1024, Length: 1024},
blobChunks := []database.BlobChunk{
{BlobID: blob1.ID, ChunkHash: chunk1Hash, Offset: 0, Length: 1024},
{BlobID: blob1.ID, ChunkHash: chunk2Hash, Offset: 1024, Length: 1024},
// chunk2 is shared between the blobs
{BlobID: blob2.ID, ChunkHash: chunk2Hash, Offset: 0, Length: 1024},
{BlobID: blob2.ID, ChunkHash: chunk3Hash, Offset: 1024, Length: 1024},
}
for _, bc := range blobChunks {
@@ -189,6 +231,7 @@ func TestBlobChunkRepositoryMultipleBlobs(t *testing.T) {
if err != nil {
t.Fatalf("failed to get blob1 chunks: %v", err)
}
if len(chunks) != 2 {
t.Errorf("expected 2 chunks for blob1, got %d", len(chunks))
}
@@ -198,15 +241,17 @@ func TestBlobChunkRepositoryMultipleBlobs(t *testing.T) {
if err != nil {
t.Fatalf("failed to get blob2 chunks: %v", err)
}
if len(chunks) != 2 {
t.Errorf("expected 2 chunks for blob2, got %d", len(chunks))
}
// Verify shared chunk
bc, err := repos.BlobChunks.GetByChunkHash(ctx, "chunk2")
bc, err := repos.BlobChunks.GetByChunkHash(ctx, chunk2Hash)
if err != nil {
t.Fatalf("failed to get shared chunk: %v", err)
}
if bc == nil {
t.Fatal("expected shared chunk, got nil")
}
+121 -84
View File
@@ -3,31 +3,39 @@ package database
import (
"context"
"database/sql"
"errors"
"fmt"
"time"
"sneak.berlin/go/vaultik/internal/log"
)
// BlobRepository provides access to the blobs table, which tracks the
// packed, encrypted storage units uploaded to the destination.
type BlobRepository struct {
db *DB
}
// NewBlobRepository creates a BlobRepository backed by db.
func NewBlobRepository(db *DB) *BlobRepository {
return &BlobRepository{db: db}
}
// Create inserts a blob row, using tx when non-nil.
func (r *BlobRepository) Create(ctx context.Context, tx *sql.Tx, blob *Blob) error {
query := `
INSERT INTO blobs (id, blob_hash, created_ts, finished_ts, uncompressed_size, compressed_size, uploaded_ts)
INSERT INTO blobs (id, blob_hash, created_ts, finished_ts,
uncompressed_size, compressed_size, uploaded_ts)
VALUES (?, ?, ?, ?, ?, ?, ?)
`
var finishedTS, uploadedTS *int64
if blob.FinishedTS != nil {
ts := blob.FinishedTS.Unix()
finishedTS = &ts
}
if blob.UploadedTS != nil {
ts := blob.UploadedTS.Unix()
uploadedTS = &ts
@@ -49,85 +57,15 @@ func (r *BlobRepository) Create(ctx context.Context, tx *sql.Tx, blob *Blob) err
return nil
}
// GetByHash returns the blob with the given content hash, or nil if no
// such blob exists.
func (r *BlobRepository) GetByHash(ctx context.Context, hash string) (*Blob, error) {
query := `
SELECT id, blob_hash, created_ts, finished_ts, uncompressed_size, compressed_size, uploaded_ts
FROM blobs
WHERE blob_hash = ?
`
var blob Blob
var createdTSUnix int64
var finishedTSUnix, uploadedTSUnix sql.NullInt64
err := r.db.conn.QueryRowContext(ctx, query, hash).Scan(
&blob.ID,
&blob.Hash,
&createdTSUnix,
&finishedTSUnix,
&blob.UncompressedSize,
&blob.CompressedSize,
&uploadedTSUnix,
)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("querying blob: %w", err)
}
blob.CreatedTS = time.Unix(createdTSUnix, 0).UTC()
if finishedTSUnix.Valid {
ts := time.Unix(finishedTSUnix.Int64, 0).UTC()
blob.FinishedTS = &ts
}
if uploadedTSUnix.Valid {
ts := time.Unix(uploadedTSUnix.Int64, 0).UTC()
blob.UploadedTS = &ts
}
return &blob, nil
return r.getOne(ctx, "blob_hash", hash)
}
// GetByID retrieves a blob by its ID
func (r *BlobRepository) GetByID(ctx context.Context, id string) (*Blob, error) {
query := `
SELECT id, blob_hash, created_ts, finished_ts, uncompressed_size, compressed_size, uploaded_ts
FROM blobs
WHERE id = ?
`
var blob Blob
var createdTSUnix int64
var finishedTSUnix, uploadedTSUnix sql.NullInt64
err := r.db.conn.QueryRowContext(ctx, query, id).Scan(
&blob.ID,
&blob.Hash,
&createdTSUnix,
&finishedTSUnix,
&blob.UncompressedSize,
&blob.CompressedSize,
&uploadedTSUnix,
)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("querying blob: %w", err)
}
blob.CreatedTS = time.Unix(createdTSUnix, 0).UTC()
if finishedTSUnix.Valid {
ts := time.Unix(finishedTSUnix.Int64, 0).UTC()
blob.FinishedTS = &ts
}
if uploadedTSUnix.Valid {
ts := time.Unix(uploadedTSUnix.Int64, 0).UTC()
blob.UploadedTS = &ts
}
return &blob, nil
return r.getOne(ctx, "id", id)
}
// GetAll returns every blob row keyed by blob ID. Useful at restore
@@ -135,7 +73,8 @@ func (r *BlobRepository) GetByID(ctx context.Context, id string) (*Blob, error)
// into blob hashes without doing one GetByID query per chunk.
func (r *BlobRepository) GetAll(ctx context.Context) (map[string]*Blob, error) {
query := `
SELECT id, blob_hash, created_ts, finished_ts, uncompressed_size, compressed_size, uploaded_ts
SELECT id, blob_hash, created_ts, finished_ts,
uncompressed_size, compressed_size, uploaded_ts
FROM blobs
`
@@ -143,14 +82,24 @@ func (r *BlobRepository) GetAll(ctx context.Context) (map[string]*Blob, error) {
if err != nil {
return nil, fmt.Errorf("querying blobs: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
out := make(map[string]*Blob)
for rows.Next() {
var blob Blob
var createdTSUnix int64
var finishedTSUnix, uploadedTSUnix sql.NullInt64
if err := rows.Scan(
var (
blob Blob
createdTSUnix int64
finishedTSUnix, uploadedTSUnix sql.NullInt64
)
err := rows.Scan(
&blob.ID,
&blob.Hash,
&createdTSUnix,
@@ -158,25 +107,36 @@ func (r *BlobRepository) GetAll(ctx context.Context) (map[string]*Blob, error) {
&blob.UncompressedSize,
&blob.CompressedSize,
&uploadedTSUnix,
); err != nil {
)
if err != nil {
return nil, fmt.Errorf("scanning blob: %w", err)
}
blob.CreatedTS = time.Unix(createdTSUnix, 0).UTC()
if finishedTSUnix.Valid {
ts := time.Unix(finishedTSUnix.Int64, 0).UTC()
blob.FinishedTS = &ts
}
if uploadedTSUnix.Valid {
ts := time.Unix(uploadedTSUnix.Int64, 0).UTC()
blob.UploadedTS = &ts
}
out[blob.ID.String()] = &blob
}
return out, rows.Err()
}
// UpdateFinished updates a blob when it's finalized
func (r *BlobRepository) UpdateFinished(ctx context.Context, tx *sql.Tx, id string, hash string, uncompressedSize, compressedSize int64) error {
func (r *BlobRepository) UpdateFinished(
ctx context.Context,
tx *sql.Tx,
id string,
hash string,
uncompressedSize, compressedSize int64,
) error {
query := `
UPDATE blobs
SET blob_hash = ?, finished_ts = ?, uncompressed_size = ?, compressed_size = ?
@@ -184,6 +144,7 @@ func (r *BlobRepository) UpdateFinished(ctx context.Context, tx *sql.Tx, id stri
`
now := time.Now().UTC().Unix()
var err error
if tx != nil {
_, err = tx.ExecContext(ctx, query, hash, now, uncompressedSize, compressedSize, id)
@@ -199,7 +160,9 @@ func (r *BlobRepository) UpdateFinished(ctx context.Context, tx *sql.Tx, id stri
}
// UpdateUploaded marks a blob as uploaded
func (r *BlobRepository) UpdateUploaded(ctx context.Context, tx *sql.Tx, id string) error {
func (r *BlobRepository) UpdateUploaded(
ctx context.Context, tx *sql.Tx, id string,
) error {
query := `
UPDATE blobs
SET uploaded_ts = ?
@@ -207,6 +170,7 @@ func (r *BlobRepository) UpdateUploaded(ctx context.Context, tx *sql.Tx, id stri
`
now := time.Now().UTC().Unix()
var err error
if tx != nil {
_, err = tx.ExecContext(ctx, query, now, id)
@@ -243,3 +207,76 @@ func (r *BlobRepository) DeleteOrphaned(ctx context.Context) error {
return nil
}
// DeleteUnuploaded deletes blob rows whose upload never completed
// (uploaded_ts IS NULL) and returns how many were removed. Their
// blob_chunks rows are removed by the ON DELETE CASCADE foreign key.
// A blob is only ever attached to a snapshot once its upload has been
// recorded, so an un-uploaded blob is never referenced by a completed
// snapshot: dropping it discards chunk rows that point at data which
// was never stored remotely, so the affected content is re-chunked and
// re-uploaded on the next run.
func (r *BlobRepository) DeleteUnuploaded(ctx context.Context) (int64, error) {
query := `DELETE FROM blobs WHERE uploaded_ts IS NULL`
result, err := r.db.ExecWithLog(ctx, query)
if err != nil {
return 0, fmt.Errorf("deleting un-uploaded blobs: %w", err)
}
rowsAffected, _ := result.RowsAffected()
if rowsAffected > 0 {
log.Debug("Deleted un-uploaded blobs", "count", rowsAffected)
}
return rowsAffected, nil
}
// getOne fetches a single blob row matched on the given column, or
// (nil, nil) when no row matches.
func (r *BlobRepository) getOne(
ctx context.Context, column, value string,
) (*Blob, error) {
query := `
SELECT id, blob_hash, created_ts, finished_ts,
uncompressed_size, compressed_size, uploaded_ts
FROM blobs
WHERE ` + column + ` = ?`
var (
blob Blob
createdTSUnix int64
finishedTSUnix, uploadedTSUnix sql.NullInt64
)
err := r.db.conn.QueryRowContext(ctx, query, value).Scan(
&blob.ID,
&blob.Hash,
&createdTSUnix,
&finishedTSUnix,
&blob.UncompressedSize,
&blob.CompressedSize,
&uploadedTSUnix,
)
if errors.Is(err, sql.ErrNoRows) {
return nil, nil //nolint:nilnil // nil,nil signals not-found; callers check nil
}
if err != nil {
return nil, fmt.Errorf("querying blob: %w", err)
}
blob.CreatedTS = time.Unix(createdTSUnix, 0).UTC()
if finishedTSUnix.Valid {
ts := time.Unix(finishedTSUnix.Int64, 0).UTC()
blob.FinishedTS = &ts
}
if uploadedTSUnix.Valid {
ts := time.Unix(uploadedTSUnix.Int64, 0).UTC()
blob.UploadedTS = &ts
}
return &blob, nil
}
+45 -7
View File
@@ -1,22 +1,25 @@
package database
package database_test
import (
"context"
"testing"
"time"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
func TestBlobRepository(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := NewBlobRepository(db)
repo := database.NewBlobRepository(db)
// Test Create
blob := &Blob{
blob := &database.Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash("blobhash123"),
CreatedTS: time.Now().Truncate(time.Second),
@@ -32,14 +35,18 @@ func TestBlobRepository(t *testing.T) {
if err != nil {
t.Fatalf("failed to get blob: %v", err)
}
if retrieved == nil {
t.Fatal("expected blob, got nil")
}
if retrieved.Hash != blob.Hash {
t.Errorf("blob hash mismatch: got %s, want %s", retrieved.Hash, blob.Hash)
}
if !retrieved.CreatedTS.Equal(blob.CreatedTS) {
t.Errorf("created timestamp mismatch: got %v, want %v", retrieved.CreatedTS, blob.CreatedTS)
t.Errorf("created timestamp mismatch: got %v, want %v",
retrieved.CreatedTS, blob.CreatedTS)
}
// Test GetByID
@@ -47,26 +54,51 @@ func TestBlobRepository(t *testing.T) {
if err != nil {
t.Fatalf("failed to get blob by ID: %v", err)
}
if retrievedByID == nil {
t.Fatal("expected blob, got nil")
}
if retrievedByID.ID != blob.ID {
t.Errorf("blob ID mismatch: got %s, want %s", retrievedByID.ID, blob.ID)
}
// Test with second blob
blob2 := &Blob{
blob2 := &database.Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash("blobhash456"),
CreatedTS: time.Now().Truncate(time.Second),
}
err = repo.Create(ctx, nil, blob2)
if err != nil {
t.Fatalf("failed to create second blob: %v", err)
}
}
func TestBlobRepositoryUpdates(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewBlobRepository(db)
blob := &database.Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash("blobhash123"),
CreatedTS: time.Now().Truncate(time.Second),
}
err := repo.Create(ctx, nil, blob)
if err != nil {
t.Fatalf("failed to create blob: %v", err)
}
// Test UpdateFinished
now := time.Now()
err = repo.UpdateFinished(ctx, nil, blob.ID.String(), blob.Hash.String(), 1000, 500)
if err != nil {
t.Fatalf("failed to update blob as finished: %v", err)
@@ -77,12 +109,15 @@ func TestBlobRepository(t *testing.T) {
if err != nil {
t.Fatalf("failed to get updated blob: %v", err)
}
if updated.FinishedTS == nil {
t.Fatal("expected finished timestamp to be set")
}
if updated.UncompressedSize != 1000 {
t.Errorf("expected uncompressed size 1000, got %d", updated.UncompressedSize)
}
if updated.CompressedSize != 500 {
t.Errorf("expected compressed size 500, got %d", updated.CompressedSize)
}
@@ -98,6 +133,7 @@ func TestBlobRepository(t *testing.T) {
if err != nil {
t.Fatalf("failed to get uploaded blob: %v", err)
}
if uploaded.UploadedTS == nil {
t.Fatal("expected uploaded timestamp to be set")
}
@@ -108,13 +144,15 @@ func TestBlobRepository(t *testing.T) {
}
func TestBlobRepositoryDuplicate(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := NewBlobRepository(db)
repo := database.NewBlobRepository(db)
blob := &Blob{
blob := &database.Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash("duplicate_blob"),
CreatedTS: time.Now().Truncate(time.Second),
+64 -23
View File
@@ -1,3 +1,4 @@
//nolint:testpackage // inspects the unexported database connection
package database
import (
@@ -9,23 +10,13 @@ import (
"sneak.berlin/go/vaultik/internal/types"
)
// TestCascadeDeleteDebug tests cascade delete with debug output
func TestCascadeDeleteDebug(t *testing.T) {
db, cleanup := setupTestDB(t)
defer cleanup()
// createCascadeFixtures creates a file with three chunk mappings for the
// cascade-delete test.
func createCascadeFixtures(t *testing.T, repos *Repositories) *File {
t.Helper()
ctx := context.Background()
repos := NewRepositories(db)
// Check if foreign keys are enabled
var fkEnabled int
err := db.conn.QueryRow("PRAGMA foreign_keys").Scan(&fkEnabled)
if err != nil {
t.Fatal(err)
}
t.Logf("Foreign keys enabled: %d", fkEnabled)
// Create a file
file := &File{
Path: "/cascade-test.txt",
MTime: time.Now().Truncate(time.Second),
@@ -34,18 +25,21 @@ func TestCascadeDeleteDebug(t *testing.T) {
UID: 1000,
GID: 1000,
}
err = repos.Files.Create(ctx, nil, file)
err := repos.Files.Create(ctx, nil, file)
if err != nil {
t.Fatalf("failed to create file: %v", err)
}
t.Logf("Created file with ID: %s", file.ID)
// Create chunks and file-chunk mappings
for i := 0; i < 3; i++ {
for i := range 3 {
chunk := &Chunk{
ChunkHash: types.ChunkHash(fmt.Sprintf("cascade-chunk-%d", i)),
Size: 1024,
}
err = repos.Chunks.Create(ctx, nil, chunk)
if err != nil {
t.Fatalf("failed to create chunk: %v", err)
@@ -56,33 +50,73 @@ func TestCascadeDeleteDebug(t *testing.T) {
Idx: i,
ChunkHash: chunk.ChunkHash,
}
err = repos.FileChunks.Create(ctx, nil, fc)
if err != nil {
t.Fatalf("failed to create file chunk: %v", err)
}
t.Logf("Created file chunk mapping: file_id=%s, idx=%d, chunk=%s", fc.FileID, fc.Idx, fc.ChunkHash)
t.Logf("Created file chunk mapping: file_id=%s, idx=%d, chunk=%s",
fc.FileID, fc.Idx, fc.ChunkHash)
}
// Verify file chunks exist
fileChunks, err := repos.FileChunks.GetByFileID(ctx, file.ID)
return file
}
// logCascadeDebugInfo logs foreign-key state and the file_chunks table
// definition for cascade-delete debugging.
func logCascadeDebugInfo(ctx context.Context, t *testing.T, db *DB) {
t.Helper()
// Check if foreign keys are enabled
var fkEnabled int
err := db.conn.QueryRowContext(ctx, "PRAGMA foreign_keys").Scan(&fkEnabled)
if err != nil {
t.Fatal(err)
}
t.Logf("File chunks before delete: %d", len(fileChunks))
t.Logf("Foreign keys enabled: %d", fkEnabled)
// Check the foreign key constraint
var fkInfo string
err = db.conn.QueryRow(`
err = db.conn.QueryRowContext(ctx, `
SELECT sql FROM sqlite_master
WHERE type='table' AND name='file_chunks'
`).Scan(&fkInfo)
if err != nil {
t.Fatal(err)
}
t.Logf("file_chunks table definition:\n%s", fkInfo)
}
// TestCascadeDeleteDebug tests cascade delete with debug output
func TestCascadeDeleteDebug(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := NewRepositories(db)
logCascadeDebugInfo(ctx, t, db)
file := createCascadeFixtures(t, repos)
// Verify file chunks exist
fileChunks, err := repos.FileChunks.GetByFileID(ctx, file.ID)
if err != nil {
t.Fatal(err)
}
t.Logf("File chunks before delete: %d", len(fileChunks))
// Delete the file
t.Log("Deleting file...")
err = repos.Files.DeleteByID(ctx, nil, file.ID)
if err != nil {
t.Fatalf("failed to delete file: %v", err)
@@ -93,6 +127,7 @@ func TestCascadeDeleteDebug(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if deletedFile != nil {
t.Error("file should have been deleted")
} else {
@@ -104,21 +139,27 @@ func TestCascadeDeleteDebug(t *testing.T) {
if err != nil {
t.Fatal(err)
}
t.Logf("File chunks after delete: %d", len(fileChunks))
// Manually check the database
var count int
err = db.conn.QueryRow("SELECT COUNT(*) FROM file_chunks WHERE file_id = ?", file.ID).Scan(&count)
err = db.conn.QueryRowContext(ctx,
"SELECT COUNT(*) FROM file_chunks WHERE file_id = ?", file.ID,
).Scan(&count)
if err != nil {
t.Fatal(err)
}
t.Logf("Manual count of file_chunks for deleted file: %d", count)
if len(fileChunks) != 0 {
t.Errorf("expected 0 file chunks after cascade delete, got %d", len(fileChunks))
// List the remaining chunks
for _, fc := range fileChunks {
t.Logf("Remaining chunk: file_id=%s, idx=%d, chunk=%s", fc.FileID, fc.Idx, fc.ChunkHash)
t.Logf("Remaining chunk: file_id=%s, idx=%d, chunk=%s",
fc.FileID, fc.Idx, fc.ChunkHash)
}
}
}
+112 -48
View File
@@ -4,19 +4,26 @@ import (
"context"
"database/sql"
"fmt"
"strings"
"sneak.berlin/go/vaultik/internal/types"
)
// ChunkFileRepository provides access to the chunk_files table, the
// reverse mapping from chunks to the files that contain them.
type ChunkFileRepository struct {
db *DB
}
// NewChunkFileRepository creates a ChunkFileRepository backed by db.
func NewChunkFileRepository(db *DB) *ChunkFileRepository {
return &ChunkFileRepository{db: db}
}
func (r *ChunkFileRepository) Create(ctx context.Context, tx *sql.Tx, cf *ChunkFile) error {
// Create inserts a chunk_files row (idempotently), using tx when non-nil.
func (r *ChunkFileRepository) Create(
ctx context.Context, tx *sql.Tx, cf *ChunkFile,
) error {
query := `
INSERT INTO chunk_files (chunk_hash, file_id, file_offset, length)
VALUES (?, ?, ?, ?)
@@ -25,9 +32,11 @@ func (r *ChunkFileRepository) Create(ctx context.Context, tx *sql.Tx, cf *ChunkF
var err error
if tx != nil {
_, err = tx.ExecContext(ctx, query, cf.ChunkHash.String(), cf.FileID.String(), cf.FileOffset, cf.Length)
_, err = tx.ExecContext(ctx, query,
cf.ChunkHash.String(), cf.FileID.String(), cf.FileOffset, cf.Length)
} else {
_, err = r.db.ExecWithLog(ctx, query, cf.ChunkHash.String(), cf.FileID.String(), cf.FileOffset, cf.Length)
_, err = r.db.ExecWithLog(ctx, query,
cf.ChunkHash.String(), cf.FileID.String(), cf.FileOffset, cf.Length)
}
if err != nil {
@@ -37,7 +46,10 @@ func (r *ChunkFileRepository) Create(ctx context.Context, tx *sql.Tx, cf *ChunkF
return nil
}
func (r *ChunkFileRepository) GetByChunkHash(ctx context.Context, chunkHash types.ChunkHash) ([]*ChunkFile, error) {
// GetByChunkHash returns all chunk_files rows for the given chunk hash.
func (r *ChunkFileRepository) GetByChunkHash(
ctx context.Context, chunkHash types.ChunkHash,
) ([]*ChunkFile, error) {
query := `
SELECT chunk_hash, file_id, file_offset, length
FROM chunk_files
@@ -48,12 +60,21 @@ func (r *ChunkFileRepository) GetByChunkHash(ctx context.Context, chunkHash type
if err != nil {
return nil, fmt.Errorf("querying chunk files: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanChunkFiles(rows)
}
func (r *ChunkFileRepository) GetByFilePath(ctx context.Context, filePath string) ([]*ChunkFile, error) {
// GetByFilePath returns all chunk_files rows for the file at the given path.
func (r *ChunkFileRepository) GetByFilePath(
ctx context.Context, filePath string,
) ([]*ChunkFile, error) {
query := `
SELECT cf.chunk_hash, cf.file_id, cf.file_offset, cf.length
FROM chunk_files cf
@@ -65,13 +86,21 @@ func (r *ChunkFileRepository) GetByFilePath(ctx context.Context, filePath string
if err != nil {
return nil, fmt.Errorf("querying chunk files: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanChunkFiles(rows)
}
// GetByFileID retrieves chunk files by file ID
func (r *ChunkFileRepository) GetByFileID(ctx context.Context, fileID types.FileID) ([]*ChunkFile, error) {
func (r *ChunkFileRepository) GetByFileID(
ctx context.Context, fileID types.FileID,
) ([]*ChunkFile, error) {
query := `
SELECT chunk_hash, file_id, file_offset, length
FROM chunk_files
@@ -82,34 +111,21 @@ func (r *ChunkFileRepository) GetByFileID(ctx context.Context, fileID types.File
if err != nil {
return nil, fmt.Errorf("querying chunk files: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanChunkFiles(rows)
}
// scanChunkFiles is a helper that scans chunk file rows
func (r *ChunkFileRepository) scanChunkFiles(rows *sql.Rows) ([]*ChunkFile, error) {
var chunkFiles []*ChunkFile
for rows.Next() {
var cf ChunkFile
var chunkHashStr, fileIDStr string
err := rows.Scan(&chunkHashStr, &fileIDStr, &cf.FileOffset, &cf.Length)
if err != nil {
return nil, fmt.Errorf("scanning chunk file: %w", err)
}
cf.ChunkHash = types.ChunkHash(chunkHashStr)
cf.FileID, err = types.ParseFileID(fileIDStr)
if err != nil {
return nil, fmt.Errorf("parsing file ID: %w", err)
}
chunkFiles = append(chunkFiles, &cf)
}
return chunkFiles, rows.Err()
}
// DeleteByFileID deletes all chunk_files entries for a given file ID
func (r *ChunkFileRepository) DeleteByFileID(ctx context.Context, tx *sql.Tx, fileID types.FileID) error {
func (r *ChunkFileRepository) DeleteByFileID(
ctx context.Context, tx *sql.Tx, fileID types.FileID,
) error {
query := `DELETE FROM chunk_files WHERE file_id = ?`
var err error
@@ -127,7 +143,11 @@ func (r *ChunkFileRepository) DeleteByFileID(ctx context.Context, tx *sql.Tx, fi
}
// DeleteByFileIDs deletes all chunk_files for multiple files in a single statement.
func (r *ChunkFileRepository) DeleteByFileIDs(ctx context.Context, tx *sql.Tx, fileIDs []types.FileID) error {
//
//nolint:dupl // symmetric implementation for a parallel association table
func (r *ChunkFileRepository) DeleteByFileIDs(
ctx context.Context, tx *sql.Tx, fileIDs []types.FileID,
) error {
if len(fileIDs) == 0 {
return nil
}
@@ -136,14 +156,15 @@ func (r *ChunkFileRepository) DeleteByFileIDs(ctx context.Context, tx *sql.Tx, f
const batchSize = 500
for i := 0; i < len(fileIDs); i += batchSize {
end := i + batchSize
if end > len(fileIDs) {
end = len(fileIDs)
}
end := min(i+batchSize, len(fileIDs))
batch := fileIDs[i:end]
query := "DELETE FROM chunk_files WHERE file_id IN (?" + repeatPlaceholder(len(batch)-1) + ")"
args := make([]interface{}, len(batch))
//nolint:gosec // G202: concatenates constant SQL and "?" placeholders only
query := "DELETE FROM chunk_files WHERE file_id IN (?" +
repeatPlaceholder(len(batch)-1) + ")"
args := make([]any, len(batch))
for j, id := range batch {
args[j] = id.String()
}
@@ -154,6 +175,7 @@ func (r *ChunkFileRepository) DeleteByFileIDs(ctx context.Context, tx *sql.Tx, f
} else {
_, err = r.db.ExecWithLog(ctx, query, args...)
}
if err != nil {
return fmt.Errorf("batch deleting chunk_files: %w", err)
}
@@ -163,30 +185,43 @@ func (r *ChunkFileRepository) DeleteByFileIDs(ctx context.Context, tx *sql.Tx, f
}
// CreateBatch inserts multiple chunk_files in a single statement for efficiency.
func (r *ChunkFileRepository) CreateBatch(ctx context.Context, tx *sql.Tx, cfs []ChunkFile) error {
func (r *ChunkFileRepository) CreateBatch(
ctx context.Context, tx *sql.Tx, cfs []ChunkFile,
) error {
if len(cfs) == 0 {
return nil
}
// Each ChunkFile has 4 values, so batch at 200 to be safe with SQLite's variable limit
// Each chunk_files row binds this many SQL variables.
const chunkFileCols = 4
// Batch at 200 rows to be safe with SQLite's variable limit.
const batchSize = 200
for i := 0; i < len(cfs); i += batchSize {
end := i + batchSize
if end > len(cfs) {
end = len(cfs)
}
end := min(i+batchSize, len(cfs))
batch := cfs[i:end]
query := "INSERT INTO chunk_files (chunk_hash, file_id, file_offset, length) VALUES "
args := make([]interface{}, 0, len(batch)*4)
args := make([]any, 0, len(batch)*chunkFileCols)
var querySb183 strings.Builder
for j, cf := range batch {
if j > 0 {
query += ", "
querySb183.WriteString(", ")
}
query += "(?, ?, ?, ?)"
args = append(args, cf.ChunkHash.String(), cf.FileID.String(), cf.FileOffset, cf.Length)
querySb183.WriteString("(?, ?, ?, ?)")
args = append(args,
cf.ChunkHash.String(), cf.FileID.String(), cf.FileOffset, cf.Length)
}
query += querySb183.String() //nolint:gosec // G202: appends "?" placeholders only
query += " ON CONFLICT(chunk_hash, file_id) DO NOTHING"
var err error
@@ -195,6 +230,7 @@ func (r *ChunkFileRepository) CreateBatch(ctx context.Context, tx *sql.Tx, cfs [
} else {
_, err = r.db.ExecWithLog(ctx, query, args...)
}
if err != nil {
return fmt.Errorf("batch inserting chunk_files: %w", err)
}
@@ -202,3 +238,31 @@ func (r *ChunkFileRepository) CreateBatch(ctx context.Context, tx *sql.Tx, cfs [
return nil
}
// scanChunkFiles is a helper that scans chunk file rows.
func (r *ChunkFileRepository) scanChunkFiles(rows *sql.Rows) ([]*ChunkFile, error) {
var chunkFiles []*ChunkFile
for rows.Next() {
var (
cf ChunkFile
chunkHashStr, fileIDStr string
)
err := rows.Scan(&chunkHashStr, &fileIDStr, &cf.FileOffset, &cf.Length)
if err != nil {
return nil, fmt.Errorf("scanning chunk file: %w", err)
}
cf.ChunkHash = types.ChunkHash(chunkHashStr)
cf.FileID, err = types.ParseFileID(fileIDStr)
if err != nil {
return nil, fmt.Errorf("parsing file ID: %w", err)
}
chunkFiles = append(chunkFiles, &cf)
}
return chunkFiles, rows.Err()
}
+114 -99
View File
@@ -1,120 +1,139 @@
package database
package database_test
import (
"context"
"testing"
"time"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
const chunk4Hash = "chunk4"
// verifyChunkFilePair asserts that the chunk-file rows cover both test
// files at their expected offsets.
func verifyChunkFilePair(
t *testing.T, chunkFiles []*database.ChunkFile,
file1ID, file2ID types.FileID,
) {
t.Helper()
foundFile1 := false
foundFile2 := false
for _, cf := range chunkFiles {
if cf.FileID == file1ID && cf.FileOffset == 0 {
foundFile1 = true
}
if cf.FileID == file2ID && cf.FileOffset == 2048 {
foundFile2 = true
}
}
if !foundFile1 || !foundFile2 {
t.Error("not all expected files found")
}
}
// createChunkFileTestFiles creates the two files used by the chunk-file
// repository tests.
func createChunkFileTestFiles(
t *testing.T, fileRepo *database.FileRepository,
) (*database.File, *database.File) {
t.Helper()
testTime := time.Now().Truncate(time.Second)
file1 := &database.File{
Path: testFilePath1,
MTime: testTime,
Size: 1024,
Mode: 0644,
UID: 1000,
GID: 1000,
LinkTarget: "",
}
file2 := &database.File{
Path: testFilePath2,
MTime: testTime,
Size: 1024,
Mode: 0644,
UID: 1000,
GID: 1000,
LinkTarget: "",
}
mustCreateFile(t, fileRepo, file1)
mustCreateFile(t, fileRepo, file2)
return file1, file2
}
func TestChunkFileRepository(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := NewChunkFileRepository(db)
fileRepo := NewFileRepository(db)
chunksRepo := NewChunkRepository(db)
repo := database.NewChunkFileRepository(db)
fileRepo := database.NewFileRepository(db)
repos := database.NewRepositories(db)
// Create test files first
testTime := time.Now().Truncate(time.Second)
file1 := &File{
Path: "/file1.txt",
MTime: testTime,
Size: 1024,
Mode: 0644,
UID: 1000,
GID: 1000,
LinkTarget: "",
}
err := fileRepo.Create(ctx, nil, file1)
if err != nil {
t.Fatalf("failed to create file1: %v", err)
}
file2 := &File{
Path: "/file2.txt",
MTime: testTime,
Size: 1024,
Mode: 0644,
UID: 1000,
GID: 1000,
LinkTarget: "",
}
err = fileRepo.Create(ctx, nil, file2)
if err != nil {
t.Fatalf("failed to create file2: %v", err)
}
// Create chunk first
chunk := &Chunk{
ChunkHash: types.ChunkHash("chunk1"),
Size: 1024,
}
err = chunksRepo.Create(ctx, nil, chunk)
if err != nil {
t.Fatalf("failed to create chunk: %v", err)
}
file1, file2 := createChunkFileTestFiles(t, fileRepo)
mustCreateChunks(t, repos, chunk1Hash)
// Test Create
cf1 := &ChunkFile{
ChunkHash: types.ChunkHash("chunk1"),
cf1 := &database.ChunkFile{
ChunkHash: types.ChunkHash(chunk1Hash),
FileID: file1.ID,
FileOffset: 0,
Length: 1024,
}
err = repo.Create(ctx, nil, cf1)
err := repo.Create(ctx, nil, cf1)
if err != nil {
t.Fatalf("failed to create chunk file: %v", err)
}
// Add same chunk in different file (deduplication scenario)
cf2 := &ChunkFile{
ChunkHash: types.ChunkHash("chunk1"),
cf2 := &database.ChunkFile{
ChunkHash: types.ChunkHash(chunk1Hash),
FileID: file2.ID,
FileOffset: 2048,
Length: 1024,
}
err = repo.Create(ctx, nil, cf2)
if err != nil {
t.Fatalf("failed to create second chunk file: %v", err)
}
// Test GetByChunkHash
chunkFiles, err := repo.GetByChunkHash(ctx, "chunk1")
chunkFiles, err := repo.GetByChunkHash(ctx, chunk1Hash)
if err != nil {
t.Fatalf("failed to get chunk files: %v", err)
}
if len(chunkFiles) != 2 {
t.Errorf("expected 2 files for chunk, got %d", len(chunkFiles))
}
// Verify both files are returned
foundFile1 := false
foundFile2 := false
for _, cf := range chunkFiles {
if cf.FileID == file1.ID && cf.FileOffset == 0 {
foundFile1 = true
}
if cf.FileID == file2.ID && cf.FileOffset == 2048 {
foundFile2 = true
}
}
if !foundFile1 || !foundFile2 {
t.Error("not all expected files found")
}
verifyChunkFilePair(t, chunkFiles, file1.ID, file2.ID)
// Test GetByFileID
chunkFiles, err = repo.GetByFileID(ctx, file1.ID)
if err != nil {
t.Fatalf("failed to get chunks by file ID: %v", err)
}
if len(chunkFiles) != 1 {
t.Errorf("expected 1 chunk for file, got %d", len(chunkFiles))
}
if chunkFiles[0].ChunkHash != types.ChunkHash("chunk1") {
if chunkFiles[0].ChunkHash != types.ChunkHash(chunk1Hash) {
t.Errorf("wrong chunk hash: expected chunk1, got %s", chunkFiles[0].ChunkHash)
}
@@ -126,60 +145,53 @@ func TestChunkFileRepository(t *testing.T) {
}
func TestChunkFileRepositoryComplexDeduplication(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := NewChunkFileRepository(db)
fileRepo := NewFileRepository(db)
chunksRepo := NewChunkRepository(db)
repo := database.NewChunkFileRepository(db)
fileRepo := database.NewFileRepository(db)
repos := database.NewRepositories(db)
// Create test files
testTime := time.Now().Truncate(time.Second)
file1 := &File{Path: "/file1.txt", MTime: testTime, Size: 3072, Mode: 0644, UID: 1000, GID: 1000}
file2 := &File{Path: "/file2.txt", MTime: testTime, Size: 3072, Mode: 0644, UID: 1000, GID: 1000}
file3 := &File{Path: "/file3.txt", MTime: testTime, Size: 2048, Mode: 0644, UID: 1000, GID: 1000}
if err := fileRepo.Create(ctx, nil, file1); err != nil {
t.Fatalf("failed to create file1: %v", err)
file1 := &database.File{
Path: testFilePath1, MTime: testTime, Size: 3072,
Mode: 0644, UID: 1000, GID: 1000,
}
if err := fileRepo.Create(ctx, nil, file2); err != nil {
t.Fatalf("failed to create file2: %v", err)
file2 := &database.File{
Path: testFilePath2, MTime: testTime, Size: 3072,
Mode: 0644, UID: 1000, GID: 1000,
}
if err := fileRepo.Create(ctx, nil, file3); err != nil {
t.Fatalf("failed to create file3: %v", err)
file3 := &database.File{
Path: "/file3.txt", MTime: testTime, Size: 2048,
Mode: 0644, UID: 1000, GID: 1000,
}
// Create chunks first
chunks := []types.ChunkHash{"chunk1", "chunk2", "chunk3", "chunk4"}
for _, chunkHash := range chunks {
chunk := &Chunk{
ChunkHash: chunkHash,
Size: 1024,
}
err := chunksRepo.Create(ctx, nil, chunk)
if err != nil {
t.Fatalf("failed to create chunk %s: %v", chunkHash, err)
}
}
mustCreateFile(t, fileRepo, file1)
mustCreateFile(t, fileRepo, file2)
mustCreateFile(t, fileRepo, file3)
mustCreateChunks(t, repos, chunk1Hash, chunk2Hash, chunk3Hash, chunk4Hash)
// Simulate a scenario where multiple files share chunks
// File1: chunk1, chunk2, chunk3
// File2: chunk2, chunk3, chunk4
// File3: chunk1, chunk4
chunkFiles := []ChunkFile{
chunkFiles := []database.ChunkFile{
// File1
{ChunkHash: types.ChunkHash("chunk1"), FileID: file1.ID, FileOffset: 0, Length: 1024},
{ChunkHash: types.ChunkHash("chunk2"), FileID: file1.ID, FileOffset: 1024, Length: 1024},
{ChunkHash: types.ChunkHash("chunk3"), FileID: file1.ID, FileOffset: 2048, Length: 1024},
{ChunkHash: chunk1Hash, FileID: file1.ID, FileOffset: 0, Length: 1024},
{ChunkHash: chunk2Hash, FileID: file1.ID, FileOffset: 1024, Length: 1024},
{ChunkHash: chunk3Hash, FileID: file1.ID, FileOffset: 2048, Length: 1024},
// File2
{ChunkHash: types.ChunkHash("chunk2"), FileID: file2.ID, FileOffset: 0, Length: 1024},
{ChunkHash: types.ChunkHash("chunk3"), FileID: file2.ID, FileOffset: 1024, Length: 1024},
{ChunkHash: types.ChunkHash("chunk4"), FileID: file2.ID, FileOffset: 2048, Length: 1024},
{ChunkHash: chunk2Hash, FileID: file2.ID, FileOffset: 0, Length: 1024},
{ChunkHash: chunk3Hash, FileID: file2.ID, FileOffset: 1024, Length: 1024},
{ChunkHash: chunk4Hash, FileID: file2.ID, FileOffset: 2048, Length: 1024},
// File3
{ChunkHash: types.ChunkHash("chunk1"), FileID: file3.ID, FileOffset: 0, Length: 1024},
{ChunkHash: types.ChunkHash("chunk4"), FileID: file3.ID, FileOffset: 1024, Length: 1024},
{ChunkHash: chunk1Hash, FileID: file3.ID, FileOffset: 0, Length: 1024},
{ChunkHash: chunk4Hash, FileID: file3.ID, FileOffset: 1024, Length: 1024},
}
for _, cf := range chunkFiles {
@@ -190,19 +202,21 @@ func TestChunkFileRepositoryComplexDeduplication(t *testing.T) {
}
// Test chunk1 (used by file1 and file3)
files, err := repo.GetByChunkHash(ctx, "chunk1")
files, err := repo.GetByChunkHash(ctx, chunk1Hash)
if err != nil {
t.Fatalf("failed to get files for chunk1: %v", err)
}
if len(files) != 2 {
t.Errorf("expected 2 files for chunk1, got %d", len(files))
}
// Test chunk2 (used by file1 and file2)
files, err = repo.GetByChunkHash(ctx, "chunk2")
files, err = repo.GetByChunkHash(ctx, chunk2Hash)
if err != nil {
t.Fatalf("failed to get files for chunk2: %v", err)
}
if len(files) != 2 {
t.Errorf("expected 2 files for chunk2, got %d", len(files))
}
@@ -212,6 +226,7 @@ func TestChunkFileRepositoryComplexDeduplication(t *testing.T) {
if err != nil {
t.Fatalf("failed to get chunks for file2: %v", err)
}
if len(file2Chunks) != 3 {
t.Errorf("expected 3 chunks for file2, got %d", len(file2Chunks))
}
+48 -9
View File
@@ -3,19 +3,25 @@ package database
import (
"context"
"database/sql"
"errors"
"fmt"
"strings"
"sneak.berlin/go/vaultik/internal/log"
)
// ChunkRepository provides access to the chunks table, which tracks
// content-defined chunks by hash and size.
type ChunkRepository struct {
db *DB
}
// NewChunkRepository creates a ChunkRepository backed by db.
func NewChunkRepository(db *DB) *ChunkRepository {
return &ChunkRepository{db: db}
}
// Create inserts a chunk row (idempotently), using tx when non-nil.
func (r *ChunkRepository) Create(ctx context.Context, tx *sql.Tx, chunk *Chunk) error {
query := `
INSERT INTO chunks (chunk_hash, size)
@@ -37,6 +43,8 @@ func (r *ChunkRepository) Create(ctx context.Context, tx *sql.Tx, chunk *Chunk)
return nil
}
// GetByHash returns the chunk with the given hash, or nil if it is not
// known to the index.
func (r *ChunkRepository) GetByHash(ctx context.Context, hash string) (*Chunk, error) {
query := `
SELECT chunk_hash, size
@@ -51,9 +59,10 @@ func (r *ChunkRepository) GetByHash(ctx context.Context, hash string) (*Chunk, e
&chunk.Size,
)
if err == sql.ErrNoRows {
return nil, nil
if errors.Is(err, sql.ErrNoRows) {
return nil, nil //nolint:nilnil // nil,nil signals not-found; callers check nil
}
if err != nil {
return nil, fmt.Errorf("querying chunk: %w", err)
}
@@ -61,7 +70,11 @@ func (r *ChunkRepository) GetByHash(ctx context.Context, hash string) (*Chunk, e
return &chunk, nil
}
func (r *ChunkRepository) GetByHashes(ctx context.Context, hashes []string) ([]*Chunk, error) {
// GetByHashes returns the chunks whose hashes appear in hashes, ordered by
// chunk hash. Unknown hashes are silently omitted from the result.
func (r *ChunkRepository) GetByHashes(
ctx context.Context, hashes []string,
) ([]*Chunk, error) {
if len(hashes) == 0 {
return nil, nil
}
@@ -71,23 +84,38 @@ func (r *ChunkRepository) GetByHashes(ctx context.Context, hashes []string) ([]*
FROM chunks
WHERE chunk_hash IN (`
args := make([]interface{}, len(hashes))
args := make([]any, len(hashes))
var querySb75 strings.Builder
for i, hash := range hashes {
if i > 0 {
query += ", "
querySb75.WriteString(", ")
}
query += "?"
querySb75.WriteString("?")
args[i] = hash
}
query += querySb75.String() //nolint:gosec // G202: appends "?" placeholders only
query += ") ORDER BY chunk_hash"
rows, err := r.db.conn.QueryContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("querying chunks: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var chunks []*Chunk
for rows.Next() {
var chunk Chunk
@@ -105,7 +133,11 @@ func (r *ChunkRepository) GetByHashes(ctx context.Context, hashes []string) ([]*
return chunks, rows.Err()
}
func (r *ChunkRepository) ListUnpacked(ctx context.Context, limit int) ([]*Chunk, error) {
// ListUnpacked returns up to limit chunks that are not yet stored in any
// blob, ordered by chunk hash.
func (r *ChunkRepository) ListUnpacked(
ctx context.Context, limit int,
) ([]*Chunk, error) {
query := `
SELECT c.chunk_hash, c.size
FROM chunks c
@@ -119,9 +151,16 @@ func (r *ChunkRepository) ListUnpacked(ctx context.Context, limit int) ([]*Chunk
if err != nil {
return nil, fmt.Errorf("querying unpacked chunks: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var chunks []*Chunk
for rows.Next() {
var chunk Chunk
+31 -3
View File
@@ -5,20 +5,48 @@ import (
"fmt"
)
// List returns every chunk in the index, ordered by chunk hash.
func (r *ChunkRepository) List(ctx context.Context) ([]*Chunk, error) {
query := `
return r.list(ctx, `
SELECT chunk_hash, size
FROM chunks
ORDER BY chunk_hash
`
`)
}
// ListInUploadedBlobs returns the chunks that are stored in a blob whose
// upload has completed (uploaded_ts set), ordered by chunk hash. These
// are the only chunks a backup may safely deduplicate against: a chunk
// recorded solely in a blob that was never uploaded refers to data that
// is not in remote storage, so trusting it would silently drop that data
// from later snapshots.
func (r *ChunkRepository) ListInUploadedBlobs(ctx context.Context) ([]*Chunk, error) {
return r.list(ctx, `
SELECT DISTINCT c.chunk_hash, c.size
FROM chunks c
JOIN blob_chunks bc ON c.chunk_hash = bc.chunk_hash
JOIN blobs b ON bc.blob_id = b.id
WHERE b.uploaded_ts IS NOT NULL
ORDER BY c.chunk_hash
`)
}
// list runs a chunk-selecting query and scans the (chunk_hash, size) rows.
func (r *ChunkRepository) list(ctx context.Context, query string) ([]*Chunk, error) {
rows, err := r.db.conn.QueryContext(ctx, query)
if err != nil {
return nil, fmt.Errorf("querying chunks: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var chunks []*Chunk
for rows.Next() {
var chunk Chunk
+21 -6
View File
@@ -1,21 +1,24 @@
package database
package database_test
import (
"context"
"testing"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
func TestChunkRepository(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := NewChunkRepository(db)
repo := database.NewChunkRepository(db)
// Test Create
chunk := &Chunk{
chunk := &database.Chunk{
ChunkHash: types.ChunkHash("chunkhash123"),
Size: 4096,
}
@@ -30,12 +33,15 @@ func TestChunkRepository(t *testing.T) {
if err != nil {
t.Fatalf("failed to get chunk: %v", err)
}
if retrieved == nil {
t.Fatal("expected chunk, got nil")
}
if retrieved.ChunkHash != chunk.ChunkHash {
t.Errorf("chunk hash mismatch: got %s, want %s", retrieved.ChunkHash, chunk.ChunkHash)
}
if retrieved.Size != chunk.Size {
t.Errorf("size mismatch: got %d, want %d", retrieved.Size, chunk.Size)
}
@@ -47,19 +53,23 @@ func TestChunkRepository(t *testing.T) {
}
// Test GetByHashes
chunk2 := &Chunk{
chunk2 := &database.Chunk{
ChunkHash: types.ChunkHash("chunkhash456"),
Size: 8192,
}
err = repo.Create(ctx, nil, chunk2)
if err != nil {
t.Fatalf("failed to create second chunk: %v", err)
}
chunks, err := repo.GetByHashes(ctx, []string{chunk.ChunkHash.String(), chunk2.ChunkHash.String()})
chunks, err := repo.GetByHashes(ctx, []string{
chunk.ChunkHash.String(), chunk2.ChunkHash.String(),
})
if err != nil {
t.Fatalf("failed to get chunks by hashes: %v", err)
}
if len(chunks) != 2 {
t.Errorf("expected 2 chunks, got %d", len(chunks))
}
@@ -69,23 +79,27 @@ func TestChunkRepository(t *testing.T) {
if err != nil {
t.Fatalf("failed to list unpacked chunks: %v", err)
}
if len(unpacked) != 2 {
t.Errorf("expected 2 unpacked chunks, got %d", len(unpacked))
}
}
func TestChunkRepositoryNotFound(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := NewChunkRepository(db)
repo := database.NewChunkRepository(db)
// Test GetByHash with non-existent hash
chunk, err := repo.GetByHash(ctx, "nonexistent")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if chunk != nil {
t.Error("expected nil for non-existent chunk")
}
@@ -95,6 +109,7 @@ func TestChunkRepositoryNotFound(t *testing.T) {
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if chunks != nil {
t.Error("expected nil for empty hash list")
}
+251 -70
View File
@@ -3,8 +3,10 @@
//
// Blobs in Vaultik are the final storage units uploaded to S3. Each blob is a
// large (up to 10GB) file containing many compressed and encrypted chunks from
// multiple source files. Blobs are content-addressed, meaning their filename
// is derived from their SHA256 hash after compression and encryption.
// multiple source files. Blobs are content-addressed: the filename in S3 is
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data
// before compression and encryption (not from the stored bytes). See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
//
// Schema is managed via numbered SQL migrations embedded in the schema/
// directory. Migration 000.sql bootstraps the schema_migrations tracking
@@ -15,17 +17,24 @@ import (
"context"
"database/sql"
"embed"
"errors"
"fmt"
"net/url"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
// Register the pure-Go sqlite driver.
_ "modernc.org/sqlite"
"sneak.berlin/go/vaultik/internal/log"
)
// errInvalidMigrationFilename is returned when an embedded migration file
// does not follow the "<version>[_<description>].sql" naming pattern.
var errInvalidMigrationFilename = errors.New("invalid migration filename")
//go:embed schema/*.sql
var schemaFS embed.FS
@@ -51,26 +60,28 @@ type DB struct {
func ParseMigrationVersion(filename string) (int, error) {
name := strings.TrimSuffix(filename, filepath.Ext(filename))
if name == "" {
return 0, fmt.Errorf("invalid migration filename %q: empty name", filename)
return 0, fmt.Errorf("%w %q: empty name", errInvalidMigrationFilename, filename)
}
// Split on underscore to separate version from description.
// If there's no underscore, the entire stem is the version.
versionStr := name
if idx := strings.IndexByte(name, '_'); idx >= 0 {
versionStr = name[:idx]
if before, _, ok := strings.Cut(name, "_"); ok {
versionStr = before
}
if versionStr == "" {
return 0, fmt.Errorf("invalid migration filename %q: empty version prefix", filename)
return 0, fmt.Errorf(
"%w %q: empty version prefix", errInvalidMigrationFilename, filename,
)
}
// Validate the version is purely numeric.
for _, ch := range versionStr {
if ch < '0' || ch > '9' {
return 0, fmt.Errorf(
"invalid migration filename %q: version %q contains non-numeric character %q",
filename, versionStr, string(ch),
"%w %q: version %q contains non-numeric character %q",
errInvalidMigrationFilename, filename, versionStr, string(ch),
)
}
}
@@ -98,61 +109,93 @@ func New(ctx context.Context, path string) (*DB, error) {
// First attempt with standard WAL mode
log.Debug("Attempting to open database with WAL mode", "path", path)
conn, err := sql.Open(
"sqlite",
path+"?_journal_mode=WAL&_synchronous=NORMAL&_busy_timeout=10000&_locking_mode=NORMAL&_foreign_keys=ON",
path+"?_journal_mode=WAL&_synchronous=NORMAL&_busy_timeout=10000"+
"&_locking_mode=NORMAL&_foreign_keys=ON",
)
if err == nil {
// Set connection pool settings
// SQLite can handle multiple readers but only one writer at a time.
// Setting MaxOpenConns to 1 ensures all writes are serialized through
// a single connection, preventing SQLITE_BUSY errors.
conn.SetMaxOpenConns(1)
conn.SetMaxIdleConns(1)
configureConnPool(conn)
if err := conn.PingContext(ctx); err == nil {
err = conn.PingContext(ctx)
if err == nil {
// Success on first try
log.Debug("Database opened successfully with WAL mode", "path", path)
// Enable foreign keys explicitly
if _, err := conn.ExecContext(ctx, "PRAGMA foreign_keys = ON"); err != nil {
log.Warn("Failed to enable foreign keys", "error", err)
return finishOpen(ctx, conn, path)
}
db := &DB{conn: conn, path: path}
if err := applyMigrations(ctx, conn); err != nil {
_ = conn.Close()
return nil, fmt.Errorf("applying migrations: %w", err)
}
return db, nil
}
log.Debug("Failed to ping database, closing connection", "path", path, "error", err)
log.Debug(
"Failed to ping database, closing connection",
"path", path, "error", err,
)
_ = conn.Close()
}
// If first attempt failed, try with TRUNCATE mode to clear any locks
return openWithRecovery(ctx, path)
}
// configureConnPool serializes all database access through one connection.
// SQLite can handle multiple readers but only one writer at a time; setting
// MaxOpenConns to 1 ensures all writes go through a single connection,
// preventing SQLITE_BUSY errors.
func configureConnPool(conn *sql.DB) {
conn.SetMaxOpenConns(1)
conn.SetMaxIdleConns(1)
}
// finishOpen enables foreign keys, wraps the connection, and applies any
// pending migrations. On migration failure the connection is closed.
func finishOpen(ctx context.Context, conn *sql.DB, path string) (*DB, error) {
// Enable foreign keys explicitly
_, err := conn.ExecContext(ctx, "PRAGMA foreign_keys = ON")
if err != nil {
log.Warn("Failed to enable foreign keys", "path", path, "error", err)
}
db := &DB{conn: conn, path: path}
err = applyMigrations(ctx, conn)
if err != nil {
_ = conn.Close()
return nil, fmt.Errorf("applying migrations: %w", err)
}
return db, nil
}
// openWithRecovery retries opening the database in TRUNCATE journal mode to
// clear stale locks, then switches back to WAL mode.
func openWithRecovery(ctx context.Context, path string) (*DB, error) {
log.Info(
"Database appears locked, attempting recovery with TRUNCATE mode",
"path", path,
)
conn, err = sql.Open(
conn, err := sql.Open(
"sqlite",
path+"?_journal_mode=TRUNCATE&_synchronous=NORMAL&_busy_timeout=10000&_foreign_keys=ON",
path+"?_journal_mode=TRUNCATE&_synchronous=NORMAL&_busy_timeout=10000"+
"&_foreign_keys=ON",
)
if err != nil {
return nil, fmt.Errorf("opening database in recovery mode: %w", err)
}
// Set connection pool settings
// SQLite can handle multiple readers but only one writer at a time.
// Setting MaxOpenConns to 1 ensures all writes are serialized through
// a single connection, preventing SQLITE_BUSY errors.
conn.SetMaxOpenConns(1)
conn.SetMaxIdleConns(1)
configureConnPool(conn)
err = conn.PingContext(ctx)
if err != nil {
log.Debug(
"Failed to ping database in recovery mode, closing",
"path", path, "error", err,
)
if err := conn.PingContext(ctx); err != nil {
log.Debug("Failed to ping database in recovery mode, closing", "path", path, "error", err)
_ = conn.Close()
return nil, fmt.Errorf(
"database still locked after recovery attempt: %w",
err,
@@ -163,35 +206,173 @@ func New(ctx context.Context, path string) (*DB, error) {
// Switch back to WAL mode
log.Debug("Switching database back to WAL mode", "path", path)
if _, err := conn.ExecContext(ctx, "PRAGMA journal_mode=WAL"); err != nil {
_, err = conn.ExecContext(ctx, "PRAGMA journal_mode=WAL")
if err != nil {
log.Warn("Failed to switch back to WAL mode", "path", path, "error", err)
}
// Ensure foreign keys are enabled
if _, err := conn.ExecContext(ctx, "PRAGMA foreign_keys=ON"); err != nil {
log.Warn("Failed to enable foreign keys", "path", path, "error", err)
}
db := &DB{conn: conn, path: path}
if err := applyMigrations(ctx, conn); err != nil {
_ = conn.Close()
return nil, fmt.Errorf("applying migrations: %w", err)
db, err := finishOpen(ctx, conn, path)
if err != nil {
return nil, err
}
log.Debug("Database connection established successfully", "path", path)
return db, nil
}
// errUntrustedSnapshotSchema is returned when a downloaded snapshot
// database carries schema objects the real schema never defines, or is
// missing a table the restore and deep-verify queries read.
var errUntrustedSnapshotSchema = errors.New(
"downloaded snapshot database has an untrusted schema")
// snapshotReadOnlyDSN builds the driver DSN that opens a materialized
// snapshot database file read-only. mode=ro opens the file read-only at
// the OS level, query_only rejects any write the engine is asked to make,
// and trusted_schema=OFF refuses to run application code named in the
// schema. The file: URI form is required for the driver to honour the
// mode parameter.
func snapshotReadOnlyDSN(path string) string {
u := url.URL{
Scheme: "file",
Path: path,
RawQuery: "mode=ro&_pragma=query_only(true)&_pragma=trusted_schema(false)",
}
return u.String()
}
// OpenReadOnly opens an already-materialized SQLite file for read-only
// querying of a snapshot database downloaded from the store, used by
// restore and deep verify. Unlike New it never applies schema migrations
// and never writes: the connection is opened read-only with query_only
// and trusted_schema=OFF. It refuses any file whose schema carries a
// trigger, view or virtual table, or lacks an expected table, so a forged
// file cannot redefine what the restore queries return. The caller owns
// the file and must remove it.
func OpenReadOnly(ctx context.Context, path string) (*DB, error) {
conn, err := sql.Open("sqlite", snapshotReadOnlyDSN(path))
if err != nil {
return nil, fmt.Errorf("opening read-only database: %w", err)
}
configureConnPool(conn)
err = conn.PingContext(ctx)
if err != nil {
_ = conn.Close()
return nil, fmt.Errorf("opening read-only database: %w", err)
}
err = verifySnapshotSchema(ctx, conn)
if err != nil {
_ = conn.Close()
return nil, err
}
return &DB{conn: conn, path: path}, nil
}
// verifySnapshotSchema rejects a downloaded database whose schema is not
// the plain table set the real schema defines. Any trigger, view or
// virtual table, or a missing expected table, fails the open.
func verifySnapshotSchema(ctx context.Context, conn *sql.DB) error {
// expectedSnapshotTables are the tables the restore and deep-verify
// queries read. A downloaded database missing any of them is not a
// genuine snapshot database and is refused.
expectedSnapshotTables := []string{
"blob_chunks",
"blobs",
"chunks",
"file_chunks",
"files",
}
rows, err := conn.QueryContext(
ctx, "SELECT type, name, sql FROM sqlite_master")
if err != nil {
return fmt.Errorf("reading snapshot schema: %w", err)
}
defer func() { _ = rows.Close() }()
present := make(map[string]struct{})
for rows.Next() {
var objType, name string
var objSQL sql.NullString
err = rows.Scan(&objType, &name, &objSQL)
if err != nil {
return fmt.Errorf("reading snapshot schema: %w", err)
}
switch objType {
case "trigger", "view":
return fmt.Errorf(
"%w: unexpected %s %q", errUntrustedSnapshotSchema, objType, name)
case "table":
if isVirtualTableSQL(objSQL.String) {
return fmt.Errorf(
"%w: unexpected virtual table %q",
errUntrustedSnapshotSchema, name)
}
present[name] = struct{}{}
}
}
err = rows.Err()
if err != nil {
return fmt.Errorf("reading snapshot schema: %w", err)
}
for _, table := range expectedSnapshotTables {
if _, ok := present[table]; !ok {
return fmt.Errorf(
"%w: missing table %q", errUntrustedSnapshotSchema, table)
}
}
return nil
}
// isVirtualTableSQL reports whether a sqlite_master row's SQL defines a
// virtual table. Virtual tables are recorded with type 'table' but a
// "CREATE VIRTUAL TABLE" definition and can run module code, so they are
// refused alongside triggers and views.
func isVirtualTableSQL(createSQL string) bool {
return strings.HasPrefix(
strings.ToUpper(strings.TrimSpace(createSQL)), "CREATE VIRTUAL TABLE")
}
// NewTestDB creates an in-memory SQLite database for testing purposes.
// The database is automatically initialized with the schema and is ready
// for use. Each call creates a new independent database instance.
func NewTestDB() (*DB, error) {
return New(context.Background(), ":memory:")
}
// Close closes the database connection.
// It ensures all pending operations are completed before closing.
// Returns an error if the database connection cannot be closed properly.
func (db *DB) Close() error {
log.Debug("Closing database connection", "path", db.path)
if err := db.conn.Close(); err != nil {
err := db.conn.Close()
if err != nil {
log.Error("Failed to close database", "path", db.path, "error", err)
return fmt.Errorf("failed to close database: %w", err)
}
log.Debug("Database connection closed successfully", "path", db.path)
return nil
}
@@ -227,22 +408,25 @@ func (db *DB) BeginTx(
func (db *DB) ExecWithLog(
ctx context.Context,
query string,
args ...interface{},
args ...any,
) (sql.Result, error) {
LogSQL("Execute", query, args...)
return db.conn.ExecContext(ctx, query, args...)
}
// QueryRowWithLog executes a query that returns at most one row with SQL logging.
// This is useful for queries that modify data and return values (e.g., INSERT ... RETURNING).
// SQLite handles its own locking internally.
// The query and args parameters follow the same format as sql.DB.QueryRowContext.
// QueryRowWithLog executes a query that returns at most one row with SQL
// logging. This is useful for queries that modify data and return values
// (e.g., INSERT ... RETURNING). SQLite handles its own locking internally.
// The query and args parameters follow the same format as
// sql.DB.QueryRowContext.
func (db *DB) QueryRowWithLog(
ctx context.Context,
query string,
args ...interface{},
args ...any,
) *sql.Row {
LogSQL("QueryRow", query, args...)
return db.conn.QueryRowContext(ctx, query, args...)
}
@@ -302,7 +486,8 @@ func bootstrapMigrationsTable(ctx context.Context, db *sql.DB) error {
// the schema_migrations table via 000.sql, then iterates through remaining
// migration files in order.
func applyMigrations(ctx context.Context, db *sql.DB) error {
if err := bootstrapMigrationsTable(ctx, db); err != nil {
err := bootstrapMigrationsTable(ctx, db)
if err != nil {
return err
}
@@ -362,30 +547,26 @@ func applyMigrations(ctx context.Context, db *sql.DB) error {
return nil
}
// NewTestDB creates an in-memory SQLite database for testing purposes.
// The database is automatically initialized with the schema and is ready for use.
// Each call creates a new independent database instance.
func NewTestDB() (*DB, error) {
return New(context.Background(), ":memory:")
}
// repeatPlaceholder generates a string of ", ?" repeated n times for IN clause construction.
// For example, repeatPlaceholder(2) returns ", ?, ?".
// repeatPlaceholder generates a string of ", ?" repeated n times for IN
// clause construction. For example, repeatPlaceholder(2) returns ", ?, ?".
func repeatPlaceholder(n int) string {
if n <= 0 {
return ""
}
return strings.Repeat(", ?", n)
}
// LogSQL logs SQL queries and their arguments when debug mode is enabled.
// Debug mode is activated by setting the GODEBUG environment variable to include "vaultik".
// This is useful for troubleshooting database operations and understanding query patterns.
// Debug mode is activated by setting the GODEBUG environment variable to
// include "vaultik". This is useful for troubleshooting database operations
// and understanding query patterns.
//
// The operation parameter describes the type of SQL operation (e.g., "Execute", "Query").
// The query parameter is the SQL statement being executed.
// The args parameter contains the query arguments that will be interpolated.
func LogSQL(operation, query string, args ...interface{}) {
// The operation parameter describes the type of SQL operation (e.g.,
// "Execute", "Query"). The query parameter is the SQL statement being
// executed. The args parameter contains the query arguments that will be
// interpolated.
func LogSQL(operation, query string, args ...any) {
if strings.Contains(os.Getenv("GODEBUG"), "vaultik") {
log.Debug(
"SQL "+operation,
+91 -26
View File
@@ -1,3 +1,4 @@
//nolint:testpackage // exercises unexported migration internals
package database
import (
@@ -9,6 +10,8 @@ import (
)
func TestDatabase(t *testing.T) {
t.Parallel()
ctx := context.Background()
dbPath := filepath.Join(t.TempDir(), "test.db")
@@ -16,8 +19,10 @@ func TestDatabase(t *testing.T) {
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() {
if err := db.Close(); err != nil {
err := db.Close()
if err != nil {
t.Errorf("failed to close database: %v", err)
}
}()
@@ -37,7 +42,10 @@ func TestDatabase(t *testing.T) {
for _, table := range tables {
var name string
err := db.conn.QueryRow("SELECT name FROM sqlite_master WHERE type='table' AND name=?", table).Scan(&name)
err := db.conn.QueryRowContext(ctx,
"SELECT name FROM sqlite_master WHERE type='table' AND name=?", table,
).Scan(&name)
if err != nil {
t.Errorf("table %s does not exist: %v", table, err)
}
@@ -45,6 +53,8 @@ func TestDatabase(t *testing.T) {
}
func TestDatabaseInvalidPath(t *testing.T) {
t.Parallel()
ctx := context.Background()
// Test with invalid path
@@ -55,6 +65,8 @@ func TestDatabaseInvalidPath(t *testing.T) {
}
func TestDatabaseConcurrentAccess(t *testing.T) {
t.Parallel()
ctx := context.Background()
dbPath := filepath.Join(t.TempDir(), "test.db")
@@ -62,8 +74,10 @@ func TestDatabaseConcurrentAccess(t *testing.T) {
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() {
if err := db.Close(); err != nil {
err := db.Close()
if err != nil {
t.Errorf("failed to close database: %v", err)
}
}()
@@ -73,18 +87,20 @@ func TestDatabaseConcurrentAccess(t *testing.T) {
index int
err error
}
results := make(chan result, 10)
for i := 0; i < 10; i++ {
for i := range 10 {
go func(i int) {
_, err := db.ExecWithLog(ctx, "INSERT INTO chunks (chunk_hash, size) VALUES (?, ?)",
_, err := db.ExecWithLog(ctx,
"INSERT INTO chunks (chunk_hash, size) VALUES (?, ?)",
fmt.Sprintf("hash%d", i), i*1024)
results <- result{index: i, err: err}
}(i)
}
// Wait for all goroutines and check results
for i := 0; i < 10; i++ {
for range 10 {
r := <-results
if r.err != nil {
t.Fatalf("concurrent insert %d failed: %v", r.index, r.err)
@@ -93,16 +109,20 @@ func TestDatabaseConcurrentAccess(t *testing.T) {
// Verify all inserts succeeded
var count int
err = db.conn.QueryRowContext(ctx, "SELECT COUNT(*) FROM chunks").Scan(&count)
if err != nil {
t.Fatalf("failed to count chunks: %v", err)
}
if count != 10 {
t.Errorf("expected 10 chunks, got %d", count)
}
}
func TestParseMigrationVersion(t *testing.T) {
t.Parallel()
tests := []struct {
name string
filename string
@@ -112,8 +132,14 @@ func TestParseMigrationVersion(t *testing.T) {
{name: "valid 000.sql", filename: "000.sql", wantVer: 0, wantError: false},
{name: "valid 001.sql", filename: "001.sql", wantVer: 1, wantError: false},
{name: "valid 099.sql", filename: "099.sql", wantVer: 99, wantError: false},
{name: "valid with description", filename: "001_initial_schema.sql", wantVer: 1, wantError: false},
{name: "valid large version", filename: "123_big_migration.sql", wantVer: 123, wantError: false},
{
name: "valid with description", filename: "001_initial_schema.sql",
wantVer: 1, wantError: false,
},
{
name: "valid large version", filename: "123_big_migration.sql",
wantVer: 123, wantError: false,
},
{name: "invalid alpha version", filename: "abc.sql", wantVer: 0, wantError: true},
{name: "invalid mixed chars", filename: "12a.sql", wantVer: 0, wantError: true},
{name: "invalid no extension", filename: "schema.sql", wantVer: 0, wantError: true},
@@ -122,33 +148,46 @@ func TestParseMigrationVersion(t *testing.T) {
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got, err := ParseMigrationVersion(tc.filename)
if tc.wantError {
if err == nil {
t.Errorf("ParseMigrationVersion(%q) = %d, nil; want error", tc.filename, got)
t.Errorf("ParseMigrationVersion(%q) = %d, nil; want error",
tc.filename, got)
}
return
}
if err != nil {
t.Errorf("ParseMigrationVersion(%q) unexpected error: %v", tc.filename, err)
t.Errorf("ParseMigrationVersion(%q) unexpected error: %v",
tc.filename, err)
return
}
if got != tc.wantVer {
t.Errorf("ParseMigrationVersion(%q) = %d; want %d", tc.filename, got, tc.wantVer)
t.Errorf("ParseMigrationVersion(%q) = %d; want %d",
tc.filename, got, tc.wantVer)
}
})
}
}
func TestApplyMigrations_Idempotent(t *testing.T) {
t.Parallel()
ctx := context.Background()
conn, err := sql.Open("sqlite", ":memory:?_foreign_keys=ON")
if err != nil {
t.Fatalf("failed to open database: %v", err)
}
defer func() {
if err := conn.Close(); err != nil {
err := conn.Close()
if err != nil {
t.Errorf("failed to close database: %v", err)
}
}()
@@ -157,41 +196,56 @@ func TestApplyMigrations_Idempotent(t *testing.T) {
conn.SetMaxIdleConns(1)
// First run: apply all migrations.
if err := applyMigrations(ctx, conn); err != nil {
err = applyMigrations(ctx, conn)
if err != nil {
t.Fatalf("first applyMigrations failed: %v", err)
}
// Count rows in schema_migrations after first run.
var countBefore int
if err := conn.QueryRowContext(ctx, "SELECT COUNT(*) FROM schema_migrations").Scan(&countBefore); err != nil {
err = conn.QueryRowContext(ctx,
"SELECT COUNT(*) FROM schema_migrations",
).Scan(&countBefore)
if err != nil {
t.Fatalf("failed to count schema_migrations after first run: %v", err)
}
// Second run: must be a no-op.
if err := applyMigrations(ctx, conn); err != nil {
err = applyMigrations(ctx, conn)
if err != nil {
t.Fatalf("second applyMigrations failed: %v", err)
}
// Count rows in schema_migrations after second run — must be unchanged.
var countAfter int
if err := conn.QueryRowContext(ctx, "SELECT COUNT(*) FROM schema_migrations").Scan(&countAfter); err != nil {
err = conn.QueryRowContext(ctx,
"SELECT COUNT(*) FROM schema_migrations",
).Scan(&countAfter)
if err != nil {
t.Fatalf("failed to count schema_migrations after second run: %v", err)
}
if countBefore != countAfter {
t.Errorf("schema_migrations row count changed: before=%d, after=%d", countBefore, countAfter)
t.Errorf("schema_migrations row count changed: before=%d, after=%d",
countBefore, countAfter)
}
}
func TestBootstrapMigrationsTable_FreshDatabase(t *testing.T) {
t.Parallel()
ctx := context.Background()
conn, err := sql.Open("sqlite", ":memory:?_foreign_keys=ON")
if err != nil {
t.Fatalf("failed to open database: %v", err)
}
defer func() {
if err := conn.Close(); err != nil {
err := conn.Close()
if err != nil {
t.Errorf("failed to close database: %v", err)
}
}()
@@ -201,38 +255,49 @@ func TestBootstrapMigrationsTable_FreshDatabase(t *testing.T) {
// Verify schema_migrations does NOT exist yet.
var tableBefore int
if err := conn.QueryRowContext(ctx,
err = conn.QueryRowContext(ctx,
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='schema_migrations'",
).Scan(&tableBefore); err != nil {
).Scan(&tableBefore)
if err != nil {
t.Fatalf("failed to check for table before bootstrap: %v", err)
}
if tableBefore != 0 {
t.Fatal("schema_migrations table should not exist before bootstrap")
}
// Run bootstrap.
if err := bootstrapMigrationsTable(ctx, conn); err != nil {
err = bootstrapMigrationsTable(ctx, conn)
if err != nil {
t.Fatalf("bootstrapMigrationsTable failed: %v", err)
}
// Verify schema_migrations now exists.
var tableAfter int
if err := conn.QueryRowContext(ctx,
err = conn.QueryRowContext(ctx,
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='schema_migrations'",
).Scan(&tableAfter); err != nil {
).Scan(&tableAfter)
if err != nil {
t.Fatalf("failed to check for table after bootstrap: %v", err)
}
if tableAfter != 1 {
t.Fatalf("schema_migrations table should exist after bootstrap, got count=%d", tableAfter)
t.Fatalf("schema_migrations table should exist after bootstrap, got count=%d",
tableAfter)
}
// Verify version 0 row exists.
var version int
if err := conn.QueryRowContext(ctx,
err = conn.QueryRowContext(ctx,
"SELECT version FROM schema_migrations WHERE version = 0",
).Scan(&version); err != nil {
).Scan(&version)
if err != nil {
t.Fatalf("version 0 row not found in schema_migrations: %v", err)
}
if version != 0 {
t.Errorf("expected version 0, got %d", version)
}
+2 -10
View File
@@ -1,20 +1,12 @@
package database
import (
"database/sql"
"fmt"
"os"
)
// Fatal prints an error message to stderr and exits with status 1
func Fatal(format string, args ...interface{}) {
// Fatalf prints an error message to stderr and exits with status 1
func Fatalf(format string, args ...any) {
fmt.Fprintf(os.Stderr, "FATAL: "+format+"\n", args...)
os.Exit(1)
}
// CloseRows closes rows and exits on error
func CloseRows(rows *sql.Rows) {
if err := rows.Close(); err != nil {
Fatal("failed to close rows: %v", err)
}
}
+125 -51
View File
@@ -4,19 +4,26 @@ import (
"context"
"database/sql"
"fmt"
"strings"
"sneak.berlin/go/vaultik/internal/types"
)
// FileChunkRepository provides access to the file_chunks table, which maps
// files to their ordered constituent chunks.
type FileChunkRepository struct {
db *DB
}
// NewFileChunkRepository creates a FileChunkRepository backed by db.
func NewFileChunkRepository(db *DB) *FileChunkRepository {
return &FileChunkRepository{db: db}
}
func (r *FileChunkRepository) Create(ctx context.Context, tx *sql.Tx, fc *FileChunk) error {
// Create inserts a file_chunks row (idempotently), using tx when non-nil.
func (r *FileChunkRepository) Create(
ctx context.Context, tx *sql.Tx, fc *FileChunk,
) error {
query := `
INSERT INTO file_chunks (file_id, idx, chunk_hash)
VALUES (?, ?, ?)
@@ -27,7 +34,8 @@ func (r *FileChunkRepository) Create(ctx context.Context, tx *sql.Tx, fc *FileCh
if tx != nil {
_, err = tx.ExecContext(ctx, query, fc.FileID.String(), fc.Idx, fc.ChunkHash.String())
} else {
_, err = r.db.ExecWithLog(ctx, query, fc.FileID.String(), fc.Idx, fc.ChunkHash.String())
_, err = r.db.ExecWithLog(ctx, query,
fc.FileID.String(), fc.Idx, fc.ChunkHash.String())
}
if err != nil {
@@ -37,7 +45,10 @@ func (r *FileChunkRepository) Create(ctx context.Context, tx *sql.Tx, fc *FileCh
return nil
}
func (r *FileChunkRepository) GetByPath(ctx context.Context, path string) ([]*FileChunk, error) {
// GetByPath returns the ordered chunks of the file at the given path.
func (r *FileChunkRepository) GetByPath(
ctx context.Context, path string,
) ([]*FileChunk, error) {
query := `
SELECT fc.file_id, fc.idx, fc.chunk_hash
FROM file_chunks fc
@@ -50,13 +61,21 @@ func (r *FileChunkRepository) GetByPath(ctx context.Context, path string) ([]*Fi
if err != nil {
return nil, fmt.Errorf("querying file chunks: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanFileChunks(rows)
}
// GetByFileID retrieves file chunks by file ID
func (r *FileChunkRepository) GetByFileID(ctx context.Context, fileID types.FileID) ([]*FileChunk, error) {
func (r *FileChunkRepository) GetByFileID(
ctx context.Context, fileID types.FileID,
) ([]*FileChunk, error) {
query := `
SELECT file_id, idx, chunk_hash
FROM file_chunks
@@ -68,13 +87,21 @@ func (r *FileChunkRepository) GetByFileID(ctx context.Context, fileID types.File
if err != nil {
return nil, fmt.Errorf("querying file chunks: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanFileChunks(rows)
}
// GetByPathTx retrieves file chunks within a transaction
func (r *FileChunkRepository) GetByPathTx(ctx context.Context, tx *sql.Tx, path string) ([]*FileChunk, error) {
func (r *FileChunkRepository) GetByPathTx(
ctx context.Context, tx *sql.Tx, path string,
) ([]*FileChunk, error) {
query := `
SELECT fc.file_id, fc.idx, fc.chunk_hash
FROM file_chunks fc
@@ -84,40 +111,33 @@ func (r *FileChunkRepository) GetByPathTx(ctx context.Context, tx *sql.Tx, path
`
LogSQL("GetByPathTx", query, path)
rows, err := tx.QueryContext(ctx, query, path)
if err != nil {
return nil, fmt.Errorf("querying file chunks: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
fileChunks, err := r.scanFileChunks(rows)
LogSQL("GetByPathTx", "Complete", path, "count", len(fileChunks))
return fileChunks, err
}
// scanFileChunks is a helper that scans file chunk rows
func (r *FileChunkRepository) scanFileChunks(rows *sql.Rows) ([]*FileChunk, error) {
var fileChunks []*FileChunk
for rows.Next() {
var fc FileChunk
var fileIDStr, chunkHashStr string
err := rows.Scan(&fileIDStr, &fc.Idx, &chunkHashStr)
if err != nil {
return nil, fmt.Errorf("scanning file chunk: %w", err)
}
fc.FileID, err = types.ParseFileID(fileIDStr)
if err != nil {
return nil, fmt.Errorf("parsing file ID: %w", err)
}
fc.ChunkHash = types.ChunkHash(chunkHashStr)
fileChunks = append(fileChunks, &fc)
}
return fileChunks, rows.Err()
}
func (r *FileChunkRepository) DeleteByPath(ctx context.Context, tx *sql.Tx, path string) error {
query := `DELETE FROM file_chunks WHERE file_id = (SELECT id FROM files WHERE path = ?)`
// DeleteByPath deletes all file_chunks rows for the file at the given path.
func (r *FileChunkRepository) DeleteByPath(
ctx context.Context, tx *sql.Tx, path string,
) error {
query := `
DELETE FROM file_chunks
WHERE file_id = (SELECT id FROM files WHERE path = ?)
`
var err error
if tx != nil {
@@ -134,7 +154,9 @@ func (r *FileChunkRepository) DeleteByPath(ctx context.Context, tx *sql.Tx, path
}
// DeleteByFileID deletes all chunks for a file by its UUID
func (r *FileChunkRepository) DeleteByFileID(ctx context.Context, tx *sql.Tx, fileID types.FileID) error {
func (r *FileChunkRepository) DeleteByFileID(
ctx context.Context, tx *sql.Tx, fileID types.FileID,
) error {
query := `DELETE FROM file_chunks WHERE file_id = ?`
var err error
@@ -152,7 +174,11 @@ func (r *FileChunkRepository) DeleteByFileID(ctx context.Context, tx *sql.Tx, fi
}
// DeleteByFileIDs deletes all chunks for multiple files in a single statement.
func (r *FileChunkRepository) DeleteByFileIDs(ctx context.Context, tx *sql.Tx, fileIDs []types.FileID) error {
//
//nolint:dupl // symmetric implementation for a parallel association table
func (r *FileChunkRepository) DeleteByFileIDs(
ctx context.Context, tx *sql.Tx, fileIDs []types.FileID,
) error {
if len(fileIDs) == 0 {
return nil
}
@@ -161,14 +187,15 @@ func (r *FileChunkRepository) DeleteByFileIDs(ctx context.Context, tx *sql.Tx, f
const batchSize = 500
for i := 0; i < len(fileIDs); i += batchSize {
end := i + batchSize
if end > len(fileIDs) {
end = len(fileIDs)
}
end := min(i+batchSize, len(fileIDs))
batch := fileIDs[i:end]
query := "DELETE FROM file_chunks WHERE file_id IN (?" + repeatPlaceholder(len(batch)-1) + ")"
args := make([]interface{}, len(batch))
//nolint:gosec // G202: concatenates constant SQL and "?" placeholders only
query := "DELETE FROM file_chunks WHERE file_id IN (?" +
repeatPlaceholder(len(batch)-1) + ")"
args := make([]any, len(batch))
for j, id := range batch {
args[j] = id.String()
}
@@ -179,6 +206,7 @@ func (r *FileChunkRepository) DeleteByFileIDs(ctx context.Context, tx *sql.Tx, f
} else {
_, err = r.db.ExecWithLog(ctx, query, args...)
}
if err != nil {
return fmt.Errorf("batch deleting file_chunks: %w", err)
}
@@ -189,32 +217,44 @@ func (r *FileChunkRepository) DeleteByFileIDs(ctx context.Context, tx *sql.Tx, f
// CreateBatch inserts multiple file_chunks in a single statement for efficiency.
// Batches are automatically split to stay within SQLite's variable limit.
func (r *FileChunkRepository) CreateBatch(ctx context.Context, tx *sql.Tx, fcs []FileChunk) error {
func (r *FileChunkRepository) CreateBatch(
ctx context.Context, tx *sql.Tx, fcs []FileChunk,
) error {
if len(fcs) == 0 {
return nil
}
// SQLite has a limit on variables (typically 999 or 32766).
// Each FileChunk has 3 values, so batch at 300 to be safe.
// Each file_chunks row binds this many SQL variables.
const fileChunkCols = 3
// SQLite has a limit on variables (typically 999 or 32766), so batch
// at 300 rows to be safe.
const batchSize = 300
for i := 0; i < len(fcs); i += batchSize {
end := i + batchSize
if end > len(fcs) {
end = len(fcs)
}
end := min(i+batchSize, len(fcs))
batch := fcs[i:end]
// Build the query with multiple value sets
query := "INSERT INTO file_chunks (file_id, idx, chunk_hash) VALUES "
args := make([]interface{}, 0, len(batch)*3)
args := make([]any, 0, len(batch)*fileChunkCols)
var querySb211 strings.Builder
for j, fc := range batch {
if j > 0 {
query += ", "
querySb211.WriteString(", ")
}
query += "(?, ?, ?)"
querySb211.WriteString("(?, ?, ?)")
args = append(args, fc.FileID.String(), fc.Idx, fc.ChunkHash.String())
}
query += querySb211.String() //nolint:gosec // G202: appends "?" placeholders only
query += " ON CONFLICT(file_id, idx) DO NOTHING"
var err error
@@ -223,6 +263,7 @@ func (r *FileChunkRepository) CreateBatch(ctx context.Context, tx *sql.Tx, fcs [
} else {
_, err = r.db.ExecWithLog(ctx, query, args...)
}
if err != nil {
return fmt.Errorf("batch inserting file_chunks: %w", err)
}
@@ -232,17 +273,50 @@ func (r *FileChunkRepository) CreateBatch(ctx context.Context, tx *sql.Tx, fcs [
}
// GetByFile is an alias for GetByPath for compatibility
func (r *FileChunkRepository) GetByFile(ctx context.Context, path string) ([]*FileChunk, error) {
func (r *FileChunkRepository) GetByFile(
ctx context.Context, path string,
) ([]*FileChunk, error) {
LogSQL("GetByFile", "Starting", path)
result, err := r.GetByPath(ctx, path)
LogSQL("GetByFile", "Complete", path, "count", len(result))
return result, err
}
// GetByFileTx retrieves file chunks within a transaction
func (r *FileChunkRepository) GetByFileTx(ctx context.Context, tx *sql.Tx, path string) ([]*FileChunk, error) {
func (r *FileChunkRepository) GetByFileTx(
ctx context.Context, tx *sql.Tx, path string,
) ([]*FileChunk, error) {
LogSQL("GetByFileTx", "Starting", path)
result, err := r.GetByPathTx(ctx, tx, path)
LogSQL("GetByFileTx", "Complete", path, "count", len(result))
return result, err
}
// scanFileChunks is a helper that scans file chunk rows
func (r *FileChunkRepository) scanFileChunks(rows *sql.Rows) ([]*FileChunk, error) {
var fileChunks []*FileChunk
for rows.Next() {
var (
fc FileChunk
fileIDStr, chunkHashStr string
)
err := rows.Scan(&fileIDStr, &fc.Idx, &chunkHashStr)
if err != nil {
return nil, fmt.Errorf("scanning file chunk: %w", err)
}
fc.FileID, err = types.ParseFileID(fileIDStr)
if err != nil {
return nil, fmt.Errorf("parsing file ID: %w", err)
}
fc.ChunkHash = types.ChunkHash(chunkHashStr)
fileChunks = append(fileChunks, &fc)
}
return fileChunks, rows.Err()
}
+78 -45
View File
@@ -1,4 +1,4 @@
package database
package database_test
import (
"context"
@@ -6,21 +6,25 @@ import (
"testing"
"time"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
func TestFileChunkRepository(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := NewFileChunkRepository(db)
fileRepo := NewFileRepository(db)
repo := database.NewFileChunkRepository(db)
fileRepo := database.NewFileRepository(db)
repos := database.NewRepositories(db)
// Create test file first
testTime := time.Now().Truncate(time.Second)
file := &File{
Path: "/test/file.txt",
file := &database.File{
Path: testFileTxt,
MTime: testTime,
Size: 3072,
Mode: 0644,
@@ -28,63 +32,51 @@ func TestFileChunkRepository(t *testing.T) {
GID: 1000,
LinkTarget: "",
}
err := fileRepo.Create(ctx, nil, file)
if err != nil {
t.Fatalf("failed to create file: %v", err)
}
// Create chunks first
chunks := []types.ChunkHash{"chunk1", "chunk2", "chunk3"}
chunkRepo := NewChunkRepository(db)
for _, chunkHash := range chunks {
chunk := &Chunk{
ChunkHash: chunkHash,
Size: 1024,
}
err = chunkRepo.Create(ctx, nil, chunk)
if err != nil {
t.Fatalf("failed to create chunk %s: %v", chunkHash, err)
}
}
mustCreateFile(t, fileRepo, file)
mustCreateChunks(t, repos, chunk1Hash, chunk2Hash, chunk3Hash)
// Test Create
fc1 := &FileChunk{
fc1 := &database.FileChunk{
FileID: file.ID,
Idx: 0,
ChunkHash: types.ChunkHash("chunk1"),
ChunkHash: types.ChunkHash(chunk1Hash),
}
err = repo.Create(ctx, nil, fc1)
err := repo.Create(ctx, nil, fc1)
if err != nil {
t.Fatalf("failed to create file chunk: %v", err)
}
// Add more chunks for the same file
fc2 := &FileChunk{
fc2 := &database.FileChunk{
FileID: file.ID,
Idx: 1,
ChunkHash: types.ChunkHash("chunk2"),
ChunkHash: types.ChunkHash(chunk2Hash),
}
err = repo.Create(ctx, nil, fc2)
if err != nil {
t.Fatalf("failed to create second file chunk: %v", err)
}
fc3 := &FileChunk{
fc3 := &database.FileChunk{
FileID: file.ID,
Idx: 2,
ChunkHash: types.ChunkHash("chunk3"),
ChunkHash: types.ChunkHash(chunk3Hash),
}
err = repo.Create(ctx, nil, fc3)
if err != nil {
t.Fatalf("failed to create third file chunk: %v", err)
}
// Test GetByFile
fileChunks, err := repo.GetByFile(ctx, "/test/file.txt")
fileChunks, err := repo.GetByFile(ctx, testFileTxt)
if err != nil {
t.Fatalf("failed to get file chunks: %v", err)
}
if len(fileChunks) != 3 {
t.Errorf("expected 3 chunks, got %d", len(fileChunks))
}
@@ -101,6 +93,41 @@ func TestFileChunkRepository(t *testing.T) {
if err != nil {
t.Fatalf("failed to create duplicate file chunk: %v", err)
}
}
func TestFileChunkRepositoryDeleteByFileID(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileChunkRepository(db)
fileRepo := database.NewFileRepository(db)
repos := database.NewRepositories(db)
file := &database.File{
Path: testFileTxt,
MTime: time.Now().Truncate(time.Second),
Size: 1024,
Mode: 0644,
UID: 1000,
GID: 1000,
}
mustCreateFile(t, fileRepo, file)
mustCreateChunks(t, repos, chunk1Hash)
fc := &database.FileChunk{
FileID: file.ID,
Idx: 0,
ChunkHash: types.ChunkHash(chunk1Hash),
}
err := repo.Create(ctx, nil, fc)
if err != nil {
t.Fatalf("failed to create file chunk: %v", err)
}
// Test DeleteByFileID
err = repo.DeleteByFileID(ctx, nil, file.ID)
@@ -108,30 +135,33 @@ func TestFileChunkRepository(t *testing.T) {
t.Fatalf("failed to delete file chunks: %v", err)
}
fileChunks, err = repo.GetByFileID(ctx, file.ID)
fileChunks, err := repo.GetByFileID(ctx, file.ID)
if err != nil {
t.Fatalf("failed to get deleted file chunks: %v", err)
}
if len(fileChunks) != 0 {
t.Errorf("expected 0 chunks after delete, got %d", len(fileChunks))
}
}
func TestFileChunkRepositoryMultipleFiles(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := NewFileChunkRepository(db)
fileRepo := NewFileRepository(db)
repo := database.NewFileChunkRepository(db)
fileRepo := database.NewFileRepository(db)
// Create test files
testTime := time.Now().Truncate(time.Second)
filePaths := []string{"/file1.txt", "/file2.txt", "/file3.txt"}
files := make([]*File, len(filePaths))
filePaths := []string{testFilePath1, testFilePath2, "/file3.txt"}
files := make([]*database.File, len(filePaths))
for i, path := range filePaths {
file := &File{
file := &database.File{
Path: types.FilePath(path),
MTime: testTime,
Size: 2048,
@@ -140,22 +170,23 @@ func TestFileChunkRepositoryMultipleFiles(t *testing.T) {
GID: 1000,
LinkTarget: "",
}
err := fileRepo.Create(ctx, nil, file)
if err != nil {
t.Fatalf("failed to create file %s: %v", path, err)
}
mustCreateFile(t, fileRepo, file)
files[i] = file
}
// Create all chunks first
chunkRepo := NewChunkRepository(db)
chunkRepo := database.NewChunkRepository(db)
for i := range files {
for j := 0; j < 2; j++ {
for j := range 2 {
chunkHash := types.ChunkHash(fmt.Sprintf("file%d_chunk%d", i, j))
chunk := &Chunk{
chunk := &database.Chunk{
ChunkHash: chunkHash,
Size: 1024,
}
err := chunkRepo.Create(ctx, nil, chunk)
if err != nil {
t.Fatalf("failed to create chunk %s: %v", chunkHash, err)
@@ -165,12 +196,13 @@ func TestFileChunkRepositoryMultipleFiles(t *testing.T) {
// Create chunks for multiple files
for i, file := range files {
for j := 0; j < 2; j++ {
fc := &FileChunk{
for j := range 2 {
fc := &database.FileChunk{
FileID: file.ID,
Idx: j,
ChunkHash: types.ChunkHash(fmt.Sprintf("file%d_chunk%d", i, j)),
}
err := repo.Create(ctx, nil, fc)
if err != nil {
t.Fatalf("failed to create file chunk: %v", err)
@@ -184,6 +216,7 @@ func TestFileChunkRepositoryMultipleFiles(t *testing.T) {
if err != nil {
t.Fatalf("failed to get chunks for file %d: %v", i, err)
}
if len(chunks) != 2 {
t.Errorf("expected 2 chunks for file %d, got %d", i, len(chunks))
}
+164 -99
View File
@@ -3,21 +3,29 @@ package database
import (
"context"
"database/sql"
"errors"
"fmt"
"strings"
"time"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/types"
)
// FileRepository provides access to the files table, which stores file
// metadata (path, times, permissions, ownership, symlink targets).
type FileRepository struct {
db *DB
}
// NewFileRepository creates a FileRepository backed by db.
func NewFileRepository(db *DB) *FileRepository {
return &FileRepository{db: db}
}
// Create inserts or updates a file row (upsert on path), using tx when
// non-nil. The file's ID is generated when zero and updated from the
// database's RETURNING clause.
func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) error {
// Generate UUID if not provided
if file.ID.IsZero() {
@@ -38,13 +46,25 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
RETURNING id
`
var idStr string
var err error
var (
idStr string
err error
)
if tx != nil {
LogSQL("Execute", query, file.ID.String(), file.Path.String(), file.SourcePath.String(), file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID, file.LinkTarget.String())
err = tx.QueryRowContext(ctx, query, file.ID.String(), file.Path.String(), file.SourcePath.String(), file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID, file.LinkTarget.String()).Scan(&idStr)
LogSQL("Execute", query,
file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String())
err = tx.QueryRowContext(ctx, query,
file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String()).Scan(&idStr)
} else {
err = r.db.QueryRowWithLog(ctx, query, file.ID.String(), file.Path.String(), file.SourcePath.String(), file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID, file.LinkTarget.String()).Scan(&idStr)
err = r.db.QueryRowWithLog(ctx, query,
file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String()).Scan(&idStr)
}
if err != nil {
@@ -60,6 +80,8 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
return nil
}
// GetByPath returns the file at the given path, or nil if the path is not
// in the index.
func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, error) {
query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
@@ -68,9 +90,10 @@ func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, err
`
file, err := r.scanFile(r.db.conn.QueryRowContext(ctx, query, path))
if err == sql.ErrNoRows {
return nil, nil
if errors.Is(err, sql.ErrNoRows) {
return nil, nil //nolint:nilnil // nil,nil signals not-found; callers check nil
}
if err != nil {
return nil, fmt.Errorf("querying file: %w", err)
}
@@ -87,9 +110,10 @@ func (r *FileRepository) GetByID(ctx context.Context, id types.FileID) (*File, e
`
file, err := r.scanFile(r.db.conn.QueryRowContext(ctx, query, id.String()))
if err == sql.ErrNoRows {
return nil, nil
if errors.Is(err, sql.ErrNoRows) {
return nil, nil //nolint:nilnil // nil,nil signals not-found; callers check nil
}
if err != nil {
return nil, fmt.Errorf("querying file: %w", err)
}
@@ -97,7 +121,11 @@ func (r *FileRepository) GetByID(ctx context.Context, id types.FileID) (*File, e
return file, nil
}
func (r *FileRepository) GetByPathTx(ctx context.Context, tx *sql.Tx, path string) (*File, error) {
// GetByPathTx returns the file at the given path within a transaction, or
// nil if the path is not in the index.
func (r *FileRepository) GetByPathTx(
ctx context.Context, tx *sql.Tx, path string,
) (*File, error) {
query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
FROM files
@@ -108,9 +136,10 @@ func (r *FileRepository) GetByPathTx(ctx context.Context, tx *sql.Tx, path strin
file, err := r.scanFile(tx.QueryRowContext(ctx, query, path))
LogSQL("GetByPathTx Scan complete", query, path)
if err == sql.ErrNoRows {
return nil, nil
if errors.Is(err, sql.ErrNoRows) {
return nil, nil //nolint:nilnil // nil,nil signals not-found; callers check nil
}
if err != nil {
return nil, fmt.Errorf("querying file: %w", err)
}
@@ -118,79 +147,16 @@ func (r *FileRepository) GetByPathTx(ctx context.Context, tx *sql.Tx, path strin
return file, nil
}
// scanFile is a helper that scans a single file row
func (r *FileRepository) scanFile(row *sql.Row) (*File, error) {
var file File
var idStr, pathStr, sourcePathStr string
var mtimeUnix int64
var linkTarget sql.NullString
err := row.Scan(
&idStr,
&pathStr,
&sourcePathStr,
&mtimeUnix,
&file.Size,
&file.Mode,
&file.UID,
&file.GID,
&linkTarget,
)
if err != nil {
return nil, err
}
file.ID, err = types.ParseFileID(idStr)
if err != nil {
return nil, fmt.Errorf("parsing file ID: %w", err)
}
file.Path = types.FilePath(pathStr)
file.SourcePath = types.SourcePath(sourcePathStr)
file.MTime = time.Unix(mtimeUnix, 0).UTC()
if linkTarget.Valid {
file.LinkTarget = types.FilePath(linkTarget.String)
}
return &file, nil
// fileRowScanner abstracts *sql.Row and *sql.Rows for scanning a file row.
type fileRowScanner interface {
Scan(dest ...any) error
}
// scanFileRows is a helper that scans a file row from rows iterator
func (r *FileRepository) scanFileRows(rows *sql.Rows) (*File, error) {
var file File
var idStr, pathStr, sourcePathStr string
var mtimeUnix int64
var linkTarget sql.NullString
err := rows.Scan(
&idStr,
&pathStr,
&sourcePathStr,
&mtimeUnix,
&file.Size,
&file.Mode,
&file.UID,
&file.GID,
&linkTarget,
)
if err != nil {
return nil, err
}
file.ID, err = types.ParseFileID(idStr)
if err != nil {
return nil, fmt.Errorf("parsing file ID: %w", err)
}
file.Path = types.FilePath(pathStr)
file.SourcePath = types.SourcePath(sourcePathStr)
file.MTime = time.Unix(mtimeUnix, 0).UTC()
if linkTarget.Valid {
file.LinkTarget = types.FilePath(linkTarget.String)
}
return &file, nil
}
func (r *FileRepository) ListModifiedSince(ctx context.Context, since time.Time) ([]*File, error) {
// ListModifiedSince returns all files whose recorded mtime is at or after
// since, ordered by path.
func (r *FileRepository) ListModifiedSince(
ctx context.Context, since time.Time,
) ([]*File, error) {
query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
FROM files
@@ -202,20 +168,29 @@ func (r *FileRepository) ListModifiedSince(ctx context.Context, since time.Time)
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var files []*File
for rows.Next() {
file, err := r.scanFileRows(rows)
if err != nil {
return nil, fmt.Errorf("scanning file: %w", err)
}
files = append(files, file)
}
return files, rows.Err()
}
// Delete removes the file row at the given path, using tx when non-nil.
func (r *FileRepository) Delete(ctx context.Context, tx *sql.Tx, path string) error {
query := `DELETE FROM files WHERE path = ?`
@@ -234,7 +209,9 @@ func (r *FileRepository) Delete(ctx context.Context, tx *sql.Tx, path string) er
}
// DeleteByID deletes a file by its UUID
func (r *FileRepository) DeleteByID(ctx context.Context, tx *sql.Tx, id types.FileID) error {
func (r *FileRepository) DeleteByID(
ctx context.Context, tx *sql.Tx, id types.FileID,
) error {
query := `DELETE FROM files WHERE id = ?`
var err error
@@ -251,7 +228,11 @@ func (r *FileRepository) DeleteByID(ctx context.Context, tx *sql.Tx, id types.Fi
return nil
}
func (r *FileRepository) ListByPrefix(ctx context.Context, prefix string) ([]*File, error) {
// ListByPrefix returns all files whose path starts with prefix, ordered by
// path.
func (r *FileRepository) ListByPrefix(
ctx context.Context, prefix string,
) ([]*File, error) {
query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
FROM files
@@ -263,14 +244,22 @@ func (r *FileRepository) ListByPrefix(ctx context.Context, prefix string) ([]*Fi
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var files []*File
for rows.Next() {
file, err := r.scanFileRows(rows)
if err != nil {
return nil, fmt.Errorf("scanning file: %w", err)
}
files = append(files, file)
}
@@ -289,14 +278,22 @@ func (r *FileRepository) ListAll(ctx context.Context) ([]*File, error) {
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
defer CloseRows(rows)
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var files []*File
for rows.Next() {
file, err := r.scanFileRows(rows)
if err != nil {
return nil, fmt.Errorf("scanning file: %w", err)
}
files = append(files, file)
}
@@ -305,30 +302,47 @@ func (r *FileRepository) ListAll(ctx context.Context) ([]*File, error) {
// CreateBatch inserts or updates multiple files in a single statement for efficiency.
// File IDs must be pre-generated before calling this method.
func (r *FileRepository) CreateBatch(ctx context.Context, tx *sql.Tx, files []*File) error {
func (r *FileRepository) CreateBatch(
ctx context.Context, tx *sql.Tx, files []*File,
) error {
if len(files) == 0 {
return nil
}
// Each File has 9 values, so batch at 100 to be safe with SQLite's variable limit
// Each files row binds this many SQL variables.
const fileCols = 9
// Batch at 100 rows to be safe with SQLite's variable limit.
const batchSize = 100
for i := 0; i < len(files); i += batchSize {
end := i + batchSize
if end > len(files) {
end = len(files)
}
end := min(i+batchSize, len(files))
batch := files[i:end]
query := `INSERT INTO files (id, path, source_path, mtime, size, mode, uid, gid, link_target) VALUES `
args := make([]interface{}, 0, len(batch)*9)
query := `INSERT INTO files
(id, path, source_path, mtime, size, mode, uid, gid, link_target)
VALUES `
args := make([]any, 0, len(batch)*fileCols)
var querySb325 strings.Builder
for j, f := range batch {
if j > 0 {
query += ", "
querySb325.WriteString(", ")
}
query += "(?, ?, ?, ?, ?, ?, ?, ?, ?)"
args = append(args, f.ID.String(), f.Path.String(), f.SourcePath.String(), f.MTime.Unix(), f.Size, f.Mode, f.UID, f.GID, f.LinkTarget.String())
querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?)")
args = append(args,
f.ID.String(), f.Path.String(), f.SourcePath.String(),
f.MTime.Unix(), f.Size, f.Mode, f.UID, f.GID,
f.LinkTarget.String())
}
query += querySb325.String() //nolint:gosec // G202: appends "?" placeholders only
query += ` ON CONFLICT(path) DO UPDATE SET
source_path = excluded.source_path,
mtime = excluded.mtime,
@@ -344,6 +358,7 @@ func (r *FileRepository) CreateBatch(ctx context.Context, tx *sql.Tx, files []*F
} else {
_, err = r.db.ExecWithLog(ctx, query, args...)
}
if err != nil {
return fmt.Errorf("batch inserting files: %w", err)
}
@@ -374,3 +389,53 @@ func (r *FileRepository) DeleteOrphaned(ctx context.Context) error {
return nil
}
// scanFile is a helper that scans a single file row
func (r *FileRepository) scanFile(row *sql.Row) (*File, error) {
return r.scanFileFrom(row)
}
// scanFileRows is a helper that scans a file row from rows iterator
func (r *FileRepository) scanFileRows(rows *sql.Rows) (*File, error) {
return r.scanFileFrom(rows)
}
// scanFileFrom scans one file row from any row scanner.
func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
var (
file File
idStr, pathStr, sourcePathStr string
mtimeUnix int64
linkTarget sql.NullString
)
err := row.Scan(
&idStr,
&pathStr,
&sourcePathStr,
&mtimeUnix,
&file.Size,
&file.Mode,
&file.UID,
&file.GID,
&linkTarget,
)
if err != nil {
return nil, err
}
file.ID, err = types.ParseFileID(idStr)
if err != nil {
return nil, fmt.Errorf("parsing file ID: %w", err)
}
file.Path = types.FilePath(pathStr)
file.SourcePath = types.SourcePath(sourcePathStr)
file.MTime = time.Unix(mtimeUnix, 0).UTC()
if linkTarget.Valid {
file.LinkTarget = types.FilePath(linkTarget.String)
}
return &file, nil
}
+65 -35
View File
@@ -1,43 +1,32 @@
package database
package database_test
import (
"context"
"database/sql"
"fmt"
"errors"
"os"
"path/filepath"
"testing"
"time"
"sneak.berlin/go/vaultik/internal/database"
)
func setupTestDB(t *testing.T) (*DB, func()) {
ctx := context.Background()
dbPath := filepath.Join(t.TempDir(), "test.db")
db, err := New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
cleanup := func() {
if err := db.Close(); err != nil {
t.Errorf("failed to close database: %v", err)
}
}
return db, cleanup
}
// errTestRollback is the sentinel returned from transaction bodies to
// force a rollback in tests.
var errTestRollback = errors.New("test rollback")
func TestFileRepository(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := NewFileRepository(db)
repo := database.NewFileRepository(db)
// Test Create
file := &File{
Path: "/test/file.txt",
file := &database.File{
Path: testFileTxt,
MTime: time.Now().Truncate(time.Second),
Size: 1024,
Mode: 0644,
@@ -56,18 +45,23 @@ func TestFileRepository(t *testing.T) {
if err != nil {
t.Fatalf("failed to get file: %v", err)
}
if retrieved == nil {
t.Fatal("expected file, got nil")
}
if retrieved.Path != file.Path {
t.Errorf("path mismatch: got %s, want %s", retrieved.Path, file.Path)
}
if !retrieved.MTime.Equal(file.MTime) {
t.Errorf("mtime mismatch: got %v, want %v", retrieved.MTime, file.MTime)
}
if retrieved.Size != file.Size {
t.Errorf("size mismatch: got %d, want %d", retrieved.Size, file.Size)
}
if retrieved.Mode != file.Mode {
t.Errorf("mode mismatch: got %o, want %o", retrieved.Mode, file.Mode)
}
@@ -75,6 +69,7 @@ func TestFileRepository(t *testing.T) {
// Test Update (upsert)
file.Size = 2048
file.MTime = time.Now().Truncate(time.Second)
err = repo.Create(ctx, nil, file)
if err != nil {
t.Fatalf("failed to update file: %v", err)
@@ -84,15 +79,41 @@ func TestFileRepository(t *testing.T) {
if err != nil {
t.Fatalf("failed to get updated file: %v", err)
}
if retrieved.Size != 2048 {
t.Errorf("size not updated: got %d, want %d", retrieved.Size, 2048)
}
}
func TestFileRepositoryListDelete(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
file := &database.File{
Path: testFileTxt,
MTime: time.Now().Truncate(time.Second),
Size: 1024,
Mode: 0644,
UID: 1000,
GID: 1000,
}
err := repo.Create(ctx, nil, file)
if err != nil {
t.Fatalf("failed to create file: %v", err)
}
// Test ListModifiedSince
files, err := repo.ListModifiedSince(ctx, time.Now().Add(-1*time.Hour))
if err != nil {
t.Fatalf("failed to list files: %v", err)
}
if len(files) != 1 {
t.Errorf("expected 1 file, got %d", len(files))
}
@@ -103,24 +124,27 @@ func TestFileRepository(t *testing.T) {
t.Fatalf("failed to delete file: %v", err)
}
retrieved, err = repo.GetByPath(ctx, file.Path.String())
retrieved, err := repo.GetByPath(ctx, file.Path.String())
if err != nil {
t.Fatalf("error getting deleted file: %v", err)
}
if retrieved != nil {
t.Error("expected nil for deleted file")
}
}
func TestFileRepositorySymlink(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := NewFileRepository(db)
repo := database.NewFileRepository(db)
// Test symlink
symlink := &File{
symlink := &database.File{
Path: "/test/link",
MTime: time.Now().Truncate(time.Second),
Size: 0,
@@ -139,25 +163,30 @@ func TestFileRepositorySymlink(t *testing.T) {
if err != nil {
t.Fatalf("failed to get symlink: %v", err)
}
if !retrieved.IsSymlink() {
t.Error("expected IsSymlink() to be true")
}
if retrieved.LinkTarget != symlink.LinkTarget {
t.Errorf("link target mismatch: got %s, want %s", retrieved.LinkTarget, symlink.LinkTarget)
t.Errorf("link target mismatch: got %s, want %s",
retrieved.LinkTarget, symlink.LinkTarget)
}
}
func TestFileRepositoryTransaction(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := NewRepositories(db)
repos := database.NewRepositories(db)
// Test transaction rollback
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
file := &File{
Path: "/test/tx_file.txt",
file := &database.File{
Path: testTxFile,
MTime: time.Now().Truncate(time.Second),
Size: 1024,
Mode: 0644,
@@ -165,23 +194,24 @@ func TestFileRepositoryTransaction(t *testing.T) {
GID: 1000,
}
if err := repos.Files.Create(ctx, tx, file); err != nil {
err := repos.Files.Create(ctx, tx, file)
if err != nil {
return err
}
// Return error to trigger rollback
return fmt.Errorf("test rollback")
return errTestRollback
})
if err == nil || err.Error() != "test rollback" {
if !errors.Is(err, errTestRollback) {
t.Fatalf("expected rollback error, got: %v", err)
}
// Verify file was not created
retrieved, err := repos.Files.GetByPath(ctx, "/test/tx_file.txt")
retrieved, err := repos.Files.GetByPath(ctx, testTxFile)
if err != nil {
t.Fatalf("error checking for file: %v", err)
}
if retrieved != nil {
t.Error("file should not exist after rollback")
}
@@ -0,0 +1,81 @@
package database
import (
"context"
"path/filepath"
"testing"
"sneak.berlin/go/vaultik/internal/types"
)
// Common fixture values shared by the internal repository tests.
const (
internalTestHost = "test-host"
internalTestSnapshotID = "test-snapshot"
internalTestFilePath = "/test.txt"
internalTestFile1 = "/file1.txt"
internalTestFile2 = "/file2.txt"
// countFilesQuery counts the rows of the files table.
countFilesQuery = "SELECT COUNT(*) FROM files"
)
// mustCreateFileRow inserts the file row, failing the test on error.
func mustCreateFileRow(t *testing.T, repos *Repositories, file *File) {
t.Helper()
err := repos.Files.Create(context.Background(), nil, file)
if err != nil {
t.Fatalf("failed to create file %s: %v", file.Path, err)
}
}
// mustAddFileToSnapshot associates a file with a snapshot, failing the
// test on error.
func mustAddFileToSnapshot(
t *testing.T, repos *Repositories, snapshotID string, fileID types.FileID,
) {
t.Helper()
err := repos.Snapshots.AddFileByID(context.Background(), nil, snapshotID, fileID)
if err != nil {
t.Fatal(err)
}
}
// setupTestDB creates an on-disk test database in a per-test temp
// directory and returns it along with a cleanup func that closes it.
func setupTestDB(t *testing.T) (*DB, func()) {
t.Helper()
ctx := context.Background()
dbPath := filepath.Join(t.TempDir(), "test.db")
db, err := New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
cleanup := func() {
err := db.Close()
if err != nil {
t.Errorf("failed to close database: %v", err)
}
}
return db, cleanup
}
// countRow runs a single-integer COUNT-style query and returns the value.
func countRow(t *testing.T, db *DB, query string, args ...any) int {
t.Helper()
var count int
err := db.conn.QueryRowContext(context.Background(), query, args...).Scan(&count)
if err != nil {
t.Fatal(err)
}
return count
}
+52
View File
@@ -0,0 +1,52 @@
package database_test
import (
"context"
"path/filepath"
"testing"
"sneak.berlin/go/vaultik/internal/database"
)
// Common fixture values shared by the repository tests.
const (
testFilePath1 = "/file1.txt"
testFilePath2 = "/file2.txt"
testFileTxt = "/test/file.txt"
testTxFile = "/test/tx_file.txt"
testHostname = "test-host"
testVersion = "1.0.0"
)
// mustCreateFile inserts the given file row, failing the test on error.
func mustCreateFile(t *testing.T, repo *database.FileRepository, file *database.File) {
t.Helper()
err := repo.Create(context.Background(), nil, file)
if err != nil {
t.Fatalf("failed to create file %s: %v", file.Path, err)
}
}
// setupTestDB creates an on-disk test database in a per-test temp
// directory and returns it along with a cleanup func that closes it.
func setupTestDB(t *testing.T) (*database.DB, func()) {
t.Helper()
ctx := context.Background()
dbPath := filepath.Join(t.TempDir(), "test.db")
db, err := database.New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
cleanup := func() {
err := db.Close()
if err != nil {
t.Errorf("failed to close database: %v", err)
}
}
return db, cleanup
}
+58
View File
@@ -0,0 +1,58 @@
package database
import (
"context"
"database/sql"
"errors"
"fmt"
)
// LocalMetaKeyStorageURL is the key under which the destination store's
// URL is recorded when a mutating command first binds the local index
// to a specific backup destination.
const LocalMetaKeyStorageURL = "storage_url"
// LocalMetaRepository provides keyed access to host-local settings
// stored in the local_meta table.
type LocalMetaRepository struct {
db *DB
}
// NewLocalMetaRepository creates a LocalMetaRepository backed by db.
func NewLocalMetaRepository(db *DB) *LocalMetaRepository {
return &LocalMetaRepository{db: db}
}
// Get returns the value stored at key, or the empty string if the key
// is not set. A missing key is not an error — the caller distinguishes
// "unset" (bind on first use) from "set to something" (compare).
func (r *LocalMetaRepository) Get(ctx context.Context, key string) (string, error) {
var value string
err := r.db.conn.QueryRowContext(ctx,
"SELECT value FROM local_meta WHERE key = ?", key,
).Scan(&value)
if errors.Is(err, sql.ErrNoRows) {
return "", nil
}
if err != nil {
return "", fmt.Errorf("reading local_meta %q: %w", key, err)
}
return value, nil
}
// Set writes key=value, replacing any prior value.
func (r *LocalMetaRepository) Set(ctx context.Context, key, value string) error {
_, err := r.db.ExecWithLog(ctx,
`INSERT INTO local_meta (key, value) VALUES (?, ?)
ON CONFLICT(key) DO UPDATE SET value = excluded.value`,
key, value,
)
if err != nil {
return fmt.Errorf("writing local_meta %q: %w", key, err)
}
return nil
}
+64
View File
@@ -0,0 +1,64 @@
package database_test
import (
"context"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
)
func TestLocalMetaEmptyOnFresh(t *testing.T) {
t.Parallel()
db, err := database.NewTestDB()
require.NoError(t, err)
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
got, err := repos.LocalMeta.Get(context.Background(), database.LocalMetaKeyStorageURL)
require.NoError(t, err)
require.Empty(t, got, "fresh DB must return empty for unset keys, not error")
}
func TestLocalMetaSetGetRoundTrip(t *testing.T) {
t.Parallel()
db, err := database.NewTestDB()
require.NoError(t, err)
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
ctx := context.Background()
require.NoError(t, repos.LocalMeta.Set(
ctx, database.LocalMetaKeyStorageURL, "file:///mnt/backups"))
got, err := repos.LocalMeta.Get(ctx, database.LocalMetaKeyStorageURL)
require.NoError(t, err)
require.Equal(t, "file:///mnt/backups", got)
}
func TestLocalMetaSetOverwrites(t *testing.T) {
t.Parallel()
db, err := database.NewTestDB()
require.NoError(t, err)
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
ctx := context.Background()
require.NoError(t, repos.LocalMeta.Set(
ctx, database.LocalMetaKeyStorageURL, "s3://old"))
require.NoError(t, repos.LocalMeta.Set(
ctx, database.LocalMetaKeyStorageURL, "s3://new"))
got, err := repos.LocalMeta.Get(ctx, database.LocalMetaKeyStorageURL)
require.NoError(t, err)
require.Equal(t, "s3://new", got)
}
+20 -12
View File
@@ -1,5 +1,3 @@
// Package database provides data models and repository interfaces for the Vaultik backup system.
// It includes types for files, chunks, blobs, snapshots, and their relationships.
package database
import (
@@ -15,7 +13,10 @@ import (
type File struct {
ID types.FileID // UUID primary key
Path types.FilePath // Absolute path of the file
SourcePath types.SourcePath // The source directory this file came from (for restore path stripping)
// SourcePath is the source directory this file came from (used for
// restore path stripping).
SourcePath types.SourcePath
MTime time.Time
Size int64
Mode uint32
@@ -50,13 +51,16 @@ type Chunk struct {
// Blob represents a blob record in the database.
// A blob is Vaultik's final storage unit - a large file (up to 10GB) containing
// many compressed and encrypted chunks from multiple source files.
// Blobs are content-addressed, meaning their filename in S3 is derived from
// the SHA256 hash of their compressed and encrypted content.
// The blob creation process is: chunks are accumulated -> compressed with zstd
// -> encrypted with age -> hashed -> uploaded to S3 with the hash as filename.
// Blobs are content-addressed: the filename in S3 is
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data
// before compression and encryption (not from the stored bytes). See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
type Blob struct {
ID types.BlobID // UUID assigned when blob creation starts
Hash types.BlobHash // SHA256 of final compressed+encrypted content (empty until finalized)
// Hash is hex(SHA256(SHA256(uncompressed blob contents)))
// (empty until finalized); see the type comment above.
Hash types.BlobHash
CreatedTS time.Time // When blob creation started
FinishedTS *time.Time // When blob was finalized (nil if still packing)
UncompressedSize int64 // Total size of raw chunks before compression
@@ -75,9 +79,10 @@ type BlobChunk struct {
Length int64
}
// ChunkFile represents the reverse mapping showing which files contain a specific chunk.
// This is used during deduplication to identify all files that share a chunk,
// which is important for garbage collection and integrity verification.
// ChunkFile represents the reverse mapping showing which files contain a
// specific chunk. This is used during deduplication to identify all files
// that share a chunk, which is important for garbage collection and
// integrity verification.
type ChunkFile struct {
ChunkHash types.ChunkHash
FileID types.FileID
@@ -97,7 +102,10 @@ type Snapshot struct {
ChunkCount int64
BlobCount int64
TotalSize int64 // Total size of all referenced files
BlobSize int64 // Total size of all referenced blobs (compressed and encrypted)
// BlobSize is the total size of all referenced blobs (compressed and
// encrypted).
BlobSize int64
BlobUncompressedSize int64 // Total uncompressed size of all referenced blobs
CompressionRatio float64 // Compression ratio (BlobSize / BlobUncompressedSize)
CompressionLevel int // Compression level used for this snapshot
+60 -3
View File
@@ -2,6 +2,7 @@ package database
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
@@ -11,7 +12,51 @@ import (
"sneak.berlin/go/vaultik/internal/log"
)
// indexDirPerm restricts the local index directory to the owning user;
// the index describes the backed-up file tree and must stay private.
const indexDirPerm = 0o700
// indexFilePerm restricts the index file to the owning user; it lists every
// backed-up path and chunk hash and must stay private.
const indexFilePerm = 0o600
// ensureIndexFileMode makes the index file owner-only before the SQLite
// driver opens it: it creates the file 0600 if absent, or chmods an existing
// one to 0600. Doing this first matters because SQLite creates its -wal and
// -shm side files with the mode of the main database file, so a private main
// file yields private side files. The driver treats a zero-byte file as an
// empty database, so pre-creating it here is safe.
func ensureIndexFileMode(path string) error {
info, err := os.Stat(path)
switch {
case err == nil:
if info.Mode().Perm() == indexFilePerm {
return nil
}
err = os.Chmod(path, indexFilePerm)
if err != nil {
return fmt.Errorf("restricting index file permissions: %w", err)
}
return nil
case errors.Is(err, os.ErrNotExist):
//nolint:gosec // G304: the index path is operator-configured by design
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY, indexFilePerm)
if err != nil {
return fmt.Errorf("creating index file: %w", err)
}
return f.Close()
default:
return fmt.Errorf("checking index file: %w", err)
}
}
// Module provides database dependencies
//
//nolint:gochecknoglobals // fx module definitions are package globals by convention
var Module = fx.Module("database",
fx.Provide(
provideDatabase,
@@ -22,23 +67,35 @@ var Module = fx.Module("database",
func provideDatabase(lc fx.Lifecycle, cfg *config.Config) (*DB, error) {
// Ensure the index directory exists
indexDir := filepath.Dir(cfg.IndexPath)
if err := os.MkdirAll(indexDir, 0700); err != nil {
err := os.MkdirAll(indexDir, indexDirPerm)
if err != nil {
return nil, fmt.Errorf("creating index directory: %w", err)
}
err = ensureIndexFileMode(cfg.IndexPath)
if err != nil {
return nil, err
}
db, err := New(context.Background(), cfg.IndexPath)
if err != nil {
return nil, fmt.Errorf("opening database: %w", err)
}
lc.Append(fx.Hook{
OnStop: func(ctx context.Context) error {
OnStop: func(_ context.Context) error {
log.Debug("Database module OnStop hook called")
if err := db.Close(); err != nil {
err := db.Close()
if err != nil {
log.Error("Failed to close database in OnStop hook", "error", err)
return err
}
log.Debug("Database closed successfully in OnStop hook")
return nil
},
})
+100
View File
@@ -0,0 +1,100 @@
package database
import (
"os"
"path/filepath"
"syscall"
"testing"
"go.uber.org/fx/fxtest"
"sneak.berlin/go/vaultik/internal/config"
)
// TestProvideDatabaseFreshIndexMode verifies that provideDatabase creates a
// missing index file owner-only (0600), even under a lenient 022 umask that
// would otherwise leave a freshly created file world-readable.
//
//nolint:paralleltest // syscall.Umask is process-global; parallel tests would clash
func TestProvideDatabaseFreshIndexMode(t *testing.T) {
restore := syscall.Umask(0o022)
defer syscall.Umask(restore)
indexPath := filepath.Join(t.TempDir(), "index.sqlite")
openIndex(t, indexPath)
assertPerm(t, indexPath, 0o600)
}
// TestProvideDatabaseExistingIndexMode verifies that provideDatabase tightens
// an existing world-readable index (0644) in a group/other-readable directory
// down to owner-only (0600).
//
//nolint:paralleltest // syscall.Umask is process-global; parallel tests would clash
func TestProvideDatabaseExistingIndexMode(t *testing.T) {
restore := syscall.Umask(0o022)
defer syscall.Umask(restore)
dir := filepath.Join(t.TempDir(), "data")
//nolint:gosec // G301: the test intentionally uses a 0755 directory
err := os.MkdirAll(dir, 0o755)
if err != nil {
t.Fatalf("creating index directory: %v", err)
}
//nolint:gosec // G302: the test intentionally uses a 0755 directory
err = os.Chmod(dir, 0o755)
if err != nil {
t.Fatalf("relaxing index directory permissions: %v", err)
}
indexPath := filepath.Join(dir, "index.sqlite")
//nolint:gosec // G306: the test intentionally starts from a 0644 index
err = os.WriteFile(indexPath, nil, 0o644)
if err != nil {
t.Fatalf("creating pre-existing index: %v", err)
}
//nolint:gosec // G302: the test intentionally starts from a 0644 index
err = os.Chmod(indexPath, 0o644)
if err != nil {
t.Fatalf("relaxing pre-existing index permissions: %v", err)
}
openIndex(t, indexPath)
assertPerm(t, indexPath, 0o600)
}
// openIndex runs provideDatabase against indexPath and closes the resulting
// database before returning.
func openIndex(t *testing.T, indexPath string) {
t.Helper()
cfg := &config.Config{IndexPath: indexPath}
db, err := provideDatabase(fxtest.NewLifecycle(t), cfg)
if err != nil {
t.Fatalf("provideDatabase: %v", err)
}
err = db.Close()
if err != nil {
t.Fatalf("closing database: %v", err)
}
}
// assertPerm fails the test unless path has exactly the given permission bits.
func assertPerm(t *testing.T, path string, want os.FileMode) {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
got := info.Mode().Perm()
if got != want {
t.Fatalf("permissions of %s = %#o, want %#o", path, got, want)
}
}
+145
View File
@@ -0,0 +1,145 @@
//nolint:testpackage // exercises unexported read-only open internals
package database
import (
"context"
"database/sql"
"errors"
"path/filepath"
"testing"
)
// genuineSnapshotDB writes a real snapshot database (the full schema
// applied) to a fresh file and returns its path.
func genuineSnapshotDB(t *testing.T) string {
t.Helper()
path := filepath.Join(t.TempDir(), "snapshot.db")
db, err := New(context.Background(), path)
if err != nil {
t.Fatalf("creating snapshot database: %v", err)
}
err = db.Close()
if err != nil {
t.Fatalf("closing snapshot database: %v", err)
}
return path
}
// forgedDB creates an empty database file and runs the given statements
// against it read-write, so a test can plant schema objects the real
// schema never defines.
func forgedDB(t *testing.T, stmts ...string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "forged.db")
db, err := sql.Open("sqlite", path)
if err != nil {
t.Fatalf("opening forged database: %v", err)
}
for _, stmt := range stmts {
_, err = db.ExecContext(context.Background(), stmt)
if err != nil {
t.Fatalf("executing %q: %v", stmt, err)
}
}
err = db.Close()
if err != nil {
t.Fatalf("closing forged database: %v", err)
}
return path
}
func TestOpenReadOnlyAcceptsGenuineSnapshot(t *testing.T) {
t.Parallel()
db, err := OpenReadOnly(context.Background(), genuineSnapshotDB(t))
if err != nil {
t.Fatalf("OpenReadOnly refused a genuine snapshot database: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
}
func TestOpenReadOnlyRefusesWrites(t *testing.T) {
t.Parallel()
db, err := OpenReadOnly(context.Background(), genuineSnapshotDB(t))
if err != nil {
t.Fatalf("OpenReadOnly: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
// A schema write depends on no table columns, so the only reason it
// can fail is that the database is open read-only.
_, err = db.Conn().ExecContext(context.Background(),
"CREATE TABLE probe_readonly (x)")
if err == nil {
t.Fatal("expected a write to a read-only snapshot database to fail")
}
}
func TestOpenReadOnlyRejectsView(t *testing.T) {
t.Parallel()
path := forgedDB(t, "CREATE VIEW files AS SELECT 1 AS path")
_, err := OpenReadOnly(context.Background(), path)
if !errors.Is(err, errUntrustedSnapshotSchema) {
t.Fatalf("expected a view named files to be refused, got %v", err)
}
}
func TestOpenReadOnlyRejectsTrigger(t *testing.T) {
t.Parallel()
path := forgedDB(t,
"CREATE TABLE files (path TEXT)",
"CREATE TRIGGER t AFTER INSERT ON files BEGIN SELECT 1; END")
_, err := OpenReadOnly(context.Background(), path)
if !errors.Is(err, errUntrustedSnapshotSchema) {
t.Fatalf("expected a trigger to be refused, got %v", err)
}
}
func TestOpenReadOnlyRejectsMissingTable(t *testing.T) {
t.Parallel()
// Only one of the expected tables is present.
path := forgedDB(t, "CREATE TABLE files (path TEXT)")
_, err := OpenReadOnly(context.Background(), path)
if !errors.Is(err, errUntrustedSnapshotSchema) {
t.Fatalf("expected a missing expected table to be refused, got %v", err)
}
}
func TestIsVirtualTableSQL(t *testing.T) {
t.Parallel()
cases := []struct {
sql string
want bool
}{
{"CREATE VIRTUAL TABLE t USING fts5(x)", true},
{" create virtual table t using fts5(x)", true},
{"CREATE TABLE t (x)", false},
{"CREATE VIEW t AS SELECT 1", false},
{"", false},
}
for _, c := range cases {
if got := isVirtualTableSQL(c.sql); got != c.want {
t.Errorf("isVirtualTableSQL(%q) = %v, want %v", c.sql, got, c.want)
}
}
}
+19 -8
View File
@@ -19,6 +19,7 @@ type Repositories struct {
ChunkFiles *ChunkFileRepository
Snapshots *SnapshotRepository
Uploads *UploadRepository
LocalMeta *LocalMetaRepository
}
// NewRepositories creates a new Repositories instance with all repository types.
@@ -34,6 +35,7 @@ func NewRepositories(db *DB) *Repositories {
ChunkFiles: NewChunkFileRepository(db),
Snapshots: NewSnapshotRepository(db),
Uploads: NewUploadRepository(db.conn),
LocalMeta: NewLocalMetaRepository(db),
}
}
@@ -48,21 +50,26 @@ type TxFunc func(ctx context.Context, tx *sql.Tx) error
// This method should be used for all write operations to ensure atomicity.
func (r *Repositories) WithTx(ctx context.Context, fn TxFunc) error {
LogSQL("WithTx", "Beginning transaction", "")
tx, err := r.db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("beginning transaction: %w", err)
}
LogSQL("WithTx", "Transaction started", "")
defer func() {
if p := recover(); p != nil {
if rollbackErr := tx.Rollback(); rollbackErr != nil {
Fatal("failed to rollback transaction: %v", rollbackErr)
rollbackErr := tx.Rollback()
if rollbackErr != nil {
Fatalf("failed to rollback transaction: %v", rollbackErr)
}
panic(p)
} else if err != nil {
if rollbackErr := tx.Rollback(); rollbackErr != nil {
Fatal("failed to rollback transaction: %v", rollbackErr)
rollbackErr := tx.Rollback()
if rollbackErr != nil {
Fatalf("failed to rollback transaction: %v", rollbackErr)
}
}
}()
@@ -88,6 +95,7 @@ func (r *Repositories) WithReadTx(ctx context.Context, fn TxFunc) error {
opts := &sql.TxOptions{
ReadOnly: true,
}
tx, err := r.db.BeginTx(ctx, opts)
if err != nil {
return fmt.Errorf("beginning read transaction: %w", err)
@@ -95,13 +103,16 @@ func (r *Repositories) WithReadTx(ctx context.Context, fn TxFunc) error {
defer func() {
if p := recover(); p != nil {
if rollbackErr := tx.Rollback(); rollbackErr != nil {
Fatal("failed to rollback transaction: %v", rollbackErr)
rollbackErr := tx.Rollback()
if rollbackErr != nil {
Fatalf("failed to rollback transaction: %v", rollbackErr)
}
panic(p)
} else if err != nil {
if rollbackErr := tx.Rollback(); rollbackErr != nil {
Fatal("failed to rollback transaction: %v", rollbackErr)
rollbackErr := tx.Rollback()
if rollbackErr != nil {
Fatalf("failed to rollback transaction: %v", rollbackErr)
}
}
}()
+104 -54
View File
@@ -1,124 +1,162 @@
package database
package database_test
import (
"context"
"database/sql"
"fmt"
"errors"
"testing"
"time"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
func TestRepositoriesTransaction(t *testing.T) {
db, cleanup := setupTestDB(t)
defer cleanup()
// errIntentionalRollback forces a transaction rollback in tests.
var errIntentionalRollback = errors.New("intentional rollback")
ctx := context.Background()
repos := NewRepositories(db)
// Test successful transaction with multiple operations
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
// Create a file
file := &File{
Path: "/test/tx_file.txt",
// createTxTestData returns a transaction body that creates a file with
// two chunks packed into one blob.
func createTxTestData(
repos *database.Repositories,
) func(context.Context, *sql.Tx) error {
return func(ctx context.Context, tx *sql.Tx) error {
file := &database.File{
Path: testTxFile,
MTime: time.Now().Truncate(time.Second),
Size: 1024,
Mode: 0644,
UID: 1000,
GID: 1000,
}
if err := repos.Files.Create(ctx, tx, file); err != nil {
err := repos.Files.Create(ctx, tx, file)
if err != nil {
return err
}
err = createTxFileChunks(ctx, tx, repos, file.ID)
if err != nil {
return err
}
return createTxBlob(ctx, tx, repos)
}
}
// createTxFileChunks creates the two test chunks and maps them to the file.
func createTxFileChunks(
ctx context.Context, tx *sql.Tx,
repos *database.Repositories, fileID types.FileID,
) error {
// Create chunks
chunk1 := &Chunk{
chunk1 := &database.Chunk{
ChunkHash: types.ChunkHash("tx_chunk1"),
Size: 512,
}
if err := repos.Chunks.Create(ctx, tx, chunk1); err != nil {
err := repos.Chunks.Create(ctx, tx, chunk1)
if err != nil {
return err
}
chunk2 := &Chunk{
chunk2 := &database.Chunk{
ChunkHash: types.ChunkHash("tx_chunk2"),
Size: 512,
}
if err := repos.Chunks.Create(ctx, tx, chunk2); err != nil {
err = repos.Chunks.Create(ctx, tx, chunk2)
if err != nil {
return err
}
// Map chunks to file
fc1 := &FileChunk{
FileID: file.ID,
fc1 := &database.FileChunk{
FileID: fileID,
Idx: 0,
ChunkHash: chunk1.ChunkHash,
}
if err := repos.FileChunks.Create(ctx, tx, fc1); err != nil {
err = repos.FileChunks.Create(ctx, tx, fc1)
if err != nil {
return err
}
fc2 := &FileChunk{
FileID: file.ID,
fc2 := &database.FileChunk{
FileID: fileID,
Idx: 1,
ChunkHash: chunk2.ChunkHash,
}
if err := repos.FileChunks.Create(ctx, tx, fc2); err != nil {
return err
}
// Create blob
blob := &Blob{
return repos.FileChunks.Create(ctx, tx, fc2)
}
// createTxBlob creates the test blob and maps both chunks into it.
func createTxBlob(
ctx context.Context, tx *sql.Tx, repos *database.Repositories,
) error {
blob := &database.Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash("tx_blob1"),
CreatedTS: time.Now().Truncate(time.Second),
}
if err := repos.Blobs.Create(ctx, tx, blob); err != nil {
err := repos.Blobs.Create(ctx, tx, blob)
if err != nil {
return err
}
// Map chunks to blob
bc1 := &BlobChunk{
bc1 := &database.BlobChunk{
BlobID: blob.ID,
ChunkHash: chunk1.ChunkHash,
ChunkHash: types.ChunkHash("tx_chunk1"),
Offset: 0,
Length: 512,
}
if err := repos.BlobChunks.Create(ctx, tx, bc1); err != nil {
err = repos.BlobChunks.Create(ctx, tx, bc1)
if err != nil {
return err
}
bc2 := &BlobChunk{
bc2 := &database.BlobChunk{
BlobID: blob.ID,
ChunkHash: chunk2.ChunkHash,
ChunkHash: types.ChunkHash("tx_chunk2"),
Offset: 512,
Length: 512,
}
if err := repos.BlobChunks.Create(ctx, tx, bc2); err != nil {
return err
}
return nil
})
return repos.BlobChunks.Create(ctx, tx, bc2)
}
func TestRepositoriesTransaction(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
err := repos.WithTx(ctx, createTxTestData(repos))
if err != nil {
t.Fatalf("transaction failed: %v", err)
}
// Verify all data was committed
file, err := repos.Files.GetByPath(ctx, "/test/tx_file.txt")
file, err := repos.Files.GetByPath(ctx, testTxFile)
if err != nil {
t.Fatalf("failed to get file: %v", err)
}
if file == nil {
t.Error("expected file after transaction")
}
chunks, err := repos.FileChunks.GetByFile(ctx, "/test/tx_file.txt")
chunks, err := repos.FileChunks.GetByFile(ctx, testTxFile)
if err != nil {
t.Fatalf("failed to get file chunks: %v", err)
}
if len(chunks) != 2 {
t.Errorf("expected 2 file chunks, got %d", len(chunks))
}
@@ -127,22 +165,25 @@ func TestRepositoriesTransaction(t *testing.T) {
if err != nil {
t.Fatalf("failed to get blob: %v", err)
}
if blob == nil {
t.Error("expected blob after transaction")
}
}
func TestRepositoriesTransactionRollback(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := NewRepositories(db)
repos := database.NewRepositories(db)
// Test transaction rollback
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
// Create a file
file := &File{
file := &database.File{
Path: "/test/rollback_file.txt",
MTime: time.Now().Truncate(time.Second),
Size: 1024,
@@ -150,24 +191,27 @@ func TestRepositoriesTransactionRollback(t *testing.T) {
UID: 1000,
GID: 1000,
}
if err := repos.Files.Create(ctx, tx, file); err != nil {
err := repos.Files.Create(ctx, tx, file)
if err != nil {
return err
}
// Create a chunk
chunk := &Chunk{
chunk := &database.Chunk{
ChunkHash: types.ChunkHash("rollback_chunk"),
Size: 1024,
}
if err := repos.Chunks.Create(ctx, tx, chunk); err != nil {
err = repos.Chunks.Create(ctx, tx, chunk)
if err != nil {
return err
}
// Return error to trigger rollback
return fmt.Errorf("intentional rollback")
return errIntentionalRollback
})
if err == nil || err.Error() != "intentional rollback" {
if !errors.Is(err, errIntentionalRollback) {
t.Fatalf("expected rollback error, got: %v", err)
}
@@ -176,6 +220,7 @@ func TestRepositoriesTransactionRollback(t *testing.T) {
if err != nil {
t.Fatalf("error checking for file: %v", err)
}
if file != nil {
t.Error("file should not exist after rollback")
}
@@ -184,20 +229,23 @@ func TestRepositoriesTransactionRollback(t *testing.T) {
if err != nil {
t.Fatalf("error checking for chunk: %v", err)
}
if chunk != nil {
t.Error("chunk should not exist after rollback")
}
}
func TestRepositoriesReadTransaction(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := NewRepositories(db)
repos := database.NewRepositories(db)
// First, create some data
file := &File{
file := &database.File{
Path: "/test/read_file.txt",
MTime: time.Now().Truncate(time.Second),
Size: 1024,
@@ -205,22 +253,25 @@ func TestRepositoriesReadTransaction(t *testing.T) {
UID: 1000,
GID: 1000,
}
err := repos.Files.Create(ctx, nil, file)
if err != nil {
t.Fatalf("failed to create file: %v", err)
}
// Test read-only transaction
var retrievedFile *File
var retrievedFile *database.File
err = repos.WithReadTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
var err error
retrievedFile, err = repos.Files.GetByPathTx(ctx, tx, "/test/read_file.txt")
if err != nil {
return err
}
// Try to write in read-only transaction (should fail)
_ = repos.Files.Create(ctx, tx, &File{
_ = repos.Files.Create(ctx, tx, &database.File{
Path: "/test/should_fail.txt",
MTime: time.Now(),
Size: 0,
@@ -232,7 +283,6 @@ func TestRepositoriesReadTransaction(t *testing.T) {
return nil
})
if err != nil {
t.Fatalf("read transaction failed: %v", err)
}

Some files were not shown because too many files have changed in this diff Show More