6 Commits
Author SHA1 Message Date
clawbot eed117fe25 Route direct-stdout command output through internal/ui (closes #149)
check / check (pull_request) Successful in 1m27s
check / check (push) Successful in 3m18s
version, info, remote info, config, and database delete wrote plain text straight to stdout, so they were unstyled and ignored --quiet. Output is now governed by internal/ui in two buckets. Status lines and confirmations (config init, config set, the database delete prompt) go through the ui message methods and are silenced by --quiet. The data a command exists to produce is written plain -- the version/info/remote-info reports, the snapshot list table, config get values, and the --json documents -- and is never suppressed, since a script depends on it and a marker would corrupt a table or document. The database delete confirmation prompt is always shown. Pure-cli commands reach ui through a small commandUI helper.

Model: opus-4-8
2026-09-22 20:28:50 +02:00
clawbot dd7a610c23 Mark a snapshot complete only after its metadata export succeeds (closes #177)
check / check (pull_request) Successful in 1m42s
check / check (push) Successful in 3m35s
finalizeSnapshotMetadata marked the snapshot complete and then exported its metadata. A crash after completion but before/during the export left the local index showing the snapshot complete while the destination had no manifest or database, and PruneDatabase (which drops only NULL completed_at rows) kept it: a silently unrestorable snapshot.

Reorder so completion is recorded last. CompleteSnapshot is split into PopulateSnapshotBlobs (before the export) and MarkSnapshotComplete (after it). An interrupted export now leaves the snapshot incomplete, so the next run PruneDatabase drops it and re-backs-up the data; the reverse tiny window leaves a restorable snapshot the index reports honestly as remote-only. Update REPOSTRUCTURE.md guarantee 4 and the ARCHITECTURE.md flow. Add a fault-injection test driving the full create path.

Model: opus-4-8
2026-09-22 20:00:41 +02:00
clawbot 1548c0f933 Correct the security claims in docs and comments, and record the accepted risks (closes #171)
check / check (pull_request) Successful in 1m58s
check / check (push) Successful in 2m49s
Docs and comments only; no behaviour change. Corrects ten overclaims the security review found: snapshot names are hashed but the hash uses no secret, so a guessed hostname and name can be confirmed; a blob is named by hex(SHA256(SHA256(uncompressed contents))), stated once in docs/REPOSTRUCTURE.md and referenced elsewhere; double hashing does not hide known content (blob packing does); age uses ChaCha20-Poly1305, not XChaCha20; encryption is required, not optional; a snapshot is marked complete before its metadata is uploaded; the export comment now matches its only caller; deep verify detects corruption, not authorship; adding a recipient does not reach existing data; restore examples target a user-owned directory.

Adds an Accepted Risks subsection under Security Considerations with the seven documented risks, cross-referenced from the README.

Model: opus-4-8
2026-09-22 19:28:31 +02:00
clawbot c3bec7d3aa Reject a metadata database truncated to the age header and nonce (closes #152)
check / check (push) Successful in 2m5s
check / check (pull_request) Successful in 2m28s
An object holding just the age header and its 16-byte nonce decrypts without error: the truncated read surfaces as io.ErrUnexpectedEOF at the age layer, which the zstd decoder maps to a clean EOF at frame start. blobgen then reported zero bytes and no error, so a truncated stream was indistinguishable from a valid empty one, and the metadata database export slipped through -- restore built a fresh schema on the empty file and reported success.

blobgen.Reader.Read now, on EOF, reads once more from the age reader and surfaces io.ErrUnexpectedEOF unless that read is (0, io.EOF), the state a genuine end leaves. downloadSnapshotDB additionally rejects a zero-length decrypted database before any schema is built.

Model: opus-4-8
2026-09-22 18:11:57 +02:00
clawbot 1244c9e48d Bound download expansion and escape control chars on the terminal (closes #164)
check / check (pull_request) Successful in 1m47s
check / check (push) Successful in 3m11s
Objects fetched from the store are untrusted; several decode paths let one expand or print without limit.

- blobgen.LimitReader errors past a byte cap (not io.LimitReader silent EOF). DecodeManifest reads through caps on both compressed input and decompressed output, far above any real manifest, so json.Decode cannot buffer a compressible bomb. FetchAndDecryptBlob bounds decompression to the blob recorded uncompressed_size (not the restoring host blob_size_limit).
- downloadSnapshotDB streams straight from storage to its temp file with io.Copy, replacing two ReadAll calls that held the whole database twice.
- FetchBlob drops the per-blob Stat round-trip, its expectedSize parameter and returned size, all of which only fed a debug log.
- TTYHandler and ui.Writer escape control characters in messages, attribute keys/values, and rendered identifiers/paths before colour codes are applied, so a crafted value cannot drive the terminal.

Model: opus-4-8
2026-09-22 17:00:35 +02:00
clawbot 82c51a5337 Validate blob hashes, offsets and lengths from the destination (closes #155)
check / check (pull_request) Successful in 1m22s
check / check (push) Successful in 3m30s
A blob hash read back from the downloaded snapshot database or the store listing was trusted unchecked. A hostile remote could set a hash such as "aa/../../etc" and have a decrypted blob written outside the cache directory, or feed a short or negative value that panicked a command.

blobDiskCache.path now refuses any key with a path separator, and ReadAt rejects a negative offset or length, bounding so a sum cannot overflow past the check. A new isBlobHash helper gates FetchBlob, shallow and deep verify, and restore: buildBlobIndexes rejects every hash from the snapshot database before any fetch. The blobs/ and metadata/ listings skip a non-conforming name, and short-hash prefixes in log and error text go through a panic-safe shortHash helper.

Model: opus-4-8
2026-09-22 16:11:28 +02:00
45 changed files with 1442 additions and 241 deletions
+17 -13
View File
@@ -74,16 +74,16 @@ Maps files to their constituent chunks:
#### Blob (`database.Blob`) #### Blob (`database.Blob`)
The final storage unit uploaded to S3. Contains many compressed and encrypted chunks: The final storage unit uploaded to S3. Contains many compressed and encrypted chunks:
- `ID`: UUID assigned at creation - `ID`: UUID assigned at creation
- `Hash`: SHA256 of final compressed+encrypted content - `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs))
- `UncompressedSize`: Total raw chunk data before compression - `UncompressedSize`: Total raw chunk data before compression
- `CompressedSize`: Size after zstd compression and age encryption - `CompressedSize`: Size after zstd compression and age encryption
- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps - `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
Blob creation process: Blob creation process:
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB) 1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. Compressed with zstd 2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
3. Encrypted with age (recipients configured in config) 3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and streamed to storage
4. SHA256 hash computed → becomes filename in S3 4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` 5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}`
#### BlobChunk (`database.BlobChunk`) #### BlobChunk (`database.BlobChunk`)
@@ -284,8 +284,10 @@ Manages snapshot lifecycle and metadata export.
Key methods: Key methods:
- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record - `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record
- `CompleteSnapshot(ctx, snapshotID)`Mark snapshot complete - `PopulateSnapshotBlobs(ctx, snapshotID)`Record every blob the snapshot references
- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3 - `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3
- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a successful export
- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark complete (no export between)
### `internal/database` ### `internal/database`
SQLite database for local index. Single-writer mode for thread safety. SQLite database for local index. Single-writer mode for thread safety.
@@ -335,16 +337,18 @@ CreateSnapshot(opts)
├─► SnapshotManager.UpdateSnapshotStatsExtended() ├─► SnapshotManager.UpdateSnapshotStatsExtended()
├─► SnapshotManager.CompleteSnapshot() ├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs
─► SnapshotManager.ExportSnapshotMetadata() ─► SnapshotManager.ExportSnapshotMetadata()
│ │
│ ├─► Copy database to temp file
│ ├─► Clean to only current snapshot data (VACUUM)
│ ├─► Compress binary SQLite with zstd
│ ├─► Encrypt with age
│ ├─► Upload db.zst.age to storage
│ └─► Upload manifest.json.zst to storage
├─► Copy database to temp file └─► SnapshotManager.MarkSnapshotComplete() // only after the export succeeds
├─► Clean to only current snapshot data (VACUUM)
├─► Compress binary SQLite with zstd
├─► Encrypt with age
├─► Upload db.zst.age to storage
└─► Upload manifest.json.zst to storage
``` ```
## Deduplication Strategy ## Deduplication Strategy
+42 -25
View File
@@ -38,7 +38,7 @@ vaultik snapshot list
Features: Features:
* modern encryption ([age](https://age-encryption.org/), X25519 + XChaCha20-Poly1305) * modern encryption ([age](https://age-encryption.org/), X25519 + ChaCha20-Poly1305)
* content-defined chunking with deduplication (FastCDC) * content-defined chunking with deduplication (FastCDC)
* incremental backups (only changed files are re-chunked) * incremental backups (only changed files are re-chunked)
* multithreaded zstd compression at configurable levels * multithreaded zstd compression at configurable levels
@@ -79,11 +79,13 @@ vaultik snapshot verify <snapshot-id>
# with one or more identities, is accepted # with one or more identities, is accepted
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
# deep verify (downloads and cryptographically verifies every blob) # deep verify (downloads every blob, decrypts it, and re-hashes it to
# detect corruption — this checks integrity, not who wrote the blob)
vaultik snapshot verify --deep <snapshot-id> vaultik snapshot verify --deep <snapshot-id>
# restore (requires the private key) # restore (requires the private key). Restore into a new directory you own,
vaultik snapshot restore <snapshot-id> /tmp/restored # writable only by you — not a shared location like /tmp
vaultik snapshot restore <snapshot-id> ~/vaultik-restore
# daily cron job: back up, keep a 4-week rolling window of snapshots # daily cron job: back up, keep a 4-week rolling window of snapshots
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w # 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
@@ -128,12 +130,13 @@ full `remote_key` from `snapshot list --json` — to restore and verify:
# keeps the key out of your shell history) # keeps the key out of your shell history)
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
# restore everything to /tmp/restored, then check every restored file's # restore everything to a new directory you own (writable only by you, not a
# chunk hashes # shared location like /tmp), then check every restored file's chunk hashes
vaultik snapshot restore --verify <remote-key> /tmp/restored vaultik snapshot restore --verify <remote-key> ~/vaultik-restore
# optionally, deep-verify the snapshot against the store (downloads and # optionally, deep-verify the snapshot against the store (downloads every
# cryptographically checks every blob) # blob, decrypts it, and re-hashes it to detect corruption — this checks
# integrity, not who wrote the blob)
vaultik snapshot verify --deep <remote-key> vaultik snapshot verify --deep <remote-key>
``` ```
@@ -449,14 +452,19 @@ Snapshot IDs follow the human-readable format
`<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g. `<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g.
`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the `server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the
destination store in plaintext. Each snapshot's metadata directory is named destination store in plaintext. Each snapshot's metadata directory is named
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a listing with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a plain
of the store reveals no hostname or snapshot name. The backup time is not listing of the store shows no hostname or snapshot name. The hash uses no
hidden: manifest.json.zst carries a plaintext timestamp, and object secret, though, so an observer who guesses a candidate hostname and snapshot
name can hash it and confirm the snapshot is present; the remote key keeps
names out of a listing but does not hide them from a guess. The backup time is
not hidden either: manifest.json.zst carries a plaintext timestamp, and object
modification times are visible at the storage layer regardless. For example, modification times are visible at the storage layer regardless. For example,
`server1_home_2025-06-01T12:00:00Z` is stored under `server1_home_2025-06-01T12:00:00Z` is stored under
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`. `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`.
See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation. derivation, and [Security Considerations](docs/REPOSTRUCTURE.md#security-considerations)
(including [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks)) for what the
format does and does not protect.
### data flow ### data flow
@@ -497,7 +505,7 @@ derivation.
### encryption ### encryption
* Asymmetric encryption using age (X25519 + XChaCha20-Poly1305) * Asymmetric encryption using age (X25519 + ChaCha20-Poly1305)
* Only the public key is needed on the source host * Only the public key is needed on the source host
* Each blob and each metadata database is encrypted independently * Each blob and each metadata database is encrypted independently
* Multiple recipients supported (encrypt to multiple keys) * Multiple recipients supported (encrypt to multiple keys)
@@ -637,17 +645,26 @@ priority.
## output style ## output style
The operational narration of the long-running commands — the Begin, Every command's user-facing output is governed by `internal/ui`, in one
Complete, Progress, and status lines of `snapshot create`, `prune`, of two ways. Color is enabled when stdout is a TTY and the `NO_COLOR`
`snapshot restore`, and the like — goes through helpers in `internal/ui` environment variable is unset (https://no-color.org/).
and conforms to the uniform style below. Some commands instead write
plain text straight to stdout (`version`, `info`, `config`, the * **Status, progress, warnings, and errors** go through the `internal/ui`
`database delete` prompt, and the `snapshot list` table); that output is message methods below: marker-prefixed, colored on a TTY, and — except
unstyled and does not honor `--quiet`. Routing it through `internal/ui` warnings and errors — silenced by `--quiet`. This is the operational
is tracked in narration of the long-running commands (`snapshot create`, `prune`,
[issue #149](https://git.eeqj.de/sneak/vaultik/issues/149). Color is `snapshot restore`, and the like) and the confirmations of
enabled when stdout is a TTY and the `NO_COLOR` environment variable is `config init`, `config set`, and `database delete`.
unset (https://no-color.org/). * **The data a command exists to produce** is written plain, with no
marker and no color, because a marker would corrupt a table or a
parsed document. This covers the `version`, `info`, and `remote info`
reports, the `snapshot list` table, `config get` values, and every
`--json` document. `--quiet` silences the human reports and tables
(`version`, `info`, `remote info`, `snapshot list`) but never the
machine-consumed `config get` value or the `--json` documents, which a
script depends on. The `database delete` confirmation prompt is also
written this way and always shown: it is an interactive exchange the
operator must see.
`internal/ui` writes to stdout; it is the output the user asked for. `internal/ui` writes to stdout; it is the output the user asked for.
Structured log records are a different thing and go through Structured log records are a different thing and go through
+15
View File
@@ -25,6 +25,21 @@ release" is exactly the contradiction
# Completed Steps # Completed Steps
- 2026-09-22: Routed the last direct-to-stdout command output through
`internal/ui`
([issue #149](https://git.eeqj.de/sneak/vaultik/issues/149)). The
`version`, `info`, `remote info`, `config`, and `database delete`
commands wrote plain text straight to stdout, so they were unstyled and
ignored `--quiet`. Output now falls in two buckets, both governed by
`internal/ui`: status lines and confirmations go through its message
methods (styled, and `--quiet` silences them), while the data a command
exists to produce — the reports, the `snapshot list` table, `config get`
values, and the `--json` documents — is written plain. `--quiet`
silences the human reports and tables but never the `config get` value
or the `--json` documents, which a script depends on, and the
`database delete` confirmation prompt is always shown. The README
output-style section now states this rule.
- 2026-09-22: Validated blob hashes, offsets and lengths read back from - 2026-09-22: Validated blob hashes, offsets and lengths read back from
the destination before using them the destination before using them
([issue #155](https://git.eeqj.de/sneak/vaultik/issues/155)). A blob ([issue #155](https://git.eeqj.de/sneak/vaultik/issues/155)). A blob
+3
View File
@@ -306,6 +306,9 @@ storage_url: "rclone://myremote/path/to/backups"
# Multiple chunks are packed into blobs up to this size # Multiple chunks are packed into blobs up to this size
# Must be at least four times chunk_size (the largest chunk the chunker can # Must be at least four times chunk_size (the largest chunk the chunker can
# emit); a smaller limit would let a single-chunk blob exceed it. # emit); a smaller limit would let a single-chunk blob exceed it.
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
# not visible in its size; lowering the limit toward chunk_size exposes them.
# Supports: 1GB, 10G, 500MB, 1GiB, etc. # Supports: 1GB, 10G, 500MB, 1GiB, etc.
# Default: 10GB # Default: 10GB
#blob_size_limit: 10GB #blob_size_limit: 10GB
+2 -2
View File
@@ -90,7 +90,7 @@ Stores information about packed, compressed, and encrypted blob files.
**Columns:** **Columns:**
- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts - `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts
- `blob_hash` (TEXT UNIQUE) - SHA256 hash of final blob (NULL until finalized) - `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (NULL until finalized); see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs)
- `created_ts` (INTEGER NOT NULL) - Creation timestamp - `created_ts` (INTEGER NOT NULL) - Creation timestamp
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress) - `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression - `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
@@ -216,7 +216,7 @@ After a snapshot is completed:
5. Upload to S3 as `metadata/{remote-key}/db.zst.age` 5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst` 6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation). The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext. The hash uses no secret, so a guessed hostname and snapshot name can still be confirmed against a listing; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its [Accepted Risks](REPOSTRUCTURE.md#accepted-risks).
### 4. Restore Process ### 4. Restore Process
+18 -5
View File
@@ -35,7 +35,7 @@ The metadata subdirectory is named with the **remote key**, a one-way hash of th
- **What it contains**: Packed collections of content-defined chunks from files - **What it contains**: Packed collections of content-defined chunks from files
- **Format**: Zstandard compressed, then Age encrypted - **Format**: Zstandard compressed, then Age encrypted
- **Encryption**: Always encrypted with Age using the configured recipients - **Encryption**: Always encrypted with Age using the configured recipients
- **Naming**: Content-addressed using SHA256 hash of the encrypted blob - **Naming**: Content-addressed. The blob's name is `hex(SHA256(SHA256(uncompressed blob contents)))` — the double SHA-256 of the concatenated chunk data, computed before compression and encryption, not a hash of the stored (compressed, encrypted) bytes. One consequence: only a holder of the age private key can check a stored blob's integrity, because matching a blob to its name means decrypting and decompressing it first — which is what `restore` and `verify --deep` do. Implemented in `internal/blobgen` (`DoubleSHA256`). This is the canonical description of blob naming; other documents and comments point here.
### Why Encrypted ### Why Encrypted
Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy. Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy.
@@ -59,14 +59,14 @@ This ID reveals the hostname, the configured snapshot name, and the backup time,
### Remote Key Derivation ### Remote Key Derivation
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string with no structure a remote observer can reverse. Implemented in `internal/snapshot/remotekey.go`. The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string. The hash is not reversible, but it uses no secret: an observer who guesses a candidate hostname and snapshot name can hash it the same way and confirm whether that snapshot is present. The remote key keeps names out of a plain listing; it does not hide them from a guess. Implemented in `internal/snapshot/remotekey.go`.
Worked example: Worked example:
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z` - Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa` - Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/` - Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
Because the hash is one-way, a listing of the destination store reveals neither the hostname nor the snapshot name of any backup. The same remote key is stored in the manifest's `snapshot_id` field. A plain listing of the destination store therefore shows only these hashes, not the hostname or snapshot name of any backup — but because the hash uses no secret, a guessed hostname and snapshot name can be hashed and confirmed against the listing. The same remote key is stored in the manifest's `snapshot_id` field.
### Files in Each Snapshot Directory ### Files in Each Snapshot Directory
@@ -124,23 +124,36 @@ From the unencrypted data, an observer of the destination store can determine:
- **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear - **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear
- How many blobs each snapshot references, and the total compressed size - How many blobs each snapshot references, and the total compressed size
- The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns) - The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)
- **Whether a guessed hostname and snapshot name are present** — the remote key is an unkeyed hash, so an observer holding candidate names can hash each one and match it against the directory listing. The human ID is never published, so it cannot be read off directly, but it can be confirmed by guessing.
Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless. Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless.
An observer cannot determine: An observer cannot determine:
- The hostname or snapshot name of any backup (the directory name and the manifest `snapshot_id` are one-way hashes of the human ID) - The hostname or snapshot name of any backup by reading it off the store — the directory name and the manifest `snapshot_id` are unkeyed hashes of the human ID, so the text is never published (though a guessed name can be confirmed, as above)
- File names or paths - File names or paths
- File contents - File contents
- File permissions or ownership - File permissions or ownership
- Directory structure - Directory structure
- Which chunks belong to which files - Which chunks belong to which files
### Accepted Risks
These are known, deliberate properties of the format and the tooling, recorded so an operator can weigh them rather than discover them.
1. **No proof of authorship.** Restore and `verify --deep` prove that data decrypts with the age private key and matches its unkeyed content hashes. They do not prove who wrote it: anyone who knows a recipient public key and can replace objects on the destination can substitute a snapshot they built. The recipient string is not stored at the destination, but a compromised backed-up host has it. Defences live on the destination side — bucket versioning or object lock, credentials for the source host that cannot delete or overwrite existing versions, and pruning from a trusted host. Note that S3 `PutObject` overwrites an existing key, so PUT permission alone is not append-only.
2. **Compression reveals sizes.** Blobs and `db.zst.age` are zstd-compressed then age-encrypted; the manifest is compressed only. age does not pad, so an object's size is the exact compressed length of its contents. All new chunks packed into one blob share a single zstd stream (8 MiB window, 4 MiB at compression levels 1-2), and a blob is closed at `blob_size_limit` and at the end of each configured path. Because a stored chunk is never packed again, someone who can write into a backed-up file and watch blob sizes learns something only when their controlled data and a secret land in the same chunk of a file that keeps changing. Advice: back up any outsider-writable directory as its own snapshot.
3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of 2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them.
4. **Decrypted data on local disk.** Several commands stage plaintext under `$TMPDIR`: `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` and removed on success, but a `kill -9` or a power loss leaves them behind — delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` should be trusted to the same degree as the restore target.
5. **Store permissions differ per command.** The backed-up host needs only PUT to run `snapshot create`: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — `snapshot verify`, `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`, `snapshot remove`, and `remote nuke` also delete. The recommended cron line uses `--prune`, which runs `prune` on the backed-up host, so granting that host `--prune` gives it credentials that can delete its own backups. To keep the source host to PUT only, prune from a separate trusted host instead.
6. **Changing recipients does not re-encrypt existing data.** Deduplicated chunks and same-named blobs already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run `vaultik database delete` and take a full backup to a fresh destination or prefix.
7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients. Long-lived ciphertext held by a third party (the destination operator) has no fallback if X25519 is ever broken: there is no second recipient type and no post-quantum option.
## Consistency Guarantees ## Consistency Guarantees
1. **Blobs are immutable** - Once written, a blob is never modified 1. **Blobs are immutable** - Once written, a blob is never modified
2. **Blobs are written before metadata** - A snapshot's metadata is only written after all its blobs are successfully uploaded 2. **Blobs are written before metadata** - A snapshot's metadata is only written after all its blobs are successfully uploaded
3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst are written as complete files 3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst are written as complete files
4. **Snapshots are marked complete in local DB only after metadata upload** - Ensures consistency between local and remote state 4. **A snapshot is marked complete in the local DB only after its metadata is uploaded** - `finalizeSnapshotMetadata` runs `ExportSnapshotMetadata` first and records completion (`MarkSnapshotComplete`) only once the export succeeds (see the backup data flow in [ARCHITECTURE.md](../ARCHITECTURE.md)). A crash during the export therefore leaves the snapshot incomplete, so the next backup's `PruneDatabase` drops it and re-backs-up its data, rather than leaving a completed-looking row in the local index with no matching metadata on the destination store. (A crash in the brief moment after the export succeeds but before completion is recorded leaves a fully-restorable snapshot on the destination that the local index drops as incomplete on the next run; `snapshot list` then reports it honestly as remote-only, which is the safe direction: the destination copy stays restorable.)
## Pruning Safety ## Pruning Safety
+8 -6
View File
@@ -1,14 +1,16 @@
// Package blob handles the creation of blobs - the final storage units for Vaultik. // Package blob handles the creation of blobs - the final storage units for Vaultik.
// A blob is a large file (up to 10GB) containing many compressed and encrypted chunks // A blob is a large file (up to 10GB) containing many compressed and encrypted chunks
// from multiple source files. Blobs are content-addressed, meaning their filename // from multiple source files. Blobs are content-addressed: a blob's filename is
// is derived from the SHA256 hash of their compressed and encrypted content. // hex(SHA256(SHA256(uncompressed blob contents))), computed from the concatenated
// chunk data before compression and encryption, not from the stored bytes. See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
// //
// The blob creation process: // The blob creation process:
// 1. Chunks are accumulated from multiple files // 1. Chunks are accumulated from multiple files
// 2. The collection is compressed using zstd // 2. Each chunk's uncompressed bytes are fed to a running SHA-256 and, in the same
// 3. The compressed data is encrypted using age // pass, compressed with zstd and encrypted with age into the temp file
// 4. The encrypted blob is hashed to create its content-addressed name // 3. On finalize, the name is the double SHA-256 of that uncompressed content
// 5. The blob is uploaded to S3 using the hash as the filename // 4. The blob is uploaded to S3 using the name as the filename
// //
// This design optimizes storage efficiency by batching many small chunks into // This design optimizes storage efficiency by batching many small chunks into
// larger blobs, reducing the number of S3 operations and associated costs. // larger blobs, reducing the number of S3 operations and associated costs.
+49
View File
@@ -0,0 +1,49 @@
package blobgen
import (
"errors"
"io"
)
// ErrOutputTooLarge is returned by a reader from LimitReader once it has
// been asked for more than its limit. It bounds how far an untrusted
// compressed stream may expand, so a small, highly compressible object
// from the store cannot decompress without limit.
var ErrOutputTooLarge = errors.New("output exceeds size limit")
// LimitReader returns a reader that yields at most limit bytes from r and
// then fails with ErrOutputTooLarge. Unlike io.LimitReader, which reports
// a silent io.EOF at the limit (indistinguishable from a stream that
// simply ended), this fails, so a caller decoding or copying the stream
// sees an error rather than a truncated value. A stream of exactly limit
// bytes reads back cleanly to EOF; the first byte beyond it is the error.
func LimitReader(r io.Reader, limit int64) io.Reader {
// remaining counts down from limit+1: the extra byte is the one that,
// if it ever arrives, proves the stream is longer than the limit.
return &limitReader{r: r, remaining: limit + 1}
}
type limitReader struct {
r io.Reader
remaining int64
}
func (l *limitReader) Read(p []byte) (int, error) {
if l.remaining <= 0 {
return 0, ErrOutputTooLarge
}
if int64(len(p)) > l.remaining {
p = p[:l.remaining]
}
n, err := l.r.Read(p)
l.remaining -= int64(n)
if l.remaining <= 0 {
// The (limit+1)th byte was just read: the stream is too long.
return n, ErrOutputTooLarge
}
return n, err
}
+43
View File
@@ -0,0 +1,43 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestLimitReaderPassesExactSize checks that a stream of exactly the limit
// reads back cleanly to EOF: the bound must not reject a legitimate blob
// whose plaintext equals its recorded size.
func TestLimitReaderPassesExactSize(t *testing.T) {
t.Parallel()
const n = 1000
r := blobgen.LimitReader(bytes.NewReader(bytes.Repeat([]byte("a"), n)), n)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.Len(t, got, n)
}
// TestLimitReaderFailsPastLimit feeds a large, highly compressible run of
// zeros — the decompressed output a zip bomb would produce — through a
// small limit and checks it fails within the bound rather than passing
// the whole stream through.
func TestLimitReaderFailsPastLimit(t *testing.T) {
t.Parallel()
const limit = 1000
r := blobgen.LimitReader(
bytes.NewReader(bytes.Repeat([]byte{0}, limit*1000)), limit)
n, err := io.Copy(io.Discard, r)
require.ErrorIs(t, err, blobgen.ErrOutputTooLarge)
require.LessOrEqual(t, n, int64(limit)+1,
"reader must stop within one byte of the limit")
}
+17
View File
@@ -2,6 +2,7 @@ package blobgen
import ( import (
"crypto/sha256" "crypto/sha256"
"errors"
"fmt" "fmt"
"hash" "hash"
"io" "io"
@@ -56,6 +57,22 @@ func (r *Reader) Read(p []byte) (int, error) {
n, err := r.teeReader.Read(p) n, err := r.teeReader.Read(p)
r.bytesRead += int64(n) r.bytesRead += int64(n)
// When the ciphertext is cut right after the age header plus its
// 16-byte nonce, the age reader's first read fails with
// io.ErrUnexpectedEOF, and the zstd decoder maps that to a clean
// io.EOF at frame start. That makes a truncated stream look like a
// valid empty one. Distinguish the two: on EOF, read once more from
// the age reader. A genuine end leaves it at (0, io.EOF); a truncated
// stream leaves its stored io.ErrUnexpectedEOF, which we surface.
if errors.Is(err, io.EOF) {
var probe [1]byte
m, ageErr := r.decryptor.Read(probe[:])
if m != 0 || !errors.Is(ageErr, io.EOF) {
return n, io.ErrUnexpectedEOF
}
}
return n, err return n, err
} }
+70
View File
@@ -0,0 +1,70 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestReaderRejectsHeaderNonceTruncation guards against a stream cut right
// after the age header plus its 16-byte nonce. age.Decrypt still succeeds on
// such an object, and the zstd decoder maps the age reader's
// io.ErrUnexpectedEOF to a clean io.EOF at frame start, so without the extra
// check the truncated stream would read as a valid empty one. Reading it must
// now fail.
func TestReaderRejectsHeaderNonceTruncation(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
// Encrypting empty plaintext yields header + nonce(16) + a single
// 16-byte final chunk tag. Dropping the trailing tag leaves exactly the
// age header plus its nonce — the truncation point that triggers the bug.
var full bytes.Buffer
w, err := age.Encrypt(&full, identity.Recipient())
require.NoError(t, err)
require.NoError(t, w.Close())
truncated := full.Bytes()[:full.Len()-16]
reader, err := blobgen.NewReader(bytes.NewReader(truncated), identity)
require.NoError(t, err)
defer func() { _ = reader.Close() }()
_, err = io.ReadAll(reader)
require.Error(t, err)
require.ErrorIs(t, err, io.ErrUnexpectedEOF)
}
// TestReaderReadsGenuinelyEmptyBlob confirms the truncation check does not
// reject a legitimately empty payload: a blob written with no data must round
// trip back to zero bytes with no error.
func TestReaderReadsGenuinelyEmptyBlob(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
var encrypted bytes.Buffer
writer, err := blobgen.NewWriter(
&encrypted, 3, []string{identity.Recipient().String()})
require.NoError(t, err)
require.NoError(t, writer.Close())
reader, err := blobgen.NewReader(bytes.NewReader(encrypted.Bytes()), identity)
require.NoError(t, err)
defer func() { _ = reader.Close() }()
data, err := io.ReadAll(reader)
require.NoError(t, err)
require.Empty(t, data)
}
+13 -7
View File
@@ -16,11 +16,17 @@ import (
) )
// DoubleSHA256 returns the double SHA-256 of content whose single SHA-256 // DoubleSHA256 returns the double SHA-256 of content whose single SHA-256
// digest is sum: it hashes that digest once more. Stored objects are named by // digest is sum: it hashes that digest once more. Stored objects — a blob, and
// this second hash so that a name never reveals whether known content is // the metadata database export — are named by this second hash.
// present — an attacker who knows a plaintext, and thus its SHA-256, still //
// cannot derive the stored name without hashing the digest again. Both a blob // The second hash does not hide whether known content is stored: an attacker
// and the metadata database export are named this way. // who can reproduce an object's entire plaintext computes the same name simply
// by hashing twice, exactly as this code does. What limits that is blob
// packing, not the double hash — a blob's name covers all of its concatenated
// chunk plaintext, so a name can be confirmed only by someone who can
// reproduce the whole blob (a snapshot made entirely of known content, or a
// known file large enough to fill blobs on its own). An ordinary file that
// shares a blob with other, unknown data cannot be confirmed this way.
func DoubleSHA256(sum []byte) []byte { func DoubleSHA256(sum []byte) []byte {
h := sha256.Sum256(sum) h := sha256.Sum256(sum)
@@ -147,8 +153,8 @@ func (w *Writer) Close() error {
// ContentID returns the double SHA-256 of the uncompressed input data: the // ContentID returns the double SHA-256 of the uncompressed input data: the
// name under which this content is stored. It is the second hash of the // name under which this content is stored. It is the second hash of the
// running SHA-256, via DoubleSHA256; see that function for why content is // running SHA-256, via DoubleSHA256; see that function for what naming content
// named this way rather than by its plain SHA-256. // this way does and does not hide.
func (w *Writer) ContentID() []byte { func (w *Writer) ContentID() []byte {
return DoubleSHA256(w.hasher.Sum(nil)) return DoubleSHA256(w.hasher.Sum(nil))
} }
+30 -19
View File
@@ -4,7 +4,6 @@ import (
"bytes" "bytes"
"errors" "errors"
"fmt" "fmt"
"io"
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
@@ -13,6 +12,7 @@ import (
"github.com/spf13/cobra" "github.com/spf13/cobra"
"gopkg.in/yaml.v3" "gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/ui"
) )
// configFileMode is the permission set for freshly written config files; // configFileMode is the permission set for freshly written config files;
@@ -46,8 +46,11 @@ const defaultConfigTemplate = `# vaultik configuration
# ─── REQUIRED ──────────────────────────────────────────────────────────────── # ─── REQUIRED ────────────────────────────────────────────────────────────────
# Age recipient public keys for encryption. # Age recipient public keys for encryption.
# Backups are encrypted to ALL listed recipients. Any one of the corresponding # Backups are encrypted to ALL listed recipients; any one of the corresponding
# private keys can decrypt. Generate a keypair with: # private keys can decrypt. Adding a recipient later does not re-encrypt data
# already stored: deduplicated chunks and existing blobs stay encrypted to the
# earlier recipients, so a newly added key cannot restore them on its own (see
# docs/REPOSTRUCTURE.md, Accepted Risks). Generate a keypair with:
# age-keygen -o vaultik_backup_private_key.txt # age-keygen -o vaultik_backup_private_key.txt
# grep 'public key' vaultik_backup_private_key.txt # grep 'public key' vaultik_backup_private_key.txt
age_recipients: age_recipients:
@@ -262,7 +265,7 @@ The config is written to the path from --config, $VAULTIK_CONFIG, or
the platform default config directory (e.g. ~/Library/Application Support/ the platform default config directory (e.g. ~/Library/Application Support/
on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`, on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`,
Args: cobra.NoArgs, Args: cobra.NoArgs,
RunE: func(_ *cobra.Command, _ []string) error { RunE: func(cmd *cobra.Command, _ []string) error {
path := configPathForInit() path := configPathForInit()
_, err := os.Stat(path) _, err := os.Stat(path)
@@ -282,8 +285,11 @@ on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`,
return fmt.Errorf("writing config file: %w", err) return fmt.Errorf("writing config file: %w", err)
} }
_, _ = fmt.Fprintf(os.Stdout, "Config written to %s\n", path) // A written-confirmation, not scriptable output: route it
_, _ = fmt.Fprintln(os.Stdout, // through the UI so it is styled and --quiet silences it.
out := commandUI(cmd)
out.Infof("Config written to %s.", path)
out.Infof(
"Edit it to set your age_recipients, snapshots, and storage_url.") "Edit it to set your age_recipients, snapshots, and storage_url.")
return nil return nil
@@ -325,7 +331,7 @@ func newConfigGetCommand() *cobra.Command {
Use: "get <key>", Use: "get <key>",
Short: "Print a config value by dotted path (e.g. storage_url, compression_level)", Short: "Print a config value by dotted path (e.g. storage_url, compression_level)",
Args: cobra.ExactArgs(1), Args: cobra.ExactArgs(1),
RunE: func(_ *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
path, err := ResolveConfigPath() path, err := ResolveConfigPath()
if err != nil { if err != nil {
return err return err
@@ -341,8 +347,13 @@ func newConfigGetCommand() *cobra.Command {
return err return err
} }
// The value is scriptable output: it must stay machine-plain
// (no marker, no color) and is never silenced by --quiet, so it
// is written straight to stdout rather than through the UI.
w := cmd.OutOrStdout()
if node.Kind == yaml.ScalarNode { if node.Kind == yaml.ScalarNode {
_, _ = fmt.Fprintln(os.Stdout, node.Value) _, _ = fmt.Fprintln(w, node.Value)
return nil return nil
} }
@@ -352,7 +363,7 @@ func newConfigGetCommand() *cobra.Command {
return fmt.Errorf("marshaling value: %w", err) return fmt.Errorf("marshaling value: %w", err)
} }
_, _ = fmt.Fprint(os.Stdout, string(out)) _, _ = fmt.Fprint(w, string(out))
return nil return nil
}, },
@@ -374,23 +385,23 @@ Examples:
vaultik config set compression_level 9 vaultik config set compression_level 9
vaultik config set s3.bucket mybucket # legacy S3 fields still supported`, vaultik config set s3.bucket mybucket # legacy S3 fields still supported`,
Args: cobra.ExactArgs(configSetArgs), Args: cobra.ExactArgs(configSetArgs),
RunE: func(_ *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
path, err := ResolveConfigPath() path, err := ResolveConfigPath()
if err != nil { if err != nil {
return err return err
} }
return writeConfigSet(os.Stdout, path, args[0], args[1]) return writeConfigSet(commandUI(cmd), path, args[0], args[1])
}, },
} }
} }
// writeConfigSet applies key=value to the config at path, writes it back // writeConfigSet applies key=value to the config at path, writes it back
// owner-only, and confirms the write by printing just the key name to w. // owner-only, and confirms the write by naming just the key through the
// The value is never echoed: it may be a secret such as // UI writer (styled, and silenced by --quiet). The value is never
// s3.secret_access_key, and captured stdout or a pasted terminal would // echoed: it may be a secret such as s3.secret_access_key, and captured
// then leak it. // stdout or a pasted terminal would then leak it.
func writeConfigSet(w io.Writer, path, key, value string) error { func writeConfigSet(out *ui.Writer, path, key, value string) error {
root, err := loadYAMLFile(path) root, err := loadYAMLFile(path)
if err != nil { if err != nil {
return err return err
@@ -401,12 +412,12 @@ func writeConfigSet(w io.Writer, path, key, value string) error {
return err return err
} }
out, err := marshalConfigYAML(root) data, err := marshalConfigYAML(root)
if err != nil { if err != nil {
return fmt.Errorf("marshaling config: %w", err) return fmt.Errorf("marshaling config: %w", err)
} }
err = os.WriteFile(path, out, configFileMode) err = os.WriteFile(path, data, configFileMode)
if err != nil { if err != nil {
return fmt.Errorf("writing config file: %w", err) return fmt.Errorf("writing config file: %w", err)
} }
@@ -422,7 +433,7 @@ func writeConfigSet(w io.Writer, path, key, value string) error {
} }
} }
_, _ = fmt.Fprintln(w, key) out.Infof("Set %s.", key)
return nil return nil
} }
+11 -8
View File
@@ -9,6 +9,7 @@ import (
"gopkg.in/yaml.v3" "gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/ui"
) )
// TestDefaultConfigTemplateParses ensures the init template is valid YAML // TestDefaultConfigTemplateParses ensures the init template is valid YAML
@@ -246,19 +247,20 @@ func TestWriteConfigSetHidesSecret(t *testing.T) {
t.Fatalf("seed config: %v", err) t.Fatalf("seed config: %v", err)
} }
var out bytes.Buffer var buf bytes.Buffer
err = writeConfigSet(&out, path, "s3.secret_access_key", secret) err = writeConfigSet(ui.NewWithColor(&buf, false), path,
"s3.secret_access_key", secret)
if err != nil { if err != nil {
t.Fatalf("writeConfigSet: %v", err) t.Fatalf("writeConfigSet: %v", err)
} }
if strings.Contains(out.String(), secret) { if strings.Contains(buf.String(), secret) {
t.Errorf("output echoed the secret value: %q", out.String()) t.Errorf("output echoed the secret value: %q", buf.String())
} }
if !strings.Contains(out.String(), "s3.secret_access_key") { if !strings.Contains(buf.String(), "s3.secret_access_key") {
t.Errorf("output did not confirm the key name: %q", out.String()) t.Errorf("output did not confirm the key name: %q", buf.String())
} }
} }
@@ -277,9 +279,10 @@ func TestWriteConfigSetTightensMode(t *testing.T) {
t.Fatalf("seed config: %v", err) t.Fatalf("seed config: %v", err)
} }
var out bytes.Buffer var buf bytes.Buffer
err = writeConfigSet(&out, path, "compression_level", "9") err = writeConfigSet(ui.NewWithColor(&buf, false), path,
"compression_level", "9")
if err != nil { if err != nil {
t.Fatalf("writeConfigSet: %v", err) t.Fatalf("writeConfigSet: %v", err)
} }
+12 -11
View File
@@ -48,7 +48,7 @@ storage destination on that run.
Use --force to skip the confirmation prompt.`, Use --force to skip the confirmation prompt.`,
Args: cobra.NoArgs, Args: cobra.NoArgs,
RunE: func(_ *cobra.Command, _ []string) error { RunE: func(cmd *cobra.Command, _ []string) error {
// Resolve config path // Resolve config path
configPath, err := ResolveConfigPath() configPath, err := ResolveConfigPath()
if err != nil { if err != nil {
@@ -62,26 +62,31 @@ Use --force to skip the confirmation prompt.`,
} }
dbPath := cfg.IndexPath dbPath := cfg.IndexPath
out := commandUI(cmd)
// Check if database exists // Check if database exists
_, err = os.Stat(dbPath) _, err = os.Stat(dbPath)
if os.IsNotExist(err) { if os.IsNotExist(err) {
_, _ = fmt.Fprintf(os.Stdout, "Database does not exist: %s\n", dbPath) out.Infof("Local state database does not exist: %s.", dbPath)
return nil return nil
} }
// Confirm unless --force // Confirm unless --force. The prompt and its immediate result
// are an interactive exchange the operator must see, so they go
// straight to stdout rather than through the UI and --quiet does
// not silence them.
if !force { if !force {
_, _ = fmt.Fprintf(os.Stdout, w := cmd.OutOrStdout()
_, _ = fmt.Fprintf(w,
"This will delete the local state database at:\n %s\n\n", dbPath) "This will delete the local state database at:\n %s\n\n", dbPath)
_, _ = fmt.Fprint(os.Stdout, "Are you sure? Type 'yes' to confirm: ") _, _ = fmt.Fprint(w, "Are you sure? Type 'yes' to confirm: ")
var confirm string var confirm string
_, err = fmt.Scanln(&confirm) _, err = fmt.Scanln(&confirm)
if err != nil || confirm != "yes" { if err != nil || confirm != "yes" {
_, _ = fmt.Fprintln(os.Stdout, "Aborted.") _, _ = fmt.Fprintln(w, "Aborted.")
//nolint:nilerr // a failed/aborted confirmation is a clean abort //nolint:nilerr // a failed/aborted confirmation is a clean abort
return nil return nil
@@ -100,11 +105,7 @@ Use --force to skip the confirmation prompt.`,
_ = os.Remove(walPath) // Ignore errors - files may not exist _ = os.Remove(walPath) // Ignore errors - files may not exist
_ = os.Remove(shmPath) _ = os.Remove(shmPath)
rootFlags := GetRootFlags() out.Infof("Local state database deleted: %s.", dbPath)
if !rootFlags.Quiet {
_, _ = fmt.Fprintf(os.Stdout, "Database deleted: %s\n", dbPath)
}
log.Info("Local state database deleted", "path", dbPath) log.Info("Local state database deleted", "path", dbPath)
return nil return nil
+206
View File
@@ -0,0 +1,206 @@
package cli //nolint:testpackage // sets the unexported rootFlags directly
import (
"bytes"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/spf13/cobra"
)
// setRootFlags overrides the global rootFlags for the duration of one
// test and restores it afterward. These tests must not run in parallel:
// the flags are process-global, so the whole struct is saved and put
// back rather than left mutated for the next test.
func setRootFlags(t *testing.T, f RootFlags) {
t.Helper()
old := rootFlags
rootFlags = f
t.Cleanup(func() { rootFlags = old })
}
// seedFile writes content to a fresh file and returns its path.
func seedFile(t *testing.T, dir, name, content string) string {
t.Helper()
path := filepath.Join(dir, name)
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("seeding %s: %v", name, err)
}
return path
}
// mustExecute runs a command with its output captured and fails the test
// if it errors, returning what the command printed.
func mustExecute(t *testing.T, cmd *cobra.Command, args ...string) string {
t.Helper()
var out bytes.Buffer
cmd.SetOut(&out)
cmd.SetArgs(args)
err := cmd.Execute()
if err != nil {
t.Fatalf("%s failed: %v", cmd.Name(), err)
}
return out.String()
}
// TestVersionQuietSuppressesReport checks that --quiet silences the whole
// version report: it is human-facing output, not a scriptable value.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestVersionQuietSuppressesReport(t *testing.T) {
setRootFlags(t, RootFlags{Quiet: true})
out := mustExecute(t, NewVersionCommand())
if out != "" {
t.Errorf("--quiet version printed %q, want nothing", out)
}
}
// TestConfigGetIgnoresQuiet checks that a config value is printed even
// under --quiet: it is scriptable output a caller depends on, so --quiet
// must not suppress it, and it stays machine-plain (no marker, no color).
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigGetIgnoresQuiet(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "storage_url: file:///mnt/x\n")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigGetCommand(), "storage_url")
if out != "file:///mnt/x\n" {
t.Errorf("config get --quiet = %q, want the plain value", out)
}
}
// TestConfigSetQuietSuppressesConfirmation checks that --quiet silences
// the confirmation line while still writing the value to the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigSetQuietSuppressesConfirmation(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "compression_level: 3\n")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigSetCommand(), "compression_level", "9")
if out != "" {
t.Errorf("--quiet config set printed %q, want nothing", out)
}
data, err := os.ReadFile(path) //nolint:gosec // G304: test-controlled path
if err != nil {
t.Fatalf("reading config back: %v", err)
}
if !strings.Contains(string(data), "compression_level: 9") {
t.Errorf("config set did not write the value under --quiet:\n%s", data)
}
}
// TestConfigSetConfirmsWhenNotQuiet checks that the confirmation names
// the key (styled) when --quiet is not set.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigSetConfirmsWhenNotQuiet(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "compression_level: 3\n")
setRootFlags(t, RootFlags{ConfigPath: path})
out := mustExecute(t, newConfigSetCommand(), "compression_level", "9")
if !strings.Contains(out, "compression_level") {
t.Errorf("config set did not confirm the key: %q", out)
}
}
// TestConfigInitQuietSuppressesConfirmation checks that --quiet silences
// the "config written" confirmation while still writing the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigInitQuietSuppressesConfirmation(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "new-config.yml")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigInitCommand())
if out != "" {
t.Errorf("--quiet config init printed %q, want nothing", out)
}
_, err := os.Stat(path)
if err != nil {
t.Errorf("config init did not write the file under --quiet: %v", err)
}
}
// seedDatabaseDeleteConfig writes a valid config whose index_path is a
// seeded database file, and returns both paths.
func seedDatabaseDeleteConfig(t *testing.T, dir string) (string, string) {
t.Helper()
dbPath := seedFile(t, dir, "index.sqlite", "not-a-real-db")
cfg := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"), filepath.Join(dir, "store"), dbPath)
cfgPath := seedFile(t, dir, "config.yml", cfg)
return dbPath, cfgPath
}
// TestDatabaseDeleteQuietSuppressesMessage checks that --quiet silences
// the "database deleted" line while still removing the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestDatabaseDeleteQuietSuppressesMessage(t *testing.T) {
dir := t.TempDir()
dbPath, cfgPath := seedDatabaseDeleteConfig(t, dir)
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: cfgPath})
out := mustExecute(t, newDatabaseDeleteCommand(), "--force")
if out != "" {
t.Errorf("--quiet database delete printed %q, want nothing", out)
}
_, err := os.Stat(dbPath)
if !os.IsNotExist(err) {
t.Errorf("database delete did not remove the file: stat err = %v", err)
}
}
// TestDatabaseDeleteReportsWhenNotQuiet checks that the deletion is
// reported when --quiet is not set.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestDatabaseDeleteReportsWhenNotQuiet(t *testing.T) {
dir := t.TempDir()
_, cfgPath := seedDatabaseDeleteConfig(t, dir)
setRootFlags(t, RootFlags{ConfigPath: cfgPath})
out := mustExecute(t, newDatabaseDeleteCommand(), "--force")
if !strings.Contains(out, "deleted") {
t.Errorf("database delete did not report the deletion: %q", out)
}
}
+15
View File
@@ -9,6 +9,7 @@ import (
"github.com/adrg/xdg" "github.com/adrg/xdg"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"sneak.berlin/go/vaultik/internal/ui"
) )
// errConfigNotFound is wrapped by all config-resolution failures. // errConfigNotFound is wrapped by all config-resolution failures.
@@ -81,6 +82,20 @@ func GetRootFlags() RootFlags {
return rootFlags return rootFlags
} }
// commandUI returns a UI writer for a command's stdout, in quiet mode
// when the global --quiet flag is set. This is how the pure-cli
// commands (version, config, database) reach internal/ui: color follows
// the writer (a TTY gets color, a captured test buffer does not), and
// --quiet silences the same message classes it silences everywhere else.
func commandUI(cmd *cobra.Command) *ui.Writer {
w := ui.New(cmd.OutOrStdout())
if GetRootFlags().Quiet {
w.SetQuiet(true)
}
return w
}
// ResolveConfigPath resolves the config file path from flags, environment, or default. // ResolveConfigPath resolves the config file path from flags, environment, or default.
// Search order: --config flag, VAULTIK_CONFIG env, XDG config dir, // Search order: --config flag, VAULTIK_CONFIG env, XDG config dir,
// /etc/vaultik/config.yml. // /etc/vaultik/config.yml.
+2 -1
View File
@@ -192,7 +192,8 @@ func newSnapshotVerifyCommand() *cobra.Command {
Long: "Checks that every blob the snapshot's manifest lists is present\n" + Long: "Checks that every blob the snapshot's manifest lists is present\n" +
"in storage with the size the manifest records, and that the\n" + "in storage with the size the manifest records, and that the\n" +
"snapshot's encrypted database is present. It does not read blob\n" + "snapshot's encrypted database is present. It does not read blob\n" +
"contents; use --deep to download and cryptographically verify them.\n\n" + "contents; use --deep to download, decrypt, and re-hash every blob\n" +
"to detect corruption -- integrity, not who wrote it.\n\n" +
"The snapshot may be named by its ID or, on a host with no local\n" + "The snapshot may be named by its ID or, on a host with no local\n" +
"index, by the remote key that 'snapshot list' prints for a\n" + "index, by the remote key that 'snapshot list' prints for a\n" +
"remote-only snapshot (an unambiguous leading part is enough).", "remote-only snapshot (an unambiguous leading part is enough).",
+14 -5
View File
@@ -2,11 +2,11 @@ package cli
import ( import (
"fmt" "fmt"
"io"
"runtime" "runtime"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"sneak.berlin/go/vaultik/internal/globals" "sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/ui"
) )
// NewVersionCommand creates the version command // NewVersionCommand creates the version command
@@ -17,16 +17,25 @@ func NewVersionCommand() *cobra.Command {
Long: `Print version, git commit, and build information for vaultik.`, Long: `Print version, git commit, and build information for vaultik.`,
Args: cobra.NoArgs, Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) { Run: func(cmd *cobra.Command, _ []string) {
writeVersion(cmd.OutOrStdout()) writeVersion(commandUI(cmd))
}, },
} }
return cmd return cmd
} }
// writeVersion prints the version report. It takes a writer rather than // writeVersion prints the version report through the UI writer. The
// using os.Stdout directly so the output can be asserted on in tests. // report is the output this command exists to produce, so it is written
func writeVersion(w io.Writer) { // plain (markers would corrupt the aligned report) via the writer's
// underlying stdout; --quiet silences it like any other non-error
// output.
func writeVersion(out *ui.Writer) {
if out.Quiet() {
return
}
w := out.Out()
_, _ = fmt.Fprintf(w, "vaultik %s\n", globals.Version) _, _ = fmt.Fprintf(w, "vaultik %s\n", globals.Version)
_, _ = fmt.Fprintf(w, " commit: %s\n", globals.Commit) _, _ = fmt.Fprintf(w, " commit: %s\n", globals.Commit)
_, _ = fmt.Fprintf(w, " build date: %s\n", globals.CommitDate) _, _ = fmt.Fprintf(w, " build date: %s\n", globals.CommitDate)
+4 -4
View File
@@ -34,9 +34,9 @@ func runVersionCommand(t *testing.T) string {
// the report is the version the binary was actually built with. The // the report is the version the binary was actually built with. The
// test binary carries no -ldflags, so that is the "dev" default -- the // test binary carries no -ldflags, so that is the "dev" default -- the
// same string an untagged `make vaultik` build stamps a prefix of. // same string an untagged `make vaultik` build stamps a prefix of.
//
//nolint:paralleltest // executes a command that reads the global rootFlags
func TestVersionCommandReportsBuildVersion(t *testing.T) { func TestVersionCommandReportsBuildVersion(t *testing.T) {
t.Parallel()
out := runVersionCommand(t) out := runVersionCommand(t)
wantFirst := "vaultik " + globals.Version wantFirst := "vaultik " + globals.Version
@@ -55,9 +55,9 @@ func TestVersionCommandReportsBuildVersion(t *testing.T) {
// being exactly "dev", so once untagged builds started carrying their // being exactly "dev", so once untagged builds started carrying their
// commit sha it would have gone silent and an unreleased binary would // commit sha it would have gone silent and an unreleased binary would
// have looked like a release. // have looked like a release.
//
//nolint:paralleltest // executes a command that reads the global rootFlags
func TestVersionCommandFlagsDevelopmentBuild(t *testing.T) { func TestVersionCommandFlagsDevelopmentBuild(t *testing.T) {
t.Parallel()
if !globals.IsDevVersion(globals.Version) { if !globals.IsDevVersion(globals.Version) {
t.Skipf("test binary was stamped with release version %q", t.Skipf("test binary was stamped with release version %q",
globals.Version) globals.Version)
+4 -2
View File
@@ -3,8 +3,10 @@
// //
// Blobs in Vaultik are the final storage units uploaded to S3. Each blob is a // Blobs in Vaultik are the final storage units uploaded to S3. Each blob is a
// large (up to 10GB) file containing many compressed and encrypted chunks from // large (up to 10GB) file containing many compressed and encrypted chunks from
// multiple source files. Blobs are content-addressed, meaning their filename // multiple source files. Blobs are content-addressed: the filename in S3 is
// is derived from their SHA256 hash after compression and encryption. // hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data
// before compression and encryption (not from the stored bytes). See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
// //
// Schema is managed via numbered SQL migrations embedded in the schema/ // Schema is managed via numbered SQL migrations embedded in the schema/
// directory. Migration 000.sql bootstraps the schema_migrations tracking // directory. Migration 000.sql bootstraps the schema_migrations tracking
+6 -6
View File
@@ -51,15 +51,15 @@ type Chunk struct {
// Blob represents a blob record in the database. // Blob represents a blob record in the database.
// A blob is Vaultik's final storage unit - a large file (up to 10GB) containing // A blob is Vaultik's final storage unit - a large file (up to 10GB) containing
// many compressed and encrypted chunks from multiple source files. // many compressed and encrypted chunks from multiple source files.
// Blobs are content-addressed, meaning their filename in S3 is derived from // Blobs are content-addressed: the filename in S3 is
// the SHA256 hash of their compressed and encrypted content. // hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data
// The blob creation process is: chunks are accumulated -> compressed with zstd // before compression and encryption (not from the stored bytes). See
// -> encrypted with age -> hashed -> uploaded to S3 with the hash as filename. // blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
type Blob struct { type Blob struct {
ID types.BlobID // UUID assigned when blob creation starts ID types.BlobID // UUID assigned when blob creation starts
// Hash is the SHA256 of the final compressed+encrypted content // Hash is hex(SHA256(SHA256(uncompressed blob contents)))
// (empty until finalized). // (empty until finalized); see the type comment above.
Hash types.BlobHash Hash types.BlobHash
CreatedTS time.Time // When blob creation started CreatedTS time.Time // When blob creation started
FinishedTS *time.Time // When blob was finalized (nil if still packing) FinishedTS *time.Time // When blob was finalized (nil if still packing)
+39
View File
@@ -0,0 +1,39 @@
package log_test
import (
"bytes"
"log/slog"
"strings"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
)
// TestTTYHandlerEscapesControlCharacters logs a message and an attribute
// value that each carry an ESC and a newline — the shape a crafted path or
// storage error from the destination would take — and checks neither raw
// byte reaches the output. The handler's own colour codes (ESC ... m) are
// stripped first; any ESC left after that came from the untrusted value.
func TestTTYHandlerEscapesControlCharacters(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
logger := slog.New(log.NewTTYHandler(&buf, debugHandlerOptions()))
logger.Info("start\x1b[31mZAP\nend", "target", "a\x1b[31mZAP\nb")
out := buf.String()
// The only newline is the line terminator; the injected ones were escaped.
require.Equal(t, 1, strings.Count(out, "\n"),
"a newline in the message or a value must be escaped, not emitted raw")
// After the handler's own colour codes are removed, no ESC survives.
stripped := ansiEscape.ReplaceAllString(out, "")
require.NotContains(t, stripped, "\x1b",
"a raw ESC from the message or a value must not reach the terminal")
// The escaped form is what appears instead.
require.Contains(t, out, `\x1b`)
}
+29 -4
View File
@@ -5,9 +5,11 @@ import (
"fmt" "fmt"
"io" "io"
"log/slog" "log/slog"
"strconv"
"strings" "strings"
"sync" "sync"
"time" "time"
"unicode"
) )
// groupSeparator joins an open group path to an attribute key. This // groupSeparator joins an open group path to an attribute key. This
@@ -116,11 +118,14 @@ func (h *TTYHandler) Handle(_ context.Context, r slog.Record) error {
levelColor = colorReset levelColor = colorReset
} }
// Print main message // Print main message. The message is escaped before the colour codes
// are written around it: it can carry text from an untrusted source
// (a storage error, for one), and a raw control character would
// otherwise reach the terminal.
_, _ = fmt.Fprintf(h.out, "%s%s%s %s%s%s %s%s%s", _, _ = fmt.Fprintf(h.out, "%s%s%s %s%s%s %s%s%s",
colorGray, timestamp, colorReset, colorGray, timestamp, colorReset,
levelColor, level, colorReset, levelColor, level, colorReset,
colorBold, r.Message, colorReset) colorBold, sanitize(r.Message), colorReset)
// Attributes carried by the handler come first, then the record's // Attributes carried by the handler come first, then the record's
// own. Handler attributes were qualified when they were added; the // own. Handler attributes were qualified when they were added; the
@@ -260,9 +265,29 @@ func (h *TTYHandler) writeAttr(a slog.Attr) {
// Future kinds also use the plain string form. // Future kinds also use the plain string form.
} }
// Escape the key and value before the colour codes are written around
// them. Both can carry text from an untrusted source — a manifest
// timestamp, a storage error, a path or symlink target read back from
// the snapshot database — so a control character in one of them must
// be rendered as an escape sequence rather than reaching the terminal,
// where it could move the cursor or inject its own colours.
_, _ = fmt.Fprintf(h.out, " %s%s%s=%s%s%s", _, _ = fmt.Fprintf(h.out, " %s%s%s=%s%s%s",
colorCyan, a.Key, colorReset, colorCyan, sanitize(a.Key), colorReset,
colorBlue, value, colorReset) colorBlue, sanitize(value), colorReset)
}
// sanitize returns s unchanged when every rune in it is printable, and a
// double-quoted, backslash-escaped form (\n, \x1b, …) otherwise. It is
// applied to untrusted text before any colour code is written, so a
// control character can never reach the terminal raw.
func sanitize(s string) string {
for _, r := range s {
if !unicode.IsPrint(r) {
return strconv.Quote(s)
}
}
return s
} }
// formatDuration formats a duration in a human-readable way // formatDuration formats a duration in a human-readable way
+30 -5
View File
@@ -7,6 +7,19 @@ import (
"io" "io"
"github.com/klauspost/compress/zstd" "github.com/klauspost/compress/zstd"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// Manifest size bounds. A manifest lists one small entry per blob, and
// blobs are large (the default target is 10 GB), so even a manifest for a
// petabyte-scale backup is a few megabytes. These caps are far above any
// manifest the writer can emit, yet stop a crafted, highly compressible
// manifest from expanding without limit when decoded: the manifest is
// fetched from the store, which is not trusted, and json.Decode buffers
// the whole value in memory.
const (
manifestMaxCompressed = 256 * 1024 * 1024 // 256 MiB
manifestMaxDecompressed = 1024 * 1024 * 1024 // 1 GiB
) )
// Manifest represents the structure of a snapshot's blob manifest // Manifest represents the structure of a snapshot's blob manifest
@@ -28,19 +41,31 @@ type BlobInfo struct {
CompressedSize int64 `json:"compressed_size"` CompressedSize int64 `json:"compressed_size"`
} }
// DecodeManifest decodes a manifest from a reader containing compressed JSON // DecodeManifest decodes a manifest from a reader containing compressed
// JSON, reading through byte limits on both the compressed input and the
// decompressed output so an untrusted manifest cannot exhaust memory.
func DecodeManifest(r io.Reader) (*Manifest, error) { func DecodeManifest(r io.Reader) (*Manifest, error) {
// Decompress using zstd return decodeManifest(r, manifestMaxCompressed, manifestMaxDecompressed)
zr, err := zstd.NewReader(r) }
// decodeManifest is DecodeManifest with explicit limits, so tests can drive
// the bounds with small inputs instead of gigabyte-scale ones.
func decodeManifest(
r io.Reader, maxCompressed, maxDecompressed int64,
) (*Manifest, error) {
// Decompress using zstd, bounding how many compressed bytes are read.
zr, err := zstd.NewReader(blobgen.LimitReader(r, maxCompressed))
if err != nil { if err != nil {
return nil, fmt.Errorf("creating zstd reader: %w", err) return nil, fmt.Errorf("creating zstd reader: %w", err)
} }
defer zr.Close() defer zr.Close()
// Decode JSON manifest // Decode JSON manifest, bounding how far the compressed input may
// expand: json.Decode buffers the whole value, so without this a
// small, highly compressible manifest could expand to gigabytes.
var manifest Manifest var manifest Manifest
err = json.NewDecoder(zr).Decode(&manifest) err = json.NewDecoder(blobgen.LimitReader(zr, maxDecompressed)).Decode(&manifest)
if err != nil { if err != nil {
return nil, fmt.Errorf("decoding manifest: %w", err) return nil, fmt.Errorf("decoding manifest: %w", err)
} }
+79
View File
@@ -0,0 +1,79 @@
//nolint:testpackage // exercises the unexported decodeManifest bounds
package snapshot
import (
"bytes"
"strings"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// testSnapshotID is a stand-in snapshot ID reused across the bound cases.
const testSnapshotID = "host_home_2026-01-01T00:00:00Z"
// TestDecodeManifestRoundTrip is the baseline: with generous bounds a
// manifest the writer produced decodes back unchanged.
func TestDecodeManifestRoundTrip(t *testing.T) {
t.Parallel()
want := &Manifest{
SnapshotID: testSnapshotID,
Timestamp: "2026-01-01T00:00:00Z",
BlobCount: 2,
TotalCompressedSize: 42,
Blobs: []BlobInfo{
{Hash: "aa", CompressedSize: 21},
{Hash: "bb", CompressedSize: 21},
},
}
compressed, err := EncodeManifest(want, 3)
require.NoError(t, err)
got, err := decodeManifest(
bytes.NewReader(compressed), manifestMaxCompressed, manifestMaxDecompressed)
require.NoError(t, err)
require.Equal(t, want, got)
}
// TestDecodeManifestBoundsDecompressedOutput feeds a valid but highly
// compressible manifest — one whose timestamp is a megabyte of the same
// character — through a small decompressed bound. The compressed form is
// tiny, so only the decompressed bound stops it; decoding must fail within
// that bound rather than expanding the value in memory.
func TestDecodeManifestBoundsDecompressedOutput(t *testing.T) {
t.Parallel()
bomb := &Manifest{
SnapshotID: testSnapshotID,
Timestamp: strings.Repeat("a", 1<<20),
}
compressed, err := EncodeManifest(bomb, 3)
require.NoError(t, err)
require.Less(t, len(compressed), 4096,
"the compressible manifest must be small compressed")
_, err = decodeManifest(bytes.NewReader(compressed), 1<<20, 4096)
require.ErrorIs(t, err, blobgen.ErrOutputTooLarge)
}
// TestDecodeManifestBoundsCompressedInput checks the compressed-input
// bound fires independently: a valid manifest with a generous decompressed
// bound but a tiny compressed bound still fails.
func TestDecodeManifestBoundsCompressedInput(t *testing.T) {
t.Parallel()
manifest := &Manifest{
SnapshotID: testSnapshotID,
Timestamp: strings.Repeat("a", 4096),
}
compressed, err := EncodeManifest(manifest, 3)
require.NoError(t, err)
_, err = decodeManifest(bytes.NewReader(compressed), 8, manifestMaxDecompressed)
require.Error(t, err)
}
+1 -1
View File
@@ -119,7 +119,7 @@ type ScannerConfig struct {
Storage storage.Storer Storage storage.Storer
MaxBlobSize int64 MaxBlobSize int64
CompressionLevel int CompressionLevel int
AgeRecipients []string // Optional, empty means no encryption AgeRecipients []string // required; output is always encrypted
EnableProgress bool // Enable the live progress reporter (ETAs, throughput) EnableProgress bool // Enable the live progress reporter (ETAs, throughput)
UI *ui.Writer // Where user-facing scanner messages go; nil = discard UI *ui.Writer // Where user-facing scanner messages go; nil = discard
Exclude []string // Glob patterns for files/directories to exclude Exclude []string // Glob patterns for files/directories to exclude
+58 -19
View File
@@ -24,7 +24,7 @@ package snapshot
// 7. Close the temporary database // 7. Close the temporary database
// 8. VACUUM the database to remove deleted data and compact (security critical) // 8. VACUUM the database to remove deleted data and compact (security critical)
// 9. Compress the binary database with zstd // 9. Compress the binary database with zstd
// 10. Encrypt the compressed database with age (if encryption is enabled) // 10. Encrypt the compressed database with age (always; recipients are required)
// 11. Upload to S3 as: metadata/{snapshot-id}/db.zst.age // 11. Upload to S3 as: metadata/{snapshot-id}/db.zst.age
// 12. Reopen the main database // 12. Reopen the main database
// //
@@ -201,11 +201,14 @@ func (sm *SnapshotManager) UpdateSnapshotStatsExtended(
}) })
} }
// CompleteSnapshot marks a snapshot as completed and ensures snapshot_blobs // PopulateSnapshotBlobs ensures snapshot_blobs holds an entry for every
// is populated with every blob holding any chunk referenced by the // blob that stores a chunk referenced by the snapshot's files, including
// snapshot's files (including deduplicated blobs uploaded by prior // blobs deduplicated from earlier snapshots. Without it, a fully
// snapshots). Without this, fully-deduplicated snapshots are unrestorable. // deduplicated snapshot would record no blobs and be unrestorable.
func (sm *SnapshotManager) CompleteSnapshot( //
// This must run before ExportSnapshotMetadata: the blob manifest and the
// trimmed metadata database are both built from snapshot_blobs.
func (sm *SnapshotManager) PopulateSnapshotBlobs(
ctx context.Context, snapshotID string, ctx context.Context, snapshotID string,
) error { ) error {
err := sm.repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error { err := sm.repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
@@ -219,6 +222,25 @@ func (sm *SnapshotManager) CompleteSnapshot(
"snapshot_id", snapshotID, "added", added) "snapshot_id", snapshotID, "added", added)
} }
return nil
})
if err != nil {
return fmt.Errorf("populating snapshot blobs: %w", err)
}
return nil
}
// MarkSnapshotComplete records the snapshot's completion timestamp. On the
// backup path this runs only after ExportSnapshotMetadata has succeeded, so
// the local index never marks a snapshot complete while the destination
// holds no manifest or database for it. A crash before this point leaves the
// snapshot incomplete, and the next run's PruneDatabase drops it. See
// https://git.eeqj.de/sneak/vaultik/issues/177.
func (sm *SnapshotManager) MarkSnapshotComplete(
ctx context.Context, snapshotID string,
) error {
err := sm.repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
return sm.repos.Snapshots.MarkComplete(ctx, tx, snapshotID) return sm.repos.Snapshots.MarkComplete(ctx, tx, snapshotID)
}) })
if err != nil { if err != nil {
@@ -230,6 +252,22 @@ func (sm *SnapshotManager) CompleteSnapshot(
return nil return nil
} }
// CompleteSnapshot populates snapshot_blobs and then marks the snapshot
// complete. The backup path (finalizeSnapshotMetadata) instead calls the two
// halves separately, with the metadata export between them, so completion is
// recorded only after a successful export. This convenience is for callers
// that do not interleave an export.
func (sm *SnapshotManager) CompleteSnapshot(
ctx context.Context, snapshotID string,
) error {
err := sm.PopulateSnapshotBlobs(ctx, snapshotID)
if err != nil {
return err
}
return sm.MarkSnapshotComplete(ctx, snapshotID)
}
// ExportSnapshotMetadata exports snapshot metadata to S3 // ExportSnapshotMetadata exports snapshot metadata to S3
// //
// This method executes the complete snapshot metadata export process: // This method executes the complete snapshot metadata export process:
@@ -238,14 +276,12 @@ func (sm *SnapshotManager) CompleteSnapshot(
// 3. Cleans the copy to contain only current snapshot data // 3. Cleans the copy to contain only current snapshot data
// 4. Dumps the cleaned database to SQL // 4. Dumps the cleaned database to SQL
// 5. Compresses the SQL dump with zstd // 5. Compresses the SQL dump with zstd
// 6. Encrypts the compressed data (if encryption is enabled) // 6. Encrypts the compressed data with age (always; recipients are required)
// 7. Uploads to S3 at: snapshots/{snapshot-id}.sql.zst[.age] // 7. Uploads to S3 at: snapshots/{snapshot-id}.sql.zst[.age]
// //
// The caller is responsible for: // The only caller (finalizeSnapshotMetadata) does not close the main database
// - Ensuring the main database is closed before calling this method // before calling this method: the index is copied at dbPath while it is still
// - Reopening the main database after this method returns // open, and every step here operates on that copy, never on the live index.
//
// This ensures database consistency during the copy operation.
func (sm *SnapshotManager) ExportSnapshotMetadata( func (sm *SnapshotManager) ExportSnapshotMetadata(
ctx context.Context, dbPath string, snapshotID string, ctx context.Context, dbPath string, snapshotID string,
) error { ) error {
@@ -415,9 +451,11 @@ func (sm *SnapshotManager) prepareExportDB(
// uploadSnapshotArtifacts uploads the database backup and blob manifest // uploadSnapshotArtifacts uploads the database backup and blob manifest
// to remote storage at metadata/<remote-key>/, where remote-key is the // to remote storage at metadata/<remote-key>/, where remote-key is the
// double-SHA256 derivation of the snapshot ID (see RemoteSnapshotKey). // double-SHA256 derivation of the snapshot ID (see RemoteSnapshotKey).
// We never write the human-readable snapshot ID into any unencrypted // The human-readable snapshot ID is never written into an unencrypted part
// part of remote storage so a listing of the destination bucket leaks // of remote storage, so a plain listing shows only the hashed key, not the
// no host, configuration, or scheduling information. // hostname or snapshot name. The hash uses no secret, so a guessed hostname
// and snapshot name can still be confirmed against a listing, and the backup
// time is public: the manifest carries a plaintext timestamp.
func (sm *SnapshotManager) uploadSnapshotArtifacts( func (sm *SnapshotManager) uploadSnapshotArtifacts(
ctx context.Context, snapshotID string, dbData, manifestData []byte, ctx context.Context, snapshotID string, dbData, manifestData []byte,
) error { ) error {
@@ -814,10 +852,11 @@ func (sm *SnapshotManager) generateBlobManifest(
} }
// Create manifest. SnapshotID in the unencrypted manifest is the // Create manifest. SnapshotID in the unencrypted manifest is the
// double-SHA256 remote key (see RemoteSnapshotKey), not the human ID, // double-SHA256 remote key (see RemoteSnapshotKey), not the human ID, so
// so neither this field nor the directory name reveals the hostname or // neither this field nor the directory name spells out the hostname or
// snapshot name. Timestamp below is written in the clear, so the backup // snapshot name — but the key uses no secret, so a guessed hostname and
// time is observable to anyone who can read the manifest. // snapshot name can be confirmed. Timestamp below is written in the clear,
// so the backup time is observable to anyone who can read the manifest.
manifest := &Manifest{ manifest := &Manifest{
SnapshotID: RemoteSnapshotKey(snapshotID), SnapshotID: RemoteSnapshotKey(snapshotID),
Timestamp: time.Now().UTC().Format(time.RFC3339), Timestamp: time.Now().UTC().Format(time.RFC3339),
+4 -2
View File
@@ -146,8 +146,10 @@ type SnapshotID string
// Used for content-addressing and deduplication of file chunks. // Used for content-addressing and deduplication of file chunks.
type ChunkHash string type ChunkHash string
// BlobHash is the SHA256 hash of a blob's compressed and encrypted content. // BlobHash is hex(SHA256(SHA256(uncompressed blob contents))), computed before
// This is used as the filename in S3 storage for content-addressed retrieval. // compression and encryption (see blobgen.DoubleSHA256 and
// docs/REPOSTRUCTURE.md). It is used as the filename in S3 storage for
// content-addressed retrieval.
type BlobHash string type BlobHash string
// FilePath represents an absolute path to a file or directory. // FilePath represents an absolute path to a file or directory.
+22 -3
View File
@@ -23,7 +23,9 @@ import (
"fmt" "fmt"
"io" "io"
"os" "os"
"strconv"
"time" "time"
"unicode"
"github.com/dustin/go-humanize" "github.com/dustin/go-humanize"
"golang.org/x/term" "golang.org/x/term"
@@ -225,17 +227,17 @@ func (w *Writer) Hex(s string) string {
short = s[:hexAbbrevLen] + "..." short = s[:hexAbbrevLen] + "..."
} }
return w.paint(ansiCyan, short) return w.paint(ansiCyan, sanitize(short))
} }
// Snapshot colorizes a snapshot ID (full, no abbreviation). // Snapshot colorizes a snapshot ID (full, no abbreviation).
func (w *Writer) Snapshot(id string) string { func (w *Writer) Snapshot(id string) string {
return w.paint(ansiCyan+ansiBold, id) return w.paint(ansiCyan+ansiBold, sanitize(id))
} }
// Path colorizes a filesystem path. // Path colorizes a filesystem path.
func (w *Writer) Path(p string) string { func (w *Writer) Path(p string) string {
return w.paint(ansiBlue, p) return w.paint(ansiBlue, sanitize(p))
} }
// Size colorizes a byte count using humanize.Bytes. // Size colorizes a byte count using humanize.Bytes.
@@ -310,6 +312,23 @@ func (w *Writer) paint(color, s string) string {
return color + s + ansiReset return color + s + ansiReset
} }
// sanitize returns s unchanged when every rune in it is printable, and a
// double-quoted, backslash-escaped form (\n, \x1b, …) otherwise. The
// string value formatters escape their argument through this before
// painting: identifiers, paths and symlink targets they render come from
// the snapshot database, which is not trusted, and escaping must happen
// before colour is applied — the painted result already contains the
// escape codes the raw text would otherwise be indistinguishable from.
func sanitize(s string) string {
for _, r := range s {
if !unicode.IsPrint(r) {
return strconv.Quote(s)
}
}
return s
}
// emit writes "<prefix> <body>\n" with the prefix painted in prefixColor // emit writes "<prefix> <body>\n" with the prefix painted in prefixColor
// and the body optionally painted in bodyColor (empty = no body color). // and the body optionally painted in bodyColor (empty = no body color).
func (w *Writer) emit(prefixColor, prefix, bodyColor, format string, args []any) { func (w *Writer) emit(prefixColor, prefix, bodyColor, format string, args []any) {
+32
View File
@@ -0,0 +1,32 @@
package ui_test
import (
"strings"
"testing"
)
// TestValueFormattersEscapeControlCharacters checks that a path carrying an
// ESC and a newline — the shape a symlink target read back from the
// snapshot database could take — is escaped before it reaches the output.
// Colour is off here, so the only way a control byte could appear is from
// the value itself.
func TestValueFormattersEscapeControlCharacters(t *testing.T) {
t.Parallel()
w, buf := newTestWriter(false)
w.Infof("restoring %s", w.Path("a\x1b[31mZAP\nb"))
out := buf.String()
if strings.ContainsRune(out, '\x1b') {
t.Fatalf("raw ESC from a value survived in output: %q", out)
}
if strings.Count(out, "\n") != 1 {
t.Fatalf("a newline in a value must be escaped, not emitted raw: %q", out)
}
if !strings.Contains(out, `\x1b`) {
t.Fatalf("expected the escaped form of ESC in output: %q", out)
}
}
+16 -34
View File
@@ -6,11 +6,9 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"time"
"filippo.io/age" "filippo.io/age"
"sneak.berlin/go/vaultik/internal/blobgen" "sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/log"
) )
// errBlobHashMismatch is returned when a fetched blob's content hash does // errBlobHashMismatch is returned when a fetched blob's content hash does
@@ -30,13 +28,14 @@ var errBlobNotFullyRead = errors.New(
// redundant SHA-256 computation. // redundant SHA-256 computation.
type hashVerifyReader struct { type hashVerifyReader struct {
reader *blobgen.Reader // underlying decrypted blob reader (has internal hasher) reader *blobgen.Reader // underlying decrypted blob reader (has internal hasher)
limited io.Reader // reader bounded to the blob's recorded plaintext size
fetcher io.ReadCloser // raw fetched stream (closed on Close) fetcher io.ReadCloser // raw fetched stream (closed on Close)
blobHash string // expected double-SHA-256 hex blobHash string // expected double-SHA-256 hex
done bool // EOF reached done bool // EOF reached
} }
func (h *hashVerifyReader) Read(p []byte) (int, error) { func (h *hashVerifyReader) Read(p []byte) (int, error) {
n, err := h.reader.Read(p) n, err := h.limited.Read(p)
if errors.Is(err, io.EOF) { if errors.Is(err, io.EOF) {
h.done = true h.done = true
} }
@@ -73,11 +72,17 @@ func (h *hashVerifyReader) Close() error {
// returns a streaming reader that computes the double-SHA-256 hash on the fly. // returns a streaming reader that computes the double-SHA-256 hash on the fly.
// The hash is verified when the returned reader is closed (after fully reading). // The hash is verified when the returned reader is closed (after fully reading).
// This avoids buffering the entire blob in memory. // This avoids buffering the entire blob in memory.
//
// maxPlaintextSize is the blob's uncompressed_size as recorded in the
// snapshot database. Decompression stops with blobgen.ErrOutputTooLarge
// once the plaintext exceeds it, so a tampered blob cannot expand without
// limit — using the recorded size, not the restoring host's
// blob_size_limit, since that config may differ from the backup host's.
func (v *Vaultik) FetchAndDecryptBlob( func (v *Vaultik) FetchAndDecryptBlob(
ctx context.Context, blobHash string, expectedSize int64, ctx context.Context, blobHash string, maxPlaintextSize int64,
identities ...age.Identity, identities ...age.Identity,
) (io.ReadCloser, error) { ) (io.ReadCloser, error) {
rc, _, err := v.FetchBlob(ctx, blobHash, expectedSize) rc, err := v.FetchBlob(ctx, blobHash)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -91,52 +96,29 @@ func (v *Vaultik) FetchAndDecryptBlob(
return &hashVerifyReader{ return &hashVerifyReader{
reader: reader, reader: reader,
limited: blobgen.LimitReader(reader, maxPlaintextSize),
fetcher: rc, fetcher: rc,
blobHash: blobHash, blobHash: blobHash,
}, nil }, nil
} }
// FetchBlob downloads a blob and returns a reader for the encrypted data. // FetchBlob downloads a blob and returns a reader for the encrypted data.
// Times the Storage.Get and Storage.Stat round-trips separately at
// debug level so we can see whether the size-only Stat (which is an
// extra request on every fetch) is hurting throughput.
func (v *Vaultik) FetchBlob( func (v *Vaultik) FetchBlob(
ctx context.Context, blobHash string, expectedSize int64, ctx context.Context, blobHash string,
) (io.ReadCloser, int64, error) { ) (io.ReadCloser, error) {
// blobHash reaches here from the snapshot database, which is not // blobHash reaches here from the snapshot database, which is not
// trusted. Reject a malformed hash before it is spliced into a storage // trusted. Reject a malformed hash before it is spliced into a storage
// path (blobHash[:2]/blobHash[2:4]) or a fetch is attempted. // path (blobHash[:2]/blobHash[2:4]) or a fetch is attempted.
if !isBlobHash(blobHash) { if !isBlobHash(blobHash) {
return nil, 0, fmt.Errorf("%w: %s", errInvalidBlobHash, shortHash(blobHash)) return nil, fmt.Errorf("%w: %s", errInvalidBlobHash, shortHash(blobHash))
} }
blobPath := fmt.Sprintf("blobs/%s/%s/%s", blobHash[:2], blobHash[2:4], blobHash) blobPath := fmt.Sprintf("blobs/%s/%s/%s", blobHash[:2], blobHash[2:4], blobHash)
t0 := time.Now()
rc, err := v.Storage.Get(ctx, blobPath) rc, err := v.Storage.Get(ctx, blobPath)
getDur := time.Since(t0)
if err != nil { if err != nil {
return nil, 0, fmt.Errorf("downloading blob %s: %w", shortHash(blobHash), err) return nil, fmt.Errorf("downloading blob %s: %w", shortHash(blobHash), err)
} }
t0 = time.Now() return rc, nil
info, err := v.Storage.Stat(ctx, blobPath)
statDur := time.Since(t0)
if err != nil {
_ = rc.Close()
return nil, 0, fmt.Errorf("stat blob %s: %w", shortHash(blobHash), err)
}
log.Debug("FetchBlob round-trips",
"hash", shortHash(blobHash),
"ms_storage_get", getDur.Milliseconds(),
"ms_storage_stat", statDur.Milliseconds(),
"expected_size", expectedSize,
"stat_size", info.Size,
)
return rc, info.Size, nil
} }
+50
View File
@@ -0,0 +1,50 @@
package vaultik_test
import (
"context"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestFetchAndDecryptBlobBoundsPlaintext feeds a small, highly
// compressible blob (256 KiB of zeros) whose decompressed size far exceeds
// the plaintext bound passed to FetchAndDecryptBlob. Decompression must
// stop with blobgen.ErrOutputTooLarge within the bound rather than
// expanding the whole blob into the restore cache.
func TestFetchAndDecryptBlobBoundsPlaintext(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
plaintext := make([]byte, 256*1024)
encryptedData, correctHash := buildHashTestBlob(t, identity, plaintext)
mockStorage := NewMockStorer()
blobPath := "blobs/" + correctHash[:2] + "/" +
correctHash[2:4] + "/" + correctHash
mockStorage.mu.Lock()
mockStorage.data[blobPath] = encryptedData
mockStorage.mu.Unlock()
tv := vaultik.NewForTesting(mockStorage)
const maxPlaintext = 1024
rc, err := tv.FetchAndDecryptBlob(
context.Background(), correctHash, maxPlaintext, identity)
require.NoError(t, err)
n, copyErr := io.Copy(io.Discard, rc)
_ = rc.Close()
require.ErrorIs(t, copyErr, blobgen.ErrOutputTooLarge)
require.LessOrEqual(t, n, int64(maxPlaintext)+1,
"decompression must stop within the recorded plaintext bound")
}
+4 -4
View File
@@ -73,7 +73,7 @@ func TestFetchBlobRejectsMalformedHash(t *testing.T) {
strings.Repeat("A", 64), // uppercase hex is not accepted strings.Repeat("A", 64), // uppercase hex is not accepted
strings.Repeat("g", 64), // not hex strings.Repeat("g", 64), // not hex
} { } {
_, _, err := tv.FetchBlob(ctx, bad, 0) _, err := tv.FetchBlob(ctx, bad)
if err == nil { if err == nil {
t.Fatalf("expected error for malformed hash %q, got nil", bad) t.Fatalf("expected error for malformed hash %q, got nil", bad)
} }
@@ -113,7 +113,7 @@ func TestFetchAndDecryptBlobVerifiesHash(t *testing.T) {
t.Parallel() t.Parallel()
rc, err := tv.FetchAndDecryptBlob( rc, err := tv.FetchAndDecryptBlob(
ctx, correctHash, int64(len(encryptedData)), identity) ctx, correctHash, int64(len(plaintext)), identity)
if err != nil { if err != nil {
t.Fatalf("expected success, got error: %v", err) t.Fatalf("expected success, got error: %v", err)
} }
@@ -145,7 +145,7 @@ func TestFetchAndDecryptBlobVerifiesHash(t *testing.T) {
mockStorage.mu.Unlock() mockStorage.mu.Unlock()
rc, err := tv.FetchAndDecryptBlob( rc, err := tv.FetchAndDecryptBlob(
ctx, fakeHash, int64(len(encryptedData)), identity) ctx, fakeHash, int64(len(plaintext)), identity)
if err != nil { if err != nil {
t.Fatalf("unexpected error opening stream: %v", err) t.Fatalf("unexpected error opening stream: %v", err)
} }
@@ -188,7 +188,7 @@ func TestFetchAndDecryptBlobCloseBeforeEOFFails(t *testing.T) {
tv := vaultik.NewForTesting(mockStorage) tv := vaultik.NewForTesting(mockStorage)
rc, err := tv.FetchAndDecryptBlob( rc, err := tv.FetchAndDecryptBlob(
context.Background(), correctHash, int64(len(encryptedData)), identity) context.Background(), correctHash, int64(len(plaintext)), identity)
if err != nil { if err != nil {
t.Fatalf("unexpected error opening stream: %v", err) t.Fatalf("unexpected error opening stream: %v", err)
} }
+185 -3
View File
@@ -14,6 +14,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot" "sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage" "sneak.berlin/go/vaultik/internal/storage"
@@ -417,9 +418,10 @@ func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
// database is uploaded but before the manifest. The destination is left // database is uploaded but before the manifest. The destination is left
// with blobs and a database but no manifest. verify and snapshot list // with blobs and a database but no manifest. verify and snapshot list
// must report the damage honestly rather than crashing or passing. // must report the damage honestly rather than crashing or passing.
// Automatic detection and repair of this partial state on the next run // Automatic detection and repair of this partial state on the next run is
// is tracked in https://git.eeqj.de/sneak/vaultik/issues/177 and is not // covered by TestBackupCompletesOnlyAfterMetadataExport
// asserted here. // (https://git.eeqj.de/sneak/vaultik/issues/177); this test exercises the
// lower-level export path in isolation.
// //
//nolint:paralleltest // installs the global logger via log.Initialize //nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupSurvivesMetadataExportInterruption(t *testing.T) { func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
@@ -496,6 +498,186 @@ func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
"snapshot list must tolerate a partially-exported snapshot") "snapshot list must tolerate a partially-exported snapshot")
} }
// Scenario 2, repair: the process dies during the metadata export of a
// full backup run. Because completion is recorded only after the export
// succeeds (finalizeSnapshotMetadata), the interrupted snapshot is left
// incomplete rather than silently marked complete without metadata at the
// destination. Rerunning the backup must then prune the incomplete
// snapshot, produce a snapshot whose destination metadata and local index
// agree, and restore. See https://git.eeqj.de/sneak/vaultik/issues/177.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupCompletesOnlyAfterMetadataExport(t *testing.T) {
log.Initialize(log.Config{})
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background()
testFiles := writeFaultSourceTree(t, fs, dataDir)
// A full-backup config: the fault-test defaults plus the fields the
// production create path reads (index location, chunk size, and the
// named snapshot to back up).
cfg := faultTestConfig()
cfg.IndexPath = dbPath
cfg.ChunkSize = config.Size(faultChunkSize)
cfg.Snapshots = map[string]config.SnapshotConfig{
"data": {Paths: []string{dataDir}},
}
inner, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
// failManifest is on for the first backup and off for the retry, so the
// manifest upload fails once — interrupting the export mid-way — then
// succeeds.
failManifest := true
store := faultstore.New(inner)
store.OnPut = func(key string) faultstore.PutAction {
if failManifest && strings.HasSuffix(key, "manifest.json.zst") {
return faultstore.PutFail
}
return faultstore.PutNormal
}
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
opts := &vaultik.SnapshotCreateOptions{Cron: true}
// First run: the export fails at the manifest upload, so the whole
// create fails and the snapshot is left incomplete.
require.Error(t, v.CreateSnapshot(opts),
"backup must fail when the metadata export is interrupted")
incompletes, err := repos.Snapshots.GetIncompleteSnapshots(ctx)
require.NoError(t, err)
require.Len(t, incompletes, 1,
"an interrupted export must leave exactly one incomplete snapshot")
afterFirst, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err)
for _, s := range afterFirst {
require.Nil(t, s.CompletedAt,
"no snapshot may be marked complete before its metadata is exported")
}
// Second run on the same index and destination: the retry succeeds.
failManifest = false
require.NoError(t, v.CreateSnapshot(opts),
"a retry after an interrupted export must succeed")
assertRetryConsistentAndRestorable(
ctx, t, cfg, inner, repos, db, fs, restoreDir, testFiles)
}
// assertRetryConsistentAndRestorable checks the end state after the retry
// backup in TestBackupCompletesOnlyAfterMetadataExport: the interrupted
// snapshot is pruned, exactly one completed snapshot remains, its metadata
// is at the destination, and it restores to the original tree.
func assertRetryConsistentAndRestorable(
ctx context.Context, t *testing.T, cfg *config.Config,
inner storage.Storer, repos *database.Repositories, db *database.DB,
fs afero.Fs, restoreDir string, testFiles map[string][]byte,
) {
t.Helper()
incompletes, err := repos.Snapshots.GetIncompleteSnapshots(ctx)
require.NoError(t, err)
assert.Empty(t, incompletes,
"the next run's prune must drop the interrupted snapshot")
local, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err)
require.Len(t, local, 1, "exactly one snapshot must remain after the retry")
final := local[0]
require.NotNil(t, final.CompletedAt, "the retry's snapshot must be complete")
// The destination and the local index agree: the completed snapshot has
// both its metadata objects at the destination.
key := snapshot.RemoteSnapshotKey(final.ID.String())
_, err = inner.Stat(ctx, "metadata/"+key+"/manifest.json.zst")
require.NoError(t, err, "the completed snapshot's manifest must be at the destination")
_, err = inner.Stat(ctx, "metadata/"+key+"/db.zst.age")
require.NoError(t, err, "the completed snapshot's database must be at the destination")
require.NoError(t, db.Close())
// The snapshot restores from the destination alone.
reader := newReaderVaultik(ctx, cfg, inner, nil, fs)
require.NoError(t, reader.Restore(&vaultik.RestoreOptions{
SnapshotID: final.ID.String(),
TargetDir: restoreDir,
Verify: true,
}), "the retry's snapshot must be restorable")
assertRestoredTree(t, fs, restoreDir, testFiles)
}
// listRecentTestLimit is a generous cap for the handful of snapshots these
// tests create when reading the local index directly.
const listRecentTestLimit = 100
// newBackupVaultik builds a Vaultik that runs the full create path
// (CreateSnapshot) writing through storer, wiring the same scanner factory
// and snapshot manager the production dependency graph provides.
func newBackupVaultik(
ctx context.Context, cfg *config.Config, storer storage.Storer,
repos *database.Repositories, db *database.DB, fs afero.Fs,
) *vaultik.Vaultik {
v := &vaultik.Vaultik{
Globals: &globals.Globals{Version: "v", Commit: "g"},
Config: cfg,
DB: db,
Repositories: repos,
Storage: storer,
SnapshotManager: newFaultSnapshotManager(fs, storer, cfg, repos),
ScannerFactory: faultScannerFactory(cfg, repos, storer),
Fs: fs,
Stdout: io.Discard,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
v.SetContext(ctx)
return v
}
// faultScannerFactory mirrors the production provideScannerFactory, binding
// the scanner to the given store, repositories, and config so a full
// create-path backup writes through the fault-injecting store.
func faultScannerFactory(
cfg *config.Config, repos *database.Repositories, storer storage.Storer,
) snapshot.ScannerFactory {
return func(params snapshot.ScannerParams) *snapshot.Scanner {
return snapshot.NewScanner(snapshot.ScannerConfig{
FS: params.Fs,
Storage: storer,
ChunkSize: faultChunkSize,
MaxBlobSize: faultMaxBlobSize,
CompressionLevel: cfg.CompressionLevel,
AgeRecipients: cfg.AgeRecipients,
Repositories: repos,
EnableProgress: params.EnableProgress,
UI: params.UI,
Exclude: params.Exclude,
SkipErrors: params.SkipErrors,
})
}
}
// Scenario 5: the restore target runs out of space mid-file. Restore // Scenario 5: the restore target runs out of space mid-file. Restore
// must fail with an out-of-space error, and must not leave a truncated // must fail with an out-of-space error, and must not leave a truncated
// file at the target path presenting as a complete restore. Restore // file at the target path presenting as a complete restore. Restore
+23 -6
View File
@@ -13,6 +13,14 @@ import (
// ShowInfo displays system and configuration information // ShowInfo displays system and configuration information
func (v *Vaultik) ShowInfo() error { func (v *Vaultik) ShowInfo() error {
// The info report is the output this command exists to produce, so it
// is written plain (markers would corrupt the aligned report) through
// the UI writer's stdout; --quiet silences it like any other
// non-error output.
if v.UI.Quiet() {
return nil
}
// System Information // System Information
v.stdoutf("=== System Information ===\n") v.stdoutf("=== System Information ===\n")
v.stdoutf("OS/Architecture: %s/%s\n", runtime.GOOS, runtime.GOARCH) v.stdoutf("OS/Architecture: %s/%s\n", runtime.GOOS, runtime.GOARCH)
@@ -213,7 +221,12 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
result.StorageType = storageInfo.Type result.StorageType = storageInfo.Type
result.StorageLocation = storageInfo.Location result.StorageLocation = storageInfo.Location
if !jsonOutput { // The human report is written only when it is neither the --json
// document (which needs stdout to itself) nor silenced by --quiet. The
// scan still runs in both cases so --json still gets a full result.
showText := !jsonOutput && !v.UI.Quiet()
if showText {
v.stdoutf("=== Remote Storage ===\n") v.stdoutf("=== Remote Storage ===\n")
v.stdoutf("Type: %s\n", storageInfo.Type) v.stdoutf("Type: %s\n", storageInfo.Type)
v.stdoutf("Location: %s\n", storageInfo.Location) v.stdoutf("Location: %s\n", storageInfo.Location)
@@ -226,7 +239,7 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
return err return err
} }
if !jsonOutput { if showText {
v.stdoutf("Downloading %d manifest(s)...\n", len(snapshotIDs)) v.stdoutf("Downloading %d manifest(s)...\n", len(snapshotIDs))
} }
@@ -234,7 +247,7 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
v.populateRemoteInfoResult(result, snapshotMetadata, snapshotIDs, referencedBlobs) v.populateRemoteInfoResult(result, snapshotMetadata, snapshotIDs, referencedBlobs)
err = v.scanRemoteBlobStorage(result, referencedBlobs, jsonOutput) err = v.scanRemoteBlobStorage(result, referencedBlobs, showText)
if err != nil { if err != nil {
return err return err
} }
@@ -252,7 +265,9 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
return enc.Encode(result) return enc.Encode(result)
} }
if showText {
v.printRemoteInfoTable(result) v.printRemoteInfoTable(result)
}
return nil return nil
} }
@@ -362,11 +377,13 @@ func (v *Vaultik) populateRemoteInfoResult(
} }
} }
// scanRemoteBlobStorage lists all blobs on remote and computes orphan stats // scanRemoteBlobStorage lists all blobs on remote and computes orphan
// stats. showText is true only when the human report is being printed
// (not --json, not --quiet), gating the progress line.
func (v *Vaultik) scanRemoteBlobStorage( func (v *Vaultik) scanRemoteBlobStorage(
result *RemoteInfoResult, referencedBlobs map[string]int64, jsonOutput bool, result *RemoteInfoResult, referencedBlobs map[string]int64, showText bool,
) error { ) error {
if !jsonOutput { if showText {
v.stdoutf("Scanning blobs...\n") v.stdoutf("Scanning blobs...\n")
} }
+30
View File
@@ -0,0 +1,30 @@
package vaultik_test
import (
"bytes"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestShowInfo_QuietSuppressesReport checks that --quiet silences the
// whole info report. The report is human-facing status, not a scriptable
// value, so under --quiet the command produces nothing (and touches none
// of its dependencies, which is why this minimal instance suffices).
func TestShowInfo_QuietSuppressesReport(t *testing.T) {
t.Parallel()
var out bytes.Buffer
v := &vaultik.Vaultik{
Stdout: &out,
UI: ui.NewWithColor(&out, false),
}
v.UI.SetQuiet(true)
require.NoError(t, v.ShowInfo())
require.Empty(t, out.String(),
"the info report must be suppressed under --quiet")
}
+42 -34
View File
@@ -1,7 +1,6 @@
package vaultik package vaultik
import ( import (
"bytes"
"context" "context"
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
@@ -49,6 +48,10 @@ var (
"restore loop ended with files still pending") "restore loop ended with files still pending")
errSnapshotDBMismatch = errors.New( errSnapshotDBMismatch = errors.New(
"decrypted database is not the requested snapshot") "decrypted database is not the requested snapshot")
// errEmptySnapshotDB is returned when the decrypted metadata database has
// zero length, which happens when the object was truncated or replaced
// with an empty payload. Rejected before any schema is built on it.
errEmptySnapshotDB = errors.New("decrypted snapshot database is empty")
) )
// snapshotDBFilename is the name the decrypted snapshot database is // snapshotDBFilename is the name the decrypted snapshot database is
@@ -428,7 +431,7 @@ func (s *restoreSession) downloadNextBlobSet(plan *restorePlan) (bool, error) {
return false, fmt.Errorf("%w: %s", errBlobMissingFromIndex, shortHash(hash)) return false, fmt.Errorf("%w: %s", errBlobMissingFromIndex, shortHash(hash))
} }
err := s.downloadBlobToCache(hash, blob.CompressedSize) err := s.downloadBlobToCache(hash, blob.CompressedSize, blob.UncompressedSize)
if err != nil { if err != nil {
return false, fmt.Errorf("downloading blob %s: %w", shortHash(hash), err) return false, fmt.Errorf("downloading blob %s: %w", shortHash(hash), err)
} }
@@ -655,32 +658,18 @@ func (v *Vaultik) downloadSnapshotDB(
defer func() { _ = reader.Close() }() defer func() { _ = reader.Close() }()
// Read all data // Decrypt and decompress straight from the storage stream, then stream
encryptedData, err := io.ReadAll(reader) // the plaintext to a temp file. Neither the encrypted bytes nor the
if err != nil { // decrypted database is ever held whole in memory; a snapshot database
return nil, "", fmt.Errorf("reading encrypted data: %w", err) // can be large.
} blobReader, err := blobgen.NewReader(reader, identities...)
log.Debug("Downloaded encrypted database",
"size", ubytes(int64(len(encryptedData))))
// Decrypt and decompress using blobgen.Reader
blobReader, err := blobgen.NewReader(bytes.NewReader(encryptedData), identities...)
if err != nil { if err != nil {
return nil, "", fmt.Errorf("creating decryption reader: %w", err) return nil, "", fmt.Errorf("creating decryption reader: %w", err)
} }
defer func() { _ = blobReader.Close() }() defer func() { _ = blobReader.Close() }()
// Read the binary SQLite database db, tempDir, err := v.materializeSnapshotDB(blobReader)
dbData, err := io.ReadAll(blobReader)
if err != nil {
return nil, "", fmt.Errorf("decrypting and decompressing: %w", err)
}
log.Debug("Decrypted database", "size", ubytes(int64(len(dbData))))
db, tempDir, err := v.materializeSnapshotDB(dbData)
if err != nil { if err != nil {
return nil, "", err return nil, "", err
} }
@@ -729,13 +718,15 @@ func (v *Vaultik) verifySnapshotDBIdentity(
return nil return nil
} }
// materializeSnapshotDB writes the decrypted snapshot database bytes into // materializeSnapshotDB streams the decrypted snapshot database into a
// a fresh private (0700) temp directory and opens the file read-only. On // fresh private (0700) temp directory and opens the file read-only. The
// any failure it removes the directory before returning, so no decrypted // database is copied through an io.Copy buffer rather than read whole into
// metadata is left on disk when the open is interrupted or the payload is // memory. On any failure it removes the directory before returning, so no
// damaged. On success the returned directory is the caller's to remove. // decrypted metadata is left on disk when the copy is interrupted or the
// payload is damaged. On success the returned directory is the caller's to
// remove.
func (v *Vaultik) materializeSnapshotDB( func (v *Vaultik) materializeSnapshotDB(
dbData []byte, dbReader io.Reader,
) (*database.DB, string, error) { ) (*database.DB, string, error) {
tempDir, err := afero.TempDir(v.Fs, "", "vaultik-restore-") tempDir, err := afero.TempDir(v.Fs, "", "vaultik-restore-")
if err != nil { if err != nil {
@@ -752,12 +743,29 @@ func (v *Vaultik) materializeSnapshotDB(
dbPath := filepath.Join(tempDir, snapshotDBFilename) dbPath := filepath.Join(tempDir, snapshotDBFilename)
err = afero.WriteFile(v.Fs, dbPath, dbData, restoreFileMode) dbFile, err := v.Fs.OpenFile(
dbPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, restoreFileMode)
if err != nil { if err != nil {
return nil, "", fmt.Errorf("writing database file: %w", err) return nil, "", fmt.Errorf("creating database file: %w", err)
} }
log.Debug("Created restore database", "path", dbPath) written, copyErr := io.Copy(dbFile, dbReader)
closeErr := dbFile.Close()
if copyErr != nil {
return nil, "", fmt.Errorf("writing database file: %w", copyErr)
}
if closeErr != nil {
return nil, "", fmt.Errorf("closing database file: %w", closeErr)
}
log.Debug("Created restore database", "path", dbPath, "size", ubytes(written))
// Reject an empty database before OpenReadOnly builds a schema on it.
if written == 0 {
return nil, "", errEmptySnapshotDB
}
db, err := database.OpenReadOnly(v.ctx, dbPath) db, err := database.OpenReadOnly(v.ctx, dbPath)
if err != nil { if err != nil {
@@ -1212,12 +1220,12 @@ func (s *restoreSession) writeFileChunks(
// size, which is what makes multi-GB blobs tractable on machines with // size, which is what makes multi-GB blobs tractable on machines with
// less RAM than the blob. // less RAM than the blob.
func (s *restoreSession) downloadBlobToCache( func (s *restoreSession) downloadBlobToCache(
blobHash string, expectedSize int64, blobHash string, compressedSize, uncompressedSize int64,
) error { ) error {
start := time.Now() start := time.Now()
t0 := time.Now() t0 := time.Now()
rc, err := s.v.FetchAndDecryptBlob(s.ctx, blobHash, expectedSize, s.identities...) rc, err := s.v.FetchAndDecryptBlob(s.ctx, blobHash, uncompressedSize, s.identities...)
fetchSetupDur := time.Since(t0) fetchSetupDur := time.Since(t0)
if err != nil { if err != nil {
@@ -1247,7 +1255,7 @@ func (s *restoreSession) downloadBlobToCache(
log.Debug("Streamed blob into disk cache", log.Debug("Streamed blob into disk cache",
"hash", blobHash[:16], "hash", blobHash[:16],
"compressed_bytes", expectedSize, "compressed_bytes", compressedSize,
"plaintext_bytes", written, "plaintext_bytes", written,
"ms_total", time.Since(start).Milliseconds(), "ms_total", time.Since(start).Milliseconds(),
"ms_fetch_setup", fetchSetupDur.Milliseconds(), "ms_fetch_setup", fetchSetupDur.Milliseconds(),
+37 -2
View File
@@ -1,13 +1,16 @@
package vaultik //nolint:testpackage // inspects unexported snapshot-db materialization package vaultik //nolint:testpackage // inspects unexported snapshot-db materialization
import ( import (
"bytes"
"context" "context"
"os" "os"
"path/filepath" "path/filepath"
"testing" "testing"
"filippo.io/age"
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
) )
@@ -37,7 +40,7 @@ func TestMaterializeSnapshotDBPrivateDir(t *testing.T) {
v := &Vaultik{ctx: context.Background(), Fs: afero.NewOsFs()} v := &Vaultik{ctx: context.Background(), Fs: afero.NewOsFs()}
db, dir, err := v.materializeSnapshotDB(dbData) db, dir, err := v.materializeSnapshotDB(bytes.NewReader(dbData))
require.NoError(t, err) require.NoError(t, err)
t.Cleanup(func() { t.Cleanup(func() {
@@ -55,6 +58,37 @@ func TestMaterializeSnapshotDBPrivateDir(t *testing.T) {
require.Error(t, err, "materialized snapshot database must be read-only") require.Error(t, err, "materialized snapshot database must be read-only")
} }
// TestMaterializeSnapshotDBRejectsCompleteEmptyStream proves the written == 0
// guard rejects a genuinely empty but complete metadata object: a real age
// header, nonce, and final tag encrypting zero plaintext bytes. The truncation
// case is stopped earlier by the reader (io.ErrUnexpectedEOF) and never reaches
// this branch, so it needs its own input. This complete stream decrypts to zero
// bytes with a clean EOF, passes the reader, and must be refused as empty rather
// than accepted as a valid zero-table database. Reverting the guard lets the
// empty file open as a fresh schema and the test fails.
func TestMaterializeSnapshotDBRejectsCompleteEmptyStream(t *testing.T) {
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
var stream bytes.Buffer
w, err := age.Encrypt(&stream, identity.Recipient())
require.NoError(t, err)
require.NoError(t, w.Close())
blobReader, err := blobgen.NewReader(bytes.NewReader(stream.Bytes()), identity)
require.NoError(t, err)
t.Cleanup(func() { _ = blobReader.Close() })
t.Setenv("TMPDIR", t.TempDir())
v := &Vaultik{ctx: context.Background(), Fs: afero.NewOsFs()}
_, _, err = v.materializeSnapshotDB(blobReader)
require.ErrorIs(t, err, errEmptySnapshotDB)
}
// TestMaterializeSnapshotDBRemovesDirOnOpenFailure proves a failed open // TestMaterializeSnapshotDBRemovesDirOnOpenFailure proves a failed open
// leaves no temp directory behind. // leaves no temp directory behind.
func TestMaterializeSnapshotDBRemovesDirOnOpenFailure(t *testing.T) { func TestMaterializeSnapshotDBRemovesDirOnOpenFailure(t *testing.T) {
@@ -64,7 +98,8 @@ func TestMaterializeSnapshotDBRemovesDirOnOpenFailure(t *testing.T) {
v := &Vaultik{ctx: context.Background(), Fs: afero.NewOsFs()} v := &Vaultik{ctx: context.Background(), Fs: afero.NewOsFs()}
_, _, err := v.materializeSnapshotDB([]byte("this is not a sqlite database")) _, _, err := v.materializeSnapshotDB(
bytes.NewReader([]byte("this is not a sqlite database")))
require.Error(t, err) require.Error(t, err)
entries, rerr := os.ReadDir(base) entries, rerr := os.ReadDir(base)
@@ -0,0 +1,85 @@
package vaultik_test
import (
"bytes"
"context"
"io"
"path/filepath"
"testing"
"filippo.io/age"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestRestoreRejectsTruncatedMetadataDB backs up a real tree, then replaces
// the snapshot's db.zst.age with a stream cut right after the age header and
// its 16-byte nonce. age.Decrypt still accepts such an object and the zstd
// decoder turns the truncated read into a clean EOF, so before the fix restore
// built a fresh empty schema and reported success. Restore must now fail with
// io.ErrUnexpectedEOF, the error the reader raises for a truncated object.
// Asserting that specific error pins the reader fix: without it the truncation
// yields an empty database, which the identity check rejects for an unrelated
// reason, and this test would pass anyway.
func TestRestoreRejectsTruncatedMetadataDB(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(64 * 1024)
maxBlobSize := int64(512 * 1024)
setupE2ESourceTree(t, fs, dataDir, chunkSize)
ctx := context.Background()
cfg, storer, snapshotID := runFileStorageBackup(
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
// Encrypting empty plaintext to the snapshot recipient yields
// header + nonce(16) + a single 16-byte final chunk tag. Dropping the
// trailing tag leaves exactly the age header plus its nonce — the
// truncation an attacker can write over metadata without any key.
recipient, err := age.ParseX25519Recipient(testAgePublicKey)
require.NoError(t, err)
var full bytes.Buffer
w, err := age.Encrypt(&full, recipient)
require.NoError(t, err)
require.NoError(t, w.Close())
truncated := full.Bytes()[:full.Len()-16]
dbKeyPath := filepath.Join(storeDir, "metadata",
snapshot.RemoteSnapshotKey(snapshotID), "db.zst.age")
require.NoError(t, afero.WriteFile(fs, dbKeyPath, truncated, 0o644))
restoreVaultik := &vaultik.Vaultik{
Config: cfg,
Storage: storer,
Fs: fs,
Stdout: io.Discard,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
restoreVaultik.SetContext(ctx)
err = restoreVaultik.Restore(&vaultik.RestoreOptions{
SnapshotID: snapshotID,
TargetDir: restoreDir,
Verify: true,
})
require.ErrorIs(t, err, io.ErrUnexpectedEOF)
}
+20 -3
View File
@@ -69,6 +69,9 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
// Prune the database before starting: delete incomplete snapshots and orphaned data. // Prune the database before starting: delete incomplete snapshots and orphaned data.
// This ensures the database is consistent before we start a new snapshot. // This ensures the database is consistent before we start a new snapshot.
// Since we use locking, only one vaultik instance accesses the DB at a time. // Since we use locking, only one vaultik instance accesses the DB at a time.
// A snapshot whose metadata export was interrupted is left incomplete by
// finalizeSnapshotMetadata, so it is among the incomplete snapshots dropped
// here (https://git.eeqj.de/sneak/vaultik/issues/177).
_, err = v.PruneDatabase() _, err = v.PruneDatabase()
if err != nil { if err != nil {
return fmt.Errorf("prune database: %w", err) return fmt.Errorf("prune database: %w", err)
@@ -324,7 +327,12 @@ func (v *Vaultik) collectUploadStats(scanner *snapshot.Scanner, stats *snapshotS
} }
} }
// finalizeSnapshotMetadata updates stats, marks complete, and exports metadata // finalizeSnapshotMetadata updates stats, exports metadata, and only then
// marks the snapshot complete. Recording completion last is deliberate: an
// export interrupted by a crash leaves the snapshot incomplete rather than
// looking complete with no manifest or database at the destination. The next
// run's PruneDatabase drops the incomplete snapshot and re-backs-up its data.
// See https://git.eeqj.de/sneak/vaultik/issues/177.
func (v *Vaultik) finalizeSnapshotMetadata( func (v *Vaultik) finalizeSnapshotMetadata(
snapshotID string, stats *snapshotStats, snapshotID string, stats *snapshotStats,
) error { ) error {
@@ -346,9 +354,11 @@ func (v *Vaultik) finalizeSnapshotMetadata(
return fmt.Errorf("updating snapshot stats: %w", err) return fmt.Errorf("updating snapshot stats: %w", err)
} }
err = v.SnapshotManager.CompleteSnapshot(v.ctx, snapshotID) // snapshot_blobs must be populated before the export, which builds the
// manifest and the trimmed metadata database from it.
err = v.SnapshotManager.PopulateSnapshotBlobs(v.ctx, snapshotID)
if err != nil { if err != nil {
return fmt.Errorf("completing snapshot: %w", err) return fmt.Errorf("populating snapshot blobs: %w", err)
} }
err = v.SnapshotManager.ExportSnapshotMetadata( err = v.SnapshotManager.ExportSnapshotMetadata(
@@ -357,6 +367,13 @@ func (v *Vaultik) finalizeSnapshotMetadata(
return fmt.Errorf("exporting snapshot metadata: %w", err) return fmt.Errorf("exporting snapshot metadata: %w", err)
} }
// Record completion last, so an interrupted export never leaves a
// snapshot marked complete without its metadata at the destination.
err = v.SnapshotManager.MarkSnapshotComplete(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("marking snapshot complete: %w", err)
}
return nil return nil
} }
+6
View File
@@ -114,10 +114,16 @@ func (v *Vaultik) ListSnapshots(jsonOutput bool) error {
return encoder.Encode(snapshots) return encoder.Encode(snapshots)
} }
// The table is the output this command exists to produce, so it is
// written plain (markers would corrupt the columns) to the UI writer's
// stdout; --quiet silences it. Reconciliation notes below go through
// the UI methods, so their warnings still emit under --quiet.
if !v.UI.Quiet() {
err = v.printSnapshotTable(snapshots) err = v.printSnapshotTable(snapshots)
if err != nil { if err != nil {
return err return err
} }
}
if remoteErr == nil { if remoteErr == nil {
v.reportListDrift(snapshots, listing) v.reportListDrift(snapshots, listing)
@@ -0,0 +1,41 @@
package vaultik_test
import (
"testing"
"time"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
)
// TestListSnapshots_QuietSuppressesTableNotJSON is the --quiet contract
// for `snapshot list`: the human table is silenced, but the --json
// document a script depends on still emits. A local snapshot with its
// remote counterpart present is used so there are no drift notes, whose
// warnings would emit even under --quiet.
func TestListSnapshots_QuietSuppressesTableNotJSON(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
ts := time.Date(2026, 3, 1, 10, 0, 0, 0, time.UTC)
env.addLocal(t, listLocalID, ts)
env.addRemote(t, listLocalID, ts)
env.v.UI.SetQuiet(true)
// Table mode: nothing on stdout.
require.NoError(t, env.v.ListSnapshots(false))
require.Empty(t, env.stdout.String(),
"the table must be suppressed under --quiet")
// JSON mode: the document is still written despite the quiet UI.
env.stdout.Reset()
require.NoError(t, env.v.ListSnapshots(true))
rows := decodeListJSON(t, env.stdout.String())
require.Len(t, rows, 1)
require.Equal(t, listLocalID, rows[0].ID,
"the --json document must still emit under --quiet")
}
+1
View File
@@ -215,6 +215,7 @@ func NewForTesting(storage storage.Storer) *TestVaultik {
Stdout: stdout, Stdout: stdout,
Stderr: stderr, Stderr: stderr,
Stdin: stdin, Stdin: stdin,
UI: ui.NewWithColor(stdout, false),
}, },
Stdout: stdout, Stdout: stdout,
Stderr: stderr, Stderr: stderr,
+1 -1
View File
@@ -393,7 +393,7 @@ func (v *Vaultik) verifyBlob(
blobInfo snapshot.BlobInfo, db *sql.DB, identities []age.Identity, blobInfo snapshot.BlobInfo, db *sql.DB, identities []age.Identity,
) error { ) error {
// Download blob using shared fetch method // Download blob using shared fetch method
reader, _, err := v.FetchBlob(v.ctx, blobInfo.Hash, blobInfo.CompressedSize) reader, err := v.FetchBlob(v.ctx, blobInfo.Hash)
if err != nil { if err != nil {
return fmt.Errorf("failed to download: %w", err) return fmt.Errorf("failed to download: %w", err)
} }