1 Commits
Author SHA1 Message Date
sneak 141a84fd0d Wait for the interrupted operation to clean up before exit (closes #159)
check / check (pull_request) Successful in 1m22s
On SIGINT/SIGTERM the process could exit before the interrupted command's
cleanup defers ran, leaving decrypted data in the temp directory: the blob
cache (vaultik-blobcache-*) and the decrypted snapshot database
(vaultik-restore-*).

RunApp now mirrors fx's run sequence: start, block on app.Wait(), then
app.Stop(), returning only after Stop completes. fx delivers both an OS
interrupt and the finished operation's Shutdowner.Shutdown() on that one
channel. Stop runs the OnStop hooks; the operation's hook cancels the command
and waits for its goroutine to return (bounded by shutdownTimeout) before
exit. The old code returned as soon as app.Done fired, without Stop, so on a
real interrupt it unwound to os.Exit while cleanup still ran and the wait
never blocked exit.

Restore's loops check the context between chunks and blobs so the wait ends
promptly. A cli test drives RunApp through the OnStop hook and asserts the
scratch file is gone before RunApp returns.

Model: opus-4-8
2026-09-22 11:41:41 +00:00
70 changed files with 1431 additions and 3765 deletions
+15 -18
View File
@@ -74,16 +74,16 @@ Maps files to their constituent chunks:
#### Blob (`database.Blob`) #### Blob (`database.Blob`)
The final storage unit uploaded to S3. Contains many compressed and encrypted chunks: The final storage unit uploaded to S3. Contains many compressed and encrypted chunks:
- `ID`: UUID assigned at creation - `ID`: UUID assigned at creation
- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs)) - `Hash`: SHA256 of final compressed+encrypted content
- `UncompressedSize`: Total raw chunk data before compression - `UncompressedSize`: Total raw chunk data before compression
- `CompressedSize`: Size after zstd compression and age encryption - `CompressedSize`: Size after zstd compression and age encryption
- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps - `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
Blob creation process: Blob creation process:
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB) 1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256 2. Compressed with zstd
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and streamed to storage 3. Encrypted with age (recipients configured in config)
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes 4. SHA256 hash computed → becomes filename in S3
5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` 5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}`
#### BlobChunk (`database.BlobChunk`) #### BlobChunk (`database.BlobChunk`)
@@ -284,10 +284,9 @@ Manages snapshot lifecycle and metadata export.
Key methods: Key methods:
- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record - `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record
- `PopulateSnapshotBlobs(ctx, snapshotID)`Record every blob the snapshot references - `CompleteSnapshot(ctx, snapshotID)`Mark snapshot complete
- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3 - `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3
- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a successful export - `CleanupIncompleteSnapshots(ctx, hostname)` → Remove failed snapshots
- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark complete (no export between)
### `internal/database` ### `internal/database`
SQLite database for local index. Single-writer mode for thread safety. SQLite database for local index. Single-writer mode for thread safety.
@@ -308,7 +307,7 @@ Repository interfaces:
``` ```
CreateSnapshot(opts) CreateSnapshot(opts)
├─► PruneDatabase() // Critical: avoid dedup errors ├─► CleanupIncompleteSnapshots() // Critical: avoid dedup errors
├─► SnapshotManager.CreateSnapshot() // Create DB record ├─► SnapshotManager.CreateSnapshot() // Create DB record
@@ -337,18 +336,16 @@ CreateSnapshot(opts)
├─► SnapshotManager.UpdateSnapshotStatsExtended() ├─► SnapshotManager.UpdateSnapshotStatsExtended()
├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs ├─► SnapshotManager.CompleteSnapshot()
─► SnapshotManager.ExportSnapshotMetadata() ─► SnapshotManager.ExportSnapshotMetadata()
│ │
│ ├─► Copy database to temp file
│ ├─► Clean to only current snapshot data (VACUUM)
│ ├─► Compress binary SQLite with zstd
│ ├─► Encrypt with age
│ ├─► Upload db.zst.age to storage
│ └─► Upload manifest.json.zst to storage
└─► SnapshotManager.MarkSnapshotComplete() // only after the export succeeds ├─► Copy database to temp file
├─► Clean to only current snapshot data (VACUUM)
├─► Compress binary SQLite with zstd
├─► Encrypt with age
├─► Upload db.zst.age to storage
└─► Upload manifest.json.zst to storage
``` ```
## Deduplication Strategy ## Deduplication Strategy
+32 -58
View File
@@ -38,7 +38,7 @@ vaultik snapshot list
Features: Features:
* modern encryption ([age](https://age-encryption.org/), X25519 + ChaCha20-Poly1305) * modern encryption ([age](https://age-encryption.org/), X25519 + XChaCha20-Poly1305)
* content-defined chunking with deduplication (FastCDC) * content-defined chunking with deduplication (FastCDC)
* incremental backups (only changed files are re-chunked) * incremental backups (only changed files are re-chunked)
* multithreaded zstd compression at configurable levels * multithreaded zstd compression at configurable levels
@@ -71,21 +71,14 @@ Requirements that no existing tool meets:
## daily use ## daily use
```sh ```sh
# verify a snapshot (shallow: checks all blobs are present with the listed size) # verify a snapshot (shallow: checks all blobs exist)
vaultik snapshot verify <snapshot-id> vaultik snapshot verify <snapshot-id>
# put the private key file in the environment (reading it from the file # deep verify (downloads and cryptographically verifies every blob)
# keeps the key out of your shell history); the whole age-keygen file, VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot verify --deep <snapshot-id>
# with one or more identities, is accepted
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
# deep verify (downloads every blob, decrypts it, and re-hashes it to # restore (requires the private key)
# detect corruption — this checks integrity, not who wrote the blob) VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot restore <snapshot-id> /tmp/restored
vaultik snapshot verify --deep <snapshot-id>
# restore (requires the private key). Restore into a new directory you own,
# writable only by you — not a shared location like /tmp
vaultik snapshot restore <snapshot-id> ~/vaultik-restore
# daily cron job: back up, keep a 4-week rolling window of snapshots # daily cron job: back up, keep a 4-week rolling window of snapshots
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w # 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
@@ -126,18 +119,15 @@ Use that remote key — the hex printed inside `<remote only:...>`, or the
full `remote_key` from `snapshot list --json` — to restore and verify: full `remote_key` from `snapshot list --json` — to restore and verify:
```sh ```sh
# put the private key file in the environment (reading it from the file # restore everything to /tmp/restored, then check every restored file's
# keeps the key out of your shell history) # chunk hashes
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
vaultik snapshot restore --verify <remote-key> /tmp/restored
# restore everything to a new directory you own (writable only by you, not a # optionally, deep-verify the snapshot against the store (downloads and
# shared location like /tmp), then check every restored file's chunk hashes # cryptographically checks every blob)
vaultik snapshot restore --verify <remote-key> ~/vaultik-restore VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
vaultik snapshot verify --deep <remote-key>
# optionally, deep-verify the snapshot against the store (downloads every
# blob, decrypts it, and re-hashes it to detect corruption — this checks
# integrity, not who wrote the blob)
vaultik snapshot verify --deep <remote-key>
``` ```
`age_recipients` (the public key) is not needed to restore — only the `age_recipients` (the public key) is not needed to restore — only the
@@ -227,7 +217,7 @@ and `vaultik prune --json | jq .` both work as written.
### environment variables ### environment variables
* `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `snapshot restore` and `snapshot verify --deep`). May hold the whole `age-keygen` file — comments and every identity in it are accepted. Set it from the file, e.g. `export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"`, so the key is not typed into your shell history. * `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `snapshot restore` and `snapshot verify --deep`)
* `VAULTIK_CONFIG`: Path to config file (overridden by `--config`) * `VAULTIK_CONFIG`: Path to config file (overridden by `--config`)
* `VAULTIK_INDEX_PATH`: Override local SQLite index path * `VAULTIK_INDEX_PATH`: Override local SQLite index path
* `VAULTIK_CPUPROFILE`: Write a CPU profile to this path for the duration of the run (development/debugging) * `VAULTIK_CPUPROFILE`: Write a CPU profile to this path for the duration of the run (development/debugging)
@@ -322,9 +312,7 @@ local index alone, and still exits zero.
logger, so stdout stays a single parseable document. logger, so stdout stays a single parseable document.
**`snapshot verify`**: Verify snapshot integrity. **`snapshot verify`**: Verify snapshot integrity.
* Default (shallow): checks that every blob the manifest lists is present in * Default (shallow): checks that all blobs referenced in the manifest exist in storage
storage with the size the manifest records, and that the encrypted database is
present. It does not read blob contents.
* `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the * `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the
encrypted metadata database encrypted metadata database
* Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a * Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a
@@ -452,19 +440,14 @@ Snapshot IDs follow the human-readable format
`<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g. `<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g.
`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the `server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the
destination store in plaintext. Each snapshot's metadata directory is named destination store in plaintext. Each snapshot's metadata directory is named
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a plain with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a listing
listing of the store shows no hostname or snapshot name. The hash uses no of the store reveals no hostname or snapshot name. The backup time is not
secret, though, so an observer who guesses a candidate hostname and snapshot hidden: manifest.json.zst carries a plaintext timestamp, and object
name can hash it and confirm the snapshot is present; the remote key keeps
names out of a listing but does not hide them from a guess. The backup time is
not hidden either: manifest.json.zst carries a plaintext timestamp, and object
modification times are visible at the storage layer regardless. For example, modification times are visible at the storage layer regardless. For example,
`server1_home_2025-06-01T12:00:00Z` is stored under `server1_home_2025-06-01T12:00:00Z` is stored under
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`. `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`.
See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation, and [Security Considerations](docs/REPOSTRUCTURE.md#security-considerations) derivation.
(including [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks)) for what the
format does and does not protect.
### data flow ### data flow
@@ -505,7 +488,7 @@ format does and does not protect.
### encryption ### encryption
* Asymmetric encryption using age (X25519 + ChaCha20-Poly1305) * Asymmetric encryption using age (X25519 + XChaCha20-Poly1305)
* Only the public key is needed on the source host * Only the public key is needed on the source host
* Each blob and each metadata database is encrypted independently * Each blob and each metadata database is encrypted independently
* Multiple recipients supported (encrypt to multiple keys) * Multiple recipients supported (encrypt to multiple keys)
@@ -645,26 +628,17 @@ priority.
## output style ## output style
Every command's user-facing output is governed by `internal/ui`, in one The operational narration of the long-running commands — the Begin,
of two ways. Color is enabled when stdout is a TTY and the `NO_COLOR` Complete, Progress, and status lines of `snapshot create`, `prune`,
environment variable is unset (https://no-color.org/). `snapshot restore`, and the like — goes through helpers in `internal/ui`
and conforms to the uniform style below. Some commands instead write
* **Status, progress, warnings, and errors** go through the `internal/ui` plain text straight to stdout (`version`, `info`, `config`, the
message methods below: marker-prefixed, colored on a TTY, and — except `database delete` prompt, and the `snapshot list` table); that output is
warnings and errors — silenced by `--quiet`. This is the operational unstyled and does not honor `--quiet`. Routing it through `internal/ui`
narration of the long-running commands (`snapshot create`, `prune`, is tracked in
`snapshot restore`, and the like) and the confirmations of [issue #149](https://git.eeqj.de/sneak/vaultik/issues/149). Color is
`config init`, `config set`, and `database delete`. enabled when stdout is a TTY and the `NO_COLOR` environment variable is
* **The data a command exists to produce** is written plain, with no unset (https://no-color.org/).
marker and no color, because a marker would corrupt a table or a
parsed document. This covers the `version`, `info`, and `remote info`
reports, the `snapshot list` table, `config get` values, and every
`--json` document. `--quiet` silences the human reports and tables
(`version`, `info`, `remote info`, `snapshot list`) but never the
machine-consumed `config get` value or the `--json` documents, which a
script depends on. The `database delete` confirmation prompt is also
written this way and always shown: it is an interactive exchange the
operator must see.
`internal/ui` writes to stdout; it is the output the user asked for. `internal/ui` writes to stdout; it is the output the user asked for.
Structured log records are a different thing and go through Structured log records are a different thing and go through
-38
View File
@@ -25,44 +25,6 @@ release" is exactly the contradiction
# Completed Steps # Completed Steps
- 2026-09-22: Routed the last direct-to-stdout command output through
`internal/ui`
([issue #149](https://git.eeqj.de/sneak/vaultik/issues/149)). The
`version`, `info`, `remote info`, `config`, and `database delete`
commands wrote plain text straight to stdout, so they were unstyled and
ignored `--quiet`. Output now falls in two buckets, both governed by
`internal/ui`: status lines and confirmations go through its message
methods (styled, and `--quiet` silences them), while the data a command
exists to produce — the reports, the `snapshot list` table, `config get`
values, and the `--json` documents — is written plain. `--quiet`
silences the human reports and tables but never the `config get` value
or the `--json` documents, which a script depends on, and the
`database delete` confirmation prompt is always shown. The README
output-style section now states this rule.
- 2026-09-22: Validated blob hashes, offsets and lengths read back from
the destination before using them
([issue #155](https://git.eeqj.de/sneak/vaultik/issues/155)). A blob
hash taken from the downloaded database or the store listing was
trusted unchecked, so a hostile remote could drive a decrypted blob to
be written outside the cache directory (a hash like `aa/../../etc`) or
crash a command with a short or negative value. `blobDiskCache.path`
now refuses any key containing a path separator, and `ReadAt` rejects a
negative offset or length and bounds with `length > size-offset` so a
sum cannot overflow past the check. A new `isBlobHash` helper (a plain
function, not a method — the packer stores `temp-placeholder-{uuid}` as
a hash) gates `FetchBlob`, shallow and deep verify; the `blobs/` and
`metadata/` listings skip a non-conforming name with a warning; and
short-hash prefixes in log and error text go through a `shortHash`
helper that cannot panic. `verify`'s chunk reader also rejects a
negative `blob_chunks` length and streams the chunk instead of
allocating a database-supplied size. `restore.go` and
`internal/database` were left untouched to avoid colliding with the
in-flight [issue #156](https://git.eeqj.de/sneak/vaultik/issues/156)
work; the cache-path and `FetchBlob` guards already stop the unsafe
write and fetch, so restore's own `buildBlobIndexes` early check is
deferred as fail-fast defense in depth.
- 2026-09-21: Stopped an interrupted blob upload from making a later - 2026-09-21: Stopped an interrupted blob upload from making a later
backup deduplicate against data that was never stored backup deduplicate against data that was never stored
([issue #148](https://git.eeqj.de/sneak/vaultik/issues/148)). The ([issue #148](https://git.eeqj.de/sneak/vaultik/issues/148)). The
+5 -8
View File
@@ -257,16 +257,16 @@ exclude:
# Storage URL - use either this OR the s3 section below # Storage URL - use either this OR the s3 section below
# Supports: s3://bucket/prefix, file:///path, rclone://remote/path # Supports: s3://bucket/prefix, file:///path, rclone://remote/path
storage_url: "rclone://myremote/path/to/backups" storage_url: "rclone://las1stor1//srv/pool.2024.04/backups/heraklion"
# S3-compatible storage configuration # S3-compatible storage configuration
#s3: #s3:
# # S3-compatible endpoint URL # # S3-compatible endpoint URL
# # Examples: https://s3.amazonaws.com, https://storage.googleapis.com # # Examples: https://s3.amazonaws.com, https://storage.googleapis.com
# endpoint: https://s3.example.com # endpoint: http://10.100.205.122:8333
# #
# # Bucket name where backups will be stored # # Bucket name where backups will be stored
# bucket: mybucket # bucket: testbucket
# #
# # Prefix (folder) within the bucket for this host's backups # # Prefix (folder) within the bucket for this host's backups
# # Useful for organizing backups from multiple hosts # # Useful for organizing backups from multiple hosts
@@ -274,8 +274,8 @@ storage_url: "rclone://myremote/path/to/backups"
# #prefix: "hosts/myserver/" # #prefix: "hosts/myserver/"
# #
# # S3 access credentials # # S3 access credentials
# access_key_id: YOUR_ACCESS_KEY # access_key_id: Z9GT22M9YFU08WRMC5D4
# secret_access_key: YOUR_SECRET_KEY # secret_access_key: Pi0tPKjFbN4rZlRhcA4zBtEkib04yy2WcIzI+AXk
# #
# # S3 region # # S3 region
# # Default: us-east-1 # # Default: us-east-1
@@ -306,9 +306,6 @@ storage_url: "rclone://myremote/path/to/backups"
# Multiple chunks are packed into blobs up to this size # Multiple chunks are packed into blobs up to this size
# Must be at least four times chunk_size (the largest chunk the chunker can # Must be at least four times chunk_size (the largest chunk the chunker can
# emit); a smaller limit would let a single-chunk blob exceed it. # emit); a smaller limit would let a single-chunk blob exceed it.
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
# not visible in its size; lowering the limit toward chunk_size exposes them.
# Supports: 1GB, 10G, 500MB, 1GiB, etc. # Supports: 1GB, 10G, 500MB, 1GiB, etc.
# Default: 10GB # Default: 10GB
#blob_size_limit: 10GB #blob_size_limit: 10GB
+2 -2
View File
@@ -90,7 +90,7 @@ Stores information about packed, compressed, and encrypted blob files.
**Columns:** **Columns:**
- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts - `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts
- `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (NULL until finalized); see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs) - `blob_hash` (TEXT UNIQUE) - SHA256 hash of final blob (NULL until finalized)
- `created_ts` (INTEGER NOT NULL) - Creation timestamp - `created_ts` (INTEGER NOT NULL) - Creation timestamp
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress) - `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression - `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
@@ -216,7 +216,7 @@ After a snapshot is completed:
5. Upload to S3 as `metadata/{remote-key}/db.zst.age` 5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst` 6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext. The hash uses no secret, so a guessed hostname and snapshot name can still be confirmed against a listing; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its [Accepted Risks](REPOSTRUCTURE.md#accepted-risks). The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation).
### 4. Restore Process ### 4. Restore Process
+5 -18
View File
@@ -35,7 +35,7 @@ The metadata subdirectory is named with the **remote key**, a one-way hash of th
- **What it contains**: Packed collections of content-defined chunks from files - **What it contains**: Packed collections of content-defined chunks from files
- **Format**: Zstandard compressed, then Age encrypted - **Format**: Zstandard compressed, then Age encrypted
- **Encryption**: Always encrypted with Age using the configured recipients - **Encryption**: Always encrypted with Age using the configured recipients
- **Naming**: Content-addressed. The blob's name is `hex(SHA256(SHA256(uncompressed blob contents)))` — the double SHA-256 of the concatenated chunk data, computed before compression and encryption, not a hash of the stored (compressed, encrypted) bytes. One consequence: only a holder of the age private key can check a stored blob's integrity, because matching a blob to its name means decrypting and decompressing it first — which is what `restore` and `verify --deep` do. Implemented in `internal/blobgen` (`DoubleSHA256`). This is the canonical description of blob naming; other documents and comments point here. - **Naming**: Content-addressed using SHA256 hash of the encrypted blob
### Why Encrypted ### Why Encrypted
Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy. Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy.
@@ -59,14 +59,14 @@ This ID reveals the hostname, the configured snapshot name, and the backup time,
### Remote Key Derivation ### Remote Key Derivation
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string. The hash is not reversible, but it uses no secret: an observer who guesses a candidate hostname and snapshot name can hash it the same way and confirm whether that snapshot is present. The remote key keeps names out of a plain listing; it does not hide them from a guess. Implemented in `internal/snapshot/remotekey.go`. The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string with no structure a remote observer can reverse. Implemented in `internal/snapshot/remotekey.go`.
Worked example: Worked example:
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z` - Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa` - Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/` - Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
A plain listing of the destination store therefore shows only these hashes, not the hostname or snapshot name of any backup — but because the hash uses no secret, a guessed hostname and snapshot name can be hashed and confirmed against the listing. The same remote key is stored in the manifest's `snapshot_id` field. Because the hash is one-way, a listing of the destination store reveals neither the hostname nor the snapshot name of any backup. The same remote key is stored in the manifest's `snapshot_id` field.
### Files in Each Snapshot Directory ### Files in Each Snapshot Directory
@@ -124,36 +124,23 @@ From the unencrypted data, an observer of the destination store can determine:
- **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear - **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear
- How many blobs each snapshot references, and the total compressed size - How many blobs each snapshot references, and the total compressed size
- The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns) - The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)
- **Whether a guessed hostname and snapshot name are present** — the remote key is an unkeyed hash, so an observer holding candidate names can hash each one and match it against the directory listing. The human ID is never published, so it cannot be read off directly, but it can be confirmed by guessing.
Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless. Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless.
An observer cannot determine: An observer cannot determine:
- The hostname or snapshot name of any backup by reading it off the store — the directory name and the manifest `snapshot_id` are unkeyed hashes of the human ID, so the text is never published (though a guessed name can be confirmed, as above) - The hostname or snapshot name of any backup (the directory name and the manifest `snapshot_id` are one-way hashes of the human ID)
- File names or paths - File names or paths
- File contents - File contents
- File permissions or ownership - File permissions or ownership
- Directory structure - Directory structure
- Which chunks belong to which files - Which chunks belong to which files
### Accepted Risks
These are known, deliberate properties of the format and the tooling, recorded so an operator can weigh them rather than discover them.
1. **No proof of authorship.** Restore and `verify --deep` prove that data decrypts with the age private key and matches its unkeyed content hashes. They do not prove who wrote it: anyone who knows a recipient public key and can replace objects on the destination can substitute a snapshot they built. The recipient string is not stored at the destination, but a compromised backed-up host has it. Defences live on the destination side — bucket versioning or object lock, credentials for the source host that cannot delete or overwrite existing versions, and pruning from a trusted host. Note that S3 `PutObject` overwrites an existing key, so PUT permission alone is not append-only.
2. **Compression reveals sizes.** Blobs and `db.zst.age` are zstd-compressed then age-encrypted; the manifest is compressed only. age does not pad, so an object's size is the exact compressed length of its contents. All new chunks packed into one blob share a single zstd stream (8 MiB window, 4 MiB at compression levels 1-2), and a blob is closed at `blob_size_limit` and at the end of each configured path. Because a stored chunk is never packed again, someone who can write into a backed-up file and watch blob sizes learns something only when their controlled data and a secret land in the same chunk of a file that keeps changing. Advice: back up any outsider-writable directory as its own snapshot.
3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of 2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them.
4. **Decrypted data on local disk.** Several commands stage plaintext under `$TMPDIR`: `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` and removed on success, but a `kill -9` or a power loss leaves them behind — delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` should be trusted to the same degree as the restore target.
5. **Store permissions differ per command.** The backed-up host needs only PUT to run `snapshot create`: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — `snapshot verify`, `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`, `snapshot remove`, and `remote nuke` also delete. The recommended cron line uses `--prune`, which runs `prune` on the backed-up host, so granting that host `--prune` gives it credentials that can delete its own backups. To keep the source host to PUT only, prune from a separate trusted host instead.
6. **Changing recipients does not re-encrypt existing data.** Deduplicated chunks and same-named blobs already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run `vaultik database delete` and take a full backup to a fresh destination or prefix.
7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients. Long-lived ciphertext held by a third party (the destination operator) has no fallback if X25519 is ever broken: there is no second recipient type and no post-quantum option.
## Consistency Guarantees ## Consistency Guarantees
1. **Blobs are immutable** - Once written, a blob is never modified 1. **Blobs are immutable** - Once written, a blob is never modified
2. **Blobs are written before metadata** - A snapshot's metadata is only written after all its blobs are successfully uploaded 2. **Blobs are written before metadata** - A snapshot's metadata is only written after all its blobs are successfully uploaded
3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst are written as complete files 3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst are written as complete files
4. **A snapshot is marked complete in the local DB only after its metadata is uploaded** - `finalizeSnapshotMetadata` runs `ExportSnapshotMetadata` first and records completion (`MarkSnapshotComplete`) only once the export succeeds (see the backup data flow in [ARCHITECTURE.md](../ARCHITECTURE.md)). A crash during the export therefore leaves the snapshot incomplete, so the next backup's `PruneDatabase` drops it and re-backs-up its data, rather than leaving a completed-looking row in the local index with no matching metadata on the destination store. (A crash in the brief moment after the export succeeds but before completion is recorded leaves a fully-restorable snapshot on the destination that the local index drops as incomplete on the next run; `snapshot list` then reports it honestly as remote-only, which is the safe direction: the destination copy stays restorable.) 4. **Snapshots are marked complete in local DB only after metadata upload** - Ensures consistency between local and remote state
## Pruning Safety ## Pruning Safety
+7 -9
View File
@@ -1,16 +1,14 @@
// Package blob handles the creation of blobs - the final storage units for Vaultik. // Package blob handles the creation of blobs - the final storage units for Vaultik.
// A blob is a large file (up to 10GB) containing many compressed and encrypted chunks // A blob is a large file (up to 10GB) containing many compressed and encrypted chunks
// from multiple source files. Blobs are content-addressed: a blob's filename is // from multiple source files. Blobs are content-addressed, meaning their filename
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the concatenated // is derived from the SHA256 hash of their compressed and encrypted content.
// chunk data before compression and encryption, not from the stored bytes. See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
// //
// The blob creation process: // The blob creation process:
// 1. Chunks are accumulated from multiple files // 1. Chunks are accumulated from multiple files
// 2. Each chunk's uncompressed bytes are fed to a running SHA-256 and, in the same // 2. The collection is compressed using zstd
// pass, compressed with zstd and encrypted with age into the temp file // 3. The compressed data is encrypted using age
// 3. On finalize, the name is the double SHA-256 of that uncompressed content // 4. The encrypted blob is hashed to create its content-addressed name
// 4. The blob is uploaded to S3 using the name as the filename // 5. The blob is uploaded to S3 using the hash as the filename
// //
// This design optimizes storage efficiency by batching many small chunks into // This design optimizes storage efficiency by batching many small chunks into
// larger blobs, reducing the number of S3 operations and associated costs. // larger blobs, reducing the number of S3 operations and associated costs.
@@ -489,7 +487,7 @@ func (p *Packer) closeBlobWriter() (string, int64, error) {
return "", 0, fmt.Errorf("seeking to start: %w", err) return "", 0, fmt.Errorf("seeking to start: %w", err)
} }
finalHash := p.currentBlob.writer.ContentID() finalHash := p.currentBlob.writer.Sum256()
return hex.EncodeToString(finalHash), finalSize, nil return hex.EncodeToString(finalHash), finalSize, nil
} }
+89
View File
@@ -0,0 +1,89 @@
// Package blobgen implements the blob data pipeline: streaming zstd
// compression, age encryption, and SHA256 content hashing for blob
// creation, plus the matching decrypt/decompress/verify reader.
package blobgen
import (
"bytes"
"encoding/hex"
"fmt"
"io"
)
// CompressResult contains the results of compression
type CompressResult struct {
Data []byte
UncompressedSize int64
CompressedSize int64
SHA256 string
}
// CompressData compresses and encrypts data, returning the result with hash
func CompressData(
data []byte, compressionLevel int, recipients []string,
) (*CompressResult, error) {
var buf bytes.Buffer
// Create writer
w, err := NewWriter(&buf, compressionLevel, recipients)
if err != nil {
return nil, fmt.Errorf("creating writer: %w", err)
}
// Write data
_, err = w.Write(data)
if err != nil {
_ = w.Close()
return nil, fmt.Errorf("writing data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return nil, fmt.Errorf("closing writer: %w", err)
}
return &CompressResult{
Data: buf.Bytes(),
UncompressedSize: int64(len(data)),
CompressedSize: int64(buf.Len()),
SHA256: hex.EncodeToString(w.Sum256()),
}, nil
}
// CompressStream compresses and encrypts from reader to writer, returning
// the number of uncompressed bytes written and the content hash.
func CompressStream(
dst io.Writer, src io.Reader, compressionLevel int, recipients []string,
) (int64, string, error) {
// Create writer
w, err := NewWriter(dst, compressionLevel, recipients)
if err != nil {
return 0, "", fmt.Errorf("creating writer: %w", err)
}
closed := false
defer func() {
if !closed {
_ = w.Close()
}
}()
// Copy data
_, err = io.Copy(w, src)
if err != nil {
return 0, "", fmt.Errorf("copying data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return 0, "", fmt.Errorf("closing writer: %w", err)
}
closed = true
return w.BytesWritten(), hex.EncodeToString(w.Sum256()), nil
}
+80
View File
@@ -0,0 +1,80 @@
package blobgen_test
import (
"bytes"
"crypto/rand"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// testRecipient is a static age recipient for tests.
const testRecipient = "age1cplgrwj77ta54dnmydvvmzn64ltk83ankxl5sww04mrtmu62kv3s89gmvv"
// TestCompressStreamNoDoubleClose is a regression test for issue #28.
// It verifies that CompressStream does not panic or return an error due to
// double-closing the underlying blobgen.Writer. Before the fix in PR #33,
// the explicit Close() on the happy path combined with defer Close() would
// cause a double close.
func TestCompressStreamNoDoubleClose(t *testing.T) {
t.Parallel()
input := []byte("regression test data for issue #28 double-close fix")
var buf bytes.Buffer
written, hash, err := blobgen.CompressStream(
&buf, bytes.NewReader(input), 3, []string{testRecipient})
require.NoError(t, err, "CompressStream should not return an error")
assert.Positive(t, written, "expected bytes written > 0")
assert.NotEmpty(t, hash, "expected non-empty hash")
assert.Positive(t, buf.Len(), "expected non-empty output")
}
// TestCompressStreamLargeInput exercises CompressStream with a larger payload
// to ensure no double-close issues surface under heavier I/O.
func TestCompressStreamLargeInput(t *testing.T) {
t.Parallel()
data := make([]byte, 512*1024) // 512 KB
_, err := rand.Read(data)
require.NoError(t, err)
var buf bytes.Buffer
written, hash, err := blobgen.CompressStream(
&buf, bytes.NewReader(data), 3, []string{testRecipient})
require.NoError(t, err)
assert.Positive(t, written)
assert.NotEmpty(t, hash)
}
// TestCompressStreamEmptyInput verifies CompressStream handles empty input
// without double-close issues.
func TestCompressStreamEmptyInput(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
_, hash, err := blobgen.CompressStream(
&buf, strings.NewReader(""), 3, []string{testRecipient})
require.NoError(t, err)
assert.NotEmpty(t, hash)
}
// TestCompressDataNoDoubleClose mirrors the stream test for CompressData,
// ensuring the explicit Close + error-path Close pattern is also safe.
func TestCompressDataNoDoubleClose(t *testing.T) {
t.Parallel()
input := []byte("CompressData regression test for double-close")
result, err := blobgen.CompressData(input, 3, []string{testRecipient})
require.NoError(t, err)
assert.Positive(t, result.CompressedSize)
assert.Equal(t, result.UncompressedSize, int64(len(input)))
assert.NotEmpty(t, result.SHA256)
}
-119
View File
@@ -1,119 +0,0 @@
package blobgen_test
import (
"bytes"
"crypto/rand"
"errors"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// ageChunkSize is age's STREAM plaintext chunk size (64 KiB); each encrypted
// chunk adds a 16-byte ChaCha20-Poly1305 tag.
const (
ageChunkSize = 64 * 1024
ageChunkTagSize = 16
ageSegmentSize = ageChunkSize + ageChunkTagSize
ageNonceSize = 16
)
// makeIdentity returns a fresh X25519 identity and its recipient string.
func makeIdentity(t *testing.T) (*age.X25519Identity, string) {
t.Helper()
id, err := age.GenerateX25519Identity()
require.NoError(t, err)
return id, id.Recipient().String()
}
// randomBytes returns n cryptographically random bytes, which do not compress
// so the encrypted payload spans multiple age segments.
func randomBytes(t *testing.T, n int) []byte {
t.Helper()
b := make([]byte, n)
_, err := rand.Read(b)
require.NoError(t, err)
return b
}
// compressibleBytes returns n bytes of a repeating pattern, which zstd packs
// down to a small payload.
func compressibleBytes(n int) []byte {
pattern := bytes.Repeat([]byte("compressible-"), n/13+1)
return pattern[:n]
}
// encryptBlob compresses, encrypts and returns a blob for plaintext at
// compression level 1.
func encryptBlob(t *testing.T, plaintext []byte, recipients ...string) []byte {
t.Helper()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, recipients)
require.NoError(t, err)
_, err = w.Write(plaintext)
require.NoError(t, err)
require.NoError(t, w.Close())
return buf.Bytes()
}
// ageHeaderLen returns the byte length of blob's age header, i.e. the offset
// of the 16-byte payload nonce that follows it. The header ends with a MAC
// line "--- <mac>\n"; the nonce begins right after that newline.
func ageHeaderLen(t *testing.T, blob []byte) int {
t.Helper()
i := bytes.Index(blob, []byte("\n--- "))
require.GreaterOrEqual(t, i, 0, "age MAC footer line not found")
nl := bytes.IndexByte(blob[i+1:], '\n')
require.GreaterOrEqual(t, nl, 0, "newline ending MAC line not found")
return i + 1 + nl + 1
}
// requireBlobUnreadable asserts that data never decrypts to a plaintext with a
// nil error: either NewReader fails, or reading it does.
func requireBlobUnreadable(t *testing.T, data []byte, id age.Identity) {
t.Helper()
r, err := blobgen.NewReader(bytes.NewReader(data), id)
if err != nil {
return
}
_, err = io.ReadAll(r)
_ = r.Close()
require.Error(t, err, "reading a damaged blob must fail")
}
// errFailWriter is returned by failAfterWriter once its byte limit is passed.
var errFailWriter = errors.New("destination write failed")
// failAfterWriter accepts writes until more than limit bytes have been sent,
// then fails every write. It models a destination that dies mid-blob.
type failAfterWriter struct {
limit int
written int
}
func (f *failAfterWriter) Write(p []byte) (int, error) {
f.written += len(p)
if f.written > f.limit {
return 0, errFailWriter
}
return len(p), nil
}
-49
View File
@@ -1,49 +0,0 @@
package blobgen
import (
"errors"
"io"
)
// ErrOutputTooLarge is returned by a reader from LimitReader once it has
// been asked for more than its limit. It bounds how far an untrusted
// compressed stream may expand, so a small, highly compressible object
// from the store cannot decompress without limit.
var ErrOutputTooLarge = errors.New("output exceeds size limit")
// LimitReader returns a reader that yields at most limit bytes from r and
// then fails with ErrOutputTooLarge. Unlike io.LimitReader, which reports
// a silent io.EOF at the limit (indistinguishable from a stream that
// simply ended), this fails, so a caller decoding or copying the stream
// sees an error rather than a truncated value. A stream of exactly limit
// bytes reads back cleanly to EOF; the first byte beyond it is the error.
func LimitReader(r io.Reader, limit int64) io.Reader {
// remaining counts down from limit+1: the extra byte is the one that,
// if it ever arrives, proves the stream is longer than the limit.
return &limitReader{r: r, remaining: limit + 1}
}
type limitReader struct {
r io.Reader
remaining int64
}
func (l *limitReader) Read(p []byte) (int, error) {
if l.remaining <= 0 {
return 0, ErrOutputTooLarge
}
if int64(len(p)) > l.remaining {
p = p[:l.remaining]
}
n, err := l.r.Read(p)
l.remaining -= int64(n)
if l.remaining <= 0 {
// The (limit+1)th byte was just read: the stream is too long.
return n, ErrOutputTooLarge
}
return n, err
}
-43
View File
@@ -1,43 +0,0 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestLimitReaderPassesExactSize checks that a stream of exactly the limit
// reads back cleanly to EOF: the bound must not reject a legitimate blob
// whose plaintext equals its recorded size.
func TestLimitReaderPassesExactSize(t *testing.T) {
t.Parallel()
const n = 1000
r := blobgen.LimitReader(bytes.NewReader(bytes.Repeat([]byte("a"), n)), n)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.Len(t, got, n)
}
// TestLimitReaderFailsPastLimit feeds a large, highly compressible run of
// zeros — the decompressed output a zip bomb would produce — through a
// small limit and checks it fails within the bound rather than passing
// the whole stream through.
func TestLimitReaderFailsPastLimit(t *testing.T) {
t.Parallel()
const limit = 1000
r := blobgen.LimitReader(
bytes.NewReader(bytes.Repeat([]byte{0}, limit*1000)), limit)
n, err := io.Copy(io.Discard, r)
require.ErrorIs(t, err, blobgen.ErrOutputTooLarge)
require.LessOrEqual(t, n, int64(limit)+1,
"reader must stop within one byte of the limit")
}
-190
View File
@@ -1,190 +0,0 @@
package blobgen_test
import (
"bytes"
"fmt"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestNewReaderWrongIdentity covers issue case 4: opening a blob with an
// identity other than the recipient reports no matching identity.
func TestNewReaderWrongIdentity(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
other, _ := makeIdentity(t)
blob := encryptBlob(t, []byte("secret payload"), recipient)
_, err := blobgen.NewReader(bytes.NewReader(blob), other)
require.Error(t, err)
var noMatch *age.NoIdentityMatchError
assert.ErrorAs(t, err, &noMatch)
}
// TestNewReaderTruncated covers issue case 6: a multi-segment blob cut at
// several points must never read back as valid data. The point immediately
// after the header and nonce is intentionally excluded: it reads as a valid
// empty blob today and is the regression case for
// https://git.eeqj.de/sneak/vaultik/issues/152.
func TestNewReaderTruncated(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
blob := encryptBlob(t, randomBytes(t, 4*65536+123), recipient)
h := ageHeaderLen(t, blob)
require.Greater(t, len(blob), h+ageNonceSize+ageSegmentSize,
"test needs a blob of at least two age segments")
cases := []struct {
name string
size int
}{
{"inside header", h / 2},
{"inside nonce", h + 8},
{"inside first segment", h + ageNonceSize + 100},
{"end of first full segment", h + ageNonceSize + ageSegmentSize},
{"last byte removed", len(blob) - 1},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, blob[:tc.size], id)
})
}
}
// TestNewReaderCorrupted covers issue case 7: one flipped byte in each region
// of a multi-segment blob makes it unreadable.
func TestNewReaderCorrupted(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
blob := encryptBlob(t, randomBytes(t, 4*65536+123), recipient)
h := ageHeaderLen(t, blob)
firstNL := bytes.IndexByte(blob, '\n')
require.Positive(t, firstNL, "header must have a version line")
cases := []struct {
name string
pos int
}{
{"header stanza", firstNL + 5},
{"header MAC line", h - 2},
{"nonce", h + 4},
{"body segment", h + ageNonceSize + 50},
{"final tag", len(blob) - 1},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
corrupt := append([]byte(nil), blob...)
corrupt[tc.pos] ^= 0xff
requireBlobUnreadable(t, corrupt, id)
})
}
}
// TestNewReaderTrailingAndGarbage covers issue case 8: bytes appended after a
// valid blob, empty input, and random garbage each fail to read.
func TestNewReaderTrailingAndGarbage(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
valid := encryptBlob(t, []byte("small payload"), recipient)
appended := append(append([]byte(nil), valid...), []byte("trailing junk")...)
t.Run("appended bytes", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, appended, id)
})
t.Run("empty input", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, []byte{}, id)
})
t.Run("random garbage", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, randomBytes(t, 512), id)
})
}
// TestNewWriterInvalidLevel covers the rejected end of issue case 9: an
// out-of-range compression level errors and writes nothing to the destination.
func TestNewWriterInvalidLevel(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
for _, level := range []int{0, -1, 20} {
t.Run(fmt.Sprintf("level%d", level), func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, level, []string{recipient})
require.ErrorIs(t, err, blobgen.ErrInvalidCompressionLevel)
assert.Nil(t, w)
assert.Zero(t, buf.Len(), "nothing written on an invalid level")
})
}
}
// TestNewWriterInvalidRecipients covers issue case 10: nil and empty recipient
// lists and an unparsable recipient string each error.
func TestNewWriterInvalidRecipients(t *testing.T) {
t.Parallel()
cases := []struct {
name string
recipients []string
}{
{"nil list", nil},
{"empty list", []string{}},
{"invalid recipient string", []string{"not-a-recipient"}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, tc.recipients)
require.Error(t, err)
assert.Nil(t, w)
})
}
}
// TestNewWriterFailingDestination covers issue case 11: a destination that
// fails mid-blob surfaces its error from Write or Close.
func TestNewWriterFailingDestination(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
// The limit clears the age header and nonce so NewWriter succeeds, then
// trips once the compressed body starts flowing.
dst := &failAfterWriter{limit: 512}
w, err := blobgen.NewWriter(dst, 1, []string{recipient})
require.NoError(t, err)
_, writeErr := w.Write(randomBytes(t, 256*1024))
closeErr := w.Close()
assert.True(t, writeErr != nil || closeErr != nil,
"destination failure must surface from Write or Close")
}
+4 -25
View File
@@ -2,7 +2,6 @@ package blobgen
import ( import (
"crypto/sha256" "crypto/sha256"
"errors"
"fmt" "fmt"
"hash" "hash"
"io" "io"
@@ -21,12 +20,10 @@ type Reader struct {
bytesRead int64 bytesRead int64
} }
// NewReader creates a new Reader that decrypts, decompresses, and verifies // NewReader creates a new Reader that decrypts, decompresses, and verifies data
// data. Every supplied identity is offered to age.Decrypt, so a blob func NewReader(r io.Reader, identity age.Identity) (*Reader, error) {
// encrypted to any one of them can be read.
func NewReader(r io.Reader, identities ...age.Identity) (*Reader, error) {
// Create decryption reader // Create decryption reader
decReader, err := age.Decrypt(r, identities...) decReader, err := age.Decrypt(r, identity)
if err != nil { if err != nil {
return nil, fmt.Errorf("creating decryption reader: %w", err) return nil, fmt.Errorf("creating decryption reader: %w", err)
} }
@@ -57,22 +54,6 @@ func (r *Reader) Read(p []byte) (int, error) {
n, err := r.teeReader.Read(p) n, err := r.teeReader.Read(p)
r.bytesRead += int64(n) r.bytesRead += int64(n)
// When the ciphertext is cut right after the age header plus its
// 16-byte nonce, the age reader's first read fails with
// io.ErrUnexpectedEOF, and the zstd decoder maps that to a clean
// io.EOF at frame start. That makes a truncated stream look like a
// valid empty one. Distinguish the two: on EOF, read once more from
// the age reader. A genuine end leaves it at (0, io.EOF); a truncated
// stream leaves its stored io.ErrUnexpectedEOF, which we surface.
if errors.Is(err, io.EOF) {
var probe [1]byte
m, ageErr := r.decryptor.Read(probe[:])
if m != 0 || !errors.Is(ageErr, io.EOF) {
return n, io.ErrUnexpectedEOF
}
}
return n, err return n, err
} }
@@ -83,9 +64,7 @@ func (r *Reader) Close() error {
return nil return nil
} }
// Sum256 returns the single SHA-256 of the plaintext read so far. This is the // Sum256 returns the SHA256 hash of all data read
// first hash only; the stored object name is its double hash, which callers
// obtain by passing this digest to DoubleSHA256.
func (r *Reader) Sum256() []byte { func (r *Reader) Sum256() []byte {
return r.hasher.Sum(nil) return r.hasher.Sum(nil)
} }
-54
View File
@@ -1,54 +0,0 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestMultipleRecipients verifies that data written for several recipients can
// be read back by each recipient's identity. Moved from internal/crypto, which
// held the only multi-recipient test; blobgen is now the sole encryption path.
func TestMultipleRecipients(t *testing.T) {
t.Parallel()
identities := make([]*age.X25519Identity, 3)
recipients := make([]string, 3)
for i := range identities {
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
identities[i] = identity
recipients[i] = identity.Recipient().String()
}
plaintext := []byte("Secret message for multiple recipients")
var encrypted bytes.Buffer
writer, err := blobgen.NewWriter(&encrypted, 3, recipients)
require.NoError(t, err)
_, err = writer.Write(plaintext)
require.NoError(t, err)
require.NoError(t, writer.Close())
// Every recipient's identity must recover the original plaintext.
for i, identity := range identities {
reader, err := blobgen.NewReader(
bytes.NewReader(encrypted.Bytes()), identity)
require.NoError(t, err, "recipient %d should open the reader", i+1)
got, err := io.ReadAll(reader)
require.NoError(t, err, "recipient %d should read the plaintext", i+1)
require.NoError(t, reader.Close())
assert.Equal(t, plaintext, got,
"recipient %d should recover the original plaintext", i+1)
}
}
-132
View File
@@ -1,132 +0,0 @@
package blobgen_test
import (
"bytes"
"crypto/sha256"
"fmt"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// checkRoundTrip writes input through a Writer, reads it back through a Reader,
// and verifies the plaintext, the byte counts, and the content hashes.
func checkRoundTrip(
t *testing.T, id *age.X25519Identity, recipient string,
level int, input []byte,
) {
t.Helper()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, level, []string{recipient})
require.NoError(t, err)
n, err := w.Write(input)
require.NoError(t, err)
assert.Equal(t, len(input), n)
require.NoError(t, w.Close())
require.Equal(t, int64(len(input)), w.BytesWritten())
r, err := blobgen.NewReader(bytes.NewReader(buf.Bytes()), id)
require.NoError(t, err)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.NoError(t, r.Close())
assert.Equal(t, input, got, "decrypted output must equal input")
require.Equal(t, int64(len(input)), r.BytesRead())
// The hash values are checked by decrypting: the reader's single SHA-256
// is the hash of the plaintext, and hashing it once more (DoubleSHA256)
// gives the writer's ContentID.
single := sha256.Sum256(got)
assert.Equal(t, single[:], r.Sum256())
assert.Equal(t, blobgen.DoubleSHA256(r.Sum256()), w.ContentID())
}
// TestWriterReaderRoundTrip covers issue cases 1 and 2: every size round trips
// for both random and compressible data, and the reader hash, its double hash
// and the byte counts all agree.
func TestWriterReaderRoundTrip(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
// Sizes exercise the age segment boundary (64 KiB) from just below to a
// few segments above it, plus the empty and single-byte edges.
sizes := []int{0, 1, 65535, 65536, 65537, 4*65536 + 123}
kinds := []struct {
name string
fill func(*testing.T, int) []byte
}{
{"random", randomBytes},
{"compressible", func(_ *testing.T, n int) []byte {
return compressibleBytes(n)
}},
}
for _, k := range kinds {
for _, size := range sizes {
name := fmt.Sprintf("%s/%d", k.name, size)
t.Run(name, func(t *testing.T) {
t.Parallel()
checkRoundTrip(t, id, recipient, 1, k.fill(t, size))
})
}
}
}
// TestZeroLengthNoWrite covers issue case 3: a Writer closed with no Write at
// all produces the double hash of the empty input, and the blob reads back as
// empty with no error.
func TestZeroLengthNoWrite(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, []string{recipient})
require.NoError(t, err)
require.NoError(t, w.Close())
assert.Equal(t, int64(0), w.BytesWritten())
empty := sha256.Sum256(nil)
doubled := sha256.Sum256(empty[:])
assert.Equal(t, doubled[:], w.ContentID(),
"ContentID of empty input is SHA256(SHA256(\"\"))")
r, err := blobgen.NewReader(bytes.NewReader(buf.Bytes()), id)
require.NoError(t, err)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.NoError(t, r.Close())
assert.Empty(t, got, "empty blob decrypts to empty output")
assert.Equal(t, int64(0), r.BytesRead())
assert.Equal(t, empty[:], r.Sum256())
}
// TestNewWriterValidLevelsRoundTrip covers the accepted end of issue case 9:
// the boundary compression levels 1 and 19 both round trip.
func TestNewWriterValidLevelsRoundTrip(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
input := randomBytes(t, 4096)
for _, level := range []int{1, 19} {
t.Run(fmt.Sprintf("level%d", level), func(t *testing.T) {
t.Parallel()
checkRoundTrip(t, id, recipient, level, input)
})
}
}
-70
View File
@@ -1,70 +0,0 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestReaderRejectsHeaderNonceTruncation guards against a stream cut right
// after the age header plus its 16-byte nonce. age.Decrypt still succeeds on
// such an object, and the zstd decoder maps the age reader's
// io.ErrUnexpectedEOF to a clean io.EOF at frame start, so without the extra
// check the truncated stream would read as a valid empty one. Reading it must
// now fail.
func TestReaderRejectsHeaderNonceTruncation(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
// Encrypting empty plaintext yields header + nonce(16) + a single
// 16-byte final chunk tag. Dropping the trailing tag leaves exactly the
// age header plus its nonce — the truncation point that triggers the bug.
var full bytes.Buffer
w, err := age.Encrypt(&full, identity.Recipient())
require.NoError(t, err)
require.NoError(t, w.Close())
truncated := full.Bytes()[:full.Len()-16]
reader, err := blobgen.NewReader(bytes.NewReader(truncated), identity)
require.NoError(t, err)
defer func() { _ = reader.Close() }()
_, err = io.ReadAll(reader)
require.Error(t, err)
require.ErrorIs(t, err, io.ErrUnexpectedEOF)
}
// TestReaderReadsGenuinelyEmptyBlob confirms the truncation check does not
// reject a legitimately empty payload: a blob written with no data must round
// trip back to zero bytes with no error.
func TestReaderReadsGenuinelyEmptyBlob(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
var encrypted bytes.Buffer
writer, err := blobgen.NewWriter(
&encrypted, 3, []string{identity.Recipient().String()})
require.NoError(t, err)
require.NoError(t, writer.Close())
reader, err := blobgen.NewReader(bytes.NewReader(encrypted.Bytes()), identity)
require.NoError(t, err)
defer func() { _ = reader.Close() }()
data, err := io.ReadAll(reader)
require.NoError(t, err)
require.Empty(t, data)
}
+11 -27
View File
@@ -1,6 +1,3 @@
// Package blobgen implements the blob data pipeline: streaming zstd
// compression, age encryption, and SHA256 content hashing for blob
// creation, plus the matching decrypt/decompress/verify reader.
package blobgen package blobgen
import ( import (
@@ -15,24 +12,6 @@ import (
"github.com/klauspost/compress/zstd" "github.com/klauspost/compress/zstd"
) )
// DoubleSHA256 returns the double SHA-256 of content whose single SHA-256
// digest is sum: it hashes that digest once more. Stored objects — a blob, and
// the metadata database export — are named by this second hash.
//
// The second hash does not hide whether known content is stored: an attacker
// who can reproduce an object's entire plaintext computes the same name simply
// by hashing twice, exactly as this code does. What limits that is blob
// packing, not the double hash — a blob's name covers all of its concatenated
// chunk plaintext, so a name can be confirmed only by someone who can
// reproduce the whole blob (a snapshot made entirely of known content, or a
// known file large enough to fill blobs on its own). An ordinary file that
// shares a blob with other, unknown data cannot be confirmed this way.
func DoubleSHA256(sum []byte) []byte {
h := sha256.Sum256(sum)
return h[:]
}
// Zstd compression level bounds accepted by NewWriter. // Zstd compression level bounds accepted by NewWriter.
const ( const (
minCompressionLevel = 1 minCompressionLevel = 1
@@ -151,12 +130,17 @@ func (w *Writer) Close() error {
return nil return nil
} }
// ContentID returns the double SHA-256 of the uncompressed input data: the // Sum256 returns the double SHA256 hash of the uncompressed input data.
// name under which this content is stored. It is the second hash of the // Double hashing (SHA256(SHA256(data))) prevents information leakage about
// running SHA-256, via DoubleSHA256; see that function for what naming content // the plaintext - an attacker cannot confirm existence of known content
// this way does and does not hide. // by computing its hash and checking for a matching blob filename.
func (w *Writer) ContentID() []byte { func (w *Writer) Sum256() []byte {
return DoubleSHA256(w.hasher.Sum(nil)) // First hash: SHA256(plaintext)
firstHash := w.hasher.Sum(nil)
// Second hash: SHA256(firstHash) - this is the blob ID
secondHash := sha256.Sum256(firstHash)
return secondHash[:]
} }
// BytesWritten returns the number of uncompressed bytes written // BytesWritten returns the number of uncompressed bytes written
+9 -10
View File
@@ -12,10 +12,9 @@ import (
"sneak.berlin/go/vaultik/internal/blobgen" "sneak.berlin/go/vaultik/internal/blobgen"
) )
// TestWriterHashIsDoubleHash verifies that Writer.ContentID() returns // TestWriterHashIsDoubleHash verifies that Writer.Sum256() returns
// SHA256(SHA256(plaintext)). Stored objects are named by this second hash so a // the double hash SHA256(SHA256(plaintext)) for security.
// name is not the plaintext's own SHA-256; this does not stop someone who // Double hashing prevents attackers from confirming existence of known content.
// already holds the plaintext from confirming it.
func TestWriterHashIsDoubleHash(t *testing.T) { func TestWriterHashIsDoubleHash(t *testing.T) {
t.Parallel() t.Parallel()
@@ -44,7 +43,7 @@ func TestWriterHashIsDoubleHash(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
// Get the hash from the writer // Get the hash from the writer
writerHash := hex.EncodeToString(writer.ContentID()) writerHash := hex.EncodeToString(writer.Sum256())
// Calculate the expected double hash: SHA256(SHA256(plaintext)) // Calculate the expected double hash: SHA256(SHA256(plaintext))
firstHash := sha256.Sum256(testData) firstHash := sha256.Sum256(testData)
@@ -61,11 +60,11 @@ func TestWriterHashIsDoubleHash(t *testing.T) {
// The writer hash should match the double hash // The writer hash should match the double hash
assert.Equal(t, expectedDoubleHash, writerHash, assert.Equal(t, expectedDoubleHash, writerHash,
"Writer.ContentID() must be SHA256(SHA256(plaintext))") "Writer.Sum256() should return SHA256(SHA256(plaintext)) for security")
// It must be the second hash, not the plaintext's own SHA-256. // Verify it's NOT the single hash (would leak information)
assert.NotEqual(t, singleHashStr, writerHash, assert.NotEqual(t, singleHashStr, writerHash,
"Writer hash must be the double hash, not the single SHA-256") "Writer hash should not be single hash (would allow content confirmation attacks)")
} }
// TestWriterDeterministicHash verifies that the same input always produces // TestWriterDeterministicHash verifies that the same input always produces
@@ -94,8 +93,8 @@ func TestWriterDeterministicHash(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, writer2.Close()) require.NoError(t, writer2.Close())
hash1 := hex.EncodeToString(writer1.ContentID()) hash1 := hex.EncodeToString(writer1.Sum256())
hash2 := hex.EncodeToString(writer2.ContentID()) hash2 := hex.EncodeToString(writer2.Sum256())
// Hashes should be identical (deterministic) // Hashes should be identical (deterministic)
assert.Equal(t, hash1, hash2, "Same input should produce same hash") assert.Equal(t, hash1, hash2, "Same input should produce same hash")
+19 -30
View File
@@ -4,6 +4,7 @@ import (
"bytes" "bytes"
"errors" "errors"
"fmt" "fmt"
"io"
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
@@ -12,7 +13,6 @@ import (
"github.com/spf13/cobra" "github.com/spf13/cobra"
"gopkg.in/yaml.v3" "gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/ui"
) )
// configFileMode is the permission set for freshly written config files; // configFileMode is the permission set for freshly written config files;
@@ -46,11 +46,8 @@ const defaultConfigTemplate = `# vaultik configuration
# ─── REQUIRED ──────────────────────────────────────────────────────────────── # ─── REQUIRED ────────────────────────────────────────────────────────────────
# Age recipient public keys for encryption. # Age recipient public keys for encryption.
# Backups are encrypted to ALL listed recipients; any one of the corresponding # Backups are encrypted to ALL listed recipients. Any one of the corresponding
# private keys can decrypt. Adding a recipient later does not re-encrypt data # private keys can decrypt. Generate a keypair with:
# already stored: deduplicated chunks and existing blobs stay encrypted to the
# earlier recipients, so a newly added key cannot restore them on its own (see
# docs/REPOSTRUCTURE.md, Accepted Risks). Generate a keypair with:
# age-keygen -o vaultik_backup_private_key.txt # age-keygen -o vaultik_backup_private_key.txt
# grep 'public key' vaultik_backup_private_key.txt # grep 'public key' vaultik_backup_private_key.txt
age_recipients: age_recipients:
@@ -265,7 +262,7 @@ The config is written to the path from --config, $VAULTIK_CONFIG, or
the platform default config directory (e.g. ~/Library/Application Support/ the platform default config directory (e.g. ~/Library/Application Support/
on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`, on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`,
Args: cobra.NoArgs, Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error { RunE: func(_ *cobra.Command, _ []string) error {
path := configPathForInit() path := configPathForInit()
_, err := os.Stat(path) _, err := os.Stat(path)
@@ -285,11 +282,8 @@ on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`,
return fmt.Errorf("writing config file: %w", err) return fmt.Errorf("writing config file: %w", err)
} }
// A written-confirmation, not scriptable output: route it _, _ = fmt.Fprintf(os.Stdout, "Config written to %s\n", path)
// through the UI so it is styled and --quiet silences it. _, _ = fmt.Fprintln(os.Stdout,
out := commandUI(cmd)
out.Infof("Config written to %s.", path)
out.Infof(
"Edit it to set your age_recipients, snapshots, and storage_url.") "Edit it to set your age_recipients, snapshots, and storage_url.")
return nil return nil
@@ -331,7 +325,7 @@ func newConfigGetCommand() *cobra.Command {
Use: "get <key>", Use: "get <key>",
Short: "Print a config value by dotted path (e.g. storage_url, compression_level)", Short: "Print a config value by dotted path (e.g. storage_url, compression_level)",
Args: cobra.ExactArgs(1), Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(_ *cobra.Command, args []string) error {
path, err := ResolveConfigPath() path, err := ResolveConfigPath()
if err != nil { if err != nil {
return err return err
@@ -347,13 +341,8 @@ func newConfigGetCommand() *cobra.Command {
return err return err
} }
// The value is scriptable output: it must stay machine-plain
// (no marker, no color) and is never silenced by --quiet, so it
// is written straight to stdout rather than through the UI.
w := cmd.OutOrStdout()
if node.Kind == yaml.ScalarNode { if node.Kind == yaml.ScalarNode {
_, _ = fmt.Fprintln(w, node.Value) _, _ = fmt.Fprintln(os.Stdout, node.Value)
return nil return nil
} }
@@ -363,7 +352,7 @@ func newConfigGetCommand() *cobra.Command {
return fmt.Errorf("marshaling value: %w", err) return fmt.Errorf("marshaling value: %w", err)
} }
_, _ = fmt.Fprint(w, string(out)) _, _ = fmt.Fprint(os.Stdout, string(out))
return nil return nil
}, },
@@ -385,23 +374,23 @@ Examples:
vaultik config set compression_level 9 vaultik config set compression_level 9
vaultik config set s3.bucket mybucket # legacy S3 fields still supported`, vaultik config set s3.bucket mybucket # legacy S3 fields still supported`,
Args: cobra.ExactArgs(configSetArgs), Args: cobra.ExactArgs(configSetArgs),
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(_ *cobra.Command, args []string) error {
path, err := ResolveConfigPath() path, err := ResolveConfigPath()
if err != nil { if err != nil {
return err return err
} }
return writeConfigSet(commandUI(cmd), path, args[0], args[1]) return writeConfigSet(os.Stdout, path, args[0], args[1])
}, },
} }
} }
// writeConfigSet applies key=value to the config at path, writes it back // writeConfigSet applies key=value to the config at path, writes it back
// owner-only, and confirms the write by naming just the key through the // owner-only, and confirms the write by printing just the key name to w.
// UI writer (styled, and silenced by --quiet). The value is never // The value is never echoed: it may be a secret such as
// echoed: it may be a secret such as s3.secret_access_key, and captured // s3.secret_access_key, and captured stdout or a pasted terminal would
// stdout or a pasted terminal would then leak it. // then leak it.
func writeConfigSet(out *ui.Writer, path, key, value string) error { func writeConfigSet(w io.Writer, path, key, value string) error {
root, err := loadYAMLFile(path) root, err := loadYAMLFile(path)
if err != nil { if err != nil {
return err return err
@@ -412,12 +401,12 @@ func writeConfigSet(out *ui.Writer, path, key, value string) error {
return err return err
} }
data, err := marshalConfigYAML(root) out, err := marshalConfigYAML(root)
if err != nil { if err != nil {
return fmt.Errorf("marshaling config: %w", err) return fmt.Errorf("marshaling config: %w", err)
} }
err = os.WriteFile(path, data, configFileMode) err = os.WriteFile(path, out, configFileMode)
if err != nil { if err != nil {
return fmt.Errorf("writing config file: %w", err) return fmt.Errorf("writing config file: %w", err)
} }
@@ -433,7 +422,7 @@ func writeConfigSet(out *ui.Writer, path, key, value string) error {
} }
} }
out.Infof("Set %s.", key) _, _ = fmt.Fprintln(w, key)
return nil return nil
} }
+8 -11
View File
@@ -9,7 +9,6 @@ import (
"gopkg.in/yaml.v3" "gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/ui"
) )
// TestDefaultConfigTemplateParses ensures the init template is valid YAML // TestDefaultConfigTemplateParses ensures the init template is valid YAML
@@ -247,20 +246,19 @@ func TestWriteConfigSetHidesSecret(t *testing.T) {
t.Fatalf("seed config: %v", err) t.Fatalf("seed config: %v", err)
} }
var buf bytes.Buffer var out bytes.Buffer
err = writeConfigSet(ui.NewWithColor(&buf, false), path, err = writeConfigSet(&out, path, "s3.secret_access_key", secret)
"s3.secret_access_key", secret)
if err != nil { if err != nil {
t.Fatalf("writeConfigSet: %v", err) t.Fatalf("writeConfigSet: %v", err)
} }
if strings.Contains(buf.String(), secret) { if strings.Contains(out.String(), secret) {
t.Errorf("output echoed the secret value: %q", buf.String()) t.Errorf("output echoed the secret value: %q", out.String())
} }
if !strings.Contains(buf.String(), "s3.secret_access_key") { if !strings.Contains(out.String(), "s3.secret_access_key") {
t.Errorf("output did not confirm the key name: %q", buf.String()) t.Errorf("output did not confirm the key name: %q", out.String())
} }
} }
@@ -279,10 +277,9 @@ func TestWriteConfigSetTightensMode(t *testing.T) {
t.Fatalf("seed config: %v", err) t.Fatalf("seed config: %v", err)
} }
var buf bytes.Buffer var out bytes.Buffer
err = writeConfigSet(ui.NewWithColor(&buf, false), path, err = writeConfigSet(&out, path, "compression_level", "9")
"compression_level", "9")
if err != nil { if err != nil {
t.Fatalf("writeConfigSet: %v", err) t.Fatalf("writeConfigSet: %v", err)
} }
+11 -12
View File
@@ -48,7 +48,7 @@ storage destination on that run.
Use --force to skip the confirmation prompt.`, Use --force to skip the confirmation prompt.`,
Args: cobra.NoArgs, Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error { RunE: func(_ *cobra.Command, _ []string) error {
// Resolve config path // Resolve config path
configPath, err := ResolveConfigPath() configPath, err := ResolveConfigPath()
if err != nil { if err != nil {
@@ -62,31 +62,26 @@ Use --force to skip the confirmation prompt.`,
} }
dbPath := cfg.IndexPath dbPath := cfg.IndexPath
out := commandUI(cmd)
// Check if database exists // Check if database exists
_, err = os.Stat(dbPath) _, err = os.Stat(dbPath)
if os.IsNotExist(err) { if os.IsNotExist(err) {
out.Infof("Local state database does not exist: %s.", dbPath) _, _ = fmt.Fprintf(os.Stdout, "Database does not exist: %s\n", dbPath)
return nil return nil
} }
// Confirm unless --force. The prompt and its immediate result // Confirm unless --force
// are an interactive exchange the operator must see, so they go
// straight to stdout rather than through the UI and --quiet does
// not silence them.
if !force { if !force {
w := cmd.OutOrStdout() _, _ = fmt.Fprintf(os.Stdout,
_, _ = fmt.Fprintf(w,
"This will delete the local state database at:\n %s\n\n", dbPath) "This will delete the local state database at:\n %s\n\n", dbPath)
_, _ = fmt.Fprint(w, "Are you sure? Type 'yes' to confirm: ") _, _ = fmt.Fprint(os.Stdout, "Are you sure? Type 'yes' to confirm: ")
var confirm string var confirm string
_, err = fmt.Scanln(&confirm) _, err = fmt.Scanln(&confirm)
if err != nil || confirm != "yes" { if err != nil || confirm != "yes" {
_, _ = fmt.Fprintln(w, "Aborted.") _, _ = fmt.Fprintln(os.Stdout, "Aborted.")
//nolint:nilerr // a failed/aborted confirmation is a clean abort //nolint:nilerr // a failed/aborted confirmation is a clean abort
return nil return nil
@@ -105,7 +100,11 @@ Use --force to skip the confirmation prompt.`,
_ = os.Remove(walPath) // Ignore errors - files may not exist _ = os.Remove(walPath) // Ignore errors - files may not exist
_ = os.Remove(shmPath) _ = os.Remove(shmPath)
out.Infof("Local state database deleted: %s.", dbPath) rootFlags := GetRootFlags()
if !rootFlags.Quiet {
_, _ = fmt.Fprintf(os.Stdout, "Database deleted: %s\n", dbPath)
}
log.Info("Local state database deleted", "path", dbPath) log.Info("Local state database deleted", "path", dbPath)
return nil return nil
-206
View File
@@ -1,206 +0,0 @@
package cli //nolint:testpackage // sets the unexported rootFlags directly
import (
"bytes"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/spf13/cobra"
)
// setRootFlags overrides the global rootFlags for the duration of one
// test and restores it afterward. These tests must not run in parallel:
// the flags are process-global, so the whole struct is saved and put
// back rather than left mutated for the next test.
func setRootFlags(t *testing.T, f RootFlags) {
t.Helper()
old := rootFlags
rootFlags = f
t.Cleanup(func() { rootFlags = old })
}
// seedFile writes content to a fresh file and returns its path.
func seedFile(t *testing.T, dir, name, content string) string {
t.Helper()
path := filepath.Join(dir, name)
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("seeding %s: %v", name, err)
}
return path
}
// mustExecute runs a command with its output captured and fails the test
// if it errors, returning what the command printed.
func mustExecute(t *testing.T, cmd *cobra.Command, args ...string) string {
t.Helper()
var out bytes.Buffer
cmd.SetOut(&out)
cmd.SetArgs(args)
err := cmd.Execute()
if err != nil {
t.Fatalf("%s failed: %v", cmd.Name(), err)
}
return out.String()
}
// TestVersionQuietSuppressesReport checks that --quiet silences the whole
// version report: it is human-facing output, not a scriptable value.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestVersionQuietSuppressesReport(t *testing.T) {
setRootFlags(t, RootFlags{Quiet: true})
out := mustExecute(t, NewVersionCommand())
if out != "" {
t.Errorf("--quiet version printed %q, want nothing", out)
}
}
// TestConfigGetIgnoresQuiet checks that a config value is printed even
// under --quiet: it is scriptable output a caller depends on, so --quiet
// must not suppress it, and it stays machine-plain (no marker, no color).
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigGetIgnoresQuiet(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "storage_url: file:///mnt/x\n")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigGetCommand(), "storage_url")
if out != "file:///mnt/x\n" {
t.Errorf("config get --quiet = %q, want the plain value", out)
}
}
// TestConfigSetQuietSuppressesConfirmation checks that --quiet silences
// the confirmation line while still writing the value to the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigSetQuietSuppressesConfirmation(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "compression_level: 3\n")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigSetCommand(), "compression_level", "9")
if out != "" {
t.Errorf("--quiet config set printed %q, want nothing", out)
}
data, err := os.ReadFile(path) //nolint:gosec // G304: test-controlled path
if err != nil {
t.Fatalf("reading config back: %v", err)
}
if !strings.Contains(string(data), "compression_level: 9") {
t.Errorf("config set did not write the value under --quiet:\n%s", data)
}
}
// TestConfigSetConfirmsWhenNotQuiet checks that the confirmation names
// the key (styled) when --quiet is not set.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigSetConfirmsWhenNotQuiet(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "compression_level: 3\n")
setRootFlags(t, RootFlags{ConfigPath: path})
out := mustExecute(t, newConfigSetCommand(), "compression_level", "9")
if !strings.Contains(out, "compression_level") {
t.Errorf("config set did not confirm the key: %q", out)
}
}
// TestConfigInitQuietSuppressesConfirmation checks that --quiet silences
// the "config written" confirmation while still writing the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigInitQuietSuppressesConfirmation(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "new-config.yml")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigInitCommand())
if out != "" {
t.Errorf("--quiet config init printed %q, want nothing", out)
}
_, err := os.Stat(path)
if err != nil {
t.Errorf("config init did not write the file under --quiet: %v", err)
}
}
// seedDatabaseDeleteConfig writes a valid config whose index_path is a
// seeded database file, and returns both paths.
func seedDatabaseDeleteConfig(t *testing.T, dir string) (string, string) {
t.Helper()
dbPath := seedFile(t, dir, "index.sqlite", "not-a-real-db")
cfg := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"), filepath.Join(dir, "store"), dbPath)
cfgPath := seedFile(t, dir, "config.yml", cfg)
return dbPath, cfgPath
}
// TestDatabaseDeleteQuietSuppressesMessage checks that --quiet silences
// the "database deleted" line while still removing the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestDatabaseDeleteQuietSuppressesMessage(t *testing.T) {
dir := t.TempDir()
dbPath, cfgPath := seedDatabaseDeleteConfig(t, dir)
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: cfgPath})
out := mustExecute(t, newDatabaseDeleteCommand(), "--force")
if out != "" {
t.Errorf("--quiet database delete printed %q, want nothing", out)
}
_, err := os.Stat(dbPath)
if !os.IsNotExist(err) {
t.Errorf("database delete did not remove the file: stat err = %v", err)
}
}
// TestDatabaseDeleteReportsWhenNotQuiet checks that the deletion is
// reported when --quiet is not set.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestDatabaseDeleteReportsWhenNotQuiet(t *testing.T) {
dir := t.TempDir()
_, cfgPath := seedDatabaseDeleteConfig(t, dir)
setRootFlags(t, RootFlags{ConfigPath: cfgPath})
out := mustExecute(t, newDatabaseDeleteCommand(), "--force")
if !strings.Contains(out, "deleted") {
t.Errorf("database delete did not report the deletion: %q", out)
}
}
-15
View File
@@ -9,7 +9,6 @@ import (
"github.com/adrg/xdg" "github.com/adrg/xdg"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"sneak.berlin/go/vaultik/internal/ui"
) )
// errConfigNotFound is wrapped by all config-resolution failures. // errConfigNotFound is wrapped by all config-resolution failures.
@@ -82,20 +81,6 @@ func GetRootFlags() RootFlags {
return rootFlags return rootFlags
} }
// commandUI returns a UI writer for a command's stdout, in quiet mode
// when the global --quiet flag is set. This is how the pure-cli
// commands (version, config, database) reach internal/ui: color follows
// the writer (a TTY gets color, a captured test buffer does not), and
// --quiet silences the same message classes it silences everywhere else.
func commandUI(cmd *cobra.Command) *ui.Writer {
w := ui.New(cmd.OutOrStdout())
if GetRootFlags().Quiet {
w.SetQuiet(true)
}
return w
}
// ResolveConfigPath resolves the config file path from flags, environment, or default. // ResolveConfigPath resolves the config file path from flags, environment, or default.
// Search order: --config flag, VAULTIK_CONFIG env, XDG config dir, // Search order: --config flag, VAULTIK_CONFIG env, XDG config dir,
// /etc/vaultik/config.yml. // /etc/vaultik/config.yml.
+2 -6
View File
@@ -188,12 +188,8 @@ func newSnapshotVerifyCommand() *cobra.Command {
cmd := &cobra.Command{ cmd := &cobra.Command{
Use: "verify <snapshot-id>", Use: "verify <snapshot-id>",
Short: "Check a snapshot's blobs are present with the listed size", Short: "Verify snapshot integrity",
Long: "Checks that every blob the snapshot's manifest lists is present\n" + Long: "Verifies that all blobs referenced in a snapshot exist.\n\n" +
"in storage with the size the manifest records, and that the\n" +
"snapshot's encrypted database is present. It does not read blob\n" +
"contents; use --deep to download, decrypt, and re-hash every blob\n" +
"to detect corruption -- integrity, not who wrote it.\n\n" +
"The snapshot may be named by its ID or, on a host with no local\n" + "The snapshot may be named by its ID or, on a host with no local\n" +
"index, by the remote key that 'snapshot list' prints for a\n" + "index, by the remote key that 'snapshot list' prints for a\n" +
"remote-only snapshot (an unambiguous leading part is enough).", "remote-only snapshot (an unambiguous leading part is enough).",
+2 -6
View File
@@ -35,12 +35,8 @@ The snapshot may be named by its ID or, when restoring on a host with no
local index, by the remote key that 'snapshot list' prints for a local index, by the remote key that 'snapshot list' prints for a
remote-only snapshot (an unambiguous leading part is enough). remote-only snapshot (an unambiguous leading part is enough).
Requires the age private key in the VAULTIK_AGE_SECRET_KEY environment Requires the VAULTIK_AGE_SECRET_KEY environment variable to be set with
variable. The variable may hold the whole age-keygen file (comments and the age private key.
all of its identities are accepted); read it from the file rather than
typing the key, so it does not land in your shell history:
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
Examples: Examples:
# Restore entire snapshot # Restore entire snapshot
-37
View File
@@ -1,37 +0,0 @@
package cli //nolint:testpackage // exercises the unexported command constructor
import (
"strings"
"testing"
"github.com/spf13/pflag"
)
// TestRestoreCommandDoesNotTakeKeyAsArgument guards the fix for the age
// key being echoed on the command line: restore must take the key only
// from the environment, never as a flag value, and its help must show the
// file-based form rather than a literal key that would land in shell
// history.
func TestRestoreCommandDoesNotTakeKeyAsArgument(t *testing.T) {
t.Parallel()
cmd := newSnapshotRestoreCommand()
cmd.Flags().VisitAll(func(f *pflag.Flag) {
lower := strings.ToLower(f.Name)
for _, banned := range []string{"key", "secret", "age", "identity"} {
if strings.Contains(lower, banned) {
t.Errorf("restore must not accept the key as a flag; found --%s", f.Name)
}
}
})
help := cmd.Long
if strings.Contains(help, "AGE-SECRET-KEY-") {
t.Error("restore help must not show a literal age private key to type")
}
if !strings.Contains(help, "$(cat ") {
t.Error("restore help should read the key from a file, e.g. $(cat ...)")
}
}
+5 -14
View File
@@ -2,11 +2,11 @@ package cli
import ( import (
"fmt" "fmt"
"io"
"runtime" "runtime"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"sneak.berlin/go/vaultik/internal/globals" "sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/ui"
) )
// NewVersionCommand creates the version command // NewVersionCommand creates the version command
@@ -17,25 +17,16 @@ func NewVersionCommand() *cobra.Command {
Long: `Print version, git commit, and build information for vaultik.`, Long: `Print version, git commit, and build information for vaultik.`,
Args: cobra.NoArgs, Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) { Run: func(cmd *cobra.Command, _ []string) {
writeVersion(commandUI(cmd)) writeVersion(cmd.OutOrStdout())
}, },
} }
return cmd return cmd
} }
// writeVersion prints the version report through the UI writer. The // writeVersion prints the version report. It takes a writer rather than
// report is the output this command exists to produce, so it is written // using os.Stdout directly so the output can be asserted on in tests.
// plain (markers would corrupt the aligned report) via the writer's func writeVersion(w io.Writer) {
// underlying stdout; --quiet silences it like any other non-error
// output.
func writeVersion(out *ui.Writer) {
if out.Quiet() {
return
}
w := out.Out()
_, _ = fmt.Fprintf(w, "vaultik %s\n", globals.Version) _, _ = fmt.Fprintf(w, "vaultik %s\n", globals.Version)
_, _ = fmt.Fprintf(w, " commit: %s\n", globals.Commit) _, _ = fmt.Fprintf(w, " commit: %s\n", globals.Commit)
_, _ = fmt.Fprintf(w, " build date: %s\n", globals.CommitDate) _, _ = fmt.Fprintf(w, " build date: %s\n", globals.CommitDate)
+4 -4
View File
@@ -34,9 +34,9 @@ func runVersionCommand(t *testing.T) string {
// the report is the version the binary was actually built with. The // the report is the version the binary was actually built with. The
// test binary carries no -ldflags, so that is the "dev" default -- the // test binary carries no -ldflags, so that is the "dev" default -- the
// same string an untagged `make vaultik` build stamps a prefix of. // same string an untagged `make vaultik` build stamps a prefix of.
//
//nolint:paralleltest // executes a command that reads the global rootFlags
func TestVersionCommandReportsBuildVersion(t *testing.T) { func TestVersionCommandReportsBuildVersion(t *testing.T) {
t.Parallel()
out := runVersionCommand(t) out := runVersionCommand(t)
wantFirst := "vaultik " + globals.Version wantFirst := "vaultik " + globals.Version
@@ -55,9 +55,9 @@ func TestVersionCommandReportsBuildVersion(t *testing.T) {
// being exactly "dev", so once untagged builds started carrying their // being exactly "dev", so once untagged builds started carrying their
// commit sha it would have gone silent and an unreleased binary would // commit sha it would have gone silent and an unreleased binary would
// have looked like a release. // have looked like a release.
//
//nolint:paralleltest // executes a command that reads the global rootFlags
func TestVersionCommandFlagsDevelopmentBuild(t *testing.T) { func TestVersionCommandFlagsDevelopmentBuild(t *testing.T) {
t.Parallel()
if !globals.IsDevVersion(globals.Version) { if !globals.IsDevVersion(globals.Version) {
t.Skipf("test binary was stamped with release version %q", t.Skipf("test binary was stamped with release version %q",
globals.Version) globals.Version)
+20 -41
View File
@@ -135,26 +135,6 @@ func (c *Config) SnapshotNames() []string {
return names return names
} }
// Names of the two places the age secret key can be configured, used by
// AgeSecretKeySourceName for error messages that must not echo the value.
//
//nolint:gosec // G101: these are the names of the config sources, not a key
const (
ageSecretKeySourceEnv = "VAULTIK_AGE_SECRET_KEY"
ageSecretKeySourceConfig = "age_secret_key"
)
// AgeSecretKeySourceName returns the human name of where AgeSecretKey was
// configured. A Config built directly (as in tests) has no recorded
// source, so it reports the config-file field name.
func (c *Config) AgeSecretKeySourceName() string {
if c.AgeSecretKeySource != "" {
return c.AgeSecretKeySource
}
return ageSecretKeySourceConfig
}
// Config represents the application configuration for Vaultik. // Config represents the application configuration for Vaultik.
// It defines all settings for backup operations, including source directories, // It defines all settings for backup operations, including source directories,
// encryption recipients, storage configuration, and performance tuning parameters. // encryption recipients, storage configuration, and performance tuning parameters.
@@ -164,11 +144,6 @@ func (c *Config) AgeSecretKeySourceName() string {
type Config struct { type Config struct {
AgeRecipients []string `yaml:"age_recipients"` AgeRecipients []string `yaml:"age_recipients"`
AgeSecretKey string `yaml:"age_secret_key"` AgeSecretKey string `yaml:"age_secret_key"`
// AgeSecretKeySource names where AgeSecretKey was configured
// ("VAULTIK_AGE_SECRET_KEY" or "age_secret_key") so a later parse
// failure can name the source without echoing the secret value. It is
// set by Load and never read from or written to the config file.
AgeSecretKeySource string `yaml:"-"`
BlobSizeLimit Size `yaml:"blob_size_limit"` BlobSizeLimit Size `yaml:"blob_size_limit"`
ChunkSize Size `yaml:"chunk_size"` ChunkSize Size `yaml:"chunk_size"`
// Exclude holds global excludes applied to all snapshots. // Exclude holds global excludes applied to all snapshots.
@@ -279,7 +254,10 @@ func Load(path string) (*Config, error) {
cfg.IndexPath = expandTilde(envIndexPath) cfg.IndexPath = expandTilde(envIndexPath)
} }
cfg.setAgeSecretKey() // Check for environment variable override for AgeSecretKey
if envAgeSecretKey := os.Getenv("VAULTIK_AGE_SECRET_KEY"); envAgeSecretKey != "" {
cfg.AgeSecretKey = extractAgeSecretKey(envAgeSecretKey)
}
// Get hostname if not set // Get hostname if not set
if cfg.Hostname == "" { if cfg.Hostname == "" {
@@ -401,21 +379,6 @@ func validateAgeRecipient(recipient string) error {
return nil return nil
} }
// setAgeSecretKey records the age secret key and where it came from. The
// value is stored raw and parsed only where decryption happens
// (internal/vaultik), so backup, list and prune keep working whatever the
// field holds. The environment variable overrides the config-file field.
func (c *Config) setAgeSecretKey() {
if c.AgeSecretKey != "" {
c.AgeSecretKeySource = ageSecretKeySourceConfig
}
if env := os.Getenv("VAULTIK_AGE_SECRET_KEY"); env != "" {
c.AgeSecretKey = env
c.AgeSecretKeySource = ageSecretKeySourceEnv
}
}
// validateStorage validates storage configuration. // validateStorage validates storage configuration.
// If StorageURL is set, it takes precedence. S3 URLs require credentials. // If StorageURL is set, it takes precedence. S3 URLs require credentials.
// File URLs don't require any S3 configuration. // File URLs don't require any S3 configuration.
@@ -472,6 +435,22 @@ func (c *Config) validateStorageURL() error {
} }
} }
// extractAgeSecretKey extracts the AGE-SECRET-KEY from the input using
// the age library's parser, which handles comments and whitespace.
func extractAgeSecretKey(input string) string {
identities, err := age.ParseIdentities(strings.NewReader(input))
if err != nil || len(identities) == 0 {
// Fall back to trimmed input if parsing fails
return strings.TrimSpace(input)
}
// Return the string representation of the first identity
if id, ok := identities[0].(*age.X25519Identity); ok {
return id.String()
}
return strings.TrimSpace(input)
}
// Module exports the config module for fx dependency injection. // Module exports the config module for fx dependency injection.
// It provides the Config type to other modules in the application. // It provides the Config type to other modules in the application.
// //
+42 -61
View File
@@ -1,4 +1,4 @@
package config //nolint:testpackage // exercises unexported source constants package config //nolint:testpackage // exercises unexported extractAgeSecretKey
import ( import (
"errors" "errors"
@@ -87,48 +87,6 @@ func TestConfigLoad(t *testing.T) {
} }
} }
// TestExampleConfigIsScrubbedAndLoads checks that the shipped
// config.example.yml carries only neutral placeholders (no real credentials,
// private addresses, or internal host names) and still parses.
func TestExampleConfigIsScrubbedAndLoads(t *testing.T) {
t.Parallel()
examplePath := filepath.Join("..", "..", "config.example.yml")
cfg, err := Load(examplePath)
if err != nil {
t.Fatalf("Failed to load config.example.yml: %v", err)
}
if cfg.StorageURL != "rclone://myremote/path/to/backups" {
t.Errorf("Expected neutral storage_url, got '%s'", cfg.StorageURL)
}
//nolint:gosec // G304: examplePath is a fixed in-repo path, not user input
raw, err := os.ReadFile(examplePath)
if err != nil {
t.Fatalf("Failed to read config.example.yml: %v", err)
}
text := string(raw)
wantSubstrings := []string{
"YOUR_ACCESS_KEY",
"YOUR_SECRET_KEY",
"endpoint: https://",
}
for _, want := range wantSubstrings {
if !strings.Contains(text, want) {
t.Errorf("Expected config.example.yml to contain %q", want)
}
}
// A raw "http://" scheme would mean a plaintext, likely private endpoint.
if strings.Contains(text, "http://") {
t.Error("config.example.yml should not contain an http:// endpoint")
}
}
// TestConfigFromEnv tests loading config path from environment variable // TestConfigFromEnv tests loading config path from environment variable
func TestConfigFromEnv(t *testing.T) { func TestConfigFromEnv(t *testing.T) {
t.Parallel() t.Parallel()
@@ -298,31 +256,53 @@ func TestValidateAgeRecipients(t *testing.T) {
} }
} }
// TestAgeSecretKeySourceName checks the name reported for the configured // TestExtractAgeSecretKey tests extraction of AGE-SECRET-KEY from various inputs
// age secret key: the recorded source when Load set one, and the func TestExtractAgeSecretKey(t *testing.T) {
// config-file field name for a Config built directly (as in tests).
func TestAgeSecretKeySourceName(t *testing.T) {
t.Parallel() t.Parallel()
tests := []struct { tests := []struct {
name string name string
source string input string
want string expected string
}{ }{
{ {
name: "unset defaults to config field", name: "plain key",
source: "", input: testIntegrationAgePrivateKey,
want: ageSecretKeySourceConfig, expected: testIntegrationAgePrivateKey,
}, },
{ {
name: "environment source", name: "key with trailing newline",
source: ageSecretKeySourceEnv, input: testIntegrationAgePrivateKey + "\n",
want: ageSecretKeySourceEnv, expected: testIntegrationAgePrivateKey,
}, },
{ {
name: "config-file source", name: "full age-keygen output",
source: ageSecretKeySourceConfig, input: "# created: 2025-01-14T12:00:00Z\n" +
want: ageSecretKeySourceConfig, "# public key: " + testIntegrationAgePublicKey + "\n" +
testIntegrationAgePrivateKey + "\n",
expected: testIntegrationAgePrivateKey,
},
{
name: "age-keygen output with extra blank lines",
input: "# created: 2025-01-14T12:00:00Z\n" +
"# public key: " + testIntegrationAgePublicKey + "\n\n" +
testIntegrationAgePrivateKey + "\n\n",
expected: testIntegrationAgePrivateKey,
},
{
name: "key with leading whitespace",
input: " " + testIntegrationAgePrivateKey + " ",
expected: testIntegrationAgePrivateKey,
},
{
name: "empty input",
input: "",
expected: "",
},
{
name: "only comments",
input: "# this is a comment\n# another comment",
expected: "# this is a comment\n# another comment",
}, },
} }
@@ -330,9 +310,10 @@ func TestAgeSecretKeySourceName(t *testing.T) {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
cfg := &Config{AgeSecretKeySource: tt.source} result := extractAgeSecretKey(tt.input)
if got := cfg.AgeSecretKeySourceName(); got != tt.want { if result != tt.expected {
t.Errorf("AgeSecretKeySourceName() = %q, want %q", got, tt.want) t.Errorf("extractAgeSecretKey(%q) = %q, want %q",
tt.input, result, tt.expected)
} }
}) })
} }
+233
View File
@@ -0,0 +1,233 @@
// Package crypto provides thread-safe age encryption and decryption
// helpers used to protect blob and metadata content.
package crypto //nolint:revive,nolintlint // stdlib crypto unused; see #76
import (
"bytes"
"errors"
"fmt"
"io"
"sync"
"filippo.io/age"
"go.uber.org/fx"
)
// ErrNoRecipients is returned when an encryptor is created or updated
// without any recipient public keys.
var ErrNoRecipients = errors.New("at least one recipient is required")
// errInvalidRecipient is returned when a recipient string does not parse as
// an X25519 age1... public key. It omits the value, which can be sensitive.
var errInvalidRecipient = errors.New(
"not a valid X25519 age1... recipient")
// Encryptor provides thread-safe encryption using the age encryption library.
// It supports encrypting data for multiple recipients simultaneously, allowing
// any of the corresponding private keys to decrypt the data. This is useful
// for backup scenarios where multiple parties should be able to decrypt the data.
type Encryptor struct {
recipients []age.Recipient
mu sync.RWMutex
}
// NewEncryptor creates a new encryptor with the given age public keys.
// Each public key should be a valid age X25519 recipient string (e.g., "age1...")
// At least one recipient must be provided. Returns an error if any of the
// public keys are invalid or if no recipients are specified.
func NewEncryptor(publicKeys []string) (*Encryptor, error) {
if len(publicKeys) == 0 {
return nil, ErrNoRecipients
}
recipients := make([]age.Recipient, 0, len(publicKeys))
for i, key := range publicKeys {
// The key string can be sensitive (e.g. a secret key pasted by
// mistake), so the error names its position, never its value.
recipient, err := age.ParseX25519Recipient(key)
if err != nil {
return nil, fmt.Errorf("%w: recipient %d", errInvalidRecipient, i)
}
recipients = append(recipients, recipient)
}
return &Encryptor{
recipients: recipients,
}, nil
}
// Encrypt encrypts data using age encryption for all configured recipients.
// The encrypted data can be decrypted by any of the corresponding private keys.
// This method is suitable for small to medium amounts of data that fit in memory.
// For large data streams, use EncryptStream or EncryptWriter instead.
func (e *Encryptor) Encrypt(data []byte) ([]byte, error) {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
var buf bytes.Buffer
// Create encrypted writer for all recipients
w, err := age.Encrypt(&buf, recipients...)
if err != nil {
return nil, fmt.Errorf("creating encrypted writer: %w", err)
}
// Write data
_, err = w.Write(data)
if err != nil {
return nil, fmt.Errorf("writing encrypted data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return nil, fmt.Errorf("closing encrypted writer: %w", err)
}
return buf.Bytes(), nil
}
// EncryptStream encrypts data from reader to writer using age encryption.
// This method is suitable for encrypting large files or streams as it processes
// data in a streaming fashion without loading everything into memory.
// The encrypted data is written directly to the destination writer.
func (e *Encryptor) EncryptStream(dst io.Writer, src io.Reader) error {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
// Create encrypted writer for all recipients
w, err := age.Encrypt(dst, recipients...)
if err != nil {
return fmt.Errorf("creating encrypted writer: %w", err)
}
// Copy data
_, err = io.Copy(w, src)
if err != nil {
return fmt.Errorf("copying encrypted data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return fmt.Errorf("closing encrypted writer: %w", err)
}
return nil
}
// EncryptWriter creates a writer that encrypts data written to it.
// All data written to the returned WriteCloser will be encrypted and written
// to the destination writer. The caller must call Close() on the returned
// writer to ensure all encrypted data is properly flushed and finalized.
// This is useful for integrating encryption into existing writer-based pipelines.
func (e *Encryptor) EncryptWriter(dst io.Writer) (io.WriteCloser, error) {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
// Create encrypted writer for all recipients
w, err := age.Encrypt(dst, recipients...)
if err != nil {
return nil, fmt.Errorf("creating encrypted writer: %w", err)
}
return w, nil
}
// UpdateRecipients updates the recipients for future encryption operations.
// This method is thread-safe and can be called while other encryption operations
// are in progress. Existing encryption operations will continue with the old
// recipients. At least one recipient must be provided. Returns an error if any
// of the public keys are invalid or if no recipients are specified.
func (e *Encryptor) UpdateRecipients(publicKeys []string) error {
if len(publicKeys) == 0 {
return ErrNoRecipients
}
recipients := make([]age.Recipient, 0, len(publicKeys))
for i, key := range publicKeys {
// The key string can be sensitive (e.g. a secret key pasted by
// mistake), so the error names its position, never its value.
recipient, err := age.ParseX25519Recipient(key)
if err != nil {
return fmt.Errorf("%w: recipient %d", errInvalidRecipient, i)
}
recipients = append(recipients, recipient)
}
e.mu.Lock()
e.recipients = recipients
e.mu.Unlock()
return nil
}
// Decryptor provides thread-safe decryption using the age encryption library.
// It uses a private key to decrypt data that was encrypted for the corresponding
// public key.
type Decryptor struct {
identity age.Identity
mu sync.RWMutex
}
// NewDecryptor creates a new decryptor with the given age private key.
// The private key should be a valid age X25519 identity string.
// Returns an error if the private key is invalid.
func NewDecryptor(privateKey string) (*Decryptor, error) {
identity, err := age.ParseX25519Identity(privateKey)
if err != nil {
return nil, fmt.Errorf("parsing age identity: %w", err)
}
return &Decryptor{
identity: identity,
}, nil
}
// Decrypt decrypts data using age decryption.
// This method is suitable for small to medium amounts of data that fit in memory.
// For large data streams, use DecryptStream instead.
func (d *Decryptor) Decrypt(data []byte) ([]byte, error) {
d.mu.RLock()
identity := d.identity
d.mu.RUnlock()
r, err := age.Decrypt(bytes.NewReader(data), identity)
if err != nil {
return nil, fmt.Errorf("creating decrypted reader: %w", err)
}
decrypted, err := io.ReadAll(r)
if err != nil {
return nil, fmt.Errorf("reading decrypted data: %w", err)
}
return decrypted, nil
}
// DecryptStream returns a reader that decrypts data from the provided reader.
// This method is suitable for decrypting large files or streams as it processes
// data in a streaming fashion without loading everything into memory.
// The caller should close the input reader when done.
func (d *Decryptor) DecryptStream(src io.Reader) (io.Reader, error) {
d.mu.RLock()
identity := d.identity
d.mu.RUnlock()
r, err := age.Decrypt(src, identity)
if err != nil {
return nil, fmt.Errorf("creating decrypted reader: %w", err)
}
return r, nil
}
// Module exports the crypto module for fx dependency injection.
//
//nolint:gochecknoglobals // fx module definitions are package globals
var Module = fx.Module("crypto")
+198
View File
@@ -0,0 +1,198 @@
package crypto_test
import (
"bytes"
"strings"
"testing"
"filippo.io/age"
"sneak.berlin/go/vaultik/internal/crypto"
)
func TestEncryptor(t *testing.T) {
t.Parallel()
// Generate a test key pair
identity, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity: %v", err)
}
publicKey := identity.Recipient().String()
// Create encryptor
enc, err := crypto.NewEncryptor([]string{publicKey})
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Test data
plaintext := []byte("Hello, World! This is a test message.")
// Encrypt
ciphertext, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Verify it's actually encrypted (should be larger and different)
if bytes.Equal(plaintext, ciphertext) {
t.Error("ciphertext equals plaintext")
}
// Decrypt to verify
r, err := age.Decrypt(bytes.NewReader(ciphertext), identity)
if err != nil {
t.Fatalf("failed to decrypt: %v", err)
}
var decrypted bytes.Buffer
_, err = decrypted.ReadFrom(r)
if err != nil {
t.Fatalf("failed to read decrypted data: %v", err)
}
if !bytes.Equal(plaintext, decrypted.Bytes()) {
t.Error("decrypted data doesn't match original")
}
}
func TestEncryptorMultipleRecipients(t *testing.T) {
t.Parallel()
// Generate three test key pairs
identity1, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity1: %v", err)
}
identity2, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity2: %v", err)
}
identity3, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity3: %v", err)
}
publicKeys := []string{
identity1.Recipient().String(),
identity2.Recipient().String(),
identity3.Recipient().String(),
}
// Create encryptor with multiple recipients
enc, err := crypto.NewEncryptor(publicKeys)
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Test data
plaintext := []byte("Secret message for multiple recipients")
// Encrypt
ciphertext, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Verify each recipient can decrypt
identities := []age.Identity{identity1, identity2, identity3}
for i, identity := range identities {
r, err := age.Decrypt(bytes.NewReader(ciphertext), identity)
if err != nil {
t.Fatalf("recipient %d failed to decrypt: %v", i+1, err)
}
var decrypted bytes.Buffer
_, err = decrypted.ReadFrom(r)
if err != nil {
t.Fatalf("recipient %d failed to read decrypted data: %v", i+1, err)
}
if !bytes.Equal(plaintext, decrypted.Bytes()) {
t.Errorf("recipient %d: decrypted data doesn't match original", i+1)
}
}
}
func TestEncryptorUpdateRecipients(t *testing.T) {
t.Parallel()
// Generate two identities
identity1, _ := age.GenerateX25519Identity()
identity2, _ := age.GenerateX25519Identity()
publicKey1 := identity1.Recipient().String()
publicKey2 := identity2.Recipient().String()
// Create encryptor with first key
enc, err := crypto.NewEncryptor([]string{publicKey1})
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Encrypt with first key
plaintext := []byte("test data")
ciphertext1, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Update to second key
err = enc.UpdateRecipients([]string{publicKey2})
if err != nil {
t.Fatalf("failed to update recipients: %v", err)
}
// Encrypt with second key
ciphertext2, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// First ciphertext should only decrypt with first identity
_, err = age.Decrypt(bytes.NewReader(ciphertext1), identity1)
if err != nil {
t.Error("failed to decrypt with identity1")
}
_, err = age.Decrypt(bytes.NewReader(ciphertext1), identity2)
if err == nil {
t.Error("should not decrypt with identity2")
}
// Second ciphertext should only decrypt with second identity
_, err = age.Decrypt(bytes.NewReader(ciphertext2), identity2)
if err != nil {
t.Error("failed to decrypt with identity2")
}
_, err = age.Decrypt(bytes.NewReader(ciphertext2), identity1)
if err == nil {
t.Error("should not decrypt with identity1")
}
}
// TestNewEncryptorSecretKeyNotEchoed verifies that a secret key mistakenly
// passed as a recipient does not appear in the returned error. A recipient
// string can be sensitive, so the error must name only the position.
func TestNewEncryptorSecretKeyNotEchoed(t *testing.T) {
t.Parallel()
secretKey := "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GX" +
"VEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5"
_, err := crypto.NewEncryptor([]string{secretKey})
if err == nil {
t.Fatal("NewEncryptor returned nil, want error")
}
if strings.Contains(err.Error(), secretKey) {
t.Fatalf("error echoed the recipient value: %v", err)
}
}
+2 -4
View File
@@ -3,10 +3,8 @@
// //
// Blobs in Vaultik are the final storage units uploaded to S3. Each blob is a // Blobs in Vaultik are the final storage units uploaded to S3. Each blob is a
// large (up to 10GB) file containing many compressed and encrypted chunks from // large (up to 10GB) file containing many compressed and encrypted chunks from
// multiple source files. Blobs are content-addressed: the filename in S3 is // multiple source files. Blobs are content-addressed, meaning their filename
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data // is derived from their SHA256 hash after compression and encryption.
// before compression and encryption (not from the stored bytes). See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
// //
// Schema is managed via numbered SQL migrations embedded in the schema/ // Schema is managed via numbered SQL migrations embedded in the schema/
// directory. Migration 000.sql bootstraps the schema_migrations tracking // directory. Migration 000.sql bootstraps the schema_migrations tracking
+6 -6
View File
@@ -51,15 +51,15 @@ type Chunk struct {
// Blob represents a blob record in the database. // Blob represents a blob record in the database.
// A blob is Vaultik's final storage unit - a large file (up to 10GB) containing // A blob is Vaultik's final storage unit - a large file (up to 10GB) containing
// many compressed and encrypted chunks from multiple source files. // many compressed and encrypted chunks from multiple source files.
// Blobs are content-addressed: the filename in S3 is // Blobs are content-addressed, meaning their filename in S3 is derived from
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data // the SHA256 hash of their compressed and encrypted content.
// before compression and encryption (not from the stored bytes). See // The blob creation process is: chunks are accumulated -> compressed with zstd
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md. // -> encrypted with age -> hashed -> uploaded to S3 with the hash as filename.
type Blob struct { type Blob struct {
ID types.BlobID // UUID assigned when blob creation starts ID types.BlobID // UUID assigned when blob creation starts
// Hash is hex(SHA256(SHA256(uncompressed blob contents))) // Hash is the SHA256 of the final compressed+encrypted content
// (empty until finalized); see the type comment above. // (empty until finalized).
Hash types.BlobHash Hash types.BlobHash
CreatedTS time.Time // When blob creation started CreatedTS time.Time // When blob creation started
FinishedTS *time.Time // When blob was finalized (nil if still packing) FinishedTS *time.Time // When blob was finalized (nil if still packing)
-54
View File
@@ -11,18 +11,6 @@ import (
"sneak.berlin/go/vaultik/internal/types" "sneak.berlin/go/vaultik/internal/types"
) )
// Sentinel errors for the single-snapshot invariant that an exported
// per-snapshot metadata database must satisfy.
var (
// ErrNoSnapshotInDatabase means the metadata database has no snapshot
// row at all.
ErrNoSnapshotInDatabase = errors.New("database contains no snapshot")
// ErrMultipleSnapshotsInDatabase means the metadata database holds
// more than the single snapshot an export is supposed to contain.
ErrMultipleSnapshotsInDatabase = errors.New(
"database contains more than one snapshot")
)
// SnapshotRepository provides access to the snapshots table and its // SnapshotRepository provides access to the snapshots table and its
// snapshot_files / snapshot_blobs association tables. // snapshot_files / snapshot_blobs association tables.
type SnapshotRepository struct { type SnapshotRepository struct {
@@ -218,48 +206,6 @@ func (r *SnapshotRepository) GetByID(
return &snapshot, nil return &snapshot, nil
} }
// GetOnlySnapshot returns the sole snapshot in an exported per-snapshot
// metadata database. The backup path writes each snapshot's database with
// exactly one snapshot row (see cleanSnapshotDB), so restore and deep
// verify expect exactly one. Zero rows return ErrNoSnapshotInDatabase and
// more than one returns ErrMultipleSnapshotsInDatabase; callers treat
// either as a failed identity check on the downloaded database.
func (r *SnapshotRepository) GetOnlySnapshot(ctx context.Context) (*Snapshot, error) {
query := `
SELECT id, hostname, vaultik_version, vaultik_git_revision,
started_at, completed_at, file_count, chunk_count, blob_count,
total_size, blob_size, compression_ratio
FROM snapshots
LIMIT 2
`
rows, err := r.db.conn.QueryContext(ctx, query)
if err != nil {
return nil, fmt.Errorf("querying snapshots: %w", err)
}
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
snapshots, err := r.scanSnapshotRows(rows)
if err != nil {
return nil, err
}
switch len(snapshots) {
case 1:
return snapshots[0], nil
case 0:
return nil, ErrNoSnapshotInDatabase
default:
return nil, ErrMultipleSnapshotsInDatabase
}
}
// ListRecent returns up to limit snapshots, most recently started first. // ListRecent returns up to limit snapshots, most recently started first.
func (r *SnapshotRepository) ListRecent( func (r *SnapshotRepository) ListRecent(
ctx context.Context, limit int, ctx context.Context, limit int,
-39
View File
@@ -1,39 +0,0 @@
package log_test
import (
"bytes"
"log/slog"
"strings"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
)
// TestTTYHandlerEscapesControlCharacters logs a message and an attribute
// value that each carry an ESC and a newline — the shape a crafted path or
// storage error from the destination would take — and checks neither raw
// byte reaches the output. The handler's own colour codes (ESC ... m) are
// stripped first; any ESC left after that came from the untrusted value.
func TestTTYHandlerEscapesControlCharacters(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
logger := slog.New(log.NewTTYHandler(&buf, debugHandlerOptions()))
logger.Info("start\x1b[31mZAP\nend", "target", "a\x1b[31mZAP\nb")
out := buf.String()
// The only newline is the line terminator; the injected ones were escaped.
require.Equal(t, 1, strings.Count(out, "\n"),
"a newline in the message or a value must be escaped, not emitted raw")
// After the handler's own colour codes are removed, no ESC survives.
stripped := ansiEscape.ReplaceAllString(out, "")
require.NotContains(t, stripped, "\x1b",
"a raw ESC from the message or a value must not reach the terminal")
// The escaped form is what appears instead.
require.Contains(t, out, `\x1b`)
}
+4 -29
View File
@@ -5,11 +5,9 @@ import (
"fmt" "fmt"
"io" "io"
"log/slog" "log/slog"
"strconv"
"strings" "strings"
"sync" "sync"
"time" "time"
"unicode"
) )
// groupSeparator joins an open group path to an attribute key. This // groupSeparator joins an open group path to an attribute key. This
@@ -118,14 +116,11 @@ func (h *TTYHandler) Handle(_ context.Context, r slog.Record) error {
levelColor = colorReset levelColor = colorReset
} }
// Print main message. The message is escaped before the colour codes // Print main message
// are written around it: it can carry text from an untrusted source
// (a storage error, for one), and a raw control character would
// otherwise reach the terminal.
_, _ = fmt.Fprintf(h.out, "%s%s%s %s%s%s %s%s%s", _, _ = fmt.Fprintf(h.out, "%s%s%s %s%s%s %s%s%s",
colorGray, timestamp, colorReset, colorGray, timestamp, colorReset,
levelColor, level, colorReset, levelColor, level, colorReset,
colorBold, sanitize(r.Message), colorReset) colorBold, r.Message, colorReset)
// Attributes carried by the handler come first, then the record's // Attributes carried by the handler come first, then the record's
// own. Handler attributes were qualified when they were added; the // own. Handler attributes were qualified when they were added; the
@@ -265,29 +260,9 @@ func (h *TTYHandler) writeAttr(a slog.Attr) {
// Future kinds also use the plain string form. // Future kinds also use the plain string form.
} }
// Escape the key and value before the colour codes are written around
// them. Both can carry text from an untrusted source — a manifest
// timestamp, a storage error, a path or symlink target read back from
// the snapshot database — so a control character in one of them must
// be rendered as an escape sequence rather than reaching the terminal,
// where it could move the cursor or inject its own colours.
_, _ = fmt.Fprintf(h.out, " %s%s%s=%s%s%s", _, _ = fmt.Fprintf(h.out, " %s%s%s=%s%s%s",
colorCyan, sanitize(a.Key), colorReset, colorCyan, a.Key, colorReset,
colorBlue, sanitize(value), colorReset) colorBlue, value, colorReset)
}
// sanitize returns s unchanged when every rune in it is printable, and a
// double-quoted, backslash-escaped form (\n, \x1b, …) otherwise. It is
// applied to untrusted text before any colour code is written, so a
// control character can never reach the terminal raw.
func sanitize(s string) string {
for _, r := range s {
if !unicode.IsPrint(r) {
return strconv.Quote(s)
}
}
return s
} }
// formatDuration formats a duration in a human-readable way // formatDuration formats a duration in a human-readable way
+5 -30
View File
@@ -7,19 +7,6 @@ import (
"io" "io"
"github.com/klauspost/compress/zstd" "github.com/klauspost/compress/zstd"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// Manifest size bounds. A manifest lists one small entry per blob, and
// blobs are large (the default target is 10 GB), so even a manifest for a
// petabyte-scale backup is a few megabytes. These caps are far above any
// manifest the writer can emit, yet stop a crafted, highly compressible
// manifest from expanding without limit when decoded: the manifest is
// fetched from the store, which is not trusted, and json.Decode buffers
// the whole value in memory.
const (
manifestMaxCompressed = 256 * 1024 * 1024 // 256 MiB
manifestMaxDecompressed = 1024 * 1024 * 1024 // 1 GiB
) )
// Manifest represents the structure of a snapshot's blob manifest // Manifest represents the structure of a snapshot's blob manifest
@@ -41,31 +28,19 @@ type BlobInfo struct {
CompressedSize int64 `json:"compressed_size"` CompressedSize int64 `json:"compressed_size"`
} }
// DecodeManifest decodes a manifest from a reader containing compressed // DecodeManifest decodes a manifest from a reader containing compressed JSON
// JSON, reading through byte limits on both the compressed input and the
// decompressed output so an untrusted manifest cannot exhaust memory.
func DecodeManifest(r io.Reader) (*Manifest, error) { func DecodeManifest(r io.Reader) (*Manifest, error) {
return decodeManifest(r, manifestMaxCompressed, manifestMaxDecompressed) // Decompress using zstd
} zr, err := zstd.NewReader(r)
// decodeManifest is DecodeManifest with explicit limits, so tests can drive
// the bounds with small inputs instead of gigabyte-scale ones.
func decodeManifest(
r io.Reader, maxCompressed, maxDecompressed int64,
) (*Manifest, error) {
// Decompress using zstd, bounding how many compressed bytes are read.
zr, err := zstd.NewReader(blobgen.LimitReader(r, maxCompressed))
if err != nil { if err != nil {
return nil, fmt.Errorf("creating zstd reader: %w", err) return nil, fmt.Errorf("creating zstd reader: %w", err)
} }
defer zr.Close() defer zr.Close()
// Decode JSON manifest, bounding how far the compressed input may // Decode JSON manifest
// expand: json.Decode buffers the whole value, so without this a
// small, highly compressible manifest could expand to gigabytes.
var manifest Manifest var manifest Manifest
err = json.NewDecoder(blobgen.LimitReader(zr, maxDecompressed)).Decode(&manifest) err = json.NewDecoder(zr).Decode(&manifest)
if err != nil { if err != nil {
return nil, fmt.Errorf("decoding manifest: %w", err) return nil, fmt.Errorf("decoding manifest: %w", err)
} }
-79
View File
@@ -1,79 +0,0 @@
//nolint:testpackage // exercises the unexported decodeManifest bounds
package snapshot
import (
"bytes"
"strings"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// testSnapshotID is a stand-in snapshot ID reused across the bound cases.
const testSnapshotID = "host_home_2026-01-01T00:00:00Z"
// TestDecodeManifestRoundTrip is the baseline: with generous bounds a
// manifest the writer produced decodes back unchanged.
func TestDecodeManifestRoundTrip(t *testing.T) {
t.Parallel()
want := &Manifest{
SnapshotID: testSnapshotID,
Timestamp: "2026-01-01T00:00:00Z",
BlobCount: 2,
TotalCompressedSize: 42,
Blobs: []BlobInfo{
{Hash: "aa", CompressedSize: 21},
{Hash: "bb", CompressedSize: 21},
},
}
compressed, err := EncodeManifest(want, 3)
require.NoError(t, err)
got, err := decodeManifest(
bytes.NewReader(compressed), manifestMaxCompressed, manifestMaxDecompressed)
require.NoError(t, err)
require.Equal(t, want, got)
}
// TestDecodeManifestBoundsDecompressedOutput feeds a valid but highly
// compressible manifest — one whose timestamp is a megabyte of the same
// character — through a small decompressed bound. The compressed form is
// tiny, so only the decompressed bound stops it; decoding must fail within
// that bound rather than expanding the value in memory.
func TestDecodeManifestBoundsDecompressedOutput(t *testing.T) {
t.Parallel()
bomb := &Manifest{
SnapshotID: testSnapshotID,
Timestamp: strings.Repeat("a", 1<<20),
}
compressed, err := EncodeManifest(bomb, 3)
require.NoError(t, err)
require.Less(t, len(compressed), 4096,
"the compressible manifest must be small compressed")
_, err = decodeManifest(bytes.NewReader(compressed), 1<<20, 4096)
require.ErrorIs(t, err, blobgen.ErrOutputTooLarge)
}
// TestDecodeManifestBoundsCompressedInput checks the compressed-input
// bound fires independently: a valid manifest with a generous decompressed
// bound but a tiny compressed bound still fails.
func TestDecodeManifestBoundsCompressedInput(t *testing.T) {
t.Parallel()
manifest := &Manifest{
SnapshotID: testSnapshotID,
Timestamp: strings.Repeat("a", 4096),
}
compressed, err := EncodeManifest(manifest, 3)
require.NoError(t, err)
_, err = decodeManifest(bytes.NewReader(compressed), 8, manifestMaxDecompressed)
require.Error(t, err)
}
+1 -1
View File
@@ -119,7 +119,7 @@ type ScannerConfig struct {
Storage storage.Storer Storage storage.Storer
MaxBlobSize int64 MaxBlobSize int64
CompressionLevel int CompressionLevel int
AgeRecipients []string // required; output is always encrypted AgeRecipients []string // Optional, empty means no encryption
EnableProgress bool // Enable the live progress reporter (ETAs, throughput) EnableProgress bool // Enable the live progress reporter (ETAs, throughput)
UI *ui.Writer // Where user-facing scanner messages go; nil = discard UI *ui.Writer // Where user-facing scanner messages go; nil = discard
Exclude []string // Glob patterns for files/directories to exclude Exclude []string // Glob patterns for files/directories to exclude
+121 -59
View File
@@ -24,7 +24,7 @@ package snapshot
// 7. Close the temporary database // 7. Close the temporary database
// 8. VACUUM the database to remove deleted data and compact (security critical) // 8. VACUUM the database to remove deleted data and compact (security critical)
// 9. Compress the binary database with zstd // 9. Compress the binary database with zstd
// 10. Encrypt the compressed database with age (always; recipients are required) // 10. Encrypt the compressed database with age (if encryption is enabled)
// 11. Upload to S3 as: metadata/{snapshot-id}/db.zst.age // 11. Upload to S3 as: metadata/{snapshot-id}/db.zst.age
// 12. Reopen the main database // 12. Reopen the main database
// //
@@ -201,14 +201,11 @@ func (sm *SnapshotManager) UpdateSnapshotStatsExtended(
}) })
} }
// PopulateSnapshotBlobs ensures snapshot_blobs holds an entry for every // CompleteSnapshot marks a snapshot as completed and ensures snapshot_blobs
// blob that stores a chunk referenced by the snapshot's files, including // is populated with every blob holding any chunk referenced by the
// blobs deduplicated from earlier snapshots. Without it, a fully // snapshot's files (including deduplicated blobs uploaded by prior
// deduplicated snapshot would record no blobs and be unrestorable. // snapshots). Without this, fully-deduplicated snapshots are unrestorable.
// func (sm *SnapshotManager) CompleteSnapshot(
// This must run before ExportSnapshotMetadata: the blob manifest and the
// trimmed metadata database are both built from snapshot_blobs.
func (sm *SnapshotManager) PopulateSnapshotBlobs(
ctx context.Context, snapshotID string, ctx context.Context, snapshotID string,
) error { ) error {
err := sm.repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error { err := sm.repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
@@ -222,25 +219,6 @@ func (sm *SnapshotManager) PopulateSnapshotBlobs(
"snapshot_id", snapshotID, "added", added) "snapshot_id", snapshotID, "added", added)
} }
return nil
})
if err != nil {
return fmt.Errorf("populating snapshot blobs: %w", err)
}
return nil
}
// MarkSnapshotComplete records the snapshot's completion timestamp. On the
// backup path this runs only after ExportSnapshotMetadata has succeeded, so
// the local index never marks a snapshot complete while the destination
// holds no manifest or database for it. A crash before this point leaves the
// snapshot incomplete, and the next run's PruneDatabase drops it. See
// https://git.eeqj.de/sneak/vaultik/issues/177.
func (sm *SnapshotManager) MarkSnapshotComplete(
ctx context.Context, snapshotID string,
) error {
err := sm.repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
return sm.repos.Snapshots.MarkComplete(ctx, tx, snapshotID) return sm.repos.Snapshots.MarkComplete(ctx, tx, snapshotID)
}) })
if err != nil { if err != nil {
@@ -252,22 +230,6 @@ func (sm *SnapshotManager) MarkSnapshotComplete(
return nil return nil
} }
// CompleteSnapshot populates snapshot_blobs and then marks the snapshot
// complete. The backup path (finalizeSnapshotMetadata) instead calls the two
// halves separately, with the metadata export between them, so completion is
// recorded only after a successful export. This convenience is for callers
// that do not interleave an export.
func (sm *SnapshotManager) CompleteSnapshot(
ctx context.Context, snapshotID string,
) error {
err := sm.PopulateSnapshotBlobs(ctx, snapshotID)
if err != nil {
return err
}
return sm.MarkSnapshotComplete(ctx, snapshotID)
}
// ExportSnapshotMetadata exports snapshot metadata to S3 // ExportSnapshotMetadata exports snapshot metadata to S3
// //
// This method executes the complete snapshot metadata export process: // This method executes the complete snapshot metadata export process:
@@ -276,12 +238,14 @@ func (sm *SnapshotManager) CompleteSnapshot(
// 3. Cleans the copy to contain only current snapshot data // 3. Cleans the copy to contain only current snapshot data
// 4. Dumps the cleaned database to SQL // 4. Dumps the cleaned database to SQL
// 5. Compresses the SQL dump with zstd // 5. Compresses the SQL dump with zstd
// 6. Encrypts the compressed data with age (always; recipients are required) // 6. Encrypts the compressed data (if encryption is enabled)
// 7. Uploads to S3 at: snapshots/{snapshot-id}.sql.zst[.age] // 7. Uploads to S3 at: snapshots/{snapshot-id}.sql.zst[.age]
// //
// The only caller (finalizeSnapshotMetadata) does not close the main database // The caller is responsible for:
// before calling this method: the index is copied at dbPath while it is still // - Ensuring the main database is closed before calling this method
// open, and every step here operates on that copy, never on the live index. // - Reopening the main database after this method returns
//
// This ensures database consistency during the copy operation.
func (sm *SnapshotManager) ExportSnapshotMetadata( func (sm *SnapshotManager) ExportSnapshotMetadata(
ctx context.Context, dbPath string, snapshotID string, ctx context.Context, dbPath string, snapshotID string,
) error { ) error {
@@ -331,6 +295,68 @@ func (sm *SnapshotManager) ExportSnapshotMetadata(
return nil return nil
} }
// CleanupIncompleteSnapshots removes incomplete snapshots that don't have
// metadata in S3. This is critical for data safety: incomplete snapshots
// can cause deduplication to skip files that were never successfully
// backed up, resulting in data loss.
func (sm *SnapshotManager) CleanupIncompleteSnapshots(
ctx context.Context, hostname string,
) error {
log.Info("Checking for incomplete snapshots", "hostname", hostname)
// Get all incomplete snapshots for this hostname
incompleteSnapshots, err := sm.repos.Snapshots.GetIncompleteByHostname(ctx, hostname)
if err != nil {
return fmt.Errorf("getting incomplete snapshots: %w", err)
}
if len(incompleteSnapshots) == 0 {
log.Debug("No incomplete snapshots found")
return nil
}
log.Info("Found incomplete snapshots", "count", len(incompleteSnapshots))
// Check each incomplete snapshot for metadata in storage
for _, snapshot := range incompleteSnapshots {
// Check if metadata exists in storage (paths use the hashed
// remote key so we don't leak host info to the listing).
metadataKey := fmt.Sprintf("metadata/%s/db.zst",
RemoteSnapshotKey(snapshot.ID.String()))
_, err := sm.storage.Stat(ctx, metadataKey)
if err != nil {
// Metadata doesn't exist in S3 - this is an incomplete snapshot
log.Info("Cleaning up incomplete snapshot record",
"snapshot_id", snapshot.ID, "started_at", snapshot.StartedAt)
// Delete the snapshot and all its associations
err := sm.deleteSnapshot(ctx, snapshot.ID.String())
if err != nil {
return fmt.Errorf("deleting incomplete snapshot %s: %w",
snapshot.ID, err)
}
log.Info("Deleted incomplete snapshot record and associated data",
"snapshot_id", snapshot.ID)
} else {
// Metadata exists - this snapshot was completed but database wasn't updated
// This shouldn't happen in normal operation, but mark it complete
log.Warn("Found snapshot with remote metadata but incomplete in database",
"snapshot_id", snapshot.ID)
err := sm.repos.Snapshots.MarkComplete(ctx, nil, snapshot.ID.String())
if err != nil {
log.Error("Failed to mark snapshot as complete in database",
"snapshot_id", snapshot.ID, "error", err)
}
}
}
return nil
}
// CleanupOrphanedData removes files, chunks, and blobs that are no longer // CleanupOrphanedData removes files, chunks, and blobs that are no longer
// referenced by any snapshot. This should be called periodically to clean // referenced by any snapshot. This should be called periodically to clean
// up data from deleted or incomplete snapshots. // up data from deleted or incomplete snapshots.
@@ -451,11 +477,9 @@ func (sm *SnapshotManager) prepareExportDB(
// uploadSnapshotArtifacts uploads the database backup and blob manifest // uploadSnapshotArtifacts uploads the database backup and blob manifest
// to remote storage at metadata/<remote-key>/, where remote-key is the // to remote storage at metadata/<remote-key>/, where remote-key is the
// double-SHA256 derivation of the snapshot ID (see RemoteSnapshotKey). // double-SHA256 derivation of the snapshot ID (see RemoteSnapshotKey).
// The human-readable snapshot ID is never written into an unencrypted part // We never write the human-readable snapshot ID into any unencrypted
// of remote storage, so a plain listing shows only the hashed key, not the // part of remote storage so a listing of the destination bucket leaks
// hostname or snapshot name. The hash uses no secret, so a guessed hostname // no host, configuration, or scheduling information.
// and snapshot name can still be confirmed against a listing, and the backup
// time is public: the manifest carries a plaintext timestamp.
func (sm *SnapshotManager) uploadSnapshotArtifacts( func (sm *SnapshotManager) uploadSnapshotArtifacts(
ctx context.Context, snapshotID string, dbData, manifestData []byte, ctx context.Context, snapshotID string, dbData, manifestData []byte,
) error { ) error {
@@ -735,7 +759,7 @@ func (sm *SnapshotManager) compressFile(inputPath, outputPath string) error {
writerClosed = true writerClosed = true
log.Debug("Compression complete", "hash", hex.EncodeToString(writer.ContentID())) log.Debug("Compression complete", "hash", hex.EncodeToString(writer.Sum256()))
return nil return nil
} }
@@ -852,11 +876,10 @@ func (sm *SnapshotManager) generateBlobManifest(
} }
// Create manifest. SnapshotID in the unencrypted manifest is the // Create manifest. SnapshotID in the unencrypted manifest is the
// double-SHA256 remote key (see RemoteSnapshotKey), not the human ID, so // double-SHA256 remote key (see RemoteSnapshotKey), not the human ID,
// neither this field nor the directory name spells out the hostname or // so neither this field nor the directory name reveals the hostname or
// snapshot name — but the key uses no secret, so a guessed hostname and // snapshot name. Timestamp below is written in the clear, so the backup
// snapshot name can be confirmed. Timestamp below is written in the clear, // time is observable to anyone who can read the manifest.
// so the backup time is observable to anyone who can read the manifest.
manifest := &Manifest{ manifest := &Manifest{
SnapshotID: RemoteSnapshotKey(snapshotID), SnapshotID: RemoteSnapshotKey(snapshotID),
Timestamp: time.Now().UTC().Format(time.RFC3339), Timestamp: time.Now().UTC().Format(time.RFC3339),
@@ -910,6 +933,45 @@ type ExtendedBackupStats struct {
UploadDurationMs int64 // Total milliseconds spent uploading to S3 UploadDurationMs int64 // Total milliseconds spent uploading to S3
} }
// deleteSnapshot removes a snapshot and all its associations from the database
func (sm *SnapshotManager) deleteSnapshot(
ctx context.Context, snapshotID string,
) error {
// Delete snapshot_files entries
err := sm.repos.Snapshots.DeleteSnapshotFiles(ctx, snapshotID)
if err != nil {
return fmt.Errorf("deleting snapshot files: %w", err)
}
// Delete snapshot_blobs entries
err = sm.repos.Snapshots.DeleteSnapshotBlobs(ctx, snapshotID)
if err != nil {
return fmt.Errorf("deleting snapshot blobs: %w", err)
}
// Delete uploads entries (has foreign key to snapshots without CASCADE)
err = sm.repos.Snapshots.DeleteSnapshotUploads(ctx, snapshotID)
if err != nil {
return fmt.Errorf("deleting snapshot uploads: %w", err)
}
// Delete the snapshot itself
err = sm.repos.Snapshots.Delete(ctx, snapshotID)
if err != nil {
return fmt.Errorf("deleting snapshot: %w", err)
}
// Clean up orphaned data
log.Debug("Cleaning up orphaned records in main database")
err = sm.CleanupOrphanedData(ctx)
if err != nil {
return fmt.Errorf("cleaning up orphaned data: %w", err)
}
return nil
}
// deleteOtherSnapshots deletes all snapshots except the current one // deleteOtherSnapshots deletes all snapshots except the current one
func (sm *SnapshotManager) deleteOtherSnapshots( func (sm *SnapshotManager) deleteOtherSnapshots(
ctx context.Context, tx *sql.Tx, currentSnapshotID string, ctx context.Context, tx *sql.Tx, currentSnapshotID string,
+50 -7
View File
@@ -1,7 +1,7 @@
// Package types provides custom types for better type safety across the // Package types provides custom types for better type safety across the
// vaultik codebase. Using distinct types for IDs, hashes, and paths prevents // vaultik codebase. Using distinct types for IDs, hashes, paths, and
// accidental mixing of semantically different values that happen to share the // credentials prevents accidental mixing of semantically different values
// same underlying type. // that happen to share the same underlying type.
package types //nolint:revive,nolintlint // rename decision tracked in #76 package types //nolint:revive,nolintlint // rename decision tracked in #76
import ( import (
@@ -146,10 +146,8 @@ type SnapshotID string
// Used for content-addressing and deduplication of file chunks. // Used for content-addressing and deduplication of file chunks.
type ChunkHash string type ChunkHash string
// BlobHash is hex(SHA256(SHA256(uncompressed blob contents))), computed before // BlobHash is the SHA256 hash of a blob's compressed and encrypted content.
// compression and encryption (see blobgen.DoubleSHA256 and // This is used as the filename in S3 storage for content-addressed retrieval.
// docs/REPOSTRUCTURE.md). It is used as the filename in S3 storage for
// content-addressed retrieval.
type BlobHash string type BlobHash string
// FilePath represents an absolute path to a file or directory. // FilePath represents an absolute path to a file or directory.
@@ -159,6 +157,34 @@ type FilePath string
// Used during restore to strip the source prefix from paths. // Used during restore to strip the source prefix from paths.
type SourcePath string type SourcePath string
// AgeRecipient is an age public key used for encryption.
// Format: age1... (Bech32-encoded X25519 public key)
type AgeRecipient string
// AgeSecretKey is an age private key used for decryption.
// Format: AGE-SECRET-KEY-... (Bech32-encoded X25519 private key)
// This type should never be logged or serialized in plaintext.
type AgeSecretKey string
// S3Endpoint is the URL of an S3-compatible storage endpoint.
type S3Endpoint string
// BucketName is the name of an S3 bucket.
type BucketName string
// S3Prefix is the path prefix within an S3 bucket.
type S3Prefix string
// AWSRegion is an AWS region identifier (e.g., "us-east-1").
type AWSRegion string
// AWSAccessKeyID is an AWS access key ID for authentication.
type AWSAccessKeyID string
// AWSSecretAccessKey is an AWS secret access key for authentication.
// This type should never be logged or serialized in plaintext.
type AWSSecretAccessKey string
// Hostname identifies a host machine. // Hostname identifies a host machine.
type Hostname string type Hostname string
@@ -180,7 +206,24 @@ func (h ChunkHash) String() string { return string(h) }
func (h BlobHash) String() string { return string(h) } func (h BlobHash) String() string { return string(h) }
func (p FilePath) String() string { return string(p) } func (p FilePath) String() string { return string(p) }
func (p SourcePath) String() string { return string(p) } func (p SourcePath) String() string { return string(p) }
func (r AgeRecipient) String() string { return string(r) }
func (e S3Endpoint) String() string { return string(e) }
func (b BucketName) String() string { return string(b) }
func (p S3Prefix) String() string { return string(p) }
func (r AWSRegion) String() string { return string(r) }
func (k AWSAccessKeyID) String() string { return string(k) }
func (h Hostname) String() string { return string(h) } func (h Hostname) String() string { return string(h) }
func (v Version) String() string { return string(v) } func (v Version) String() string { return string(v) }
func (r GitRevision) String() string { return string(r) } func (r GitRevision) String() string { return string(r) }
func (p GlobPattern) String() string { return string(p) } func (p GlobPattern) String() string { return string(p) }
// Redacted String methods for sensitive types - prevents accidental logging
func (k AgeSecretKey) String() string { return "[REDACTED]" }
func (k AWSSecretAccessKey) String() string { return "[REDACTED]" }
// Raw returns the actual value for sensitive types when explicitly needed.
func (k AgeSecretKey) Raw() string { return string(k) }
// Raw returns the actual value for sensitive types when explicitly needed.
func (k AWSSecretAccessKey) Raw() string { return string(k) }
-136
View File
@@ -1,136 +0,0 @@
package types_test
import (
"database/sql"
"database/sql/driver"
"fmt"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/types"
)
// scannableID is the shared behaviour of the UUID-backed id types. A pointer
// to FileID or BlobID satisfies it, so both are tested through one set of
// cases.
type scannableID interface {
driver.Valuer
sql.Scanner
fmt.Stringer
IsZero() bool
}
// idKind adapts one id type to the generic tests below.
type idKind struct {
name string
newZero func() scannableID
newRandom func() scannableID
parse func(string) (scannableID, error)
}
func idKinds() []idKind {
return []idKind{
{
name: "FileID",
newZero: func() scannableID { return &types.FileID{} },
newRandom: func() scannableID {
id := types.NewFileID()
return &id
},
parse: func(s string) (scannableID, error) {
id, err := types.ParseFileID(s)
return &id, err
},
},
{
name: "BlobID",
newZero: func() scannableID { return &types.BlobID{} },
newRandom: func() scannableID {
id := types.NewBlobID()
return &id
},
parse: func(s string) (scannableID, error) {
id, err := types.ParseBlobID(s)
return &id, err
},
},
}
}
// TestIDValueScan checks that Value then Scan round trips from both a string
// and a []byte, that a NULL scans to the zero id, and that a non-string type
// and malformed text are rejected.
func TestIDValueScan(t *testing.T) {
t.Parallel()
for _, k := range idKinds() {
t.Run(k.name, func(t *testing.T) {
t.Parallel()
orig := k.newRandom()
v, err := orig.Value()
require.NoError(t, err)
s, ok := v.(string)
require.True(t, ok, "Value must yield a string")
fromString := k.newZero()
require.NoError(t, fromString.Scan(s))
assert.Equal(t, orig.String(), fromString.String())
assert.False(t, fromString.IsZero())
fromBytes := k.newZero()
require.NoError(t, fromBytes.Scan([]byte(s)))
assert.Equal(t, orig.String(), fromBytes.String())
nulled := k.newRandom()
require.NoError(t, nulled.Scan(nil))
assert.True(t, nulled.IsZero(), "NULL scans to the zero id")
require.Error(t, k.newZero().Scan(42),
"a non-string type must be rejected")
assert.Error(t, k.newZero().Scan("not-a-uuid"),
"malformed text must be rejected")
})
}
}
// TestIDParse checks that the Parse function accepts a canonical id and
// rejects malformed text.
func TestIDParse(t *testing.T) {
t.Parallel()
for _, k := range idKinds() {
t.Run(k.name, func(t *testing.T) {
t.Parallel()
canonical := k.newRandom().String()
parsed, err := k.parse(canonical)
require.NoError(t, err)
assert.Equal(t, canonical, parsed.String())
_, err = k.parse("not-a-uuid")
assert.Error(t, err)
})
}
}
// TestIDIsZero checks IsZero on the zero and on a freshly generated id.
func TestIDIsZero(t *testing.T) {
t.Parallel()
for _, k := range idKinds() {
t.Run(k.name, func(t *testing.T) {
t.Parallel()
assert.True(t, k.newZero().IsZero())
assert.False(t, k.newRandom().IsZero())
})
}
}
+3 -22
View File
@@ -23,9 +23,7 @@ import (
"fmt" "fmt"
"io" "io"
"os" "os"
"strconv"
"time" "time"
"unicode"
"github.com/dustin/go-humanize" "github.com/dustin/go-humanize"
"golang.org/x/term" "golang.org/x/term"
@@ -227,17 +225,17 @@ func (w *Writer) Hex(s string) string {
short = s[:hexAbbrevLen] + "..." short = s[:hexAbbrevLen] + "..."
} }
return w.paint(ansiCyan, sanitize(short)) return w.paint(ansiCyan, short)
} }
// Snapshot colorizes a snapshot ID (full, no abbreviation). // Snapshot colorizes a snapshot ID (full, no abbreviation).
func (w *Writer) Snapshot(id string) string { func (w *Writer) Snapshot(id string) string {
return w.paint(ansiCyan+ansiBold, sanitize(id)) return w.paint(ansiCyan+ansiBold, id)
} }
// Path colorizes a filesystem path. // Path colorizes a filesystem path.
func (w *Writer) Path(p string) string { func (w *Writer) Path(p string) string {
return w.paint(ansiBlue, sanitize(p)) return w.paint(ansiBlue, p)
} }
// Size colorizes a byte count using humanize.Bytes. // Size colorizes a byte count using humanize.Bytes.
@@ -312,23 +310,6 @@ func (w *Writer) paint(color, s string) string {
return color + s + ansiReset return color + s + ansiReset
} }
// sanitize returns s unchanged when every rune in it is printable, and a
// double-quoted, backslash-escaped form (\n, \x1b, …) otherwise. The
// string value formatters escape their argument through this before
// painting: identifiers, paths and symlink targets they render come from
// the snapshot database, which is not trusted, and escaping must happen
// before colour is applied — the painted result already contains the
// escape codes the raw text would otherwise be indistinguishable from.
func sanitize(s string) string {
for _, r := range s {
if !unicode.IsPrint(r) {
return strconv.Quote(s)
}
}
return s
}
// emit writes "<prefix> <body>\n" with the prefix painted in prefixColor // emit writes "<prefix> <body>\n" with the prefix painted in prefixColor
// and the body optionally painted in bodyColor (empty = no body color). // and the body optionally painted in bodyColor (empty = no body color).
func (w *Writer) emit(prefixColor, prefix, bodyColor, format string, args []any) { func (w *Writer) emit(prefixColor, prefix, bodyColor, format string, args []any) {
-32
View File
@@ -1,32 +0,0 @@
package ui_test
import (
"strings"
"testing"
)
// TestValueFormattersEscapeControlCharacters checks that a path carrying an
// ESC and a newline — the shape a symlink target read back from the
// snapshot database could take — is escaped before it reaches the output.
// Colour is off here, so the only way a control byte could appear is from
// the value itself.
func TestValueFormattersEscapeControlCharacters(t *testing.T) {
t.Parallel()
w, buf := newTestWriter(false)
w.Infof("restoring %s", w.Path("a\x1b[31mZAP\nb"))
out := buf.String()
if strings.ContainsRune(out, '\x1b') {
t.Fatalf("raw ESC from a value survived in output: %q", out)
}
if strings.Count(out, "\n") != 1 {
t.Fatalf("a newline in a value must be escaped, not emitted raw: %q", out)
}
if !strings.Contains(out, `\x1b`) {
t.Fatalf("expected the escaped form of ESC in output: %q", out)
}
}
+41 -26
View File
@@ -2,13 +2,16 @@ package vaultik
import ( import (
"context" "context"
"crypto/sha256"
"encoding/hex" "encoding/hex"
"errors" "errors"
"fmt" "fmt"
"io" "io"
"time"
"filippo.io/age" "filippo.io/age"
"sneak.berlin/go/vaultik/internal/blobgen" "sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/log"
) )
// errBlobHashMismatch is returned when a fetched blob's content hash does // errBlobHashMismatch is returned when a fetched blob's content hash does
@@ -28,14 +31,13 @@ var errBlobNotFullyRead = errors.New(
// redundant SHA-256 computation. // redundant SHA-256 computation.
type hashVerifyReader struct { type hashVerifyReader struct {
reader *blobgen.Reader // underlying decrypted blob reader (has internal hasher) reader *blobgen.Reader // underlying decrypted blob reader (has internal hasher)
limited io.Reader // reader bounded to the blob's recorded plaintext size
fetcher io.ReadCloser // raw fetched stream (closed on Close) fetcher io.ReadCloser // raw fetched stream (closed on Close)
blobHash string // expected double-SHA-256 hex blobHash string // expected double-SHA-256 hex
done bool // EOF reached done bool // EOF reached
} }
func (h *hashVerifyReader) Read(p []byte) (int, error) { func (h *hashVerifyReader) Read(p []byte) (int, error) {
n, err := h.limited.Read(p) n, err := h.reader.Read(p)
if errors.Is(err, io.EOF) { if errors.Is(err, io.EOF) {
h.done = true h.done = true
} }
@@ -55,10 +57,14 @@ func (h *hashVerifyReader) Close() error {
return errBlobNotFullyRead return errBlobNotFullyRead
} }
actualHashHex := hex.EncodeToString(blobgen.DoubleSHA256(h.reader.Sum256())) firstHash := h.reader.Sum256()
secondHasher := sha256.New()
secondHasher.Write(firstHash)
actualHashHex := hex.EncodeToString(secondHasher.Sum(nil))
if actualHashHex != h.blobHash { if actualHashHex != h.blobHash {
return fmt.Errorf("%w: expected %s, got %s", return fmt.Errorf("%w: expected %s, got %s",
errBlobHashMismatch, shortHash(h.blobHash), shortHash(actualHashHex)) errBlobHashMismatch, h.blobHash[:16], actualHashHex[:16])
} }
if readerErr != nil { if readerErr != nil {
@@ -72,22 +78,15 @@ func (h *hashVerifyReader) Close() error {
// returns a streaming reader that computes the double-SHA-256 hash on the fly. // returns a streaming reader that computes the double-SHA-256 hash on the fly.
// The hash is verified when the returned reader is closed (after fully reading). // The hash is verified when the returned reader is closed (after fully reading).
// This avoids buffering the entire blob in memory. // This avoids buffering the entire blob in memory.
//
// maxPlaintextSize is the blob's uncompressed_size as recorded in the
// snapshot database. Decompression stops with blobgen.ErrOutputTooLarge
// once the plaintext exceeds it, so a tampered blob cannot expand without
// limit — using the recorded size, not the restoring host's
// blob_size_limit, since that config may differ from the backup host's.
func (v *Vaultik) FetchAndDecryptBlob( func (v *Vaultik) FetchAndDecryptBlob(
ctx context.Context, blobHash string, maxPlaintextSize int64, ctx context.Context, blobHash string, expectedSize int64, identity age.Identity,
identities ...age.Identity,
) (io.ReadCloser, error) { ) (io.ReadCloser, error) {
rc, err := v.FetchBlob(ctx, blobHash) rc, _, err := v.FetchBlob(ctx, blobHash, expectedSize)
if err != nil { if err != nil {
return nil, err return nil, err
} }
reader, err := blobgen.NewReader(rc, identities...) reader, err := blobgen.NewReader(rc, identity)
if err != nil { if err != nil {
_ = rc.Close() _ = rc.Close()
@@ -96,29 +95,45 @@ func (v *Vaultik) FetchAndDecryptBlob(
return &hashVerifyReader{ return &hashVerifyReader{
reader: reader, reader: reader,
limited: blobgen.LimitReader(reader, maxPlaintextSize),
fetcher: rc, fetcher: rc,
blobHash: blobHash, blobHash: blobHash,
}, nil }, nil
} }
// FetchBlob downloads a blob and returns a reader for the encrypted data. // FetchBlob downloads a blob and returns a reader for the encrypted data.
// Times the Storage.Get and Storage.Stat round-trips separately at
// debug level so we can see whether the size-only Stat (which is an
// extra request on every fetch) is hurting throughput.
func (v *Vaultik) FetchBlob( func (v *Vaultik) FetchBlob(
ctx context.Context, blobHash string, ctx context.Context, blobHash string, expectedSize int64,
) (io.ReadCloser, error) { ) (io.ReadCloser, int64, error) {
// blobHash reaches here from the snapshot database, which is not
// trusted. Reject a malformed hash before it is spliced into a storage
// path (blobHash[:2]/blobHash[2:4]) or a fetch is attempted.
if !isBlobHash(blobHash) {
return nil, fmt.Errorf("%w: %s", errInvalidBlobHash, shortHash(blobHash))
}
blobPath := fmt.Sprintf("blobs/%s/%s/%s", blobHash[:2], blobHash[2:4], blobHash) blobPath := fmt.Sprintf("blobs/%s/%s/%s", blobHash[:2], blobHash[2:4], blobHash)
t0 := time.Now()
rc, err := v.Storage.Get(ctx, blobPath) rc, err := v.Storage.Get(ctx, blobPath)
getDur := time.Since(t0)
if err != nil { if err != nil {
return nil, fmt.Errorf("downloading blob %s: %w", shortHash(blobHash), err) return nil, 0, fmt.Errorf("downloading blob %s: %w", blobHash[:16], err)
} }
return rc, nil t0 = time.Now()
info, err := v.Storage.Stat(ctx, blobPath)
statDur := time.Since(t0)
if err != nil {
_ = rc.Close()
return nil, 0, fmt.Errorf("stat blob %s: %w", blobHash[:16], err)
}
log.Debug("FetchBlob round-trips",
"hash", blobHash[:16],
"ms_storage_get", getDur.Milliseconds(),
"ms_storage_stat", statDur.Milliseconds(),
"expected_size", expectedSize,
"stat_size", info.Size,
)
return rc, info.Size, nil
} }
-50
View File
@@ -1,50 +0,0 @@
package vaultik_test
import (
"context"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestFetchAndDecryptBlobBoundsPlaintext feeds a small, highly
// compressible blob (256 KiB of zeros) whose decompressed size far exceeds
// the plaintext bound passed to FetchAndDecryptBlob. Decompression must
// stop with blobgen.ErrOutputTooLarge within the bound rather than
// expanding the whole blob into the restore cache.
func TestFetchAndDecryptBlobBoundsPlaintext(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
plaintext := make([]byte, 256*1024)
encryptedData, correctHash := buildHashTestBlob(t, identity, plaintext)
mockStorage := NewMockStorer()
blobPath := "blobs/" + correctHash[:2] + "/" +
correctHash[2:4] + "/" + correctHash
mockStorage.mu.Lock()
mockStorage.data[blobPath] = encryptedData
mockStorage.mu.Unlock()
tv := vaultik.NewForTesting(mockStorage)
const maxPlaintext = 1024
rc, err := tv.FetchAndDecryptBlob(
context.Background(), correctHash, maxPlaintext, identity)
require.NoError(t, err)
n, copyErr := io.Copy(io.Discard, rc)
_ = rc.Close()
require.ErrorIs(t, copyErr, blobgen.ErrOutputTooLarge)
require.LessOrEqual(t, n, int64(maxPlaintext)+1,
"decompression must stop within the recorded plaintext bound")
}
+5 -34
View File
@@ -40,13 +40,13 @@ func buildHashTestBlob(
} }
// Compute the double-SHA-256 hash of the plaintext (matches // Compute the double-SHA-256 hash of the plaintext (matches
// blobgen.Writer.ContentID). // blobgen.Writer.Sum256).
firstHash := sha256.Sum256(plaintext) firstHash := sha256.Sum256(plaintext)
secondHash := sha256.Sum256(firstHash[:]) secondHash := sha256.Sum256(firstHash[:])
correctHash := hex.EncodeToString(secondHash[:]) correctHash := hex.EncodeToString(secondHash[:])
// Verify our hash matches what blobgen.Writer produces // Verify our hash matches what blobgen.Writer produces
writerHash := hex.EncodeToString(writer.ContentID()) writerHash := hex.EncodeToString(writer.Sum256())
if correctHash != writerHash { if correctHash != writerHash {
t.Fatalf("hash computation mismatch: manual=%s, writer=%s", t.Fatalf("hash computation mismatch: manual=%s, writer=%s",
correctHash, writerHash) correctHash, writerHash)
@@ -55,35 +55,6 @@ func buildHashTestBlob(
return encBuf.Bytes(), correctHash return encBuf.Bytes(), correctHash
} }
// TestFetchBlobRejectsMalformedHash verifies FetchBlob refuses a blob hash
// that is not 64 lowercase hex characters before it builds a storage path
// or issues any request. The hash reaches FetchBlob from the snapshot
// database, which is not trusted, so a value such as one containing "/.."
// must never reach the store.
func TestFetchBlobRejectsMalformedHash(t *testing.T) {
t.Parallel()
mockStorage := NewMockStorer()
tv := vaultik.NewForTesting(mockStorage)
ctx := context.Background()
for _, bad := range []string{
"aa/../../../home/u/.profile",
"abc",
strings.Repeat("A", 64), // uppercase hex is not accepted
strings.Repeat("g", 64), // not hex
} {
_, err := tv.FetchBlob(ctx, bad)
if err == nil {
t.Fatalf("expected error for malformed hash %q, got nil", bad)
}
}
if calls := mockStorage.GetCalls(); len(calls) != 0 {
t.Fatalf("storage was accessed for a malformed hash: %v", calls)
}
}
// TestFetchAndDecryptBlobVerifiesHash verifies that FetchAndDecryptBlob checks // TestFetchAndDecryptBlobVerifiesHash verifies that FetchAndDecryptBlob checks
// the double-SHA-256 hash of the decrypted plaintext against the expected blob hash. // the double-SHA-256 hash of the decrypted plaintext against the expected blob hash.
func TestFetchAndDecryptBlobVerifiesHash(t *testing.T) { func TestFetchAndDecryptBlobVerifiesHash(t *testing.T) {
@@ -113,7 +84,7 @@ func TestFetchAndDecryptBlobVerifiesHash(t *testing.T) {
t.Parallel() t.Parallel()
rc, err := tv.FetchAndDecryptBlob( rc, err := tv.FetchAndDecryptBlob(
ctx, correctHash, int64(len(plaintext)), identity) ctx, correctHash, int64(len(encryptedData)), identity)
if err != nil { if err != nil {
t.Fatalf("expected success, got error: %v", err) t.Fatalf("expected success, got error: %v", err)
} }
@@ -145,7 +116,7 @@ func TestFetchAndDecryptBlobVerifiesHash(t *testing.T) {
mockStorage.mu.Unlock() mockStorage.mu.Unlock()
rc, err := tv.FetchAndDecryptBlob( rc, err := tv.FetchAndDecryptBlob(
ctx, fakeHash, int64(len(plaintext)), identity) ctx, fakeHash, int64(len(encryptedData)), identity)
if err != nil { if err != nil {
t.Fatalf("unexpected error opening stream: %v", err) t.Fatalf("unexpected error opening stream: %v", err)
} }
@@ -188,7 +159,7 @@ func TestFetchAndDecryptBlobCloseBeforeEOFFails(t *testing.T) {
tv := vaultik.NewForTesting(mockStorage) tv := vaultik.NewForTesting(mockStorage)
rc, err := tv.FetchAndDecryptBlob( rc, err := tv.FetchAndDecryptBlob(
context.Background(), correctHash, int64(len(plaintext)), identity) context.Background(), correctHash, int64(len(encryptedData)), identity)
if err != nil { if err != nil {
t.Fatalf("unexpected error opening stream: %v", err) t.Fatalf("unexpected error opening stream: %v", err)
} }
+14 -72
View File
@@ -6,7 +6,6 @@ import (
"io" "io"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"sync" "sync"
) )
@@ -14,9 +13,6 @@ import (
var ( var (
errCacheKeyMissing = errors.New("key not in cache") errCacheKeyMissing = errors.New("key not in cache")
errCacheReadBeyondBlob = errors.New("read beyond blob size") errCacheReadBeyondBlob = errors.New("read beyond blob size")
errCacheKeyHasSeparator = errors.New(
"cache key contains a path separator")
errCacheNegativeRead = errors.New("negative offset or length")
) )
// blobCacheFileMode is the permission mode for cached blob files. // blobCacheFileMode is the permission mode for cached blob files.
@@ -78,11 +74,6 @@ func newBlobDiskCache(maxBytes int64) (*blobDiskCache, error) {
// Put writes blob data to disk cache. Entries larger than maxBytes are // Put writes blob data to disk cache. Entries larger than maxBytes are
// silently skipped. // silently skipped.
func (c *blobDiskCache) Put(key string, data []byte) error { func (c *blobDiskCache) Put(key string, data []byte) error {
p, err := c.path(key)
if err != nil {
return err
}
entrySize := int64(len(data)) entrySize := int64(len(data))
c.mu.Lock() c.mu.Lock()
@@ -96,12 +87,11 @@ func (c *blobDiskCache) Put(key string, data []byte) error {
if e, ok := c.items[key]; ok { if e, ok := c.items[key]; ok {
c.unlink(e) c.unlink(e)
c.curBytes -= e.size c.curBytes -= e.size
_ = os.Remove(p) _ = os.Remove(c.path(key))
delete(c.items, key) delete(c.items, key)
} }
err = os.WriteFile(p, data, blobCacheFileMode) err := os.WriteFile(c.path(key), data, blobCacheFileMode)
if err != nil { if err != nil {
return fmt.Errorf("writing blob to cache: %w", err) return fmt.Errorf("writing blob to cache: %w", err)
} }
@@ -129,26 +119,19 @@ func (c *blobDiskCache) Put(key string, data []byte) error {
// disk without buffering its entire plaintext (which may be tens of GB) // disk without buffering its entire plaintext (which may be tens of GB)
// in RAM. // in RAM.
func (c *blobDiskCache) PutFromReader(key string, r io.Reader) (int64, error) { func (c *blobDiskCache) PutFromReader(key string, r io.Reader) (int64, error) {
p, err := c.path(key)
if err != nil {
return 0, err
}
c.mu.Lock() c.mu.Lock()
// Remove any prior entry first; we'll re-link after the file is // Remove any prior entry first; we'll re-link after the file is
// written successfully. // written successfully.
if e, ok := c.items[key]; ok { if e, ok := c.items[key]; ok {
c.unlink(e) c.unlink(e)
c.curBytes -= e.size c.curBytes -= e.size
_ = os.Remove(p) _ = os.Remove(c.path(key))
delete(c.items, key) delete(c.items, key)
} }
c.mu.Unlock() c.mu.Unlock()
//nolint:gosec // G304: path() rejects keys with a separator
f, err := os.OpenFile( f, err := os.OpenFile(
p, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, blobCacheFileMode) c.path(key), os.O_CREATE|os.O_TRUNC|os.O_WRONLY, blobCacheFileMode)
if err != nil { if err != nil {
return 0, fmt.Errorf("creating cache file: %w", err) return 0, fmt.Errorf("creating cache file: %w", err)
} }
@@ -157,13 +140,13 @@ func (c *blobDiskCache) PutFromReader(key string, r io.Reader) (int64, error) {
closeErr := f.Close() closeErr := f.Close()
if copyErr != nil { if copyErr != nil {
_ = os.Remove(p) _ = os.Remove(c.path(key))
return written, fmt.Errorf("streaming to cache file: %w", copyErr) return written, fmt.Errorf("streaming to cache file: %w", copyErr)
} }
if closeErr != nil { if closeErr != nil {
_ = os.Remove(p) _ = os.Remove(c.path(key))
return written, fmt.Errorf("closing cache file: %w", closeErr) return written, fmt.Errorf("closing cache file: %w", closeErr)
} }
@@ -175,7 +158,7 @@ func (c *blobDiskCache) PutFromReader(key string, r io.Reader) (int64, error) {
// floor — but the restore path passes math.MaxInt64 as maxBytes // floor — but the restore path passes math.MaxInt64 as maxBytes
// so this branch is effectively unreachable there. // so this branch is effectively unreachable there.
if written > c.maxBytes { if written > c.maxBytes {
_ = os.Remove(p) _ = os.Remove(c.path(key))
return written, nil return written, nil
} }
@@ -198,11 +181,6 @@ func (c *blobDiskCache) PutFromReader(key string, r io.Reader) (int64, error) {
// Get reads a cached blob from disk. Returns data and true on hit. // Get reads a cached blob from disk. Returns data and true on hit.
func (c *blobDiskCache) Get(key string) ([]byte, bool) { func (c *blobDiskCache) Get(key string) ([]byte, bool) {
p, err := c.path(key)
if err != nil {
return nil, false
}
c.mu.Lock() c.mu.Lock()
c.getCalls++ c.getCalls++
@@ -217,8 +195,7 @@ func (c *blobDiskCache) Get(key string) ([]byte, bool) {
c.pushFront(e) c.pushFront(e)
c.mu.Unlock() c.mu.Unlock()
//nolint:gosec // G304: path() rejects keys with a separator data, err := os.ReadFile(c.path(key))
data, err := os.ReadFile(p)
if err != nil { if err != nil {
c.mu.Lock() c.mu.Lock()
if e2, ok2 := c.items[key]; ok2 && e2 == e { if e2, ok2 := c.items[key]; ok2 && e2 == e {
@@ -236,20 +213,6 @@ func (c *blobDiskCache) Get(key string) ([]byte, bool) {
// ReadAt reads a slice of a cached blob without loading the entire blob into memory. // ReadAt reads a slice of a cached blob without loading the entire blob into memory.
func (c *blobDiskCache) ReadAt(key string, offset, length int64) ([]byte, error) { func (c *blobDiskCache) ReadAt(key string, offset, length int64) ([]byte, error) {
p, err := c.path(key)
if err != nil {
return nil, err
}
// offset and length come from a blob_chunks row read back from the
// destination. A negative value must be rejected outright; the upper
// bound is checked as length > size-offset (a subtraction) so a huge
// offset+length cannot overflow int64 and slip past the check.
if offset < 0 || length < 0 {
return nil, fmt.Errorf("%w: offset=%d length=%d",
errCacheNegativeRead, offset, length)
}
c.mu.Lock() c.mu.Lock()
c.readAtCalls++ c.readAtCalls++
@@ -260,7 +223,7 @@ func (c *blobDiskCache) ReadAt(key string, offset, length int64) ([]byte, error)
return nil, fmt.Errorf("%w: %q", errCacheKeyMissing, key) return nil, fmt.Errorf("%w: %q", errCacheKeyMissing, key)
} }
if length > e.size-offset { if offset+length > e.size {
c.mu.Unlock() c.mu.Unlock()
return nil, fmt.Errorf("%w: offset=%d length=%d size=%d", return nil, fmt.Errorf("%w: offset=%d length=%d size=%d",
@@ -271,7 +234,7 @@ func (c *blobDiskCache) ReadAt(key string, offset, length int64) ([]byte, error)
c.pushFront(e) c.pushFront(e)
c.mu.Unlock() c.mu.Unlock()
f, err := os.Open(p) //nolint:gosec // G304: path() rejects keys with a separator f, err := os.Open(c.path(key))
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -313,13 +276,7 @@ func (c *blobDiskCache) Delete(key string) {
c.unlink(e) c.unlink(e)
delete(c.items, key) delete(c.items, key)
c.curBytes -= e.size c.curBytes -= e.size
_ = os.Remove(c.path(key))
// The key is already in the map, so it passed path() when it was
// inserted; the error cannot occur here.
p, err := c.path(key)
if err == nil {
_ = os.Remove(p)
}
} }
// Keys returns a snapshot of all cached keys. Safe for iteration without // Keys returns a snapshot of all cached keys. Safe for iteration without
@@ -390,18 +347,8 @@ func (c *blobDiskCache) Close() error {
return os.RemoveAll(c.dir) return os.RemoveAll(c.dir)
} }
// path returns the on-disk location of the cache file for key. The key is func (c *blobDiskCache) path(key string) string {
// a blob hash read back from the destination and is not trusted: a value return filepath.Join(c.dir, key)
// such as "aa/../../../home/u/.profile" would otherwise make filepath.Join
// escape the cache directory, so a key containing a path separator is
// refused rather than joined.
func (c *blobDiskCache) path(key string) (string, error) {
if strings.ContainsRune(key, '/') ||
strings.ContainsRune(key, filepath.Separator) {
return "", fmt.Errorf("%w: %q", errCacheKeyHasSeparator, key)
}
return filepath.Join(c.dir, key), nil
} }
func (c *blobDiskCache) unlink(e *blobDiskCacheEntry) { func (c *blobDiskCache) unlink(e *blobDiskCacheEntry) {
@@ -444,10 +391,5 @@ func (c *blobDiskCache) evictLRU() {
c.unlink(victim) c.unlink(victim)
delete(c.items, victim.key) delete(c.items, victim.key)
c.curBytes -= victim.size c.curBytes -= victim.size
_ = os.Remove(c.path(victim.key))
// victim.key was validated by path() on insertion, so this cannot err.
p, err := c.path(victim.key)
if err == nil {
_ = os.Remove(p)
}
} }
-52
View File
@@ -1,52 +0,0 @@
package vaultik
import "errors"
// blobHashHexLen is the length of a blob hash written as lowercase hex: a
// SHA-256 digest is 32 bytes, so 64 characters. Remote snapshot keys are
// SHA-256 hashes too and share this exact form.
const blobHashHexLen = 64
// shortHashLen is how many leading characters of a hash appear in log and
// error text.
const shortHashLen = 16
// errInvalidBlobHash reports a value used as a blob hash that is not
// exactly 64 lowercase hex characters. Restore, verify and prune read
// these values back from the destination, which is not trusted, so each
// one is checked before it is used to build a path or drive a read.
var errInvalidBlobHash = errors.New(
"blob hash is not 64 lowercase hex characters")
// isBlobHash reports whether s is exactly 64 lowercase hex characters.
// Every real blob hash and remote snapshot key has this form.
//
// The check is a plain function, not a method on types.BlobHash: the
// packer stores "temp-placeholder-{uuid}" as the hash of an unfinished
// blob in the local index, so the type itself must keep accepting values
// that are not hashes.
func isBlobHash(s string) bool {
if len(s) != blobHashHexLen {
return false
}
for _, r := range s {
if (r < '0' || r > '9') && (r < 'a' || r > 'f') {
return false
}
}
return true
}
// shortHash returns the leading part of a hash for log and error text. It
// never panics: a string shorter than the prefix is returned whole. A hash
// read from the destination may be malformed, and formatting one for a
// message must not crash the command.
func shortHash(s string) string {
if len(s) <= shortHashLen {
return s
}
return s[:shortHashLen]
}
@@ -1,262 +0,0 @@
package vaultik //nolint:testpackage // drives unexported input validation
import (
"context"
"errors"
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/types"
)
// These tests treat every hash, offset and length read back from the
// destination as hostile. A blob hash comes from the downloaded snapshot
// database or the store listing, neither of which is authenticated (see
// https://git.eeqj.de/sneak/vaultik/issues/155), so each is validated
// before it is used to build a path or size an allocation.
// TestBlobCacheRejectsKeyWithSeparator proves the arbitrary-file-write
// hole is closed: a blob hash that climbs out of the cache directory is
// refused and nothing is written outside it. This is the exact write the
// restore path performs, keyed by the hash from the snapshot database.
func TestBlobCacheRejectsKeyWithSeparator(t *testing.T) {
t.Parallel()
cache, err := newBlobDiskCache(1 << 20)
require.NoError(t, err)
defer func() { _ = cache.Close() }()
target := filepath.Join(t.TempDir(), "pwned")
// A hash whose relative form escapes the cache directory to target.
key, err := filepath.Rel(cache.dir, target)
require.NoError(t, err)
require.Contains(t, key, "..")
err = cache.Put(key, []byte("secret"))
require.ErrorIs(t, err, errCacheKeyHasSeparator)
_, err = cache.PutFromReader(key, strings.NewReader("secret"))
require.ErrorIs(t, err, errCacheKeyHasSeparator)
_, statErr := os.Stat(target)
require.Truef(t, os.IsNotExist(statErr),
"cache wrote outside its directory at %s", target)
}
// TestBuildBlobIndexesRejectsHostileHash proves restore refuses a snapshot
// database whose blob_hash escapes the cache directory. buildBlobIndexes is
// the first place restore reads these hashes back, and it fails there, before
// any blob is fetched or written, so a hash containing /../ cannot steer a
// later write outside the cache directory.
func TestBuildBlobIndexesRejectsHostileHash(t *testing.T) {
t.Parallel()
ctx := context.Background()
db, err := database.New(ctx, filepath.Join(t.TempDir(), "index.sqlite"))
require.NoError(t, err)
defer func() { _ = db.Close() }()
// A blob cache and a target file just outside it. The hostile hash is
// the relative path from the cache to that target, so an unguarded
// restore keyed by this hash would write there.
cache, err := newBlobDiskCache(1 << 20)
require.NoError(t, err)
defer func() { _ = cache.Close() }()
target := filepath.Join(t.TempDir(), "pwned")
hostile, err := filepath.Rel(cache.dir, target)
require.NoError(t, err)
require.Contains(t, hostile, "..")
repos := database.NewRepositories(db)
require.NoError(t, repos.Blobs.Create(ctx, nil, &database.Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash(hostile),
CreatedTS: time.Now().UTC(),
}))
v := NewForTesting(nil)
v.SetContext(ctx)
_, _, err = v.buildBlobIndexes(repos)
require.ErrorIs(t, err, errInvalidBlobHash)
_, statErr := os.Stat(target)
require.Truef(t, os.IsNotExist(statErr),
"restore wrote outside the cache directory at %s", target)
}
// TestBlobCacheReadAtRejectsBadBounds proves a blob_chunks row cannot
// drive an out-of-range or negative read. offset/length reach ReadAt
// straight from the database.
func TestBlobCacheReadAtRejectsBadBounds(t *testing.T) {
t.Parallel()
cache, err := newBlobDiskCache(1 << 20)
require.NoError(t, err)
defer func() { _ = cache.Close() }()
require.NoError(t, cache.Put("blob", make([]byte, 100)))
_, err = cache.ReadAt("blob", -1, 10)
require.ErrorIs(t, err, errCacheNegativeRead)
_, err = cache.ReadAt("blob", 0, -1)
require.ErrorIs(t, err, errCacheNegativeRead)
// A length past the end is rejected via the subtraction bound, so a
// huge offset+length cannot overflow past the check.
_, err = cache.ReadAt("blob", 50, 60)
require.ErrorIs(t, err, errCacheReadBeyondBlob)
}
// TestListAllRemoteBlobsSkipsNonConformingName proves a bogus object name
// under blobs/ (here a three-character name) is skipped rather than
// entering the blob map, so prune's later hash[:2]/hash[2:4] path build
// cannot panic on it.
func TestListAllRemoteBlobsSkipsNonConformingName(t *testing.T) {
// Initialize the global logger before t.Parallel() so the write lands
// in the serial phase and cannot race other parallel tests reading it.
log.Initialize(log.Config{})
t.Parallel()
good := strings.Repeat("a", blobHashHexLen)
store := &stubLister{objects: []storage.ObjectInfo{
{Key: "blobs/" + good[:2] + "/" + good[2:4] + "/" + good, Size: 10},
{Key: "blobs/a/b/c", Size: 3},
}}
v := &Vaultik{Storage: store}
v.SetContext(context.Background())
blobs, err := v.listAllRemoteBlobs()
require.NoError(t, err)
require.Contains(t, blobs, good)
require.NotContains(t, blobs, "c")
require.Len(t, blobs, 1)
}
// TestVerifyManifestBlobsRejectsShortHash proves a manifest (which is not
// authenticated) with a short blob hash fails cleanly instead of panicking
// on blob.Hash[:2].
func TestVerifyManifestBlobsRejectsShortHash(t *testing.T) {
t.Parallel()
v := &Vaultik{Stdout: io.Discard}
manifest := &snapshot.Manifest{
Blobs: []snapshot.BlobInfo{{Hash: "abc", CompressedSize: 1}},
}
verified, missing, mismatched, missingSize, err :=
v.verifyManifestBlobs(manifest, &VerifyOptions{JSON: true})
require.ErrorIs(t, err, errInvalidBlobHash)
require.Zero(t, verified)
require.Zero(t, missing)
require.Zero(t, mismatched)
require.Zero(t, missingSize)
}
// TestVerifyBlobChunksRejectsNegativeLength proves a blob_chunks row with a
// negative length returns an error rather than reaching make([]byte,
// length) or streaming an untrusted size.
func TestVerifyBlobChunksRejectsNegativeLength(t *testing.T) {
t.Parallel()
ctx := context.Background()
db, err := database.New(ctx, filepath.Join(t.TempDir(), "index.sqlite"))
require.NoError(t, err)
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
blobHash := strings.Repeat("b", blobHashHexLen)
blob := &database.Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash(blobHash),
CreatedTS: time.Now().UTC(),
}
require.NoError(t, repos.Blobs.Create(ctx, nil, blob))
chunkHash := strings.Repeat("c", blobHashHexLen)
require.NoError(t, repos.Chunks.Create(ctx, nil,
&database.Chunk{ChunkHash: types.ChunkHash(chunkHash), Size: 1024}))
require.NoError(t, repos.BlobChunks.Create(ctx, nil, &database.BlobChunk{
BlobID: blob.ID,
ChunkHash: types.ChunkHash(chunkHash),
Offset: 0,
Length: -1,
}))
v := NewForTesting(nil)
_, err = v.verifyBlobChunks(db.Conn(), blobHash, strings.NewReader(""))
require.ErrorIs(t, err, errNegativeChunkLength)
}
// errStubUnused marks a stubLister method a test never exercises.
var errStubUnused = errors.New("stubLister method not used in test")
// stubLister is a storage.Storer whose ListStream yields a fixed set of
// objects; every other method is unused by the tests here.
type stubLister struct {
objects []storage.ObjectInfo
}
func (s *stubLister) ListStream(
_ context.Context, prefix string,
) <-chan storage.ObjectInfo {
ch := make(chan storage.ObjectInfo, len(s.objects))
for _, o := range s.objects {
if strings.HasPrefix(o.Key, prefix) {
ch <- o
}
}
close(ch)
return ch
}
func (s *stubLister) Put(_ context.Context, _ string, _ io.Reader) error {
return errStubUnused
}
func (s *stubLister) PutWithProgress(
_ context.Context, _ string, _ io.Reader, _ int64, _ storage.ProgressCallback,
) error {
return errStubUnused
}
func (s *stubLister) Get(_ context.Context, _ string) (io.ReadCloser, error) {
return nil, errStubUnused
}
func (s *stubLister) Stat(_ context.Context, _ string) (*storage.ObjectInfo, error) {
return nil, errStubUnused
}
func (s *stubLister) Delete(_ context.Context, _ string) error {
return errStubUnused
}
func (s *stubLister) List(_ context.Context, _ string) ([]string, error) {
return nil, errStubUnused
}
func (s *stubLister) Info() storage.Info {
return storage.Info{}
}
+3 -185
View File
@@ -14,7 +14,6 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot" "sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage" "sneak.berlin/go/vaultik/internal/storage"
@@ -418,10 +417,9 @@ func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
// database is uploaded but before the manifest. The destination is left // database is uploaded but before the manifest. The destination is left
// with blobs and a database but no manifest. verify and snapshot list // with blobs and a database but no manifest. verify and snapshot list
// must report the damage honestly rather than crashing or passing. // must report the damage honestly rather than crashing or passing.
// Automatic detection and repair of this partial state on the next run is // Automatic detection and repair of this partial state on the next run
// covered by TestBackupCompletesOnlyAfterMetadataExport // is tracked in https://git.eeqj.de/sneak/vaultik/issues/177 and is not
// (https://git.eeqj.de/sneak/vaultik/issues/177); this test exercises the // asserted here.
// lower-level export path in isolation.
// //
//nolint:paralleltest // installs the global logger via log.Initialize //nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupSurvivesMetadataExportInterruption(t *testing.T) { func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
@@ -498,186 +496,6 @@ func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
"snapshot list must tolerate a partially-exported snapshot") "snapshot list must tolerate a partially-exported snapshot")
} }
// Scenario 2, repair: the process dies during the metadata export of a
// full backup run. Because completion is recorded only after the export
// succeeds (finalizeSnapshotMetadata), the interrupted snapshot is left
// incomplete rather than silently marked complete without metadata at the
// destination. Rerunning the backup must then prune the incomplete
// snapshot, produce a snapshot whose destination metadata and local index
// agree, and restore. See https://git.eeqj.de/sneak/vaultik/issues/177.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupCompletesOnlyAfterMetadataExport(t *testing.T) {
log.Initialize(log.Config{})
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background()
testFiles := writeFaultSourceTree(t, fs, dataDir)
// A full-backup config: the fault-test defaults plus the fields the
// production create path reads (index location, chunk size, and the
// named snapshot to back up).
cfg := faultTestConfig()
cfg.IndexPath = dbPath
cfg.ChunkSize = config.Size(faultChunkSize)
cfg.Snapshots = map[string]config.SnapshotConfig{
"data": {Paths: []string{dataDir}},
}
inner, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
// failManifest is on for the first backup and off for the retry, so the
// manifest upload fails once — interrupting the export mid-way — then
// succeeds.
failManifest := true
store := faultstore.New(inner)
store.OnPut = func(key string) faultstore.PutAction {
if failManifest && strings.HasSuffix(key, "manifest.json.zst") {
return faultstore.PutFail
}
return faultstore.PutNormal
}
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
opts := &vaultik.SnapshotCreateOptions{Cron: true}
// First run: the export fails at the manifest upload, so the whole
// create fails and the snapshot is left incomplete.
require.Error(t, v.CreateSnapshot(opts),
"backup must fail when the metadata export is interrupted")
incompletes, err := repos.Snapshots.GetIncompleteSnapshots(ctx)
require.NoError(t, err)
require.Len(t, incompletes, 1,
"an interrupted export must leave exactly one incomplete snapshot")
afterFirst, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err)
for _, s := range afterFirst {
require.Nil(t, s.CompletedAt,
"no snapshot may be marked complete before its metadata is exported")
}
// Second run on the same index and destination: the retry succeeds.
failManifest = false
require.NoError(t, v.CreateSnapshot(opts),
"a retry after an interrupted export must succeed")
assertRetryConsistentAndRestorable(
ctx, t, cfg, inner, repos, db, fs, restoreDir, testFiles)
}
// assertRetryConsistentAndRestorable checks the end state after the retry
// backup in TestBackupCompletesOnlyAfterMetadataExport: the interrupted
// snapshot is pruned, exactly one completed snapshot remains, its metadata
// is at the destination, and it restores to the original tree.
func assertRetryConsistentAndRestorable(
ctx context.Context, t *testing.T, cfg *config.Config,
inner storage.Storer, repos *database.Repositories, db *database.DB,
fs afero.Fs, restoreDir string, testFiles map[string][]byte,
) {
t.Helper()
incompletes, err := repos.Snapshots.GetIncompleteSnapshots(ctx)
require.NoError(t, err)
assert.Empty(t, incompletes,
"the next run's prune must drop the interrupted snapshot")
local, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err)
require.Len(t, local, 1, "exactly one snapshot must remain after the retry")
final := local[0]
require.NotNil(t, final.CompletedAt, "the retry's snapshot must be complete")
// The destination and the local index agree: the completed snapshot has
// both its metadata objects at the destination.
key := snapshot.RemoteSnapshotKey(final.ID.String())
_, err = inner.Stat(ctx, "metadata/"+key+"/manifest.json.zst")
require.NoError(t, err, "the completed snapshot's manifest must be at the destination")
_, err = inner.Stat(ctx, "metadata/"+key+"/db.zst.age")
require.NoError(t, err, "the completed snapshot's database must be at the destination")
require.NoError(t, db.Close())
// The snapshot restores from the destination alone.
reader := newReaderVaultik(ctx, cfg, inner, nil, fs)
require.NoError(t, reader.Restore(&vaultik.RestoreOptions{
SnapshotID: final.ID.String(),
TargetDir: restoreDir,
Verify: true,
}), "the retry's snapshot must be restorable")
assertRestoredTree(t, fs, restoreDir, testFiles)
}
// listRecentTestLimit is a generous cap for the handful of snapshots these
// tests create when reading the local index directly.
const listRecentTestLimit = 100
// newBackupVaultik builds a Vaultik that runs the full create path
// (CreateSnapshot) writing through storer, wiring the same scanner factory
// and snapshot manager the production dependency graph provides.
func newBackupVaultik(
ctx context.Context, cfg *config.Config, storer storage.Storer,
repos *database.Repositories, db *database.DB, fs afero.Fs,
) *vaultik.Vaultik {
v := &vaultik.Vaultik{
Globals: &globals.Globals{Version: "v", Commit: "g"},
Config: cfg,
DB: db,
Repositories: repos,
Storage: storer,
SnapshotManager: newFaultSnapshotManager(fs, storer, cfg, repos),
ScannerFactory: faultScannerFactory(cfg, repos, storer),
Fs: fs,
Stdout: io.Discard,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
v.SetContext(ctx)
return v
}
// faultScannerFactory mirrors the production provideScannerFactory, binding
// the scanner to the given store, repositories, and config so a full
// create-path backup writes through the fault-injecting store.
func faultScannerFactory(
cfg *config.Config, repos *database.Repositories, storer storage.Storer,
) snapshot.ScannerFactory {
return func(params snapshot.ScannerParams) *snapshot.Scanner {
return snapshot.NewScanner(snapshot.ScannerConfig{
FS: params.Fs,
Storage: storer,
ChunkSize: faultChunkSize,
MaxBlobSize: faultMaxBlobSize,
CompressionLevel: cfg.CompressionLevel,
AgeRecipients: cfg.AgeRecipients,
Repositories: repos,
EnableProgress: params.EnableProgress,
UI: params.UI,
Exclude: params.Exclude,
SkipErrors: params.SkipErrors,
})
}
}
// Scenario 5: the restore target runs out of space mid-file. Restore // Scenario 5: the restore target runs out of space mid-file. Restore
// must fail with an out-of-space error, and must not leave a truncated // must fail with an out-of-space error, and must not leave a truncated
// file at the target path presenting as a complete restore. Restore // file at the target path presenting as a complete restore. Restore
+6 -23
View File
@@ -13,14 +13,6 @@ import (
// ShowInfo displays system and configuration information // ShowInfo displays system and configuration information
func (v *Vaultik) ShowInfo() error { func (v *Vaultik) ShowInfo() error {
// The info report is the output this command exists to produce, so it
// is written plain (markers would corrupt the aligned report) through
// the UI writer's stdout; --quiet silences it like any other
// non-error output.
if v.UI.Quiet() {
return nil
}
// System Information // System Information
v.stdoutf("=== System Information ===\n") v.stdoutf("=== System Information ===\n")
v.stdoutf("OS/Architecture: %s/%s\n", runtime.GOOS, runtime.GOARCH) v.stdoutf("OS/Architecture: %s/%s\n", runtime.GOOS, runtime.GOARCH)
@@ -221,12 +213,7 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
result.StorageType = storageInfo.Type result.StorageType = storageInfo.Type
result.StorageLocation = storageInfo.Location result.StorageLocation = storageInfo.Location
// The human report is written only when it is neither the --json if !jsonOutput {
// document (which needs stdout to itself) nor silenced by --quiet. The
// scan still runs in both cases so --json still gets a full result.
showText := !jsonOutput && !v.UI.Quiet()
if showText {
v.stdoutf("=== Remote Storage ===\n") v.stdoutf("=== Remote Storage ===\n")
v.stdoutf("Type: %s\n", storageInfo.Type) v.stdoutf("Type: %s\n", storageInfo.Type)
v.stdoutf("Location: %s\n", storageInfo.Location) v.stdoutf("Location: %s\n", storageInfo.Location)
@@ -239,7 +226,7 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
return err return err
} }
if showText { if !jsonOutput {
v.stdoutf("Downloading %d manifest(s)...\n", len(snapshotIDs)) v.stdoutf("Downloading %d manifest(s)...\n", len(snapshotIDs))
} }
@@ -247,7 +234,7 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
v.populateRemoteInfoResult(result, snapshotMetadata, snapshotIDs, referencedBlobs) v.populateRemoteInfoResult(result, snapshotMetadata, snapshotIDs, referencedBlobs)
err = v.scanRemoteBlobStorage(result, referencedBlobs, showText) err = v.scanRemoteBlobStorage(result, referencedBlobs, jsonOutput)
if err != nil { if err != nil {
return err return err
} }
@@ -265,9 +252,7 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
return enc.Encode(result) return enc.Encode(result)
} }
if showText {
v.printRemoteInfoTable(result) v.printRemoteInfoTable(result)
}
return nil return nil
} }
@@ -377,13 +362,11 @@ func (v *Vaultik) populateRemoteInfoResult(
} }
} }
// scanRemoteBlobStorage lists all blobs on remote and computes orphan // scanRemoteBlobStorage lists all blobs on remote and computes orphan stats
// stats. showText is true only when the human report is being printed
// (not --json, not --quiet), gating the progress line.
func (v *Vaultik) scanRemoteBlobStorage( func (v *Vaultik) scanRemoteBlobStorage(
result *RemoteInfoResult, referencedBlobs map[string]int64, showText bool, result *RemoteInfoResult, referencedBlobs map[string]int64, jsonOutput bool,
) error { ) error {
if showText { if !jsonOutput {
v.stdoutf("Scanning blobs...\n") v.stdoutf("Scanning blobs...\n")
} }
-30
View File
@@ -1,30 +0,0 @@
package vaultik_test
import (
"bytes"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestShowInfo_QuietSuppressesReport checks that --quiet silences the
// whole info report. The report is human-facing status, not a scriptable
// value, so under --quiet the command produces nothing (and touches none
// of its dependencies, which is why this minimal instance suffices).
func TestShowInfo_QuietSuppressesReport(t *testing.T) {
t.Parallel()
var out bytes.Buffer
v := &vaultik.Vaultik{
Stdout: &out,
UI: ui.NewWithColor(&out, false),
}
v.UI.SetQuiet(true)
require.NoError(t, v.ShowInfo())
require.Empty(t, out.String(),
"the info report must be suppressed under --quiet")
}
+2 -15
View File
@@ -247,23 +247,10 @@ func (v *Vaultik) listAllRemoteBlobs() (map[string]int64, error) {
} }
parts := strings.Split(object.Key, "/") parts := strings.Split(object.Key, "/")
if len(parts) != blobKeyParts || parts[0] != "blobs" { if len(parts) == blobKeyParts && parts[0] == "blobs" {
continue
}
// The object name is read from the destination store and is not
// trusted. A name that is not a blob hash (e.g. a short or
// non-hex string) would panic the later hash[:2]/hash[2:4] path
// build, so skip it with a warning rather than delete it.
if !isBlobHash(parts[3]) {
log.Warn("Skipping non-conforming object under blobs/",
"key", object.Key)
continue
}
allBlobs[parts[3]] = object.Size allBlobs[parts[3]] = object.Size
} }
}
log.Info("Found blobs in storage", "count", len(allBlobs)) log.Info("Found blobs in storage", "count", len(allBlobs))
+49 -128
View File
@@ -1,6 +1,7 @@
package vaultik package vaultik
import ( import (
"bytes"
"context" "context"
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
@@ -18,7 +19,6 @@ import (
"sneak.berlin/go/vaultik/internal/blobgen" "sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/types" "sneak.berlin/go/vaultik/internal/types"
) )
@@ -29,13 +29,8 @@ var (
errDecryptionKeyRequired = errors.New( errDecryptionKeyRequired = errors.New(
"decryption key required for restore\n\n" + "decryption key required for restore\n\n" +
"Set the VAULTIK_AGE_SECRET_KEY environment variable to your " + "Set the VAULTIK_AGE_SECRET_KEY environment variable to your " +
"age private key file:\n" + "age private key:\n" +
" export VAULTIK_AGE_SECRET_KEY=\"$(cat vaultik_backup_private_key.txt)\"") " export VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...'")
// errInvalidAgeSecretKey is returned when the configured key does not
// parse as any age identity. It names the source but never the value,
// which is secret, so the message is safe to print and log.
errInvalidAgeSecretKey = errors.New(
"configured age secret key holds no usable age identity")
errBlobMissingFromIndex = errors.New("blob hash missing from blob index") errBlobMissingFromIndex = errors.New("blob hash missing from blob index")
errChunkNotInAnyBlob = errors.New("chunk not found in any blob") errChunkNotInAnyBlob = errors.New("chunk not found in any blob")
errBlobIDNotInHashIndex = errors.New("blob id missing from hash index") errBlobIDNotInHashIndex = errors.New("blob id missing from hash index")
@@ -46,12 +41,6 @@ var (
"restored file has trailing data after its last chunk") "restored file has trailing data after its last chunk")
errRestoreIncomplete = errors.New( errRestoreIncomplete = errors.New(
"restore loop ended with files still pending") "restore loop ended with files still pending")
errSnapshotDBMismatch = errors.New(
"decrypted database is not the requested snapshot")
// errEmptySnapshotDB is returned when the decrypted metadata database has
// zero length, which happens when the object was truncated or replaced
// with an empty payload. Rejected before any schema is built on it.
errEmptySnapshotDB = errors.New("decrypted snapshot database is empty")
) )
// snapshotDBFilename is the name the decrypted snapshot database is // snapshotDBFilename is the name the decrypted snapshot database is
@@ -105,7 +94,7 @@ type RestoreResult struct {
func (v *Vaultik) Restore(opts *RestoreOptions) error { func (v *Vaultik) Restore(opts *RestoreOptions) error {
startTime := time.Now() startTime := time.Now()
identities, err := v.restoreIdentities() identity, err := v.prepareRestoreIdentity()
if err != nil { if err != nil {
return err return err
} }
@@ -119,7 +108,7 @@ func (v *Vaultik) Restore(opts *RestoreOptions) error {
// Step 1: Download and decrypt the snapshot metadata database // Step 1: Download and decrypt the snapshot metadata database
log.Info("Downloading snapshot metadata...") log.Info("Downloading snapshot metadata...")
tempDB, tempDir, err := v.downloadSnapshotDB(opts.SnapshotID, identities) tempDB, tempDir, err := v.downloadSnapshotDB(opts.SnapshotID, identity)
if err != nil { if err != nil {
return fmt.Errorf("downloading snapshot database: %w", err) return fmt.Errorf("downloading snapshot database: %w", err)
} }
@@ -168,7 +157,7 @@ func (v *Vaultik) Restore(opts *RestoreOptions) error {
} }
// Step 5: Restore files // Step 5: Restore files
result, err := v.restoreAllFiles(files, repos, opts, identities, chunkToBlobMap) result, err := v.restoreAllFiles(files, repos, opts, identity, chunkToBlobMap)
if err != nil { if err != nil {
return err return err
} }
@@ -229,28 +218,21 @@ func (v *Vaultik) finishRestore(
return nil return nil
} }
// restoreIdentities parses the configured age secret key once into every // prepareRestoreIdentity validates that an age secret key is configured
// identity it contains. The value may be a single key line or a whole // and parses it.
// age-keygen file with several identities; all of them are returned so //
// blobgen (via age.Decrypt) can read a blob encrypted to any of their //nolint:ireturn // age.Identity is the decryption abstraction by design
// recipients. This is the first step of both restore and deep verify, so func (v *Vaultik) prepareRestoreIdentity() (age.Identity, error) {
// a missing or unparseable key fails before anything is downloaded. The
// error names the configuration source but never the key value.
func (v *Vaultik) restoreIdentities() ([]age.Identity, error) {
if v.Config.AgeSecretKey == "" { if v.Config.AgeSecretKey == "" {
return nil, errDecryptionKeyRequired return nil, errDecryptionKeyRequired
} }
// age.ParseIdentities skips comment and blank lines and rejects a identity, err := age.ParseX25519Identity(v.Config.AgeSecretKey)
// malformed key. Its error can quote the offending line, so it is not
// wrapped here — that would leak the secret into the message.
identities, err := age.ParseIdentities(strings.NewReader(v.Config.AgeSecretKey))
if err != nil { if err != nil {
return nil, fmt.Errorf("%w (source: %s)", return nil, fmt.Errorf("parsing age secret key: %w", err)
errInvalidAgeSecretKey, v.Config.AgeSecretKeySourceName())
} }
return identities, nil return identity, nil
} }
// restoreAllFiles processes files in blob-locality order: drain every // restoreAllFiles processes files in blob-locality order: drain every
@@ -263,7 +245,7 @@ func (v *Vaultik) restoreAllFiles(
files []*database.File, files []*database.File,
repos *database.Repositories, repos *database.Repositories,
opts *RestoreOptions, opts *RestoreOptions,
identities []age.Identity, identity age.Identity,
chunkToBlobMap map[string]*database.BlobChunk, chunkToBlobMap map[string]*database.BlobChunk,
) (*RestoreResult, error) { ) (*RestoreResult, error) {
result := &RestoreResult{} result := &RestoreResult{}
@@ -317,7 +299,7 @@ func (v *Vaultik) restoreAllFiles(
ctx: v.ctx, ctx: v.ctx,
repos: repos, repos: repos,
opts: opts, opts: opts,
identities: identities, identity: identity,
chunkToBlobMap: chunkToBlobMap, chunkToBlobMap: chunkToBlobMap,
blobByHash: blobByHash, blobByHash: blobByHash,
blobIDToHash: blobIDToHash, blobIDToHash: blobIDToHash,
@@ -428,12 +410,12 @@ func (s *restoreSession) downloadNextBlobSet(plan *restorePlan) (bool, error) {
blob, ok := s.blobByHash[hash] blob, ok := s.blobByHash[hash]
if !ok { if !ok {
return false, fmt.Errorf("%w: %s", errBlobMissingFromIndex, shortHash(hash)) return false, fmt.Errorf("%w: %s", errBlobMissingFromIndex, hash[:16])
} }
err := s.downloadBlobToCache(hash, blob.CompressedSize, blob.UncompressedSize) err := s.downloadBlobToCache(hash, blob.CompressedSize)
if err != nil { if err != nil {
return false, fmt.Errorf("downloading blob %s: %w", shortHash(hash), err) return false, fmt.Errorf("downloading blob %s: %w", hash[:16], err)
} }
s.result.BlobsDownloaded++ s.result.BlobsDownloaded++
@@ -477,16 +459,6 @@ func (v *Vaultik) buildBlobIndexes(
blobByHash := make(map[string]*database.Blob, len(blobsByID)) blobByHash := make(map[string]*database.Blob, len(blobsByID))
for id, blob := range blobsByID { for id, blob := range blobsByID {
hash := blob.Hash.String() hash := blob.Hash.String()
// The snapshot database is untrusted. A hash that is not 64
// lowercase hex characters could steer a later fetch to a path
// outside the cache directory, so reject it here, before any
// blob is downloaded.
if !isBlobHash(hash) {
return nil, nil, fmt.Errorf(
"%w: %s", errInvalidBlobHash, shortHash(hash))
}
blobIDToHash[id] = hash blobIDToHash[id] = hash
blobByHash[hash] = blob blobByHash[hash] = blob
} }
@@ -641,7 +613,7 @@ func (v *Vaultik) handleRestoreVerification(
// for a remote-only snapshot) is used as-is, so a host with no local // for a remote-only snapshot) is used as-is, so a host with no local
// index can restore the snapshots it can only see on the store. // index can restore the snapshots it can only see on the store.
func (v *Vaultik) downloadSnapshotDB( func (v *Vaultik) downloadSnapshotDB(
snapshotID string, identities []age.Identity, snapshotID string, identity age.Identity,
) (*database.DB, string, error) { ) (*database.DB, string, error) {
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID) remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
if err != nil { if err != nil {
@@ -658,75 +630,41 @@ func (v *Vaultik) downloadSnapshotDB(
defer func() { _ = reader.Close() }() defer func() { _ = reader.Close() }()
// Decrypt and decompress straight from the storage stream, then stream // Read all data
// the plaintext to a temp file. Neither the encrypted bytes nor the encryptedData, err := io.ReadAll(reader)
// decrypted database is ever held whole in memory; a snapshot database if err != nil {
// can be large. return nil, "", fmt.Errorf("reading encrypted data: %w", err)
blobReader, err := blobgen.NewReader(reader, identities...) }
log.Debug("Downloaded encrypted database",
"size", ubytes(int64(len(encryptedData))))
// Decrypt and decompress using blobgen.Reader
blobReader, err := blobgen.NewReader(bytes.NewReader(encryptedData), identity)
if err != nil { if err != nil {
return nil, "", fmt.Errorf("creating decryption reader: %w", err) return nil, "", fmt.Errorf("creating decryption reader: %w", err)
} }
defer func() { _ = blobReader.Close() }() defer func() { _ = blobReader.Close() }()
db, tempDir, err := v.materializeSnapshotDB(blobReader) // Read the binary SQLite database
dbData, err := io.ReadAll(blobReader)
if err != nil { if err != nil {
return nil, "", err return nil, "", fmt.Errorf("decrypting and decompressing: %w", err)
} }
// Confirm the decrypted database really is the snapshot named by log.Debug("Decrypted database", "size", ubytes(int64(len(dbData))))
// remoteKey before any files are read from it. On mismatch, close the
// database and remove its private directory so nothing is left behind.
err = v.verifySnapshotDBIdentity(db, snapshotID, remoteKey)
if err != nil {
_ = db.Close()
_ = v.Fs.RemoveAll(tempDir)
return nil, "", err return v.materializeSnapshotDB(dbData)
}
return db, tempDir, nil
} }
// verifySnapshotDBIdentity confirms the decrypted metadata database really // materializeSnapshotDB writes the decrypted snapshot database bytes into
// is the snapshot named by remoteKey. age decryption proves the database // a fresh private (0700) temp directory and opens the file read-only. On
// is readable, not that the object served at // any failure it removes the directory before returning, so no decrypted
// metadata/<remoteKey>/db.zst.age is the snapshot that was requested: an // metadata is left on disk when the open is interrupted or the payload is
// attacker who swaps in another valid db.zst.age (which needs no key // damaged. On success the returned directory is the caller's to remove.
// material) would otherwise redirect restore and deep verify to a
// different snapshot's contents. The exported per-snapshot database holds
// exactly one snapshot row, and a snapshot's remote key is derived from
// that row's ID, so the database is the requested one exactly when its
// sole snapshot hashes back to remoteKey. Comparing the requested
// identifier directly would not do: it may be a remote-key prefix a
// recovery host uses in place of a human snapshot ID it cannot know.
func (v *Vaultik) verifySnapshotDBIdentity(
db *database.DB, requested, remoteKey string,
) error {
repos := database.NewRepositories(db)
snap, err := repos.Snapshots.GetOnlySnapshot(v.ctx)
if err != nil {
return fmt.Errorf("checking identity of database for %s: %w", requested, err)
}
if snapshot.RemoteSnapshotKey(snap.ID.String()) != remoteKey {
return fmt.Errorf("%w: requested %s but the database is snapshot %s",
errSnapshotDBMismatch, requested, snap.ID)
}
return nil
}
// materializeSnapshotDB streams the decrypted snapshot database into a
// fresh private (0700) temp directory and opens the file read-only. The
// database is copied through an io.Copy buffer rather than read whole into
// memory. On any failure it removes the directory before returning, so no
// decrypted metadata is left on disk when the copy is interrupted or the
// payload is damaged. On success the returned directory is the caller's to
// remove.
func (v *Vaultik) materializeSnapshotDB( func (v *Vaultik) materializeSnapshotDB(
dbReader io.Reader, dbData []byte,
) (*database.DB, string, error) { ) (*database.DB, string, error) {
tempDir, err := afero.TempDir(v.Fs, "", "vaultik-restore-") tempDir, err := afero.TempDir(v.Fs, "", "vaultik-restore-")
if err != nil { if err != nil {
@@ -743,29 +681,12 @@ func (v *Vaultik) materializeSnapshotDB(
dbPath := filepath.Join(tempDir, snapshotDBFilename) dbPath := filepath.Join(tempDir, snapshotDBFilename)
dbFile, err := v.Fs.OpenFile( err = afero.WriteFile(v.Fs, dbPath, dbData, restoreFileMode)
dbPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, restoreFileMode)
if err != nil { if err != nil {
return nil, "", fmt.Errorf("creating database file: %w", err) return nil, "", fmt.Errorf("writing database file: %w", err)
} }
written, copyErr := io.Copy(dbFile, dbReader) log.Debug("Created restore database", "path", dbPath)
closeErr := dbFile.Close()
if copyErr != nil {
return nil, "", fmt.Errorf("writing database file: %w", copyErr)
}
if closeErr != nil {
return nil, "", fmt.Errorf("closing database file: %w", closeErr)
}
log.Debug("Created restore database", "path", dbPath, "size", ubytes(written))
// Reject an empty database before OpenReadOnly builds a schema on it.
if written == 0 {
return nil, "", errEmptySnapshotDB
}
db, err := database.OpenReadOnly(v.ctx, dbPath) db, err := database.OpenReadOnly(v.ctx, dbPath)
if err != nil { if err != nil {
@@ -863,7 +784,7 @@ type restoreSession struct {
ctx context.Context //nolint:containedctx // per-restore state by design ctx context.Context //nolint:containedctx // per-restore state by design
repos *database.Repositories repos *database.Repositories
opts *RestoreOptions opts *RestoreOptions
identities []age.Identity identity age.Identity
chunkToBlobMap map[string]*database.BlobChunk chunkToBlobMap map[string]*database.BlobChunk
blobByHash map[string]*database.Blob blobByHash map[string]*database.Blob
blobIDToHash map[string]string blobIDToHash map[string]string
@@ -1220,12 +1141,12 @@ func (s *restoreSession) writeFileChunks(
// size, which is what makes multi-GB blobs tractable on machines with // size, which is what makes multi-GB blobs tractable on machines with
// less RAM than the blob. // less RAM than the blob.
func (s *restoreSession) downloadBlobToCache( func (s *restoreSession) downloadBlobToCache(
blobHash string, compressedSize, uncompressedSize int64, blobHash string, expectedSize int64,
) error { ) error {
start := time.Now() start := time.Now()
t0 := time.Now() t0 := time.Now()
rc, err := s.v.FetchAndDecryptBlob(s.ctx, blobHash, uncompressedSize, s.identities...) rc, err := s.v.FetchAndDecryptBlob(s.ctx, blobHash, expectedSize, s.identity)
fetchSetupDur := time.Since(t0) fetchSetupDur := time.Since(t0)
if err != nil { if err != nil {
@@ -1255,7 +1176,7 @@ func (s *restoreSession) downloadBlobToCache(
log.Debug("Streamed blob into disk cache", log.Debug("Streamed blob into disk cache",
"hash", blobHash[:16], "hash", blobHash[:16],
"compressed_bytes", compressedSize, "compressed_bytes", expectedSize,
"plaintext_bytes", written, "plaintext_bytes", written,
"ms_total", time.Since(start).Milliseconds(), "ms_total", time.Since(start).Milliseconds(),
"ms_fetch_setup", fetchSetupDur.Milliseconds(), "ms_fetch_setup", fetchSetupDur.Milliseconds(),
-108
View File
@@ -1,108 +0,0 @@
package vaultik //nolint:testpackage // exercises unexported restoreIdentities
import (
"bytes"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/config"
)
// encryptBlobTo returns a blobgen blob of plaintext encrypted to exactly
// one recipient, so a decryptor succeeds only if it holds that recipient's
// identity.
func encryptBlobTo(t *testing.T, recipient string, plaintext []byte) []byte {
t.Helper()
var buf bytes.Buffer
writer, err := blobgen.NewWriter(&buf, 1, []string{recipient})
require.NoError(t, err)
_, err = writer.Write(plaintext)
require.NoError(t, err)
require.NoError(t, writer.Close())
return buf.Bytes()
}
// decryptBlobWith reads a blob back through the identities and returns its
// plaintext.
func decryptBlobWith(t *testing.T, blob []byte, identities []age.Identity) []byte {
t.Helper()
reader, err := blobgen.NewReader(bytes.NewReader(blob), identities...)
require.NoError(t, err)
plaintext, err := io.ReadAll(reader)
require.NoError(t, err)
require.NoError(t, reader.Close())
return plaintext
}
// TestRestoreIdentitiesAcceptsEveryIdentity proves a key file holding two
// identities yields both, so a blob encrypted only to the second
// recipient — the one the previous single-identity parse dropped — still
// decrypts.
func TestRestoreIdentitiesAcceptsEveryIdentity(t *testing.T) {
t.Parallel()
first, err := age.GenerateX25519Identity()
require.NoError(t, err)
second, err := age.GenerateX25519Identity()
require.NoError(t, err)
// A whole age-keygen-style file: comment lines plus two identity lines.
keyFile := "# public key: " + first.Recipient().String() + "\n" +
first.String() + "\n" +
"# public key: " + second.Recipient().String() + "\n" +
second.String() + "\n"
v := &Vaultik{Config: &config.Config{AgeSecretKey: keyFile}}
identities, err := v.restoreIdentities()
require.NoError(t, err)
require.Len(t, identities, 2)
plaintext := []byte("payload encrypted only to the second identity")
blob := encryptBlobTo(t, second.Recipient().String(), plaintext)
require.Equal(t, plaintext, decryptBlobWith(t, blob, identities))
}
// TestRestoreIdentitiesAcceptsTrailingNewline mirrors a YAML
// age_secret_key value that carries a trailing newline: it must still
// parse to its one identity and decrypt a blob encrypted to it.
func TestRestoreIdentitiesAcceptsTrailingNewline(t *testing.T) {
t.Parallel()
id, err := age.GenerateX25519Identity()
require.NoError(t, err)
v := &Vaultik{Config: &config.Config{AgeSecretKey: id.String() + "\n"}}
identities, err := v.restoreIdentities()
require.NoError(t, err)
require.Len(t, identities, 1)
plaintext := []byte("value with a trailing newline")
blob := encryptBlobTo(t, id.Recipient().String(), plaintext)
require.Equal(t, plaintext, decryptBlobWith(t, blob, identities))
}
// TestRestoreIdentitiesMissingKey reports the dedicated missing-key error
// rather than a parse failure, so the user is told to set the key.
func TestRestoreIdentitiesMissingKey(t *testing.T) {
t.Parallel()
v := &Vaultik{Config: &config.Config{}}
_, err := v.restoreIdentities()
require.ErrorIs(t, err, errDecryptionKeyRequired)
}
@@ -1,44 +0,0 @@
package vaultik_test
import (
"context"
"io"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestRestoreRejectsMalformedKeyBeforeDownload verifies that a malformed
// age secret key stops restore at the parse step: nothing is fetched from
// the store, and the error does not echo the key value (which is secret).
func TestRestoreRejectsMalformedKeyBeforeDownload(t *testing.T) {
t.Parallel()
const malformed = "this-is-not-a-valid-age-key"
mock := NewMockStorer()
v := &vaultik.Vaultik{
Config: &config.Config{AgeSecretKey: malformed},
Storage: mock,
Stdout: io.Discard,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
v.SetContext(context.Background())
err := v.Restore(&vaultik.RestoreOptions{
SnapshotID: "any-snapshot",
TargetDir: t.TempDir(),
})
require.Error(t, err)
require.NotContains(t, err.Error(), malformed,
"error must not echo the key value")
require.Contains(t, err.Error(), "age_secret_key",
"error should name the configuration source")
require.Empty(t, mock.GetCalls(),
"a malformed key must fail before anything is fetched")
}
@@ -1,251 +0,0 @@
package vaultik_test
import (
"context"
"io"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestRestoreAndDeepVerifyRejectSwappedDatabase proves that swapping two
// snapshots' encrypted databases on the store is caught. age decryption
// alone proves only that a database is readable; without an identity check
// restore would happily write the wrong snapshot's files and deep verify
// would report success. After the swap, restore and deep verify of A both
// fail, and the error names the snapshot the database actually holds (B).
func TestRestoreAndDeepVerifyRejectSwappedDatabase(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
storeDir := filepath.Join(tempDir, "remote")
chunkSize := int64(64 * 1024)
storer, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
ctx := context.Background()
// Two snapshots with different content, backed up into one shared
// store. Different names give them different remote keys, so their
// metadata directories are distinct and can be tampered with alone.
dataA := filepath.Join(tempDir, "srcA")
require.NoError(t, fs.MkdirAll(dataA, 0o755))
require.NoError(t, afero.WriteFile(fs, filepath.Join(dataA, "a.bin"),
bytesPattern("alpha-", int(chunkSize*2)), 0o644))
dataB := filepath.Join(tempDir, "srcB")
require.NoError(t, fs.MkdirAll(dataB, 0o755))
require.NoError(t, afero.WriteFile(fs, filepath.Join(dataB, "b.bin"),
bytesPattern("beta-", int(chunkSize*2)), 0o644))
idA := backupNamedSnapshotToStore(ctx, t, fs, dataA, storer,
filepath.Join(tempDir, "idxA.sqlite"), "alpha")
idB := backupNamedSnapshotToStore(ctx, t, fs, dataB, storer,
filepath.Join(tempDir, "idxB.sqlite"), "beta")
require.NotEqual(t, idA, idB)
keyA := snapshot.RemoteSnapshotKey(idA)
keyB := snapshot.RemoteSnapshotKey(idB)
require.NotEqual(t, keyA, keyB)
// Baseline: each snapshot verifies against its own intact metadata.
require.NoError(t, newStoreClient(ctx, t, fs, storer).RunDeepVerify(
idA, &vaultik.VerifyOptions{Deep: true}))
require.NoError(t, newStoreClient(ctx, t, fs, storer).RunDeepVerify(
idB, &vaultik.VerifyOptions{Deep: true}))
// Swap the two snapshots' encrypted databases on the store.
swapStoreFiles(t, fs,
filepath.Join(storeDir, "metadata", keyA, "db.zst.age"),
filepath.Join(storeDir, "metadata", keyB, "db.zst.age"))
// Restore of A now decrypts B's database; the identity check must
// reject it and name the snapshot it actually found.
restoreErr := newStoreClient(ctx, t, fs, storer).Restore(&vaultik.RestoreOptions{
SnapshotID: idA,
TargetDir: filepath.Join(tempDir, "restoreA"),
})
require.Error(t, restoreErr)
require.ErrorContains(t, restoreErr, idB)
// Deep verify of A must reject the swapped database for the same reason.
verifyErr := newStoreClient(ctx, t, fs, storer).RunDeepVerify(
idA, &vaultik.VerifyOptions{Deep: true})
require.Error(t, verifyErr)
require.ErrorContains(t, verifyErr, idB)
}
// TestDeepVerifyRejectsSwappedDatabaseWithEmptyManifest covers the case the
// issue calls out: swapping in a database whose blob set is empty and
// pairing it with an equally empty manifest. The manifest then agrees with
// the database, so every blob-level check passes and deep verify used to
// report success with zero blobs verified. The identity check rejects it
// before any blob check runs.
func TestDeepVerifyRejectsSwappedDatabaseWithEmptyManifest(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
storeDir := filepath.Join(tempDir, "remote")
chunkSize := int64(64 * 1024)
storer, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
ctx := context.Background()
// Snapshot A: real content, so its manifest lists blobs.
dataA := filepath.Join(tempDir, "srcA")
require.NoError(t, fs.MkdirAll(dataA, 0o755))
require.NoError(t, afero.WriteFile(fs, filepath.Join(dataA, "a.bin"),
bytesPattern("alpha-", int(chunkSize*2)), 0o644))
idA := backupNamedSnapshotToStore(ctx, t, fs, dataA, storer,
filepath.Join(tempDir, "idxA.sqlite"), "alpha")
// Snapshot C: a single empty file, so it references no blobs and its
// manifest is empty. This is the database/manifest pair an attacker
// would swap in to make the blob checks vacuously pass.
dataC := filepath.Join(tempDir, "srcC")
require.NoError(t, fs.MkdirAll(dataC, 0o755))
require.NoError(t, afero.WriteFile(fs,
filepath.Join(dataC, "empty.bin"), []byte{}, 0o644))
idC := backupNamedSnapshotToStore(ctx, t, fs, dataC, storer,
filepath.Join(tempDir, "idxC.sqlite"), "charlie")
keyA := snapshot.RemoteSnapshotKey(idA)
keyC := snapshot.RemoteSnapshotKey(idC)
// Replace A's database and manifest with C's empty pair.
copyStoreFile(t, fs,
filepath.Join(storeDir, "metadata", keyC, "db.zst.age"),
filepath.Join(storeDir, "metadata", keyA, "db.zst.age"))
copyStoreFile(t, fs,
filepath.Join(storeDir, "metadata", keyC, "manifest.json.zst"),
filepath.Join(storeDir, "metadata", keyA, "manifest.json.zst"))
verifyErr := newStoreClient(ctx, t, fs, storer).RunDeepVerify(
idA, &vaultik.VerifyOptions{Deep: true})
require.Error(t, verifyErr)
require.ErrorContains(t, verifyErr, idC)
}
// backupNamedSnapshotToStore backs up dataDir into the shared storer under
// the given snapshot name and returns the human snapshot ID. Two snapshots
// backed up under different names get different remote keys, so their
// metadata directories on the store are distinct.
func backupNamedSnapshotToStore(
ctx context.Context, t *testing.T, fs afero.Fs,
dataDir string, storer storage.Storer, dbPath, name string,
) string {
t.Helper()
const (
chunkSize = int64(64 * 1024)
maxBlobSize = int64(512 * 1024)
)
cfg := &config.Config{
AgeRecipients: []string{testAgePublicKey},
AgeSecretKey: testAgeSecretKey,
CompressionLevel: 3,
Hostname: testHostname,
}
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
sm := snapshot.NewSnapshotManager(snapshot.SnapshotManagerParams{
Repos: repos,
Storage: storer,
Config: cfg,
})
sm.SetFilesystem(fs)
scanner := snapshot.NewScanner(snapshot.ScannerConfig{
FS: fs,
Storage: storer,
ChunkSize: chunkSize,
MaxBlobSize: maxBlobSize,
CompressionLevel: cfg.CompressionLevel,
AgeRecipients: cfg.AgeRecipients,
Repositories: repos,
})
snapshotID, err := sm.CreateSnapshotWithName(
ctx, cfg.Hostname, name, "test-version", "test-git")
require.NoError(t, err)
_, err = scanner.Scan(ctx, dataDir, snapshotID)
require.NoError(t, err)
require.NoError(t, sm.CompleteSnapshot(ctx, snapshotID))
require.NoError(t, sm.ExportSnapshotMetadata(ctx, dbPath, snapshotID))
require.NoError(t, db.Close())
return snapshotID
}
// newStoreClient builds a Vaultik that reads only from the store: the
// secret key, the storer, and a filesystem, with no local index. This is
// what restore and deep verify need.
func newStoreClient(
ctx context.Context, t *testing.T, fs afero.Fs, storer storage.Storer,
) *vaultik.Vaultik {
t.Helper()
v := &vaultik.Vaultik{
Config: &config.Config{
AgeSecretKey: testAgeSecretKey,
Hostname: testHostname,
},
Storage: storer,
Fs: fs,
Stdout: io.Discard,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
v.SetContext(ctx)
return v
}
// swapStoreFiles exchanges the contents of two files on the store.
func swapStoreFiles(t *testing.T, fs afero.Fs, a, b string) {
t.Helper()
dataA, err := afero.ReadFile(fs, a)
require.NoError(t, err)
dataB, err := afero.ReadFile(fs, b)
require.NoError(t, err)
require.NoError(t, afero.WriteFile(fs, a, dataB, 0o644))
require.NoError(t, afero.WriteFile(fs, b, dataA, 0o644))
}
// copyStoreFile overwrites dst with the contents of src on the store.
func copyStoreFile(t *testing.T, fs afero.Fs, src, dst string) {
t.Helper()
data, err := afero.ReadFile(fs, src)
require.NoError(t, err)
require.NoError(t, afero.WriteFile(fs, dst, data, 0o644))
}
+2 -37
View File
@@ -1,16 +1,13 @@
package vaultik //nolint:testpackage // inspects unexported snapshot-db materialization package vaultik //nolint:testpackage // inspects unexported snapshot-db materialization
import ( import (
"bytes"
"context" "context"
"os" "os"
"path/filepath" "path/filepath"
"testing" "testing"
"filippo.io/age"
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
) )
@@ -40,7 +37,7 @@ func TestMaterializeSnapshotDBPrivateDir(t *testing.T) {
v := &Vaultik{ctx: context.Background(), Fs: afero.NewOsFs()} v := &Vaultik{ctx: context.Background(), Fs: afero.NewOsFs()}
db, dir, err := v.materializeSnapshotDB(bytes.NewReader(dbData)) db, dir, err := v.materializeSnapshotDB(dbData)
require.NoError(t, err) require.NoError(t, err)
t.Cleanup(func() { t.Cleanup(func() {
@@ -58,37 +55,6 @@ func TestMaterializeSnapshotDBPrivateDir(t *testing.T) {
require.Error(t, err, "materialized snapshot database must be read-only") require.Error(t, err, "materialized snapshot database must be read-only")
} }
// TestMaterializeSnapshotDBRejectsCompleteEmptyStream proves the written == 0
// guard rejects a genuinely empty but complete metadata object: a real age
// header, nonce, and final tag encrypting zero plaintext bytes. The truncation
// case is stopped earlier by the reader (io.ErrUnexpectedEOF) and never reaches
// this branch, so it needs its own input. This complete stream decrypts to zero
// bytes with a clean EOF, passes the reader, and must be refused as empty rather
// than accepted as a valid zero-table database. Reverting the guard lets the
// empty file open as a fresh schema and the test fails.
func TestMaterializeSnapshotDBRejectsCompleteEmptyStream(t *testing.T) {
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
var stream bytes.Buffer
w, err := age.Encrypt(&stream, identity.Recipient())
require.NoError(t, err)
require.NoError(t, w.Close())
blobReader, err := blobgen.NewReader(bytes.NewReader(stream.Bytes()), identity)
require.NoError(t, err)
t.Cleanup(func() { _ = blobReader.Close() })
t.Setenv("TMPDIR", t.TempDir())
v := &Vaultik{ctx: context.Background(), Fs: afero.NewOsFs()}
_, _, err = v.materializeSnapshotDB(blobReader)
require.ErrorIs(t, err, errEmptySnapshotDB)
}
// TestMaterializeSnapshotDBRemovesDirOnOpenFailure proves a failed open // TestMaterializeSnapshotDBRemovesDirOnOpenFailure proves a failed open
// leaves no temp directory behind. // leaves no temp directory behind.
func TestMaterializeSnapshotDBRemovesDirOnOpenFailure(t *testing.T) { func TestMaterializeSnapshotDBRemovesDirOnOpenFailure(t *testing.T) {
@@ -98,8 +64,7 @@ func TestMaterializeSnapshotDBRemovesDirOnOpenFailure(t *testing.T) {
v := &Vaultik{ctx: context.Background(), Fs: afero.NewOsFs()} v := &Vaultik{ctx: context.Background(), Fs: afero.NewOsFs()}
_, _, err := v.materializeSnapshotDB( _, _, err := v.materializeSnapshotDB([]byte("this is not a sqlite database"))
bytes.NewReader([]byte("this is not a sqlite database")))
require.Error(t, err) require.Error(t, err)
entries, rerr := os.ReadDir(base) entries, rerr := os.ReadDir(base)
@@ -1,85 +0,0 @@
package vaultik_test
import (
"bytes"
"context"
"io"
"path/filepath"
"testing"
"filippo.io/age"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestRestoreRejectsTruncatedMetadataDB backs up a real tree, then replaces
// the snapshot's db.zst.age with a stream cut right after the age header and
// its 16-byte nonce. age.Decrypt still accepts such an object and the zstd
// decoder turns the truncated read into a clean EOF, so before the fix restore
// built a fresh empty schema and reported success. Restore must now fail with
// io.ErrUnexpectedEOF, the error the reader raises for a truncated object.
// Asserting that specific error pins the reader fix: without it the truncation
// yields an empty database, which the identity check rejects for an unrelated
// reason, and this test would pass anyway.
func TestRestoreRejectsTruncatedMetadataDB(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(64 * 1024)
maxBlobSize := int64(512 * 1024)
setupE2ESourceTree(t, fs, dataDir, chunkSize)
ctx := context.Background()
cfg, storer, snapshotID := runFileStorageBackup(
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
// Encrypting empty plaintext to the snapshot recipient yields
// header + nonce(16) + a single 16-byte final chunk tag. Dropping the
// trailing tag leaves exactly the age header plus its nonce — the
// truncation an attacker can write over metadata without any key.
recipient, err := age.ParseX25519Recipient(testAgePublicKey)
require.NoError(t, err)
var full bytes.Buffer
w, err := age.Encrypt(&full, recipient)
require.NoError(t, err)
require.NoError(t, w.Close())
truncated := full.Bytes()[:full.Len()-16]
dbKeyPath := filepath.Join(storeDir, "metadata",
snapshot.RemoteSnapshotKey(snapshotID), "db.zst.age")
require.NoError(t, afero.WriteFile(fs, dbKeyPath, truncated, 0o644))
restoreVaultik := &vaultik.Vaultik{
Config: cfg,
Storage: storer,
Fs: fs,
Stdout: io.Discard,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
restoreVaultik.SetContext(ctx)
err = restoreVaultik.Restore(&vaultik.RestoreOptions{
SnapshotID: snapshotID,
TargetDir: restoreDir,
Verify: true,
})
require.ErrorIs(t, err, io.ErrUnexpectedEOF)
}
-191
View File
@@ -1,191 +0,0 @@
package vaultik_test
import (
"bytes"
"context"
"encoding/json"
"io"
"os"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestShallowVerifyDetectsWrongBlobSize backs up a real snapshot, runs
// shallow verify (which passes and reports exactly the blobs it checked),
// then grows one stored blob so its size no longer matches the manifest.
// Shallow verify must then fail, count the grown blob as a size mismatch,
// and drop it from the verified count rather than continuing to report it
// as checked.
func TestShallowVerifyDetectsWrongBlobSize(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source")
storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(32 * 1024)
maxBlobSize := int64(128 * 1024)
// Enough data to span several blobs, so the mismatch count and the
// dropped verified count are both meaningful.
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
require.NoError(t, afero.WriteFile(fs,
filepath.Join(dataDir, "data.bin"),
bytesPattern("shallow-", int(maxBlobSize*4)), 0o644))
ctx := context.Background()
cfg, storer, snapshotID := runFileStorageBackup(
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
newVerifier := func(out io.Writer) *vaultik.Vaultik {
v := &vaultik.Vaultik{
Config: cfg,
Storage: storer,
Fs: fs,
Stdout: out,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
v.SetContext(ctx)
return v
}
var out bytes.Buffer
require.NoError(t,
newVerifier(&out).VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{JSON: true}),
"shallow verify should pass on a healthy snapshot")
healthy := decodeVerifyResult(t, out.Bytes())
require.Equal(t, "ok", healthy.Status)
require.Positive(t, healthy.BlobCount)
require.Equal(t, healthy.BlobCount, healthy.Verified,
"shallow verify must report exactly the blobs it checked")
require.Zero(t, healthy.Mismatched)
// Grow one stored blob so its size no longer matches the manifest.
growOneBlob(t, fs, filepath.Join(storeDir, "blobs"))
out.Reset()
err := newVerifier(&out).VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{JSON: true})
require.Error(t, err,
"shallow verify must fail when a blob's stored size differs from the manifest")
bad := decodeVerifyResult(t, out.Bytes())
require.Equal(t, "failed", bad.Status)
require.Equal(t, 1, bad.Mismatched)
require.Equal(t, healthy.BlobCount-1, bad.Verified,
"the wrong-sized blob must not be counted as verified")
}
// TestShallowVerifyDetectsMissingDatabase backs up a real snapshot,
// confirms shallow verify passes, then deletes the snapshot's encrypted
// database. Shallow verify must fail: a snapshot without its database is
// not restorable, even when every blob is present.
func TestShallowVerifyDetectsMissingDatabase(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source")
storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(32 * 1024)
maxBlobSize := int64(128 * 1024)
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
require.NoError(t, afero.WriteFile(fs,
filepath.Join(dataDir, "data.bin"),
bytesPattern("shallow-db-", int(maxBlobSize*2)), 0o644))
ctx := context.Background()
cfg, storer, snapshotID := runFileStorageBackup(
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
newVerifier := func() *vaultik.Vaultik {
v := &vaultik.Vaultik{
Config: cfg,
Storage: storer,
Fs: fs,
Stdout: io.Discard,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
v.SetContext(ctx)
return v
}
require.NoError(t,
newVerifier().VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{}),
"shallow verify should pass on a healthy snapshot")
// The database lives under the hashed remote key, not the human ID.
dbObject := filepath.Join(storeDir, "metadata",
snapshot.RemoteSnapshotKey(snapshotID), "db.zst.age")
require.NoError(t, os.Remove(dbObject))
require.Error(t,
newVerifier().VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{}),
"shallow verify must fail when db.zst.age is absent")
}
// growOneBlob appends bytes to the first blob file found under blobsDir,
// changing its on-disk size so it no longer matches the manifest.
func growOneBlob(t *testing.T, fs afero.Fs, blobsDir string) {
t.Helper()
var blobPath string
err := afero.Walk(fs, blobsDir,
func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if blobPath == "" && !info.IsDir() {
blobPath = path
}
return nil
})
require.NoError(t, err)
require.NotEmpty(t, blobPath, "expected at least one blob on disk")
data, err := afero.ReadFile(fs, blobPath)
require.NoError(t, err)
data = append(data, []byte("extra")...)
require.NoError(t, afero.WriteFile(fs, blobPath, data, 0o644))
}
func decodeVerifyResult(t *testing.T, b []byte) vaultik.VerifyResult {
t.Helper()
var result vaultik.VerifyResult
require.NoError(t, json.Unmarshal(b, &result))
return result
}
+45 -155
View File
@@ -20,6 +20,7 @@ import (
var ( var (
errSnapshotNotInConfig = errors.New("snapshot not found in config") errSnapshotNotInConfig = errors.New("snapshot not found in config")
errNoSnapshotsInConfig = errors.New("no snapshots configured") errNoSnapshotsInConfig = errors.New("no snapshots configured")
errBlobsMissing = errors.New("blobs are missing")
errSnapshotVerifyFailed = errors.New("verification failed") errSnapshotVerifyFailed = errors.New("verification failed")
errRemoveAllNeedsForce = errors.New("--all requires --force") errRemoveAllNeedsForce = errors.New("--all requires --force")
errInvalidTableName = errors.New("invalid table name") errInvalidTableName = errors.New("invalid table name")
@@ -55,8 +56,8 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
return err return err
} }
// Clean up incomplete snapshots FIRST, before any scanning. // Clean up incomplete snapshots FIRST, before any scanning
// This is critical for data safety; PruneDatabase below does it. // This is critical for data safety - see CleanupIncompleteSnapshots for details
hostname := v.Config.Hostname hostname := v.Config.Hostname
if hostname == "" { if hostname == "" {
hostname, _ = os.Hostname() hostname, _ = os.Hostname()
@@ -69,9 +70,6 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
// Prune the database before starting: delete incomplete snapshots and orphaned data. // Prune the database before starting: delete incomplete snapshots and orphaned data.
// This ensures the database is consistent before we start a new snapshot. // This ensures the database is consistent before we start a new snapshot.
// Since we use locking, only one vaultik instance accesses the DB at a time. // Since we use locking, only one vaultik instance accesses the DB at a time.
// A snapshot whose metadata export was interrupted is left incomplete by
// finalizeSnapshotMetadata, so it is among the incomplete snapshots dropped
// here (https://git.eeqj.de/sneak/vaultik/issues/177).
_, err = v.PruneDatabase() _, err = v.PruneDatabase()
if err != nil { if err != nil {
return fmt.Errorf("prune database: %w", err) return fmt.Errorf("prune database: %w", err)
@@ -327,12 +325,7 @@ func (v *Vaultik) collectUploadStats(scanner *snapshot.Scanner, stats *snapshotS
} }
} }
// finalizeSnapshotMetadata updates stats, exports metadata, and only then // finalizeSnapshotMetadata updates stats, marks complete, and exports metadata
// marks the snapshot complete. Recording completion last is deliberate: an
// export interrupted by a crash leaves the snapshot incomplete rather than
// looking complete with no manifest or database at the destination. The next
// run's PruneDatabase drops the incomplete snapshot and re-backs-up its data.
// See https://git.eeqj.de/sneak/vaultik/issues/177.
func (v *Vaultik) finalizeSnapshotMetadata( func (v *Vaultik) finalizeSnapshotMetadata(
snapshotID string, stats *snapshotStats, snapshotID string, stats *snapshotStats,
) error { ) error {
@@ -354,11 +347,9 @@ func (v *Vaultik) finalizeSnapshotMetadata(
return fmt.Errorf("updating snapshot stats: %w", err) return fmt.Errorf("updating snapshot stats: %w", err)
} }
// snapshot_blobs must be populated before the export, which builds the err = v.SnapshotManager.CompleteSnapshot(v.ctx, snapshotID)
// manifest and the trimmed metadata database from it.
err = v.SnapshotManager.PopulateSnapshotBlobs(v.ctx, snapshotID)
if err != nil { if err != nil {
return fmt.Errorf("populating snapshot blobs: %w", err) return fmt.Errorf("completing snapshot: %w", err)
} }
err = v.SnapshotManager.ExportSnapshotMetadata( err = v.SnapshotManager.ExportSnapshotMetadata(
@@ -367,13 +358,6 @@ func (v *Vaultik) finalizeSnapshotMetadata(
return fmt.Errorf("exporting snapshot metadata: %w", err) return fmt.Errorf("exporting snapshot metadata: %w", err)
} }
// Record completion last, so an interrupted export never leaves a
// snapshot marked complete without its metadata at the destination.
err = v.SnapshotManager.MarkSnapshotComplete(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("marking snapshot complete: %w", err)
}
return nil return nil
} }
@@ -686,24 +670,11 @@ func (v *Vaultik) VerifySnapshotWithOptions(
v.printVerifyHeader(snapshotID, opts) v.printVerifyHeader(snapshotID, opts)
// Resolve the identifier to the snapshot's remote key. A human ID is // Resolve the identifier to the snapshot's remote key and download the
// hashed; a remote key (or its abbreviation, as printed for a // manifest. A human ID is hashed; a remote key (or its abbreviation,
// remote-only snapshot) is used as-is, so a host with no local index // as printed for a remote-only snapshot) is used as-is, so a host with
// can verify a snapshot it can only see on the store. The key is kept // no local index can verify a snapshot it can only see on the store.
// so we can also check for the snapshot's encrypted database below. manifest, err := v.resolveAndDownloadManifest(snapshotID)
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
if err != nil {
if opts.JSON {
result.Status = verifyStatusFailed
result.ErrorMessage = fmt.Sprintf("resolving snapshot identifier: %v", err)
return v.outputVerifyJSON(result)
}
return fmt.Errorf("resolving snapshot identifier: %w", err)
}
manifest, err := v.downloadManifestByKey(remoteKey)
if err != nil { if err != nil {
if opts.JSON { if opts.JSON {
result.Status = verifyStatusFailed result.Status = verifyStatusFailed
@@ -733,34 +704,14 @@ func (v *Vaultik) VerifySnapshotWithOptions(
v.printlnStdout() v.printlnStdout()
// Check each blob is present with the size the manifest records. // Check each blob exists
v.stdoutf("Checking blob presence and sizes...\n") v.stdoutf("Checking blob existence...\n")
} }
// A snapshot is only restorable if its encrypted database is present result.Verified, result.Missing, result.MissingSize =
// alongside the blobs. Shallow verify checks that the object exists; it v.verifyManifestBlobsExist(manifest, opts)
// does not decrypt it (that is deep verify's job).
dbPath := fmt.Sprintf("metadata/%s/db.zst.age", remoteKey)
_, dbErr := v.Storage.Stat(v.ctx, dbPath) return v.formatVerifyResult(result, manifest, opts)
if dbErr != nil {
result.DatabaseMissing = true
}
result.Verified, result.Missing, result.Mismatched, result.MissingSize, err =
v.verifyManifestBlobs(manifest, opts)
if err != nil {
if opts.JSON {
result.Status = verifyStatusFailed
result.ErrorMessage = fmt.Sprintf("verifying manifest blobs: %v", err)
return v.outputVerifyJSON(result)
}
return fmt.Errorf("verifying manifest blobs: %w", err)
}
return v.formatVerifyResult(result, opts)
} }
// printVerifyHeader prints the snapshot ID and parsed timestamp for // printVerifyHeader prints the snapshot ID and parsed timestamp for
@@ -785,34 +736,25 @@ func (v *Vaultik) printVerifyHeader(snapshotID string, opts *VerifyOptions) {
} }
} }
// verifyManifestBlobs checks that each blob in the manifest is present in // verifyManifestBlobsExist checks that each blob in the manifest exists
// storage with the size the manifest records, returning the counts of // in storage, returning the verified count, missing count, and total
// blobs that were present with the right size, absent, and present but the // missing bytes.
// wrong size, plus the total bytes of the absent blobs. It does not read func (v *Vaultik) verifyManifestBlobsExist(
// blob contents; deep verification (RunDeepVerify) does that. The size
// comparison matches the deep path (see verifyBlobExistenceFromDB).
func (v *Vaultik) verifyManifestBlobs(
manifest *snapshot.Manifest, opts *VerifyOptions, manifest *snapshot.Manifest, opts *VerifyOptions,
) (int, int, int, int64, error) { ) (int, int, int64) {
var ( var (
verified, missing, mismatched int verified, missing int
missingSize int64 missingSize int64
) )
for _, blob := range manifest.Blobs { for _, blob := range manifest.Blobs {
// The manifest is unauthenticated, so its blob hashes are checked
// before being spliced into a storage path.
if !isBlobHash(blob.Hash) {
return 0, 0, 0, 0, fmt.Errorf("%w: %s",
errInvalidBlobHash, shortHash(blob.Hash))
}
blobPath := fmt.Sprintf("blobs/%s/%s/%s", blobPath := fmt.Sprintf("blobs/%s/%s/%s",
blob.Hash[:2], blob.Hash[2:4], blob.Hash) blob.Hash[:2], blob.Hash[2:4], blob.Hash)
stat, err := v.Storage.Stat(v.ctx, blobPath) // Shallow: check existence only (deep verification is handled
switch { // by RunDeepVerify).
case err != nil: _, err := v.Storage.Stat(v.ctx, blobPath)
if err != nil {
if !opts.JSON { if !opts.JSON {
v.stdoutf(" Missing: %s (%s)\n", v.stdoutf(" Missing: %s (%s)\n",
blob.Hash, ubytes(blob.CompressedSize)) blob.Hash, ubytes(blob.CompressedSize))
@@ -820,32 +762,23 @@ func (v *Vaultik) verifyManifestBlobs(
missing++ missing++
missingSize += blob.CompressedSize missingSize += blob.CompressedSize
case stat.Size != blob.CompressedSize: } else {
if !opts.JSON {
v.stdoutf(" Wrong size: %s (store has %s, manifest lists %s)\n",
blob.Hash, ubytes(stat.Size), ubytes(blob.CompressedSize))
}
mismatched++
default:
verified++ verified++
} }
} }
return verified, missing, mismatched, missingSize, nil return verified, missing, missingSize
} }
// formatVerifyResult outputs the final verification results as JSON or // formatVerifyResult outputs the final verification results as JSON or
// human-readable text. // human-readable text.
func (v *Vaultik) formatVerifyResult( func (v *Vaultik) formatVerifyResult(
result *VerifyResult, opts *VerifyOptions, result *VerifyResult, manifest *snapshot.Manifest, opts *VerifyOptions,
) error { ) error {
failure := shallowVerifyFailure(result)
if opts.JSON { if opts.JSON {
if failure != "" { if result.Missing > 0 {
result.Status = verifyStatusFailed result.Status = verifyStatusFailed
result.ErrorMessage = failure result.ErrorMessage = fmt.Sprintf("%d blobs are missing", result.Missing)
} else { } else {
result.Status = "ok" result.Status = "ok"
} }
@@ -854,57 +787,29 @@ func (v *Vaultik) formatVerifyResult(
} }
v.stdoutf("\nVerification complete:\n") v.stdoutf("\nVerification complete:\n")
v.stdoutf(" Present with listed size: %d blobs\n", result.Verified) v.stdoutf(" Verified: %d blobs (%s)\n", result.Verified,
ubytes(manifest.TotalCompressedSize-result.MissingSize))
if result.Missing > 0 { if result.Missing > 0 {
v.stdoutf(" Missing: %d blobs (%s)\n", v.stdoutf(" Missing: %d blobs (%s)\n",
result.Missing, ubytes(result.MissingSize)) result.Missing, ubytes(result.MissingSize))
} } else {
v.stdoutf(" Missing: 0 blobs\n")
if result.Mismatched > 0 {
v.stdoutf(" Wrong size: %d blobs\n", result.Mismatched)
}
if result.DatabaseMissing {
v.stdoutf(" Encrypted database: missing\n")
} }
v.stdoutf(" Status: ") v.stdoutf(" Status: ")
if failure != "" { if result.Missing > 0 {
v.stdoutf("FAILED - %s\n", failure) v.stdoutf("FAILED - %d blobs are missing\n", result.Missing)
return fmt.Errorf("%w: %s", errSnapshotVerifyFailed, failure) return fmt.Errorf("%d %w", result.Missing, errBlobsMissing)
} }
// Report only what was actually checked: presence and size, not contents. v.stdoutf("OK - All blobs verified\n")
v.stdoutf("OK - all %d blobs listed in the manifest are present with the "+
"listed size; contents not checked (use --deep)\n", result.Verified)
return nil return nil
} }
// shallowVerifyFailure returns a human-readable description of everything
// that failed shallow verification, or the empty string if it passed.
func shallowVerifyFailure(result *VerifyResult) string {
var parts []string
if result.Missing > 0 {
parts = append(parts, fmt.Sprintf("%d blobs are missing", result.Missing))
}
if result.Mismatched > 0 {
parts = append(parts,
fmt.Sprintf("%d blobs have the wrong size", result.Mismatched))
}
if result.DatabaseMissing {
parts = append(parts, "the encrypted database is missing")
}
return strings.Join(parts, "; ")
}
// outputVerifyJSON outputs the verification result as JSON // outputVerifyJSON outputs the verification result as JSON
func (v *Vaultik) outputVerifyJSON(result *VerifyResult) error { func (v *Vaultik) outputVerifyJSON(result *VerifyResult) error {
encoder := json.NewEncoder(v.Stdout) encoder := json.NewEncoder(v.Stdout)
@@ -1347,38 +1252,23 @@ func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) {
} }
parts := strings.Split(object.Key, "/") parts := strings.Split(object.Key, "/")
if len(parts) < minSnapshotIDParts || if len(parts) >= minSnapshotIDParts &&
parts[0] != metadataDirName || parts[1] == "" { parts[0] == metadataDirName && parts[1] != "" {
continue
}
// Skip macOS resource fork files (._*) and other hidden files // Skip macOS resource fork files (._*) and other hidden files
if strings.HasPrefix(parts[1], ".") { if strings.HasPrefix(parts[1], ".") {
continue continue
} }
if !strings.HasSuffix(object.Key, "/") && if strings.HasSuffix(object.Key, "/") ||
!strings.Contains(object.Key, "/manifest.json.zst") { strings.Contains(object.Key, "/manifest.json.zst") {
continue
}
key := parts[1] key := parts[1]
// A remote snapshot key is a SHA-256 hash: 64 lowercase hex
// characters. The listing comes from the untrusted destination,
// so accept a key only in that form.
if !isBlobHash(key) {
log.Warn("Skipping non-conforming key under metadata/",
"key", object.Key)
continue
}
if !seen[key] { if !seen[key] {
seen[key] = true seen[key] = true
keys = append(keys, key) keys = append(keys, key)
} }
} }
}
}
return keys, nil return keys, nil
} }
+14
View File
@@ -69,6 +69,20 @@ func (v *Vaultik) resolveSnapshotRemoteKey(identifier string) (string, error) {
} }
} }
// resolveAndDownloadManifest resolves a snapshot identifier to its remote
// key (see resolveSnapshotRemoteKey) and downloads that snapshot's
// manifest.
func (v *Vaultik) resolveAndDownloadManifest(
identifier string,
) (*snapshot.Manifest, error) {
remoteKey, err := v.resolveSnapshotRemoteKey(identifier)
if err != nil {
return nil, err
}
return v.downloadManifestByKey(remoteKey)
}
// isRemoteKeyOrPrefix reports whether s is a full remote key or the // isRemoteKeyOrPrefix reports whether s is a full remote key or the
// leading part of one: 1 to 64 lowercase hex characters. A human snapshot // leading part of one: 1 to 64 lowercase hex characters. A human snapshot
// ID is never all hex, so this shape test is enough to tell the two apart. // ID is never all hex, so this shape test is enough to tell the two apart.
-6
View File
@@ -114,16 +114,10 @@ func (v *Vaultik) ListSnapshots(jsonOutput bool) error {
return encoder.Encode(snapshots) return encoder.Encode(snapshots)
} }
// The table is the output this command exists to produce, so it is
// written plain (markers would corrupt the columns) to the UI writer's
// stdout; --quiet silences it. Reconciliation notes below go through
// the UI methods, so their warnings still emit under --quiet.
if !v.UI.Quiet() {
err = v.printSnapshotTable(snapshots) err = v.printSnapshotTable(snapshots)
if err != nil { if err != nil {
return err return err
} }
}
if remoteErr == nil { if remoteErr == nil {
v.reportListDrift(snapshots, listing) v.reportListDrift(snapshots, listing)
@@ -1,41 +0,0 @@
package vaultik_test
import (
"testing"
"time"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
)
// TestListSnapshots_QuietSuppressesTableNotJSON is the --quiet contract
// for `snapshot list`: the human table is silenced, but the --json
// document a script depends on still emits. A local snapshot with its
// remote counterpart present is used so there are no drift notes, whose
// warnings would emit even under --quiet.
func TestListSnapshots_QuietSuppressesTableNotJSON(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
ts := time.Date(2026, 3, 1, 10, 0, 0, 0, time.UTC)
env.addLocal(t, listLocalID, ts)
env.addRemote(t, listLocalID, ts)
env.v.UI.SetQuiet(true)
// Table mode: nothing on stdout.
require.NoError(t, env.v.ListSnapshots(false))
require.Empty(t, env.stdout.String(),
"the table must be suppressed under --quiet")
// JSON mode: the document is still written despite the quiet UI.
env.stdout.Reset()
require.NoError(t, env.v.ListSnapshots(true))
rows := decodeListJSON(t, env.stdout.String())
require.Len(t, rows, 1)
require.Equal(t, listLocalID, rows[0].ID,
"the --json document must still emit under --quiet")
}
+28 -1
View File
@@ -5,6 +5,7 @@ package vaultik
import ( import (
"bytes" "bytes"
"context" "context"
"errors"
"fmt" "fmt"
"io" "io"
"os" "os"
@@ -12,6 +13,7 @@ import (
"github.com/spf13/afero" "github.com/spf13/afero"
"go.uber.org/fx" "go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/crypto"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/globals" "sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/snapshot" "sneak.berlin/go/vaultik/internal/snapshot"
@@ -19,6 +21,12 @@ import (
"sneak.berlin/go/vaultik/internal/ui" "sneak.berlin/go/vaultik/internal/ui"
) )
// Sentinel errors for misconfigured encryption settings.
var (
errNoAgeRecipients = errors.New("no age recipients configured")
errNoAgeSecretKey = errors.New("no age secret key configured")
)
// Vaultik contains all dependencies needed for vaultik operations // Vaultik contains all dependencies needed for vaultik operations
type Vaultik struct { type Vaultik struct {
Globals *globals.Globals Globals *globals.Globals
@@ -167,6 +175,26 @@ func (v *Vaultik) CanDecrypt() bool {
return v.Config.AgeSecretKey != "" return v.Config.AgeSecretKey != ""
} }
// GetEncryptor creates a new Encryptor instance based on the configured age recipients
// Returns an error if no recipients are configured
func (v *Vaultik) GetEncryptor() (*crypto.Encryptor, error) {
if len(v.Config.AgeRecipients) == 0 {
return nil, errNoAgeRecipients
}
return crypto.NewEncryptor(v.Config.AgeRecipients)
}
// GetDecryptor creates a new Decryptor instance based on the configured age secret key
// Returns an error if no secret key is configured
func (v *Vaultik) GetDecryptor() (*crypto.Decryptor, error) {
if v.Config.AgeSecretKey == "" {
return nil, errNoAgeSecretKey
}
return crypto.NewDecryptor(v.Config.AgeSecretKey)
}
// GetFilesystem returns the filesystem instance used by Vaultik // GetFilesystem returns the filesystem instance used by Vaultik
// //
//nolint:ireturn // afero.Fs is the filesystem abstraction by design //nolint:ireturn // afero.Fs is the filesystem abstraction by design
@@ -215,7 +243,6 @@ func NewForTesting(storage storage.Storer) *TestVaultik {
Stdout: stdout, Stdout: stdout,
Stderr: stderr, Stderr: stderr,
Stdin: stdin, Stdin: stdin,
UI: ui.NewWithColor(stdout, false),
}, },
Stdout: stdout, Stdout: stdout,
Stderr: stderr, Stderr: stderr,
+83 -114
View File
@@ -6,14 +6,14 @@ import (
"encoding/hex" "encoding/hex"
"errors" "errors"
"fmt" "fmt"
"hash"
"io" "io"
"os" "os"
"path/filepath" "path/filepath"
"time" "time"
"filippo.io/age" "github.com/klauspost/compress/zstd"
"sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot" "sneak.berlin/go/vaultik/internal/snapshot"
@@ -26,7 +26,6 @@ var (
"VAULTIK_AGE_SECRET_KEY not set; required for deep verification") "VAULTIK_AGE_SECRET_KEY not set; required for deep verification")
errChunksOutOfOrder = errors.New("chunks out of order") errChunksOutOfOrder = errors.New("chunks out of order")
errChunkHashMismatch = errors.New("chunk hash mismatch") errChunkHashMismatch = errors.New("chunk hash mismatch")
errNegativeChunkLength = errors.New("chunk length is negative")
errTrailingBlobData = errors.New( errTrailingBlobData = errors.New(
"blob has unexpected trailing bytes not covered by chunk list") "blob has unexpected trailing bytes not covered by chunk list")
errManifestExtraBlob = errors.New("manifest contains blob not in database") errManifestExtraBlob = errors.New("manifest contains blob not in database")
@@ -56,11 +55,6 @@ type VerifyResult struct {
Verified int `json:"verified"` Verified int `json:"verified"`
Missing int `json:"missing"` Missing int `json:"missing"`
MissingSize int64 `json:"missing_size,omitempty"` MissingSize int64 `json:"missing_size,omitempty"`
Mismatched int `json:"mismatched,omitempty"`
// DatabaseMissing is set by shallow verify when the snapshot's
// encrypted database (metadata/<key>/db.zst.age) is absent, which
// makes the snapshot unrestorable regardless of the blobs.
DatabaseMissing bool `json:"database_missing,omitempty"`
ErrorMessage string `json:"error,omitempty"` ErrorMessage string `json:"error,omitempty"`
} }
@@ -94,21 +88,13 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
errSecretKeyRequired.Error(), errSecretKeyRequired) errSecretKeyRequired.Error(), errSecretKeyRequired)
} }
// Parse the age secret key once, the same way restore does, and reuse
// the identities for the database and every blob.
identities, err := v.restoreIdentities()
if err != nil {
return v.deepVerifyFailure(result, opts, err.Error(), err)
}
log.Info("Starting snapshot verification", "snapshot_id", snapshotID, "mode", "deep") log.Info("Starting snapshot verification", "snapshot_id", snapshotID, "mode", "deep")
if !opts.JSON { if !opts.JSON {
v.stdoutf("Deep verification of snapshot: %s\n\n", snapshotID) v.stdoutf("Deep verification of snapshot: %s\n\n", snapshotID)
} }
manifest, tempDB, dbBlobs, err := v.loadVerificationData( manifest, tempDB, dbBlobs, err := v.loadVerificationData(snapshotID, opts, result)
snapshotID, opts, result, identities)
if err != nil { if err != nil {
return err return err
} }
@@ -128,8 +114,7 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
result.TotalSize = totalSize result.TotalSize = totalSize
err = v.runVerificationSteps( err = v.runVerificationSteps(manifest, dbBlobs, tempDB, opts, result, totalSize)
manifest, dbBlobs, tempDB, opts, result, totalSize, identities)
if err != nil { if err != nil {
return err return err
} }
@@ -154,7 +139,6 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
// loadVerificationData downloads manifest, database, and blob list for verification // loadVerificationData downloads manifest, database, and blob list for verification
func (v *Vaultik) loadVerificationData( func (v *Vaultik) loadVerificationData(
snapshotID string, opts *VerifyOptions, result *VerifyResult, snapshotID string, opts *VerifyOptions, result *VerifyResult,
identities []age.Identity,
) (*snapshot.Manifest, *tempDB, []snapshot.BlobInfo, error) { ) (*snapshot.Manifest, *tempDB, []snapshot.BlobInfo, error) {
// Resolve the identifier to the snapshot's remote key. A human ID is // Resolve the identifier to the snapshot's remote key. A human ID is
// hashed; a remote key (or its abbreviation, as printed for a // hashed; a remote key (or its abbreviation, as printed for a
@@ -191,10 +175,24 @@ func (v *Vaultik) loadVerificationData(
v.stdoutf("Downloading and decrypting database...\n") v.stdoutf("Downloading and decrypting database...\n")
} }
tdb, err := v.downloadVerifiedSnapshotDB( // Download and decrypt database
snapshotID, remoteKey, opts, result, identities) dbPath := fmt.Sprintf("metadata/%s/db.zst.age", remoteKey)
log.Info("Downloading encrypted database", "path", dbPath)
dbReader, err := v.Storage.Get(v.ctx, dbPath)
if err != nil { if err != nil {
return nil, nil, nil, err return nil, nil, nil, v.deepVerifyFailure(result, opts,
fmt.Sprintf("failed to download database: %v", err),
fmt.Errorf("failed to download database: %w", err))
}
defer func() { _ = dbReader.Close() }()
tdb, err := v.decryptAndLoadDatabase(dbReader)
if err != nil {
return nil, nil, nil, v.deepVerifyFailure(result, opts,
fmt.Sprintf("failed to decrypt database: %v", err),
fmt.Errorf("failed to decrypt database: %w", err))
} }
dbBlobs, err := v.getBlobsFromDatabase(tdb.db.Conn()) dbBlobs, err := v.getBlobsFromDatabase(tdb.db.Conn())
@@ -223,45 +221,6 @@ func (v *Vaultik) loadVerificationData(
return manifest, tdb, dbBlobs, nil return manifest, tdb, dbBlobs, nil
} }
// downloadVerifiedSnapshotDB downloads and decrypts the snapshot metadata
// database and confirms it really is the snapshot named by remoteKey
// before any of its rows are trusted (see verifySnapshotDBIdentity). On
// any failure it records the failure in result and returns the error the
// caller should propagate; the temp database is closed on a rejected
// identity so nothing is left on disk.
func (v *Vaultik) downloadVerifiedSnapshotDB(
snapshotID, remoteKey string, opts *VerifyOptions, result *VerifyResult,
identities []age.Identity,
) (*tempDB, error) {
dbPath := fmt.Sprintf("metadata/%s/db.zst.age", remoteKey)
log.Info("Downloading encrypted database", "path", dbPath)
dbReader, err := v.Storage.Get(v.ctx, dbPath)
if err != nil {
return nil, v.deepVerifyFailure(result, opts,
fmt.Sprintf("failed to download database: %v", err),
fmt.Errorf("failed to download database: %w", err))
}
defer func() { _ = dbReader.Close() }()
tdb, err := v.decryptAndLoadDatabase(dbReader, identities)
if err != nil {
return nil, v.deepVerifyFailure(result, opts,
fmt.Sprintf("failed to decrypt database: %v", err),
fmt.Errorf("failed to decrypt database: %w", err))
}
err = v.verifySnapshotDBIdentity(tdb.db, snapshotID, remoteKey)
if err != nil {
_ = tdb.Close()
return nil, v.deepVerifyFailure(result, opts, err.Error(), err)
}
return tdb, nil
}
// runVerificationSteps executes manifest verification, blob existence // runVerificationSteps executes manifest verification, blob existence
// check, and deep content verification. // check, and deep content verification.
func (v *Vaultik) runVerificationSteps( func (v *Vaultik) runVerificationSteps(
@@ -271,7 +230,6 @@ func (v *Vaultik) runVerificationSteps(
opts *VerifyOptions, opts *VerifyOptions,
result *VerifyResult, result *VerifyResult,
totalSize int64, totalSize int64,
identities []age.Identity,
) error { ) error {
if !opts.JSON { if !opts.JSON {
v.stdoutf("Verifying manifest against database...\n") v.stdoutf("Verifying manifest against database...\n")
@@ -298,7 +256,7 @@ func (v *Vaultik) runVerificationSteps(
len(dbBlobs), ubytes(totalSize)) len(dbBlobs), ubytes(totalSize))
} }
err = v.performDeepVerificationFromDB(dbBlobs, tdb.db.Conn(), opts, identities) err = v.performDeepVerificationFromDB(dbBlobs, tdb.db.Conn(), opts)
if err != nil { if err != nil {
return v.deepVerifyFailure(result, opts, err.Error(), err) return v.deepVerifyFailure(result, opts, err.Error(), err)
} }
@@ -323,18 +281,26 @@ func (t *tempDB) Close() error {
} }
// decryptAndLoadDatabase decrypts and loads the binary SQLite database // decryptAndLoadDatabase decrypts and loads the binary SQLite database
// from the encrypted stream. It reads through the same blobgen reader restore // from the encrypted stream.
// uses, streaming the decrypted, decompressed database to a temp file. func (v *Vaultik) decryptAndLoadDatabase(reader io.ReadCloser) (*tempDB, error) {
func (v *Vaultik) decryptAndLoadDatabase( // Get decryptor
reader io.ReadCloser, identities []age.Identity, decryptor, err := v.GetDecryptor()
) (*tempDB, error) {
// Decrypt and decompress through the shared blobgen reader.
blobReader, err := blobgen.NewReader(reader, identities...)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to create decryption reader: %w", err) return nil, fmt.Errorf("failed to get decryptor: %w", err)
} }
defer func() { _ = blobReader.Close() }() // Decrypt the stream
decryptedReader, err := decryptor.DecryptStream(reader)
if err != nil {
return nil, fmt.Errorf("failed to decrypt database: %w", err)
}
// Decompress the binary database
decompressor, err := zstd.NewReader(decryptedReader)
if err != nil {
return nil, fmt.Errorf("failed to create decompressor: %w", err)
}
defer decompressor.Close()
// Materialize the decrypted database inside a private (0700) temp // Materialize the decrypted database inside a private (0700) temp
// directory so it is never world-readable, and remove the whole // directory so it is never world-readable, and remove the whole
@@ -364,7 +330,7 @@ func (v *Vaultik) decryptAndLoadDatabase(
// Stream decompress directly to file // Stream decompress directly to file
log.Info("Decompressing database...") log.Info("Decompressing database...")
written, err := io.Copy(tempFile, blobReader) written, err := io.Copy(tempFile, decompressor)
if err != nil { if err != nil {
_ = tempFile.Close() _ = tempFile.Close()
@@ -389,40 +355,53 @@ func (v *Vaultik) decryptAndLoadDatabase(
} }
// verifyBlob downloads and verifies a single blob // verifyBlob downloads and verifies a single blob
func (v *Vaultik) verifyBlob( func (v *Vaultik) verifyBlob(blobInfo snapshot.BlobInfo, db *sql.DB) error {
blobInfo snapshot.BlobInfo, db *sql.DB, identities []age.Identity,
) error {
// Download blob using shared fetch method // Download blob using shared fetch method
reader, err := v.FetchBlob(v.ctx, blobInfo.Hash) reader, _, err := v.FetchBlob(v.ctx, blobInfo.Hash, blobInfo.CompressedSize)
if err != nil { if err != nil {
return fmt.Errorf("failed to download: %w", err) return fmt.Errorf("failed to download: %w", err)
} }
defer func() { _ = reader.Close() }() defer func() { _ = reader.Close() }()
// Decrypt and decompress through the shared blobgen reader, which hashes // Get decryptor
// the plaintext as it is read. A blob's hash — its remote name — is the decryptor, err := v.GetDecryptor()
// double SHA-256 of that plaintext (see blobgen.DoubleSHA256), not of the
// encrypted bytes.
blobReader, err := blobgen.NewReader(reader, identities...)
if err != nil { if err != nil {
return fmt.Errorf("failed to create blob reader: %w", err) return fmt.Errorf("failed to get decryptor: %w", err)
} }
defer func() { _ = blobReader.Close() }() // Decrypt blob
decryptedReader, err := decryptor.DecryptStream(reader)
if err != nil {
return fmt.Errorf("failed to decrypt: %w", err)
}
chunkCount, err := v.verifyBlobChunks(db, blobInfo.Hash, blobReader) // Decompress blob
decompressor, err := zstd.NewReader(decryptedReader)
if err != nil {
return fmt.Errorf("failed to decompress: %w", err)
}
defer decompressor.Close()
// A blob's hash — its remote name — is the double SHA256 of its
// decompressed plaintext (see blobgen.Writer.Sum256), not of the
// encrypted bytes. Hash the plaintext as chunk verification streams
// it, then compare on completion.
plaintextHasher := sha256.New()
hashedStream := io.TeeReader(decompressor, plaintextHasher)
chunkCount, err := v.verifyBlobChunks(db, blobInfo.Hash, hashedStream)
if err != nil { if err != nil {
return err return err
} }
err = v.verifyBlobFinalIntegrity(blobReader, blobInfo.Hash) err = v.verifyBlobFinalIntegrity(hashedStream, plaintextHasher, blobInfo.Hash)
if err != nil { if err != nil {
return err return err
} }
log.Info("Blob verified", log.Info("Blob verified",
"hash", shortHash(blobInfo.Hash)+"...", "hash", blobInfo.Hash[:16]+"...",
"chunks", chunkCount, "chunks", chunkCount,
"size", ubytes(blobInfo.CompressedSize), "size", ubytes(blobInfo.CompressedSize),
) )
@@ -488,24 +467,21 @@ func (v *Vaultik) verifyBlobChunks(
totalRead = offset totalRead = offset
} }
// length comes from an untrusted blob_chunks row: reject a // Read chunk data
// negative value, and hash by streaming exactly length bytes chunkData := make([]byte, length)
// rather than allocating a database-supplied size up front.
if length < 0 {
return 0, fmt.Errorf("%w: offset %d length %d",
errNegativeChunkLength, offset, length)
}
hasher := sha256.New() _, err = io.ReadFull(decompressor, chunkData)
n, err := io.CopyN(hasher, decompressor, length)
if err != nil { if err != nil {
return 0, fmt.Errorf("failed to read chunk at offset %d: %w", offset, err) return 0, fmt.Errorf("failed to read chunk at offset %d: %w", offset, err)
} }
totalRead += n totalRead += length
// Verify chunk hash
hasher := sha256.New()
hasher.Write(chunkData)
calculatedHash := hex.EncodeToString(hasher.Sum(nil)) calculatedHash := hex.EncodeToString(hasher.Sum(nil))
if calculatedHash != chunkHash { if calculatedHash != chunkHash {
return 0, fmt.Errorf("%w at offset %d: calculated %s, expected %s", return 0, fmt.Errorf("%w at offset %d: calculated %s, expected %s",
errChunkHashMismatch, offset, calculatedHash, chunkHash) errChunkHashMismatch, offset, calculatedHash, chunkHash)
@@ -525,12 +501,11 @@ func (v *Vaultik) verifyBlobChunks(
// verifyBlobFinalIntegrity checks that no trailing data exists in the // verifyBlobFinalIntegrity checks that no trailing data exists in the
// decompressed stream and that the blob hash matches the expected value. // decompressed stream and that the blob hash matches the expected value.
func (v *Vaultik) verifyBlobFinalIntegrity( func (v *Vaultik) verifyBlobFinalIntegrity(
blobReader *blobgen.Reader, expectedHash string, plaintext io.Reader, plaintextHasher hash.Hash, expectedHash string,
) error { ) error {
// Verify no remaining data in blob - if the chunk list is accurate, // Verify no remaining data in blob - if the chunk list is accurate,
// the blob should be fully consumed. Draining to EOF also completes the // the blob should be fully consumed.
// reader's plaintext hash. remaining, err := io.Copy(io.Discard, plaintext)
remaining, err := io.Copy(io.Discard, blobReader)
if err != nil { if err != nil {
return fmt.Errorf("failed to check for remaining blob data: %w", err) return fmt.Errorf("failed to check for remaining blob data: %w", err)
} }
@@ -539,9 +514,10 @@ func (v *Vaultik) verifyBlobFinalIntegrity(
return fmt.Errorf("%w: %d bytes", errTrailingBlobData, remaining) return fmt.Errorf("%w: %d bytes", errTrailingBlobData, remaining)
} }
// The blob hash is the double SHA-256 of its plaintext content. // The blob hash is the double SHA256 of its plaintext content.
calculatedBlobHash := hex.EncodeToString( firstHash := plaintextHasher.Sum(nil)
blobgen.DoubleSHA256(blobReader.Sum256())) secondHash := sha256.Sum256(firstHash)
calculatedBlobHash := hex.EncodeToString(secondHash[:])
if calculatedBlobHash != expectedHash { if calculatedBlobHash != expectedHash {
return fmt.Errorf("%w: calculated %s, expected %s", return fmt.Errorf("%w: calculated %s, expected %s",
@@ -655,12 +631,6 @@ func (v *Vaultik) verifyBlobExistenceFromDB(blobs []snapshot.BlobInfo) error {
log.Info("Verifying blob existence in S3", "blob_count", len(blobs)) log.Info("Verifying blob existence in S3", "blob_count", len(blobs))
for i, blob := range blobs { for i, blob := range blobs {
// The hash is read from the snapshot database, which is not
// trusted; check it before it is spliced into a storage path.
if !isBlobHash(blob.Hash) {
return fmt.Errorf("%w: %s", errInvalidBlobHash, shortHash(blob.Hash))
}
// Construct blob path // Construct blob path
blobPath := fmt.Sprintf("blobs/%s/%s/%s", blob.Hash[:2], blob.Hash[2:4], blob.Hash) blobPath := fmt.Sprintf("blobs/%s/%s/%s", blob.Hash[:2], blob.Hash[2:4], blob.Hash)
@@ -697,7 +667,6 @@ func (v *Vaultik) verifyBlobExistenceFromDB(blobs []snapshot.BlobInfo) error {
// each blob using the database as source. // each blob using the database as source.
func (v *Vaultik) performDeepVerificationFromDB( func (v *Vaultik) performDeepVerificationFromDB(
blobs []snapshot.BlobInfo, db *sql.DB, opts *VerifyOptions, blobs []snapshot.BlobInfo, db *sql.DB, opts *VerifyOptions,
identities []age.Identity,
) error { ) error {
// Calculate total bytes for ETA // Calculate total bytes for ETA
var totalBytesExpected int64 var totalBytesExpected int64
@@ -715,7 +684,7 @@ func (v *Vaultik) performDeepVerificationFromDB(
for i, blobInfo := range blobs { for i, blobInfo := range blobs {
// Verify individual blob // Verify individual blob
err := v.verifyBlob(blobInfo, db, identities) err := v.verifyBlob(blobInfo, db)
if err != nil { if err != nil {
return fmt.Errorf("blob %s verification failed: %w", blobInfo.Hash, err) return fmt.Errorf("blob %s verification failed: %w", blobInfo.Hash, err)
} }
+100
View File
@@ -0,0 +1,100 @@
package vaultik_test
import (
"bytes"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"io"
"testing"
"github.com/klauspost/compress/zstd"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/crypto"
)
// TestTeeReaderWithDecryption tests that TeeReader correctly hashes all encrypted
// bytes when streaming through age decryption and zstd decompression.
// This validates the verification path: hash encrypted blob -> decrypt -> decompress.
func TestTeeReaderWithDecryption(t *testing.T) {
t.Parallel()
// Test data - use random data that doesn't compress well (5MB)
testData := make([]byte, 5*1024*1024)
_, err := rand.Read(testData)
require.NoError(t, err)
// Compress the data
var compressedBuf bytes.Buffer
compressor, err := zstd.NewWriter(&compressedBuf,
zstd.WithEncoderLevel(zstd.SpeedDefault))
require.NoError(t, err)
_, err = compressor.Write(testData)
require.NoError(t, err)
err = compressor.Close()
require.NoError(t, err)
// Encrypt the compressed data
testRecipient := "age1cplgrwj77ta54dnmydvvmzn64ltk83ankxl5sww04mrt" +
"mu62kv3s89gmvv"
testSecretKey := "AGE-SECRET-KEY-1C77PYNTHXSHNNC6EYR2W52UWYXACXA5J" +
"T00J9CCW9986M3XY87PSGP89AQ"
encryptor, err := crypto.NewEncryptor([]string{testRecipient})
require.NoError(t, err)
var encryptedBuf bytes.Buffer
err = encryptor.EncryptStream(&encryptedBuf, bytes.NewReader(compressedBuf.Bytes()))
require.NoError(t, err)
encryptedData := encryptedBuf.Bytes()
// Calculate the expected hash of the encrypted data directly
expectedHash := sha256.Sum256(encryptedData)
expectedHashStr := hex.EncodeToString(expectedHash[:])
t.Logf("Encrypted data size: %d bytes", len(encryptedData))
t.Logf("Expected hash: %s", expectedHashStr)
// Now simulate what verifyBlob does: use TeeReader to hash while decrypting
decryptor, err := crypto.NewDecryptor(testSecretKey)
require.NoError(t, err)
// Create hasher and tee reader
hasher := sha256.New()
reader := bytes.NewReader(encryptedData)
teeReader := io.TeeReader(reader, hasher)
// Decrypt through the tee reader
decryptedReader, err := decryptor.DecryptStream(teeReader)
require.NoError(t, err)
// Decompress
decompressor, err := zstd.NewReader(decryptedReader)
require.NoError(t, err)
defer decompressor.Close()
// Read all decompressed data (simulating chunk verification)
decompressedData, err := io.ReadAll(decompressor)
require.NoError(t, err)
// Verify we got the original data back
assert.Equal(t, testData, decompressedData, "Decompressed data should match original")
// Drain remaining decompressed data (should be 0)
remaining, err := io.Copy(io.Discard, decompressor)
require.NoError(t, err)
assert.Equal(t, int64(0), remaining, "No remaining decompressed data")
// Calculate hash from tee reader
calculatedHashStr := hex.EncodeToString(hasher.Sum(nil))
t.Logf("Calculated hash (before drain): %s", calculatedHashStr)
// Verify the hash matches the direct hash of encrypted data
assert.Equal(t, expectedHashStr, calculatedHashStr,
"Hash calculated via TeeReader should match direct hash of encrypted data")
}