Correct the security claims in docs and comments, and record the accepted risks (closes #171)
check / check (pull_request) Successful in 2m32s
check / check (pull_request) Successful in 2m32s
Docs and comments only; no behaviour change. Corrects ten overclaims the security review found: snapshot names are hashed but the hash uses no secret, so a guessed hostname and name can be confirmed; a blob is named by hex(SHA256(SHA256(uncompressed contents))), stated once in docs/REPOSTRUCTURE.md and referenced elsewhere; double hashing does not hide known content (blob packing does); age uses ChaCha20-Poly1305, not XChaCha20; encryption is required, not optional; a snapshot is marked complete before its metadata is uploaded; the export comment now matches its only caller; deep verify detects corruption, not authorship; adding a recipient does not reach existing data; restore examples target a user-owned directory. Adds an Accepted Risks subsection under Security Considerations with the seven documented risks, cross-referenced from the README. Model: opus-4-8
This commit is contained in:
+4
-4
@@ -74,16 +74,16 @@ Maps files to their constituent chunks:
|
||||
#### Blob (`database.Blob`)
|
||||
The final storage unit uploaded to S3. Contains many compressed and encrypted chunks:
|
||||
- `ID`: UUID assigned at creation
|
||||
- `Hash`: SHA256 of final compressed+encrypted content
|
||||
- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs))
|
||||
- `UncompressedSize`: Total raw chunk data before compression
|
||||
- `CompressedSize`: Size after zstd compression and age encryption
|
||||
- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
|
||||
|
||||
Blob creation process:
|
||||
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
|
||||
2. Compressed with zstd
|
||||
3. Encrypted with age (recipients configured in config)
|
||||
4. SHA256 hash computed → becomes filename in S3
|
||||
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
|
||||
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and streamed to storage
|
||||
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
|
||||
5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}`
|
||||
|
||||
#### BlobChunk (`database.BlobChunk`)
|
||||
|
||||
@@ -38,7 +38,7 @@ vaultik snapshot list
|
||||
|
||||
Features:
|
||||
|
||||
* modern encryption ([age](https://age-encryption.org/), X25519 + XChaCha20-Poly1305)
|
||||
* modern encryption ([age](https://age-encryption.org/), X25519 + ChaCha20-Poly1305)
|
||||
* content-defined chunking with deduplication (FastCDC)
|
||||
* incremental backups (only changed files are re-chunked)
|
||||
* multithreaded zstd compression at configurable levels
|
||||
@@ -79,11 +79,13 @@ vaultik snapshot verify <snapshot-id>
|
||||
# with one or more identities, is accepted
|
||||
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
|
||||
|
||||
# deep verify (downloads and cryptographically verifies every blob)
|
||||
# deep verify (downloads every blob, decrypts it, and re-hashes it to
|
||||
# detect corruption — this checks integrity, not who wrote the blob)
|
||||
vaultik snapshot verify --deep <snapshot-id>
|
||||
|
||||
# restore (requires the private key)
|
||||
vaultik snapshot restore <snapshot-id> /tmp/restored
|
||||
# restore (requires the private key). Restore into a new directory you own,
|
||||
# writable only by you — not a shared location like /tmp
|
||||
vaultik snapshot restore <snapshot-id> ~/vaultik-restore
|
||||
|
||||
# daily cron job: back up, keep a 4-week rolling window of snapshots
|
||||
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
|
||||
@@ -128,12 +130,13 @@ full `remote_key` from `snapshot list --json` — to restore and verify:
|
||||
# keeps the key out of your shell history)
|
||||
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
|
||||
|
||||
# restore everything to /tmp/restored, then check every restored file's
|
||||
# chunk hashes
|
||||
vaultik snapshot restore --verify <remote-key> /tmp/restored
|
||||
# restore everything to a new directory you own (writable only by you, not a
|
||||
# shared location like /tmp), then check every restored file's chunk hashes
|
||||
vaultik snapshot restore --verify <remote-key> ~/vaultik-restore
|
||||
|
||||
# optionally, deep-verify the snapshot against the store (downloads and
|
||||
# cryptographically checks every blob)
|
||||
# optionally, deep-verify the snapshot against the store (downloads every
|
||||
# blob, decrypts it, and re-hashes it to detect corruption — this checks
|
||||
# integrity, not who wrote the blob)
|
||||
vaultik snapshot verify --deep <remote-key>
|
||||
```
|
||||
|
||||
@@ -449,14 +452,19 @@ Snapshot IDs follow the human-readable format
|
||||
`<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g.
|
||||
`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the
|
||||
destination store in plaintext. Each snapshot's metadata directory is named
|
||||
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a listing
|
||||
of the store reveals no hostname or snapshot name. The backup time is not
|
||||
hidden: manifest.json.zst carries a plaintext timestamp, and object
|
||||
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a plain
|
||||
listing of the store shows no hostname or snapshot name. The hash uses no
|
||||
secret, though, so an observer who guesses a candidate hostname and snapshot
|
||||
name can hash it and confirm the snapshot is present; the remote key keeps
|
||||
names out of a listing but does not hide them from a guess. The backup time is
|
||||
not hidden either: manifest.json.zst carries a plaintext timestamp, and object
|
||||
modification times are visible at the storage layer regardless. For example,
|
||||
`server1_home_2025-06-01T12:00:00Z` is stored under
|
||||
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`.
|
||||
See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
|
||||
derivation.
|
||||
derivation, and [Security Considerations](docs/REPOSTRUCTURE.md#security-considerations)
|
||||
(including [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks)) for what the
|
||||
format does and does not protect.
|
||||
|
||||
### data flow
|
||||
|
||||
@@ -497,7 +505,7 @@ derivation.
|
||||
|
||||
### encryption
|
||||
|
||||
* Asymmetric encryption using age (X25519 + XChaCha20-Poly1305)
|
||||
* Asymmetric encryption using age (X25519 + ChaCha20-Poly1305)
|
||||
* Only the public key is needed on the source host
|
||||
* Each blob and each metadata database is encrypted independently
|
||||
* Multiple recipients supported (encrypt to multiple keys)
|
||||
|
||||
@@ -306,6 +306,9 @@ storage_url: "rclone://myremote/path/to/backups"
|
||||
# Multiple chunks are packed into blobs up to this size
|
||||
# Must be at least four times chunk_size (the largest chunk the chunker can
|
||||
# emit); a smaller limit would let a single-chunk blob exceed it.
|
||||
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
|
||||
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
|
||||
# not visible in its size; lowering the limit toward chunk_size exposes them.
|
||||
# Supports: 1GB, 10G, 500MB, 1GiB, etc.
|
||||
# Default: 10GB
|
||||
#blob_size_limit: 10GB
|
||||
|
||||
+2
-2
@@ -90,7 +90,7 @@ Stores information about packed, compressed, and encrypted blob files.
|
||||
|
||||
**Columns:**
|
||||
- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts
|
||||
- `blob_hash` (TEXT UNIQUE) - SHA256 hash of final blob (NULL until finalized)
|
||||
- `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (NULL until finalized); see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs)
|
||||
- `created_ts` (INTEGER NOT NULL) - Creation timestamp
|
||||
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
|
||||
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
|
||||
@@ -216,7 +216,7 @@ After a snapshot is completed:
|
||||
5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
|
||||
6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`
|
||||
|
||||
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation).
|
||||
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext. The hash uses no secret, so a guessed hostname and snapshot name can still be confirmed against a listing; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its [Accepted Risks](REPOSTRUCTURE.md#accepted-risks).
|
||||
|
||||
### 4. Restore Process
|
||||
|
||||
|
||||
+18
-5
@@ -35,7 +35,7 @@ The metadata subdirectory is named with the **remote key**, a one-way hash of th
|
||||
- **What it contains**: Packed collections of content-defined chunks from files
|
||||
- **Format**: Zstandard compressed, then Age encrypted
|
||||
- **Encryption**: Always encrypted with Age using the configured recipients
|
||||
- **Naming**: Content-addressed using SHA256 hash of the encrypted blob
|
||||
- **Naming**: Content-addressed. The blob's name is `hex(SHA256(SHA256(uncompressed blob contents)))` — the double SHA-256 of the concatenated chunk data, computed before compression and encryption, not a hash of the stored (compressed, encrypted) bytes. One consequence: only a holder of the age private key can check a stored blob's integrity, because matching a blob to its name means decrypting and decompressing it first — which is what `restore` and `verify --deep` do. Implemented in `internal/blobgen` (`DoubleSHA256`). This is the canonical description of blob naming; other documents and comments point here.
|
||||
|
||||
### Why Encrypted
|
||||
Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy.
|
||||
@@ -59,14 +59,14 @@ This ID reveals the hostname, the configured snapshot name, and the backup time,
|
||||
|
||||
### Remote Key Derivation
|
||||
|
||||
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string with no structure a remote observer can reverse. Implemented in `internal/snapshot/remotekey.go`.
|
||||
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string. The hash is not reversible, but it uses no secret: an observer who guesses a candidate hostname and snapshot name can hash it the same way and confirm whether that snapshot is present. The remote key keeps names out of a plain listing; it does not hide them from a guess. Implemented in `internal/snapshot/remotekey.go`.
|
||||
|
||||
Worked example:
|
||||
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
|
||||
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
|
||||
- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
|
||||
|
||||
Because the hash is one-way, a listing of the destination store reveals neither the hostname nor the snapshot name of any backup. The same remote key is stored in the manifest's `snapshot_id` field.
|
||||
A plain listing of the destination store therefore shows only these hashes, not the hostname or snapshot name of any backup — but because the hash uses no secret, a guessed hostname and snapshot name can be hashed and confirmed against the listing. The same remote key is stored in the manifest's `snapshot_id` field.
|
||||
|
||||
### Files in Each Snapshot Directory
|
||||
|
||||
@@ -124,23 +124,36 @@ From the unencrypted data, an observer of the destination store can determine:
|
||||
- **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear
|
||||
- How many blobs each snapshot references, and the total compressed size
|
||||
- The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)
|
||||
- **Whether a guessed hostname and snapshot name are present** — the remote key is an unkeyed hash, so an observer holding candidate names can hash each one and match it against the directory listing. The human ID is never published, so it cannot be read off directly, but it can be confirmed by guessing.
|
||||
|
||||
Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless.
|
||||
|
||||
An observer cannot determine:
|
||||
- The hostname or snapshot name of any backup (the directory name and the manifest `snapshot_id` are one-way hashes of the human ID)
|
||||
- The hostname or snapshot name of any backup by reading it off the store — the directory name and the manifest `snapshot_id` are unkeyed hashes of the human ID, so the text is never published (though a guessed name can be confirmed, as above)
|
||||
- File names or paths
|
||||
- File contents
|
||||
- File permissions or ownership
|
||||
- Directory structure
|
||||
- Which chunks belong to which files
|
||||
|
||||
### Accepted Risks
|
||||
|
||||
These are known, deliberate properties of the format and the tooling, recorded so an operator can weigh them rather than discover them.
|
||||
|
||||
1. **No proof of authorship.** Restore and `verify --deep` prove that data decrypts with the age private key and matches its unkeyed content hashes. They do not prove who wrote it: anyone who knows a recipient public key and can replace objects on the destination can substitute a snapshot they built. The recipient string is not stored at the destination, but a compromised backed-up host has it. Defences live on the destination side — bucket versioning or object lock, credentials for the source host that cannot delete or overwrite existing versions, and pruning from a trusted host. Note that S3 `PutObject` overwrites an existing key, so PUT permission alone is not append-only.
|
||||
2. **Compression reveals sizes.** Blobs and `db.zst.age` are zstd-compressed then age-encrypted; the manifest is compressed only. age does not pad, so an object's size is the exact compressed length of its contents. All new chunks packed into one blob share a single zstd stream (8 MiB window, 4 MiB at compression levels 1-2), and a blob is closed at `blob_size_limit` and at the end of each configured path. Because a stored chunk is never packed again, someone who can write into a backed-up file and watch blob sizes learns something only when their controlled data and a secret land in the same chunk of a file that keeps changing. Advice: back up any outsider-writable directory as its own snapshot.
|
||||
3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of 2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them.
|
||||
4. **Decrypted data on local disk.** Several commands stage plaintext under `$TMPDIR`: `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` and removed on success, but a `kill -9` or a power loss leaves them behind — delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` should be trusted to the same degree as the restore target.
|
||||
5. **Store permissions differ per command.** The backed-up host needs only PUT to run `snapshot create`: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — `snapshot verify`, `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`, `snapshot remove`, and `remote nuke` also delete. The recommended cron line uses `--prune`, which runs `prune` on the backed-up host, so granting that host `--prune` gives it credentials that can delete its own backups. To keep the source host to PUT only, prune from a separate trusted host instead.
|
||||
6. **Changing recipients does not re-encrypt existing data.** Deduplicated chunks and same-named blobs already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run `vaultik database delete` and take a full backup to a fresh destination or prefix.
|
||||
7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients. Long-lived ciphertext held by a third party (the destination operator) has no fallback if X25519 is ever broken: there is no second recipient type and no post-quantum option.
|
||||
|
||||
## Consistency Guarantees
|
||||
|
||||
1. **Blobs are immutable** - Once written, a blob is never modified
|
||||
2. **Blobs are written before metadata** - A snapshot's metadata is only written after all its blobs are successfully uploaded
|
||||
3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst are written as complete files
|
||||
4. **Snapshots are marked complete in local DB only after metadata upload** - Ensures consistency between local and remote state
|
||||
4. **A snapshot is marked complete in the local DB before its metadata is uploaded, not after** - `CompleteSnapshot` runs first, then `ExportSnapshotMetadata` (see the backup data flow in [ARCHITECTURE.md](../ARCHITECTURE.md)). A crash between the two leaves a completed-looking row in the local index with no matching metadata on the destination store. `vaultik prune` reconciles this away: it drops any local snapshot whose remote metadata is missing.
|
||||
|
||||
## Pruning Safety
|
||||
|
||||
|
||||
@@ -1,14 +1,16 @@
|
||||
// Package blob handles the creation of blobs - the final storage units for Vaultik.
|
||||
// A blob is a large file (up to 10GB) containing many compressed and encrypted chunks
|
||||
// from multiple source files. Blobs are content-addressed, meaning their filename
|
||||
// is derived from the SHA256 hash of their compressed and encrypted content.
|
||||
// from multiple source files. Blobs are content-addressed: a blob's filename is
|
||||
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the concatenated
|
||||
// chunk data before compression and encryption, not from the stored bytes. See
|
||||
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
|
||||
//
|
||||
// The blob creation process:
|
||||
// 1. Chunks are accumulated from multiple files
|
||||
// 2. The collection is compressed using zstd
|
||||
// 3. The compressed data is encrypted using age
|
||||
// 4. The encrypted blob is hashed to create its content-addressed name
|
||||
// 5. The blob is uploaded to S3 using the hash as the filename
|
||||
// 2. Each chunk's uncompressed bytes are fed to a running SHA-256 and, in the same
|
||||
// pass, compressed with zstd and encrypted with age into the temp file
|
||||
// 3. On finalize, the name is the double SHA-256 of that uncompressed content
|
||||
// 4. The blob is uploaded to S3 using the name as the filename
|
||||
//
|
||||
// This design optimizes storage efficiency by batching many small chunks into
|
||||
// larger blobs, reducing the number of S3 operations and associated costs.
|
||||
|
||||
@@ -16,11 +16,17 @@ import (
|
||||
)
|
||||
|
||||
// DoubleSHA256 returns the double SHA-256 of content whose single SHA-256
|
||||
// digest is sum: it hashes that digest once more. Stored objects are named by
|
||||
// this second hash so that a name never reveals whether known content is
|
||||
// present — an attacker who knows a plaintext, and thus its SHA-256, still
|
||||
// cannot derive the stored name without hashing the digest again. Both a blob
|
||||
// and the metadata database export are named this way.
|
||||
// digest is sum: it hashes that digest once more. Stored objects — a blob, and
|
||||
// the metadata database export — are named by this second hash.
|
||||
//
|
||||
// The second hash does not hide whether known content is stored: an attacker
|
||||
// who can reproduce an object's entire plaintext computes the same name simply
|
||||
// by hashing twice, exactly as this code does. What limits that is blob
|
||||
// packing, not the double hash — a blob's name covers all of its concatenated
|
||||
// chunk plaintext, so a name can be confirmed only by someone who can
|
||||
// reproduce the whole blob (a snapshot made entirely of known content, or a
|
||||
// known file large enough to fill blobs on its own). An ordinary file that
|
||||
// shares a blob with other, unknown data cannot be confirmed this way.
|
||||
func DoubleSHA256(sum []byte) []byte {
|
||||
h := sha256.Sum256(sum)
|
||||
|
||||
@@ -147,8 +153,8 @@ func (w *Writer) Close() error {
|
||||
|
||||
// ContentID returns the double SHA-256 of the uncompressed input data: the
|
||||
// name under which this content is stored. It is the second hash of the
|
||||
// running SHA-256, via DoubleSHA256; see that function for why content is
|
||||
// named this way rather than by its plain SHA-256.
|
||||
// running SHA-256, via DoubleSHA256; see that function for what naming content
|
||||
// this way does and does not hide.
|
||||
func (w *Writer) ContentID() []byte {
|
||||
return DoubleSHA256(w.hasher.Sum(nil))
|
||||
}
|
||||
|
||||
@@ -46,8 +46,11 @@ const defaultConfigTemplate = `# vaultik configuration
|
||||
# ─── REQUIRED ────────────────────────────────────────────────────────────────
|
||||
|
||||
# Age recipient public keys for encryption.
|
||||
# Backups are encrypted to ALL listed recipients. Any one of the corresponding
|
||||
# private keys can decrypt. Generate a keypair with:
|
||||
# Backups are encrypted to ALL listed recipients; any one of the corresponding
|
||||
# private keys can decrypt. Adding a recipient later does not re-encrypt data
|
||||
# already stored: deduplicated chunks and existing blobs stay encrypted to the
|
||||
# earlier recipients, so a newly added key cannot restore them on its own (see
|
||||
# docs/REPOSTRUCTURE.md, Accepted Risks). Generate a keypair with:
|
||||
# age-keygen -o vaultik_backup_private_key.txt
|
||||
# grep 'public key' vaultik_backup_private_key.txt
|
||||
age_recipients:
|
||||
|
||||
@@ -192,7 +192,8 @@ func newSnapshotVerifyCommand() *cobra.Command {
|
||||
Long: "Checks that every blob the snapshot's manifest lists is present\n" +
|
||||
"in storage with the size the manifest records, and that the\n" +
|
||||
"snapshot's encrypted database is present. It does not read blob\n" +
|
||||
"contents; use --deep to download and cryptographically verify them.\n\n" +
|
||||
"contents; use --deep to download, decrypt, and re-hash every blob\n" +
|
||||
"to detect corruption -- integrity, not who wrote it.\n\n" +
|
||||
"The snapshot may be named by its ID or, on a host with no local\n" +
|
||||
"index, by the remote key that 'snapshot list' prints for a\n" +
|
||||
"remote-only snapshot (an unambiguous leading part is enough).",
|
||||
|
||||
@@ -3,8 +3,10 @@
|
||||
//
|
||||
// Blobs in Vaultik are the final storage units uploaded to S3. Each blob is a
|
||||
// large (up to 10GB) file containing many compressed and encrypted chunks from
|
||||
// multiple source files. Blobs are content-addressed, meaning their filename
|
||||
// is derived from their SHA256 hash after compression and encryption.
|
||||
// multiple source files. Blobs are content-addressed: the filename in S3 is
|
||||
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data
|
||||
// before compression and encryption (not from the stored bytes). See
|
||||
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
|
||||
//
|
||||
// Schema is managed via numbered SQL migrations embedded in the schema/
|
||||
// directory. Migration 000.sql bootstraps the schema_migrations tracking
|
||||
|
||||
@@ -51,15 +51,15 @@ type Chunk struct {
|
||||
// Blob represents a blob record in the database.
|
||||
// A blob is Vaultik's final storage unit - a large file (up to 10GB) containing
|
||||
// many compressed and encrypted chunks from multiple source files.
|
||||
// Blobs are content-addressed, meaning their filename in S3 is derived from
|
||||
// the SHA256 hash of their compressed and encrypted content.
|
||||
// The blob creation process is: chunks are accumulated -> compressed with zstd
|
||||
// -> encrypted with age -> hashed -> uploaded to S3 with the hash as filename.
|
||||
// Blobs are content-addressed: the filename in S3 is
|
||||
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data
|
||||
// before compression and encryption (not from the stored bytes). See
|
||||
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
|
||||
type Blob struct {
|
||||
ID types.BlobID // UUID assigned when blob creation starts
|
||||
|
||||
// Hash is the SHA256 of the final compressed+encrypted content
|
||||
// (empty until finalized).
|
||||
// Hash is hex(SHA256(SHA256(uncompressed blob contents)))
|
||||
// (empty until finalized); see the type comment above.
|
||||
Hash types.BlobHash
|
||||
CreatedTS time.Time // When blob creation started
|
||||
FinishedTS *time.Time // When blob was finalized (nil if still packing)
|
||||
|
||||
@@ -119,7 +119,7 @@ type ScannerConfig struct {
|
||||
Storage storage.Storer
|
||||
MaxBlobSize int64
|
||||
CompressionLevel int
|
||||
AgeRecipients []string // Optional, empty means no encryption
|
||||
AgeRecipients []string // required; output is always encrypted
|
||||
EnableProgress bool // Enable the live progress reporter (ETAs, throughput)
|
||||
UI *ui.Writer // Where user-facing scanner messages go; nil = discard
|
||||
Exclude []string // Glob patterns for files/directories to exclude
|
||||
|
||||
@@ -24,7 +24,7 @@ package snapshot
|
||||
// 7. Close the temporary database
|
||||
// 8. VACUUM the database to remove deleted data and compact (security critical)
|
||||
// 9. Compress the binary database with zstd
|
||||
// 10. Encrypt the compressed database with age (if encryption is enabled)
|
||||
// 10. Encrypt the compressed database with age (always; recipients are required)
|
||||
// 11. Upload to S3 as: metadata/{snapshot-id}/db.zst.age
|
||||
// 12. Reopen the main database
|
||||
//
|
||||
@@ -238,14 +238,12 @@ func (sm *SnapshotManager) CompleteSnapshot(
|
||||
// 3. Cleans the copy to contain only current snapshot data
|
||||
// 4. Dumps the cleaned database to SQL
|
||||
// 5. Compresses the SQL dump with zstd
|
||||
// 6. Encrypts the compressed data (if encryption is enabled)
|
||||
// 6. Encrypts the compressed data with age (always; recipients are required)
|
||||
// 7. Uploads to S3 at: snapshots/{snapshot-id}.sql.zst[.age]
|
||||
//
|
||||
// The caller is responsible for:
|
||||
// - Ensuring the main database is closed before calling this method
|
||||
// - Reopening the main database after this method returns
|
||||
//
|
||||
// This ensures database consistency during the copy operation.
|
||||
// The only caller (finalizeSnapshotMetadata) does not close the main database
|
||||
// before calling this method: the index is copied at dbPath while it is still
|
||||
// open, and every step here operates on that copy, never on the live index.
|
||||
func (sm *SnapshotManager) ExportSnapshotMetadata(
|
||||
ctx context.Context, dbPath string, snapshotID string,
|
||||
) error {
|
||||
@@ -415,9 +413,11 @@ func (sm *SnapshotManager) prepareExportDB(
|
||||
// uploadSnapshotArtifacts uploads the database backup and blob manifest
|
||||
// to remote storage at metadata/<remote-key>/, where remote-key is the
|
||||
// double-SHA256 derivation of the snapshot ID (see RemoteSnapshotKey).
|
||||
// We never write the human-readable snapshot ID into any unencrypted
|
||||
// part of remote storage so a listing of the destination bucket leaks
|
||||
// no host, configuration, or scheduling information.
|
||||
// The human-readable snapshot ID is never written into an unencrypted part
|
||||
// of remote storage, so a plain listing shows only the hashed key, not the
|
||||
// hostname or snapshot name. The hash uses no secret, so a guessed hostname
|
||||
// and snapshot name can still be confirmed against a listing, and the backup
|
||||
// time is public: the manifest carries a plaintext timestamp.
|
||||
func (sm *SnapshotManager) uploadSnapshotArtifacts(
|
||||
ctx context.Context, snapshotID string, dbData, manifestData []byte,
|
||||
) error {
|
||||
@@ -814,10 +814,11 @@ func (sm *SnapshotManager) generateBlobManifest(
|
||||
}
|
||||
|
||||
// Create manifest. SnapshotID in the unencrypted manifest is the
|
||||
// double-SHA256 remote key (see RemoteSnapshotKey), not the human ID,
|
||||
// so neither this field nor the directory name reveals the hostname or
|
||||
// snapshot name. Timestamp below is written in the clear, so the backup
|
||||
// time is observable to anyone who can read the manifest.
|
||||
// double-SHA256 remote key (see RemoteSnapshotKey), not the human ID, so
|
||||
// neither this field nor the directory name spells out the hostname or
|
||||
// snapshot name — but the key uses no secret, so a guessed hostname and
|
||||
// snapshot name can be confirmed. Timestamp below is written in the clear,
|
||||
// so the backup time is observable to anyone who can read the manifest.
|
||||
manifest := &Manifest{
|
||||
SnapshotID: RemoteSnapshotKey(snapshotID),
|
||||
Timestamp: time.Now().UTC().Format(time.RFC3339),
|
||||
|
||||
@@ -146,8 +146,10 @@ type SnapshotID string
|
||||
// Used for content-addressing and deduplication of file chunks.
|
||||
type ChunkHash string
|
||||
|
||||
// BlobHash is the SHA256 hash of a blob's compressed and encrypted content.
|
||||
// This is used as the filename in S3 storage for content-addressed retrieval.
|
||||
// BlobHash is hex(SHA256(SHA256(uncompressed blob contents))), computed before
|
||||
// compression and encryption (see blobgen.DoubleSHA256 and
|
||||
// docs/REPOSTRUCTURE.md). It is used as the filename in S3 storage for
|
||||
// content-addressed retrieval.
|
||||
type BlobHash string
|
||||
|
||||
// FilePath represents an absolute path to a file or directory.
|
||||
|
||||
Reference in New Issue
Block a user