Correct remote layout and privacy docs for hashed snapshot keys (closes #67) #128

Merged
clawbot merged 1 commits from issue-67-remote-layout-docs into next 2026-09-21 19:24:45 +02:00
6 changed files with 69 additions and 28 deletions
Showing only changes of commit 978fef8b2c - Show all commits
+7 -1
View File
@@ -366,11 +366,17 @@ bucket/
│ └── {full-hash} # Compressed+encrypted blob │ └── {full-hash} # Compressed+encrypted blob
└── metadata/ └── metadata/
└── {snapshot-id}/ └── {remote-key}/
├── db.zst.age # Encrypted binary SQLite database ├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Blob list (for pruning/verification) └── manifest.json.zst # Blob list (for pruning/verification)
``` ```
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
written to the store as a directory name. See
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation and a worked example.
## Thread Safety ## Thread Safety
- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`. - `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
+14 -4
View File
@@ -344,7 +344,7 @@ both are set.
├── blobs/ ├── blobs/
│ └── <aa>/<bb>/<full_blob_hash> │ └── <aa>/<bb>/<full_blob_hash>
└── metadata/ └── metadata/
└── <snapshot_id>/ └── <remote-key>/
├── db.zst.age # Encrypted binary SQLite database ├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Unencrypted blob list (for pruning) └── manifest.json.zst # Unencrypted blob list (for pruning)
``` ```
@@ -355,8 +355,18 @@ both are set.
* `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling * `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling
pruning without the private key pruning without the private key
Snapshot IDs follow the format `<hostname>_<snapshot-name>_<RFC3339-timestamp>` Snapshot IDs follow the human-readable format
(e.g. `server1_home_2025-06-01T12:00:00Z`). `<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g.
`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the
destination store in plaintext. Each snapshot's metadata directory is named
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a listing
of the store reveals no hostname or snapshot name. The backup time is not
hidden: manifest.json.zst carries a plaintext timestamp, and object
modification times are visible at the storage layer regardless. For example,
`server1_home_2025-06-01T12:00:00Z` is stored under
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`.
See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation.
### data flow ### data flow
@@ -373,7 +383,7 @@ Snapshot IDs follow the format `<hostname>_<snapshot-name>_<RFC3339-timestamp>`
**restore:** **restore:**
1. Download and decrypt `metadata/<snapshot_id>/db.zst.age` 1. Download and decrypt `metadata/<remote-key>/db.zst.age`
2. Open the binary SQLite database 2. Open the binary SQLite database
3. Query files (optionally filtered by paths) 3. Query files (optionally filtered by paths)
4. Download and decrypt required blobs 4. Download and decrypt required blobs
+4 -2
View File
@@ -194,8 +194,10 @@ After a snapshot is completed:
2. Clean temporary database to contain only current snapshot data 2. Clean temporary database to contain only current snapshot data
3. Export to SQL dump using sqlite3 3. Export to SQL dump using sqlite3
4. Compress with zstd and encrypt with age 4. Compress with zstd and encrypt with age
5. Upload to S3 as `metadata/{snapshot-id}/db.zst.age` 5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
6. Generate blob manifest and upload as `metadata/{snapshot-id}/manifest.json.zst` 6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation).
### 4. Restore Process ### 4. Restore Process
+37 -17
View File
@@ -17,11 +17,13 @@ Vaultik stores all backup data in an S3-compatible object store. The repository
│ └── <hash[2:4]>/ │ └── <hash[2:4]>/
│ └── <full-hash> │ └── <full-hash>
└── metadata/ └── metadata/
└── <snapshot-id>/ └── <remote-key>/
├── db.zst.age ├── db.zst.age
└── manifest.json.zst └── manifest.json.zst
``` ```
The metadata subdirectory is named with the **remote key**, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See [Remote Key Derivation](#remote-key-derivation).
## Blobs Directory (`blobs/`) ## Blobs Directory (`blobs/`)
### Structure ### Structure
@@ -40,9 +42,11 @@ Blobs contain the actual file data from backups and must be encrypted for securi
## Metadata Directory (`metadata/`) ## Metadata Directory (`metadata/`)
Each snapshot has its own subdirectory named with the snapshot ID. Each snapshot has its own subdirectory. The directory is **not** named with the human-readable snapshot ID; it is named with the remote key — a one-way hash of that ID. The human ID is never written to the destination store as a directory name (see [Remote Key Derivation](#remote-key-derivation)).
### Snapshot ID Format ### Snapshot ID Format
The human-readable snapshot ID is used in CLI arguments, log lines, and the local database. It is not written to the destination store.
- **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>` if no - **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>` if no
name was specified) name was specified)
- **Example**: `laptop_home_2024-01-15T14:30:52Z` - **Example**: `laptop_home_2024-01-15T14:30:52Z`
@@ -51,6 +55,19 @@ Each snapshot has its own subdirectory named with the snapshot ID.
- Snapshot name from the configured `snapshots:` map (optional) - Snapshot name from the configured `snapshots:` map (optional)
- RFC3339 UTC timestamp - RFC3339 UTC timestamp
This ID reveals the hostname, the configured snapshot name, and the backup time, so it is never used as the on-disk directory name — the remote key is used instead.
### Remote Key Derivation
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string with no structure a remote observer can reverse. Implemented in `internal/snapshot/remotekey.go`.
Worked example:
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
Because the hash is one-way, a listing of the destination store reveals neither the hostname nor the snapshot name of any backup. The same remote key is stored in the manifest's `snapshot_id` field.
### Files in Each Snapshot Directory ### Files in Each Snapshot Directory
#### `db.zst.age` - Encrypted Database #### `db.zst.age` - Encrypted Database
@@ -68,16 +85,17 @@ Each snapshot has its own subdirectory named with the snapshot ID.
- **Structure**: - **Structure**:
```json ```json
{ {
"snapshot_id": "laptop_home_2024-01-15T14:30:52Z", "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
"timestamp": "2024-01-15T14:30:52Z", "timestamp": "2025-06-01T12:00:00Z",
"blob_count": 42, "blob_count": 42,
"total_compressed_size": 1048576,
"blobs": [ "blobs": [
"cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", { "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 24576 },
"deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", { "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 32768 }
...
] ]
} }
``` ```
`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is written in the clear.
### Why Manifest is Unencrypted ### Why Manifest is Unencrypted
The manifest must be readable without the private key to enable: The manifest must be readable without the private key to enable:
@@ -86,7 +104,7 @@ The manifest must be readable without the private key to enable:
3. **Verification** - Checking blob existence without decryption 3. **Verification** - Checking blob existence without decryption
4. **Cross-snapshot deduplication analysis** - Finding shared blobs between snapshots 4. **Cross-snapshot deduplication analysis** - Finding shared blobs between snapshots
The manifest only contains blob hashes, not file names or any other sensitive information. The manifest contains the remote key, the backup timestamp, the blob count and total compressed size, and each blob's hash and compressed size. It contains no file names, paths, or other decrypted metadata.
## Security Considerations ## Security Considerations
@@ -96,19 +114,21 @@ The manifest only contains blob hashes, not file names or any other sensitive in
- **File-to-chunk mappings** (in db.zst.age) - **File-to-chunk mappings** (in db.zst.age)
### What's Not Encrypted ### What's Not Encrypted
- **Blob hashes** (in manifest.json.zst) - **The remote key** — directory names and the manifest `snapshot_id`, a one-way hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation))
- **Snapshot IDs** (directory names) - **The backup timestamp** (in manifest.json.zst)
- **Blob count per snapshot** (in manifest.json.zst) - **Blob hashes and their compressed sizes** (in manifest.json.zst)
- **Blob count and total compressed size per snapshot** (in manifest.json.zst)
### Privacy Implications ### Privacy Implications
From the unencrypted data, an observer can determine: From the unencrypted data, an observer of the destination store can determine:
- When backups were taken (from snapshot IDs) - **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear
- Which hostname created backups (from snapshot IDs) - How many blobs each snapshot references, and the total compressed size
- How many blobs each snapshot references - The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)
- Which blobs are shared between snapshots (deduplication patterns)
- The size of each encrypted blob Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless.
An observer cannot determine: An observer cannot determine:
- The hostname or snapshot name of any backup (the directory name and the manifest `snapshot_id` are one-way hashes of the human ID)
- File names or paths - File names or paths
- File contents - File contents
- File permissions or ownership - File permissions or ownership
+3 -2
View File
@@ -22,8 +22,9 @@ const remoteKeyPrefix = "vaultik|"
// //
// - the "metadata/<remote-key>/..." subdirectory on the storage // - the "metadata/<remote-key>/..." subdirectory on the storage
// backend so a directory listing of the bucket / file:// dest // backend so a directory listing of the bucket / file:// dest
// doesn't reveal hostnames, configured snapshot names, or backup // doesn't reveal hostnames or configured snapshot names. (The
// timestamps; // backup time is not hidden: the manifest.json.zst inside that
// directory carries a plaintext RFC3339 timestamp.)
// - the `snapshot_id` field of the unencrypted manifest.json.zst // - the `snapshot_id` field of the unencrypted manifest.json.zst
// for the same reason; // for the same reason;
// - any code path that needs to translate a known local snapshot ID // - any code path that needs to translate a known local snapshot ID
+4 -2
View File
@@ -840,8 +840,10 @@ func (sm *SnapshotManager) generateBlobManifest(
} }
// Create manifest. SnapshotID in the unencrypted manifest is the // Create manifest. SnapshotID in the unencrypted manifest is the
// double-SHA256 remote key, not the human ID, so the public bytes // double-SHA256 remote key (see RemoteSnapshotKey), not the human ID,
// don't reveal hostname/snapshot-name/timestamp metadata. // so neither this field nor the directory name reveals the hostname or
// snapshot name. Timestamp below is written in the clear, so the backup
// time is observable to anyone who can read the manifest.
manifest := &Manifest{ manifest := &Manifest{
SnapshotID: RemoteSnapshotKey(snapshotID), SnapshotID: RemoteSnapshotKey(snapshotID),
Timestamp: time.Now().UTC().Format(time.RFC3339), Timestamp: time.Now().UTC().Format(time.RFC3339),