Rewrite docs/REPOSTRUCTURE.md sentence by sentence
Rewritten from scratch against the code on next: the layout of the destination, what it protects, the accepted risks, the order of writes and the rules that keep prune from deleting data in use. Corrected: a restore needs only db.zst.age and the blobs it reads, not the manifest. The snapshot ID form without a name is gone, since no command creates one. Model: fable-5-1
This commit is contained in:
sneak authored and 
clawbot committed
1 parent
1d743e2255
commit
c8ad289e46
1 file changed
+311
-261
+311
-261
@@ -1,158 +1,167 @@
|
|||||||
# Vaultik S3 Repository Structure
|
# vaultik destination layout
|
||||||
|
|
||||||
This document describes the structure and organization of data stored in the S3
|
This document describes what `vaultik` stores at the destination. It covers the
|
||||||
bucket by Vaultik.
|
layout, what the layout protects, and the risks it accepts.
|
||||||
|
|
||||||
## Overview
|
The destination is the place the backups are stored. It is an S3 bucket and
|
||||||
|
prefix, a local directory or an rclone remote. The layout is the same on all
|
||||||
|
three.
|
||||||
|
|
||||||
Vaultik stores all backup data in an S3-compatible object store. The repository
|
## Terms
|
||||||
consists of two main components, plus a marker file:
|
|
||||||
|
|
||||||
1. **Blobs** - The actual backup data (content-addressed, encrypted)
|
- **Snapshot**: one backup of the paths listed under one name in the config.
|
||||||
2. **Metadata** - Snapshot information and manifests (partially encrypted)
|
- **Snapshot ID**: the name of a snapshot, in the form
|
||||||
|
`<hostname>_<snapshot-name>_<timestamp>`.
|
||||||
|
- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
|
||||||
|
content.
|
||||||
|
- **Blob**: one object under `blobs/`. It holds many chunks.
|
||||||
|
- **Local index**: the SQLite database on the backed-up machine.
|
||||||
|
- **Recipient**: an age public key that backups are encrypted to.
|
||||||
|
|
||||||
## Directory Structure
|
## Layout
|
||||||
|
|
||||||
```
|
```
|
||||||
<bucket>/<prefix>/
|
<destination>/
|
||||||
├── vaultik-destination.txt
|
├── vaultik-destination.txt
|
||||||
├── blobs/
|
├── blobs/
|
||||||
│ └── <hash[0:2]>/
|
│ └── <hash[0:2]>/
|
||||||
│ └── <hash[2:4]>/
|
│ └── <hash[2:4]>/
|
||||||
│ └── <full-hash>
|
│ └── <hash>
|
||||||
└── metadata/
|
└── metadata/
|
||||||
└── <remote-key>/
|
└── <remote-key>/
|
||||||
├── db.zst.age
|
├── db.zst.age
|
||||||
└── manifest.json.zst
|
└── manifest.json.zst
|
||||||
```
|
```
|
||||||
|
|
||||||
The metadata subdirectory is named with the **remote key**, a one-way hash of
|
The destination holds three things:
|
||||||
the snapshot ID, not with the human-readable snapshot ID itself. See
|
|
||||||
[Remote Key Derivation](#remote-key-derivation).
|
|
||||||
|
|
||||||
## Marker File (`vaultik-destination.txt`)
|
- `vaultik-destination.txt`, the marker file
|
||||||
|
- `blobs/`, which holds the file contents
|
||||||
|
- `metadata/`, which holds one directory for each snapshot
|
||||||
|
|
||||||
|
## The marker file
|
||||||
|
|
||||||
|
Every backup writes `vaultik-destination.txt` before it uploads anything else.
|
||||||
|
The file holds one fixed line of text. It names no host, path or time.
|
||||||
|
|
||||||
|
The marker tells an emptied or missing destination apart from a new one. An
|
||||||
|
unplugged volume leaves an empty directory at its mount point. Without the
|
||||||
|
marker, that directory would look like a destination whose snapshots are all
|
||||||
|
gone.
|
||||||
|
|
||||||
|
The check applies while the local index holds snapshots. When the destination
|
||||||
|
then answers that the marker does not exist, commands treat the destination as
|
||||||
|
not present:
|
||||||
|
|
||||||
|
- `snapshot create` refuses to write to it.
|
||||||
|
- `prune`, `snapshot purge` and `remote nuke` fail.
|
||||||
|
- `snapshot list` shows the local index alone.
|
||||||
|
- `snapshot remove` deletes nothing at the destination.
|
||||||
|
|
||||||
|
The check is skipped when the look-up fails for any other reason. S3 refuses
|
||||||
|
the look-up to credentials that can only write. A machine with such credentials
|
||||||
|
therefore does not notice an emptied destination. Its next backup would leave
|
||||||
|
out the data its local index records as stored. After a destination is emptied,
|
||||||
|
run `vaultik database delete` on such a machine before it backs up again.
|
||||||
|
|
||||||
Every backup writes this file at the root before it uploads anything. It holds
|
|
||||||
one fixed line of plain text and names no host, path or time. While the local
|
|
||||||
index holds snapshots stored there, a destination that answers a look-up of the
|
|
||||||
marker with not found is treated as not present: `snapshot create` refuses to
|
|
||||||
write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list`
|
|
||||||
and `remote nuke` behave as for a destination that cannot be listed. An
|
|
||||||
unplugged volume leaves an empty directory at its mount point, and without the
|
|
||||||
marker that directory would read as a destination whose snapshots are all gone.
|
|
||||||
`remote nuke` deletes the marker last.
|
`remote nuke` deletes the marker last.
|
||||||
|
|
||||||
When the look-up of the marker fails for any reason other than not found, the
|
## Blobs
|
||||||
check is skipped and the command goes on. S3 refuses it on every run to
|
|
||||||
credentials that allow only PUT, so a host with such credentials does not notice
|
|
||||||
a destination emptied since its last backup, and its next backup would leave out
|
|
||||||
the data its local index records as already stored. After the destination is
|
|
||||||
emptied, run `vaultik database delete` on such a host before it backs up again.
|
|
||||||
|
|
||||||
## Blobs Directory (`blobs/`)
|
A blob holds the content of many chunks, one after another.
|
||||||
|
|
||||||
### Structure
|
- **Path**: `blobs/<first 2 characters>/<next 2 characters>/<full hash>`.
|
||||||
|
|
||||||
- **Path format**: `blobs/<first-2-chars>/<next-2-chars>/<full-hash>`
|
|
||||||
- **Example**:
|
- **Example**:
|
||||||
`blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`
|
`blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`.
|
||||||
- **Sharding**: The two-level directory structure (using the first 4 characters
|
- **Format**: the chunk data, compressed with zstd and then encrypted with age.
|
||||||
of the hash) prevents any single directory from containing too many objects
|
- **Recipients**: the public keys in `age_recipients` when the blob was written.
|
||||||
|
|
||||||
### Content
|
The two directory levels keep any one directory from holding too many objects.
|
||||||
|
|
||||||
- **What it contains**: Packed collections of content-defined chunks from files
|
### Blob names
|
||||||
- **Format**: Zstandard compressed, then Age encrypted
|
|
||||||
- **Encryption**: Always encrypted with Age using the configured recipients
|
|
||||||
- **Naming**: Content-addressed. The blob's name is
|
|
||||||
`hex(SHA256(SHA256(uncompressed blob contents)))` — the double SHA-256 of the
|
|
||||||
concatenated chunk data, computed before compression and encryption, not a
|
|
||||||
hash of the stored (compressed, encrypted) bytes. One consequence: only a
|
|
||||||
holder of the age private key can check a stored blob's integrity, because
|
|
||||||
matching a blob to its name means decrypting and decompressing it first —
|
|
||||||
which is what `restore` and `verify --deep` do. Implemented in
|
|
||||||
`internal/blobgen` (`DoubleSHA256`). This is the canonical description of blob
|
|
||||||
naming; other documents and comments point here.
|
|
||||||
|
|
||||||
### Why Encrypted
|
A blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. The contents are
|
||||||
|
the chunk data, joined in order, before compression and encryption. The name is
|
||||||
|
not a hash of the stored bytes.
|
||||||
|
|
||||||
Blobs contain the actual file data from backups and must be encrypted for
|
This has two consequences:
|
||||||
security. The content-addressing ensures deduplication while the encryption
|
|
||||||
ensures privacy.
|
|
||||||
|
|
||||||
## Metadata Directory (`metadata/`)
|
- Checking a blob against its name needs the private key. The blob must be
|
||||||
|
decrypted and decompressed first. `snapshot restore` and
|
||||||
|
`snapshot verify --deep` do this.
|
||||||
|
- Without the key, only a blob's presence and size can be checked.
|
||||||
|
`snapshot verify` does this.
|
||||||
|
|
||||||
Each snapshot has its own subdirectory. The directory is **not** named with the
|
`blobgen.DoubleSHA256` in `internal/blobgen` computes the name.
|
||||||
human-readable snapshot ID; it is named with the remote key — a one-way hash of
|
|
||||||
that ID. The human ID is never written to the destination store as a directory
|
|
||||||
name (see [Remote Key Derivation](#remote-key-derivation)).
|
|
||||||
|
|
||||||
### Snapshot ID Format
|
### Uploads
|
||||||
|
|
||||||
The human-readable snapshot ID is used in CLI arguments, log lines, and the
|
A backup looks a blob up by name before it uploads it. It does not upload a
|
||||||
local database. It is not written to the destination store.
|
blob that is already there.
|
||||||
|
|
||||||
- **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>`
|
When the destination refuses the look-up, the backup uploads the blob. An
|
||||||
if no name was specified)
|
object with that name is then replaced by one that holds the same chunk data.
|
||||||
- **Example**: `laptop_home_2024-01-15T14:30:52Z`
|
|
||||||
- **Components**:
|
|
||||||
- Short hostname (everything before the first dot is stripped from the FQDN)
|
|
||||||
- Snapshot name from the configured `snapshots:` map (optional)
|
|
||||||
- RFC3339 UTC timestamp
|
|
||||||
|
|
||||||
This ID reveals the hostname, the configured snapshot name, and the backup time,
|
## Metadata
|
||||||
so it is never used as the on-disk directory name — the remote key is used
|
|
||||||
instead.
|
|
||||||
|
|
||||||
### Remote Key Derivation
|
Each snapshot has one directory under `metadata/`. The directory is named with
|
||||||
|
the snapshot's remote key, not with its snapshot ID.
|
||||||
|
|
||||||
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double
|
### Snapshot IDs
|
||||||
SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The
|
|
||||||
result is a 64-character hex string. The hash is not reversible, but it uses no
|
|
||||||
secret: an observer who guesses a candidate hostname and snapshot name can hash
|
|
||||||
it the same way and confirm whether that snapshot is present. The remote key
|
|
||||||
keeps names out of a plain listing; it does not hide them from a guess.
|
|
||||||
Implemented in `internal/snapshot/remotekey.go`.
|
|
||||||
|
|
||||||
Worked example:
|
A snapshot ID has the form `<hostname>_<snapshot-name>_<timestamp>`. An example
|
||||||
|
is `laptop_home_2024-01-15T14:30:52Z`. It has three parts:
|
||||||
|
|
||||||
|
- the hostname, up to its first `.`
|
||||||
|
- the snapshot name from the config
|
||||||
|
- the time the backup started, in UTC
|
||||||
|
|
||||||
|
The snapshot ID is used on the command line, in log records and in the local
|
||||||
|
index. It reveals the hostname, the snapshot name and the backup time. It is
|
||||||
|
therefore never written to the destination without encryption.
|
||||||
|
|
||||||
|
### Remote keys
|
||||||
|
|
||||||
|
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot ID)))`. It is 64 hex
|
||||||
|
characters long. `internal/snapshot/remotekey.go` computes it.
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
|
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
|
||||||
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
|
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
|
||||||
- Directory:
|
- Directory:
|
||||||
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
|
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
|
||||||
|
|
||||||
A plain listing of the destination store therefore shows only these hashes, not
|
A listing of the destination therefore shows no hostname and no snapshot name.
|
||||||
the hostname or snapshot name of any backup — but because the hash uses no
|
|
||||||
secret, a guessed hostname and snapshot name can be hashed and confirmed against
|
|
||||||
the listing. The same remote key is stored in the manifest's `snapshot_id`
|
|
||||||
field.
|
|
||||||
|
|
||||||
### Files in Each Snapshot Directory
|
The hash cannot be reversed, but it uses no secret. Someone who guesses a
|
||||||
|
snapshot ID can hash it the same way and see whether that snapshot exists. The
|
||||||
|
remote key keeps names out of a listing. It does not hide them from a guess.
|
||||||
|
|
||||||
#### `db.zst.age` - Encrypted Database
|
### `db.zst.age`
|
||||||
|
|
||||||
- **What it contains**: Pruned binary SQLite database for this snapshot
|
The snapshot's database. A restore needs it.
|
||||||
- **Format**: Binary SQLite → Zstandard compressed → Age encrypted
|
|
||||||
- **Encryption**: Encrypted with Age
|
|
||||||
- **Purpose**: Contains full file metadata, chunk mappings, and all
|
|
||||||
relationships
|
|
||||||
- **Why encrypted**: Contains sensitive metadata like file paths, permissions,
|
|
||||||
and ownership
|
|
||||||
|
|
||||||
#### `manifest.json.zst` - Unencrypted Blob Manifest
|
- **Contents**: a SQLite database that holds this snapshot only. It lists every
|
||||||
|
file with its metadata, its chunks, and the blob that holds each chunk.
|
||||||
|
- **Format**: the database file, compressed with zstd and then encrypted with
|
||||||
|
age.
|
||||||
|
|
||||||
- **What it contains**: JSON list of all blob hashes referenced by this snapshot
|
It is encrypted because it holds file paths, permissions and owners.
|
||||||
- **Format**: JSON → Zstandard compressed (NOT encrypted)
|
[DATAMODEL.md](DATAMODEL.md) describes its tables.
|
||||||
- **Encryption**: NOT encrypted
|
|
||||||
- **Purpose**: Enables pruning operations without requiring decryption keys
|
### `manifest.json.zst`
|
||||||
- **Structure**:
|
|
||||||
|
The list of the blobs the snapshot uses.
|
||||||
|
|
||||||
|
- **Format**: JSON, compressed with zstd. It is not encrypted.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
|
"snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
|
||||||
"timestamp": "2025-06-01T12:00:00Z",
|
"timestamp": "2025-06-01T12:04:31Z",
|
||||||
"blob_count": 42,
|
"blob_count": 2,
|
||||||
"total_compressed_size": 1048576,
|
"total_compressed_size": 57344,
|
||||||
"blobs": [
|
"blobs": [
|
||||||
{
|
{
|
||||||
"hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
|
"hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
|
||||||
@@ -166,189 +175,230 @@ field.
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is
|
- `snapshot_id`: the remote key, not the snapshot ID.
|
||||||
written in the clear.
|
- `timestamp`: when the manifest was written, at the end of the backup.
|
||||||
|
- `blob_count`: the number of blobs.
|
||||||
|
- `total_compressed_size`: the total size of those blobs, in bytes.
|
||||||
|
- `blobs`: the hash and the size of each blob.
|
||||||
|
|
||||||
### Why Manifest is Unencrypted
|
The manifest holds no file name and no path.
|
||||||
|
|
||||||
The manifest must be readable without the private key to enable:
|
The manifest is not encrypted so that these commands work without the private
|
||||||
|
key:
|
||||||
|
|
||||||
1. **Pruning operations** - Identifying unreferenced blobs for deletion
|
- `prune`, which must know which blobs are still in use
|
||||||
2. **Storage analysis** - Understanding space usage without decryption
|
- `snapshot verify`, which checks that each blob is present
|
||||||
3. **Verification** - Checking blob existence without decryption
|
- `snapshot list` and `remote info`, which report what the destination holds
|
||||||
4. **Cross-snapshot deduplication analysis** - Finding shared blobs between
|
|
||||||
snapshots
|
|
||||||
|
|
||||||
The manifest contains the remote key, the backup timestamp, the blob count and
|
## What is protected
|
||||||
total compressed size, and each blob's hash and compressed size. It contains no
|
|
||||||
file names, paths, or other decrypted metadata.
|
|
||||||
|
|
||||||
## Security Considerations
|
These are encrypted:
|
||||||
|
|
||||||
### What's Encrypted
|
- file contents, in the blobs
|
||||||
|
- file names, paths, permissions, owners and times, in `db.zst.age`
|
||||||
|
- which chunks make up which file, in `db.zst.age`
|
||||||
|
|
||||||
- **All file content** (in blobs)
|
These are not encrypted:
|
||||||
- **All file metadata** (paths, permissions, timestamps, ownership in
|
|
||||||
db.zst.age)
|
|
||||||
- **File-to-chunk mappings** (in db.zst.age)
|
|
||||||
|
|
||||||
### What's Not Encrypted
|
- the remote key of each snapshot
|
||||||
|
- the time each backup finished, in the manifest
|
||||||
|
- the hash and the compressed size of each blob, in the manifest
|
||||||
|
- the number of blobs and their total size for each snapshot, in the manifest
|
||||||
|
|
||||||
- **The remote key** — directory names and the manifest `snapshot_id`, a one-way
|
Someone who can read the destination therefore learns these:
|
||||||
hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation))
|
|
||||||
- **The backup timestamp** (in manifest.json.zst)
|
|
||||||
- **Blob hashes and their compressed sizes** (in manifest.json.zst)
|
|
||||||
- **Blob count and total compressed size per snapshot** (in manifest.json.zst)
|
|
||||||
|
|
||||||
### Privacy Implications
|
- when each backup finished
|
||||||
|
- how many blobs each snapshot uses, and how large they are
|
||||||
|
- which blobs two snapshots share
|
||||||
|
- whether a snapshot ID they guess exists
|
||||||
|
|
||||||
From the unencrypted data, an observer of the destination store can determine:
|
They do not learn these:
|
||||||
|
|
||||||
- **When each backup was taken** — not from the directory name, which is a
|
- any hostname or snapshot name, unless they guess it
|
||||||
one-way hash, but from the plaintext `timestamp` field in manifest.json.zst,
|
- file names or paths
|
||||||
which is published in the clear
|
- file contents
|
||||||
- How many blobs each snapshot references, and the total compressed size
|
- permissions or owners
|
||||||
- The compressed size of each blob, and which blobs are shared between snapshots
|
- the directory structure
|
||||||
(deduplication patterns)
|
- which chunks belong to which file
|
||||||
- **Whether a guessed hostname and snapshot name are present** — the remote key
|
|
||||||
is an unkeyed hash, so an observer holding candidate names can hash each one
|
|
||||||
and match it against the directory listing. The human ID is never published,
|
|
||||||
so it cannot be read off directly, but it can be confirmed by guessing.
|
|
||||||
|
|
||||||
Together these give an observer a timing-and-size profile of every snapshot.
|
The times and sizes could not be hidden by encrypting the manifest. The
|
||||||
This is an accepted, documented property of the format, not a defect: the
|
destination itself shows when each object was written and how large it is.
|
||||||
manifest is unencrypted so that pruning can run without the private key, and the
|
|
||||||
timing channel could not be closed by encrypting it anyway — object creation
|
|
||||||
times and per-object sizes stay visible at the storage layer on both `s3://` and
|
|
||||||
`file://` destinations regardless.
|
|
||||||
|
|
||||||
An observer cannot determine:
|
## Accepted risks
|
||||||
|
|
||||||
- The hostname or snapshot name of any backup by reading it off the store — the
|
These are known properties of the format. They are written down so that you can
|
||||||
directory name and the manifest `snapshot_id` are unkeyed hashes of the human
|
weigh them.
|
||||||
ID, so the text is never published (though a guessed name can be confirmed, as
|
|
||||||
above)
|
|
||||||
- File names or paths
|
|
||||||
- File contents
|
|
||||||
- File permissions or ownership
|
|
||||||
- Directory structure
|
|
||||||
- Which chunks belong to which files
|
|
||||||
|
|
||||||
### Accepted Risks
|
### No proof of who wrote the data
|
||||||
|
|
||||||
These are known, deliberate properties of the format and the tooling, recorded
|
A restore and `snapshot verify --deep` prove two things. The data decrypts with
|
||||||
so an operator can weigh them rather than discover them.
|
your private key, and it matches its hashes. They do not prove who wrote it.
|
||||||
|
|
||||||
1. **No proof of authorship.** Restore and `verify --deep` prove that data
|
The hashes use no secret. Someone who knows a recipient's public key and can
|
||||||
decrypts with the age private key and matches its unkeyed content hashes.
|
replace objects at the destination can put a snapshot of their own there. The
|
||||||
They do not prove who wrote it: anyone who knows a recipient public key and
|
public key is not stored at the destination, but a backed-up machine that is
|
||||||
can replace objects on the destination can substitute a snapshot they built.
|
broken into has it.
|
||||||
The recipient string is not stored at the destination, but a compromised
|
|
||||||
backed-up host has it. Defences live on the destination side — bucket
|
|
||||||
versioning or object lock, credentials for the source host that cannot delete
|
|
||||||
or overwrite existing versions, and pruning from a trusted host. Note that S3
|
|
||||||
`PutObject` overwrites an existing key, so PUT permission alone is not
|
|
||||||
append-only.
|
|
||||||
2. **Compression reveals sizes.** Blobs and `db.zst.age` are zstd-compressed
|
|
||||||
then age-encrypted; the manifest is compressed only. age does not pad, so an
|
|
||||||
object's size is the exact compressed length of its contents. All new chunks
|
|
||||||
packed into one blob share a single zstd stream (8 MiB window, 4 MiB at
|
|
||||||
compression levels 1-2), and a blob is closed at `blob_size_limit` and at the
|
|
||||||
end of each configured path. Because a stored chunk is never packed again,
|
|
||||||
someone who can write into a backed-up file and watch blob sizes learns
|
|
||||||
something only when their controlled data and a secret land in the same chunk
|
|
||||||
of a file that keeps changing. Advice: back up any outsider-writable
|
|
||||||
directory as its own snapshot.
|
|
||||||
3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The
|
|
||||||
default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of
|
|
||||||
2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a
|
|
||||||
blob holds hundreds of chunks, so individual chunk lengths are not visible in
|
|
||||||
the blob's size; lowering the limit toward the chunk size begins to expose
|
|
||||||
them.
|
|
||||||
4. **Decrypted data on local disk.** Several commands stage plaintext under
|
|
||||||
`$TMPDIR`: `snapshot restore` writes decrypted blobs under
|
|
||||||
`vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at
|
|
||||||
`vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at
|
|
||||||
`vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of
|
|
||||||
the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600`
|
|
||||||
and removed on success, but a `kill -9` or a power loss leaves them behind —
|
|
||||||
delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR`
|
|
||||||
should be trusted to the same degree as the restore target.
|
|
||||||
5. **Store permissions differ per command.** The backed-up host needs only PUT
|
|
||||||
to run `snapshot create`: it writes blobs and metadata and neither reads nor
|
|
||||||
deletes them. Other commands need more — `snapshot verify`,
|
|
||||||
`snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`,
|
|
||||||
`snapshot remove`, and `remote nuke` also delete. The recommended cron line
|
|
||||||
uses `--prune`, which runs `prune` on the backed-up host, so granting that
|
|
||||||
host `--prune` gives it credentials that can delete its own backups. To keep
|
|
||||||
the source host to PUT only, prune from a separate trusted host instead.
|
|
||||||
6. **Changing recipients does not re-encrypt existing data.** Deduplicated
|
|
||||||
chunks and same-named blobs already on the destination stay encrypted to the
|
|
||||||
recipients in force when they were written. A new snapshot that reuses them
|
|
||||||
cannot be restored with a newly added recipient's key alone, because those
|
|
||||||
reused objects were never encrypted to it. To make everything readable by a
|
|
||||||
new key, run `vaultik database delete` and take a full backup to a fresh
|
|
||||||
destination or prefix.
|
|
||||||
7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients.
|
|
||||||
Long-lived ciphertext held by a third party (the destination operator) has no
|
|
||||||
fallback if X25519 is ever broken: there is no second recipient type and no
|
|
||||||
post-quantum option.
|
|
||||||
|
|
||||||
## Consistency Guarantees
|
The defences are on the destination's side:
|
||||||
|
|
||||||
1. **Blobs are immutable** - Once written, a blob is never modified
|
- object versioning or object lock on the bucket
|
||||||
2. **Blobs are written before metadata** - A snapshot's metadata is only written
|
- credentials for the backed-up machine that cannot delete or overwrite
|
||||||
after all its blobs are successfully uploaded
|
- running `prune` from a trusted machine
|
||||||
3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst
|
|
||||||
are written as complete files
|
|
||||||
4. **A snapshot is marked complete in the local DB only after its metadata is
|
|
||||||
uploaded** - `finalizeSnapshotMetadata` runs `ExportSnapshotMetadata` first
|
|
||||||
and records completion (`MarkSnapshotComplete`) only once the export succeeds
|
|
||||||
(see the backup data flow in [ARCHITECTURE.md](ARCHITECTURE.md)). A crash
|
|
||||||
during the export therefore leaves the snapshot incomplete, so the next
|
|
||||||
backup's `PruneDatabase` drops it and re-backs-up its data, rather than
|
|
||||||
leaving a completed-looking row in the local index with no matching metadata
|
|
||||||
on the destination store. (A crash in the brief moment after the export
|
|
||||||
succeeds but before completion is recorded leaves a fully-restorable snapshot
|
|
||||||
on the destination that the local index drops as incomplete on the next run;
|
|
||||||
`snapshot list` then reports it honestly as remote-only, which is the safe
|
|
||||||
direction: the destination copy stays restorable.)
|
|
||||||
|
|
||||||
## Pruning Safety
|
An S3 `PutObject` replaces an existing object. Permission to write alone
|
||||||
|
therefore does not stop overwrites.
|
||||||
|
|
||||||
The prune operation is safe because:
|
### Sizes are visible
|
||||||
|
|
||||||
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs
|
age adds no padding. The size of an object is the exact size of its compressed
|
||||||
that no manifest references. Under `metadata/` it deletes only a directory
|
contents.
|
||||||
that has no `manifest.json.zst`, whose name is a remote key, and that no
|
|
||||||
local snapshot record claims
|
|
||||||
2. Manifests are unencrypted and can be read without keys
|
|
||||||
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and
|
|
||||||
exits with an error, rather than treating that snapshot's blobs as
|
|
||||||
unreferenced
|
|
||||||
4. It deletes an unreferenced blob only once the blob is older than
|
|
||||||
`prune_grace_period` (default `24h`), and a directory under `metadata/` with
|
|
||||||
no manifest only once every object in it is. A backup uploads its blobs and
|
|
||||||
its `db.zst.age` before its manifest, so a prune running meanwhile, on this
|
|
||||||
host or another sharing the destination, leaves them as long as no backup run
|
|
||||||
takes longer than the grace period. The grace period does not protect a blob
|
|
||||||
a backup reuses from an earlier run that stopped before its manifest: the
|
|
||||||
backup skips the upload of a blob already on the destination, so that blob
|
|
||||||
keeps the time of its first upload, and a prune before the backup's manifest
|
|
||||||
is written can delete it; the new snapshot then cannot be restored.
|
|
||||||
`remote nuke` has no such check
|
|
||||||
|
|
||||||
## Restoration Requirements
|
All the new chunks in one blob share one zstd stream. The stream looks back 8
|
||||||
|
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at
|
||||||
|
`blob_size_limit` and at the end of each configured path. A chunk that is
|
||||||
|
already stored is never packed again.
|
||||||
|
|
||||||
To restore from a backup, you need:
|
Someone who can write into a backed-up file and watch blob sizes can learn
|
||||||
|
something in one case. Their data and a secret must land in the same chunk, in
|
||||||
|
a file that keeps changing.
|
||||||
|
|
||||||
1. **The Age private key** - To decrypt blobs and database
|
Back up a directory that outsiders can write to as a snapshot of its own.
|
||||||
2. **The snapshot metadata** - Both files from the snapshot's metadata directory
|
|
||||||
3. **All referenced blobs** - As listed in the manifest
|
|
||||||
|
|
||||||
The restoration process:
|
### Chunk boundaries use no secret
|
||||||
|
|
||||||
1. Download and decrypt the database dump to understand file structure
|
The FastCDC parameters are fixed and public. With the default `chunk_size` of
|
||||||
2. Download and decrypt the required blobs
|
10 MB, a chunk is between 2.5 MB and 40 MB. A file of 2.5 MB or less is one
|
||||||
3. Reconstruct files from their chunks
|
chunk.
|
||||||
4. Restore file metadata (permissions, timestamps, etc.)
|
|
||||||
|
At the default `blob_size_limit` of 10 GB, a blob holds hundreds of chunks. The
|
||||||
|
size of the blob then does not show the size of any one chunk. A limit close to
|
||||||
|
`chunk_size` begins to show chunk sizes.
|
||||||
|
|
||||||
|
### Decrypted data on the local disk
|
||||||
|
|
||||||
|
These commands write unencrypted data to the temporary directory:
|
||||||
|
|
||||||
|
- `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/`. There
|
||||||
|
is no limit on their total size.
|
||||||
|
- `snapshot restore` writes the decrypted database to
|
||||||
|
`vaultik-restore-*/snapshot.db`.
|
||||||
|
- `snapshot verify --deep` writes the decrypted database to
|
||||||
|
`vaultik-verify-*/snapshot.db`.
|
||||||
|
- `snapshot create` writes a copy of the local index to
|
||||||
|
`vaultik-snapshot-*/snapshot.db`.
|
||||||
|
|
||||||
|
These files are readable only by their owner, and are removed when the command
|
||||||
|
ends. A `kill -9` or a power loss leaves them behind. Delete any leftover
|
||||||
|
`vaultik-*` entry in the temporary directory yourself.
|
||||||
|
|
||||||
|
Trust the temporary directory as much as the directory you restore into.
|
||||||
|
|
||||||
|
### Commands need different permissions
|
||||||
|
|
||||||
|
`snapshot create` needs only permission to write objects. Other commands need
|
||||||
|
more:
|
||||||
|
|
||||||
|
- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list
|
||||||
|
and read.
|
||||||
|
- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete.
|
||||||
|
- `prune` lists, reads and deletes.
|
||||||
|
|
||||||
|
`snapshot create --prune` runs a prune on the backed-up machine. That machine
|
||||||
|
then holds credentials that can delete its own backups. To keep it to
|
||||||
|
write-only credentials, run `prune` from a separate, trusted machine.
|
||||||
|
|
||||||
|
### New recipients cannot read old data
|
||||||
|
|
||||||
|
Changing `age_recipients` does not re-encrypt what is already stored. A chunk
|
||||||
|
that is already stored is not written again, so it stays encrypted to the
|
||||||
|
earlier recipients.
|
||||||
|
|
||||||
|
A new snapshot that uses such chunks cannot be restored with a new recipient's
|
||||||
|
key alone. To make everything readable by a new key, run
|
||||||
|
`vaultik database delete` and take a full backup to a new destination.
|
||||||
|
|
||||||
|
### X25519 recipients only
|
||||||
|
|
||||||
|
`vaultik` accepts age X25519 public keys only. It rejects age SSH and plugin
|
||||||
|
recipients.
|
||||||
|
|
||||||
|
The encrypted data is held by a third party for a long time. There is no second
|
||||||
|
kind of recipient to fall back on if X25519 is broken, and no post-quantum
|
||||||
|
option.
|
||||||
|
|
||||||
|
## Order of writes
|
||||||
|
|
||||||
|
A backup writes in a fixed order:
|
||||||
|
|
||||||
|
- the marker file
|
||||||
|
- the blobs
|
||||||
|
- `db.zst.age`
|
||||||
|
- `manifest.json.zst`
|
||||||
|
|
||||||
|
A snapshot's metadata is written only after all its blobs are uploaded. The
|
||||||
|
manifest is written last. A directory under `metadata/` that has no manifest
|
||||||
|
therefore belongs to a backup that did not finish.
|
||||||
|
|
||||||
|
The local index marks a snapshot complete only after both metadata files are
|
||||||
|
uploaded. A backup that stops during that upload leaves the snapshot
|
||||||
|
unfinished. The next backup removes it from the local index and stores its data
|
||||||
|
again.
|
||||||
|
|
||||||
|
One short gap remains. A backup can stop after the upload and before the
|
||||||
|
snapshot is marked complete. The destination then holds a snapshot that can be
|
||||||
|
restored, and the local index does not list it. `snapshot list` shows it as
|
||||||
|
remote only.
|
||||||
|
|
||||||
|
An upload that is cut off leaves no partial object under the final name on S3
|
||||||
|
or on a local directory. The same holds for an rclone remote that can move
|
||||||
|
objects. An rclone remote that cannot move objects can keep a partial object.
|
||||||
|
|
||||||
|
## Pruning
|
||||||
|
|
||||||
|
`prune` deletes two kinds of object at the destination:
|
||||||
|
|
||||||
|
- a blob that no manifest lists
|
||||||
|
- a directory under `metadata/` that has no manifest
|
||||||
|
|
||||||
|
These rules keep it from deleting data that is in use:
|
||||||
|
|
||||||
|
- It keeps every blob that any manifest lists.
|
||||||
|
- It reads the manifests, which are not encrypted, so it needs no private key.
|
||||||
|
- It deletes nothing when any manifest cannot be downloaded or decoded.
|
||||||
|
- It deletes a blob only once the blob is older than `prune_grace_period`.
|
||||||
|
- It deletes a directory only once every object in it is that old.
|
||||||
|
- It keeps a directory whose snapshot is in the local index.
|
||||||
|
- It leaves alone a directory whose name is not a remote key.
|
||||||
|
|
||||||
|
`prune_grace_period` defaults to `24h`. It protects a backup that is still
|
||||||
|
running. Such a backup has uploaded blobs and perhaps `db.zst.age`, but not yet
|
||||||
|
its manifest. A prune from this machine or another keeps those objects, as long
|
||||||
|
as no backup takes longer than the grace period.
|
||||||
|
|
||||||
|
The grace period does not protect one case. A backup does not upload a blob
|
||||||
|
that is already at the destination, so that blob keeps the time of its first
|
||||||
|
upload. The blob can be left over from an earlier backup that stopped before
|
||||||
|
its manifest. A prune that runs before the new manifest is written can then
|
||||||
|
delete that blob. The new snapshot cannot be restored afterwards.
|
||||||
|
|
||||||
|
To avoid this case, give each machine and each user its own destination.
|
||||||
|
|
||||||
|
`remote nuke` applies none of these rules.
|
||||||
|
|
||||||
|
## What a restore needs
|
||||||
|
|
||||||
|
- the age private key
|
||||||
|
- the snapshot's `db.zst.age`
|
||||||
|
- the blobs that hold the files to restore
|
||||||
|
|
||||||
|
A restore does not read the manifest and does not need the local index.
|
||||||
|
|
||||||
|
It works in these steps:
|
||||||
|
|
||||||
|
- It downloads and decrypts `db.zst.age`.
|
||||||
|
- It checks that the database is the one for the snapshot that was asked for.
|
||||||
|
- It downloads and decrypts the blobs it needs. It checks each blob against its
|
||||||
|
name.
|
||||||
|
- It writes each file from its chunks.
|
||||||
|
- It sets each file's permissions and times, and its owner when it runs as
|
||||||
|
root.
|
||||||
Reference in new issue
Block a user