vaultik/docs/REPOSTRUCTURE.md

# vaultik destination layout

This document describes what `vaultik` stores at the destination. It covers the
layout, what the layout protects, and the risks it accepts.

The destination is the place the backups are stored. It is an S3 bucket and
prefix, a local directory or an rclone remote. The layout is the same on all
three.

## Terms

- **Snapshot**: one backup of the paths listed under one name in the config.
- **Snapshot ID**: the name of a snapshot, in the form
  `<hostname>_<snapshot-name>_<timestamp>`.
- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
  content.
- **Blob**: one object under `blobs/`. It holds many chunks.
- **Local index**: the SQLite database on the backed-up machine.
- **Recipient**: an age public key that backups are encrypted to.

## Layout

```
<destination>/
├── vaultik-destination.txt
├── blobs/
│   └── <hash[0:2]>/
│       └── <hash[2:4]>/
│           └── <hash>
└── metadata/
    └── <remote-key>/
        ├── db.zst.age
        └── manifest.json.zst
```

The destination holds three things:

- `vaultik-destination.txt`, the marker file
- `blobs/`, which holds the file contents
- `metadata/`, which holds one directory for each snapshot

## The marker file

Every backup writes `vaultik-destination.txt` before it uploads anything else.
The file holds one fixed line of text. It names no host, path or time.

The marker tells an emptied or missing destination apart from a new one. An
unplugged volume leaves an empty directory at its mount point. Without the
marker, that directory would look like a destination whose snapshots are all
gone.

The check applies while the local index holds snapshots. When the destination
then answers that the marker does not exist, commands treat the destination as
not present:

- `snapshot create` refuses to write to it.
- `prune`, `snapshot purge` and `remote nuke` fail.
- `snapshot list` shows the local index alone.
- `snapshot remove` deletes nothing at the destination.

The check is skipped when the look-up fails for any other reason. S3 refuses
the look-up to credentials that can only write. A machine with such credentials
therefore does not notice an emptied destination. Its next backup would leave
out the data its local index records as stored. After a destination is emptied,
run `vaultik database delete` on such a machine before it backs up again.

`remote nuke` deletes the marker last.

## Blobs

A blob holds the content of many chunks, one after another.

- **Path**: `blobs/<first 2 characters>/<next 2 characters>/<full hash>`.
- **Example**:
  `blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`.
- **Format**: the chunk data, compressed with zstd and then encrypted with age.
- **Recipients**: the public keys in `age_recipients` when the blob was written.

The two directory levels keep any one directory from holding too many objects.

### Blob names

A blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. The contents are
the chunk data, joined in order, before compression and encryption. The name is
not a hash of the stored bytes.

This has two consequences:

- Checking a blob against its name needs the private key. The blob must be
  decrypted and decompressed first. `snapshot restore` and
  `snapshot verify --deep` do this.
- Without the key, only a blob's presence and size can be checked.
  `snapshot verify` does this.

`blobgen.DoubleSHA256` in `internal/blobgen` computes the name.

### Uploads

A backup looks a blob up by name before it uploads it. It does not upload a
blob that is already there.

When the destination refuses the look-up, the backup uploads the blob. An
object with that name is then replaced by one that holds the same chunk data.

## Metadata

Each snapshot has one directory under `metadata/`. The directory is named with
the snapshot's remote key, not with its snapshot ID.

### Snapshot IDs

A snapshot ID has the form `<hostname>_<snapshot-name>_<timestamp>`. An example
is `laptop_home_2024-01-15T14:30:52Z`. It has three parts:

- the hostname, up to its first `.`
- the snapshot name from the config
- the time the backup started, in UTC

The snapshot ID is used on the command line, in log records and in the local
index. It reveals the hostname, the snapshot name and the backup time. It is
therefore never written to the destination without encryption.

### Remote keys

The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot ID)))`. It is 64 hex
characters long. `internal/snapshot/remotekey.go` computes it.

For example:

- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
- Directory:
  `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`

A listing of the destination therefore shows no hostname and no snapshot name.

The hash cannot be reversed, but it uses no secret. Someone who guesses a
snapshot ID can hash it the same way and see whether that snapshot exists. The
remote key keeps names out of a listing. It does not hide them from a guess.

### `db.zst.age`

The snapshot's database. A restore needs it.

- **Contents**: a SQLite database that holds this snapshot only. It lists every
  file with its metadata, its chunks, and the blob that holds each chunk.
- **Format**: the database file, compressed with zstd and then encrypted with
  age.

It is encrypted because it holds file paths, permissions and owners.
[DATAMODEL.md](DATAMODEL.md) describes its tables.

### `manifest.json.zst`

The list of the blobs the snapshot uses.

- **Format**: JSON, compressed with zstd. It is not encrypted.

```json
{
    "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
    "timestamp": "2025-06-01T12:04:31Z",
    "blob_count": 2,
    "total_compressed_size": 57344,
    "blobs": [
        {
            "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
            "compressed_size": 24576
        },
        {
            "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678",
            "compressed_size": 32768
        }
    ]
}
```

- `snapshot_id`: the remote key, not the snapshot ID.
- `timestamp`: when the manifest was written, at the end of the backup.
- `blob_count`: the number of blobs.
- `total_compressed_size`: the total size of those blobs, in bytes.
- `blobs`: the hash and the size of each blob.

The manifest holds no file name and no path.

The manifest is not encrypted so that these commands work without the private
key:

- `prune`, which must know which blobs are still in use
- `snapshot verify`, which checks that each blob is present
- `snapshot list` and `remote info`, which report what the destination holds

## What is protected

These are encrypted:

- file contents, in the blobs
- file names, paths, permissions, owners and times, in `db.zst.age`
- which chunks make up which file, in `db.zst.age`

These are not encrypted:

- the remote key of each snapshot
- the time each backup finished, in the manifest
- the hash and the compressed size of each blob, in the manifest
- the number of blobs and their total size for each snapshot, in the manifest

Someone who can read the destination therefore learns these:

- when each backup finished
- how many blobs each snapshot uses, and how large they are
- which blobs two snapshots share
- whether a snapshot ID they guess exists

They do not learn these:

- any hostname or snapshot name, unless they guess it
- file names or paths
- file contents
- permissions or owners
- the directory structure
- which chunks belong to which file

The times and sizes could not be hidden by encrypting the manifest. The
destination itself shows when each object was written and how large it is.

## Accepted risks

These are known properties of the format. They are written down so that you can
weigh them.

### No proof of who wrote the data

A restore and `snapshot verify --deep` prove two things. The data decrypts with
your private key, and it matches its hashes. They do not prove who wrote it.

The hashes use no secret. Someone who knows a recipient's public key and can
replace objects at the destination can put a snapshot of their own there. The
public key is not stored at the destination, but a backed-up machine that is
broken into has it.

The defences are on the destination's side:

- object versioning or object lock on the bucket
- credentials for the backed-up machine that cannot delete or overwrite
- running `prune` from a trusted machine

An S3 `PutObject` replaces an existing object. Permission to write alone
therefore does not stop overwrites.

### Sizes are visible

age adds no padding. The size of an object is the exact size of its compressed
contents.

All the new chunks in one blob share one zstd stream. The stream looks back 8
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at
`blob_size_limit` and at the end of each configured path. A chunk that is
already stored is never packed again.

Someone who can write into a backed-up file and watch blob sizes can learn
something in one case. Their data and a secret must land in the same chunk, in
a file that keeps changing.

Back up a directory that outsiders can write to as a snapshot of its own.

### Chunk boundaries use no secret

The FastCDC parameters are fixed and public. With the default `chunk_size` of
10 MB, a chunk is between 2.5 MB and 40 MB. A file of 2.5 MB or less is one
chunk.

At the default `blob_size_limit` of 10 GB, a blob holds hundreds of chunks. The
size of the blob then does not show the size of any one chunk. A limit close to
`chunk_size` begins to show chunk sizes.

### Decrypted data on the local disk

These commands write unencrypted data to the temporary directory:

- `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/`. There
  is no limit on their total size.
- `snapshot restore` writes the decrypted database to
  `vaultik-restore-*/snapshot.db`.
- `snapshot verify --deep` writes the decrypted database to
  `vaultik-verify-*/snapshot.db`.
- `snapshot create` writes a copy of the local index to
  `vaultik-snapshot-*/snapshot.db`.

These files are readable only by their owner, and are removed when the command
ends. A `kill -9` or a power loss leaves them behind. Delete any leftover
`vaultik-*` entry in the temporary directory yourself.

Trust the temporary directory as much as the directory you restore into.

### Commands need different permissions

`snapshot create` needs only permission to write objects. Other commands need
more:

- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list
  and read.
- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete.
- `prune` lists, reads and deletes.

`snapshot create --prune` runs a prune on the backed-up machine. That machine
then holds credentials that can delete its own backups. To keep it to
write-only credentials, run `prune` from a separate, trusted machine.

### New recipients cannot read old data

Changing `age_recipients` does not re-encrypt what is already stored. A chunk
that is already stored is not written again, so it stays encrypted to the
earlier recipients.

A new snapshot that uses such chunks cannot be restored with a new recipient's
key alone. To make everything readable by a new key, run
`vaultik database delete` and take a full backup to a new destination.

### X25519 recipients only

`vaultik` accepts age X25519 public keys only. It rejects age SSH and plugin
recipients.

The encrypted data is held by a third party for a long time. There is no second
kind of recipient to fall back on if X25519 is broken, and no post-quantum
option.

## Order of writes

A backup writes in a fixed order:

- the marker file
- the blobs
- `db.zst.age`
- `manifest.json.zst`

A snapshot's metadata is written only after all its blobs are uploaded. The
manifest is written last. A directory under `metadata/` that has no manifest
therefore belongs to a backup that did not finish.

The local index marks a snapshot complete only after both metadata files are
uploaded. A backup that stops during that upload leaves the snapshot
unfinished. The next backup removes it from the local index and stores its data
again.

One short gap remains. A backup can stop after the upload and before the
snapshot is marked complete. The destination then holds a snapshot that can be
restored, and the local index does not list it. `snapshot list` shows it as
remote only.

An upload that is cut off leaves no partial object under the final name on S3
or on a local directory. The same holds for an rclone remote that can move
objects. An rclone remote that cannot move objects can keep a partial object.

## Pruning

`prune` deletes two kinds of object at the destination:

- a blob that no manifest lists
- a directory under `metadata/` that has no manifest

These rules keep it from deleting data that is in use:

- It keeps every blob that any manifest lists.
- It reads the manifests, which are not encrypted, so it needs no private key.
- It deletes nothing when any manifest cannot be downloaded or decoded.
- It deletes a blob only once the blob is older than `prune_grace_period`.
- It deletes a directory only once every object in it is that old.
- It keeps a directory whose snapshot is in the local index.
- It leaves alone a directory whose name is not a remote key.

`prune_grace_period` defaults to `24h`. It protects a backup that is still
running. Such a backup has uploaded blobs and perhaps `db.zst.age`, but not yet
its manifest. A prune from this machine or another keeps those objects, as long
as no backup takes longer than the grace period.

The grace period does not protect one case. A backup does not upload a blob
that is already at the destination, so that blob keeps the time of its first
upload. The blob can be left over from an earlier backup that stopped before
its manifest. A prune that runs before the new manifest is written can then
delete that blob. The new snapshot cannot be restored afterwards.

To avoid this case, give each machine and each user its own destination.

`remote nuke` applies none of these rules.

## What a restore needs

- the age private key
- the snapshot's `db.zst.age`
- the blobs that hold the files to restore

A restore does not read the manifest and does not need the local index.

It works in these steps:

- It downloads and decrypts `db.zst.age`.
- It checks that the database is the one for the snapshot that was asked for.
- It downloads and decrypts the blobs it needs. It checks each blob against its
  name.
- It writes each file from its chunks.
- It sets each file's permissions and times, and its owner when it runs as
  root.