vaultik/docs/REPOSTRUCTURE.md

vaultik destination layout

This document describes what vaultik stores at the destination. It covers the layout, what the layout protects, and the risks it accepts.

The destination is the place the backups are stored. It is an S3 bucket and prefix, a local directory or an rclone remote. The layout is the same on all three.

Terms

• Snapshot: one backup of the paths listed under one name in the config.
• Snapshot ID: the name of a snapshot, in the form <hostname>_<snapshot-name>_<timestamp>.
• Chunk: a piece of a file's content, named by the SHA-256 hash of that content.
• Blob: one object under blobs/. It holds many chunks.
• Local index: the SQLite database on the backed-up machine.
• Recipient: an age public key that backups are encrypted to.

Layout

<destination>/
├── vaultik-destination.txt
├── blobs/
│   └── <hash[0:2]>/
│       └── <hash[2:4]>/
│           └── <hash>
└── metadata/
    └── <remote-key>/
        ├── db.zst.age
        └── manifest.json.zst

The destination holds three things:

• vaultik-destination.txt, the marker file
• blobs/, which holds the file contents
• metadata/, which holds one directory for each snapshot

The marker file

Every backup writes vaultik-destination.txt before it uploads anything else. The file holds one fixed line of text. It names no host, path or time.

The marker tells an emptied or missing destination apart from a new one. An unplugged volume leaves an empty directory at its mount point. Without the marker, that directory would look like a destination whose snapshots are all gone.

The check applies while the local index holds snapshots. When the destination then answers that the marker does not exist, commands treat the destination as not present:

• snapshot create refuses to write to it.
• prune, snapshot purge and remote nuke fail.
• snapshot list shows the local index alone.
• snapshot remove deletes nothing at the destination.

The check is skipped when the look-up fails for any other reason. S3 refuses the look-up to credentials that can only write. A machine with such credentials therefore does not notice an emptied destination. Its next backup would not upload data that its local index says is stored. After a destination is emptied, run vaultik database delete on such a machine before it backs up again.

remote nuke deletes the marker last.

Blobs

A blob holds the content of many chunks, one after another.

• Path: blobs/<first 2 characters>/<next 2 characters>/<full hash>.
• Example: blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678.
• Format: the chunk data, compressed with zstd and then encrypted with age.
• Recipients: the public keys in age_recipients when the blob was written.

The two directory levels keep any one directory from holding too many objects.

Blob names

A blob's name is hex(SHA256(SHA256(uncompressed contents))). The contents are the chunk data, joined in order, before compression and encryption. The name is not a hash of the stored bytes.

This has two consequences:

• Checking a blob against its name needs the private key. The blob must be decrypted and decompressed first. snapshot restore and snapshot verify --deep do this.
• Without the key, only a blob's presence and size can be checked. snapshot verify does this.

blobgen.DoubleSHA256 in internal/blobgen computes the name.

Uploads

A backup looks a blob up by name before it uploads it. It does not upload a blob that is already there.

When the destination refuses the look-up, the backup uploads the blob. An object with that name is then replaced by one that holds the same chunk data.

Metadata

Each snapshot has one directory under metadata/. The directory is named with the snapshot's remote key, not with its snapshot ID.

Snapshot IDs

A snapshot ID has the form <hostname>_<snapshot-name>_<timestamp>. An example is laptop_home_2024-01-15T14:30:52Z. It has three parts:

• the hostname, up to its first .
• the snapshot name from the config
• the time the backup started, in UTC

The snapshot ID is used on the command line, in log records and in the local index. It reveals the hostname, the snapshot name and the backup time. It is therefore never written to the destination without encryption.

Remote keys

The remote key is hex(SHA256(SHA256("vaultik|" + snapshot ID))). It is 64 hex characters long. internal/snapshot/remotekey.go computes it.

For example:

• Snapshot ID: server1_home_2025-06-01T12:00:00Z
• Remote key: 17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa
• Directory: metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/

A listing of the destination therefore shows no hostname and no snapshot name.

The hash cannot be reversed, but it uses no secret. Someone who guesses a snapshot ID can hash it the same way and see whether that snapshot exists. The remote key keeps names out of a listing. It does not hide them from a guess.

db.zst.age

The snapshot's database. A restore needs it.

• Contents: a SQLite database. It is a copy of the local index without the rows of other snapshots. It lists every file in the snapshot with its metadata, its chunks, and the blob that holds each chunk.
• Format: the database file, compressed with zstd and then encrypted with age.

It is encrypted because it holds file paths, permissions and owners. DATAMODEL.md describes its tables.

manifest.json.zst

The list of the blobs the snapshot uses.

• Format: JSON, compressed with zstd. It is not encrypted.

{
    "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
    "timestamp": "2025-06-01T12:04:31Z",
    "blob_count": 2,
    "total_compressed_size": 57344,
    "blobs": [
        {
            "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
            "compressed_size": 24576
        },
        {
            "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678",
            "compressed_size": 32768
        }
    ]
}

• snapshot_id: the remote key, not the snapshot ID.
• timestamp: when the manifest was written, at the end of the backup.
• blob_count: the number of blobs.
• total_compressed_size: the total size of those blobs, in bytes.
• blobs: the hash and the size of each blob.

The manifest holds no file name and no path.

The manifest is not encrypted so that these commands work without the private key:

• prune, which must know which blobs are still in use
• snapshot verify, which checks that each blob is present
• snapshot list and remote info, which report what the destination holds

What is protected

These are encrypted:

• file contents, in the blobs
• in db.zst.age:
  • file names and paths
  • permissions and owners
  • modification times
  • which chunks make up which file

These are not encrypted:

• the remote key of each snapshot
• the time each backup finished, in the manifest
• the hash and the compressed size of each blob, in the manifest
• the number of blobs and their total size for each snapshot, in the manifest

Someone who can read the destination therefore learns these:

• when each backup finished
• how many blobs each snapshot uses, and how large they are
• which blobs two snapshots share
• whether a snapshot ID they guess exists

They do not learn these:

• any hostname or snapshot name, unless they guess it
• file names or paths
• file contents
• permissions or owners
• the directory structure
• which chunks belong to which file

The times and sizes could not be hidden by encrypting the manifest. The destination itself shows when each object was written and how large it is.

Accepted risks

These are known properties of the format. They are written down so that you can weigh them.

No proof of who wrote the data

A restore and snapshot verify --deep prove two things. The data decrypts with your private key, and it matches its hashes. They do not prove who wrote it.

The hashes use no secret. Someone who knows a recipient's public key and can replace objects at the destination can put a snapshot of their own there. The public key is not stored at the destination, but a backed-up machine that is broken into has it.

The defences are on the destination's side:

• object versioning or object lock on the bucket
• credentials for the backed-up machine that cannot delete or overwrite
• running prune from a trusted machine

An S3 PutObject replaces an existing object. Permission to write alone therefore does not stop overwrites.

Sizes are visible

An object's size follows exactly from the size of its compressed contents.

All the new chunks in one blob share one zstd stream. The stream looks back 8 MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at blob_size_limit and at the end of each configured path. A chunk that is already stored is never packed again.

Data that an outsider writes compresses smaller when it matches a secret in the same chunk, so the blob gets smaller. The blob's size therefore shows the outsider whether their data matched the secret.

This works only when all of these hold:

• The outsider can write into a backed-up file.
• The outsider can see the blob sizes at the destination.
• The outsider's data and the secret land in the same chunk.
• The file keeps changing.

Back up a directory that outsiders can write to as a snapshot of its own.

Chunk boundaries use no secret

The FastCDC parameters are fixed and public. chunk_size sets the average chunk size. No chunk is larger than four times this size. Only a file's last chunk can be smaller than a quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A file of 2.5 MB or less is one chunk.

At the default blob_size_limit of 10 GB, a blob holds hundreds of chunks. The size of the blob then does not show the size of any one chunk. A limit close to chunk_size begins to show chunk sizes.

Decrypted data on the local disk

These commands write unencrypted data to the temporary directory:

• snapshot restore writes decrypted blobs under vaultik-blobcache-*/. There is no limit on their total size.
• snapshot restore writes the decrypted database to vaultik-restore-*/snapshot.db.
• snapshot verify --deep writes the decrypted database to vaultik-verify-*/snapshot.db.
• snapshot create writes a copy of the local index to vaultik-snapshot-*/snapshot.db.

These files are readable only by their owner, and are removed when the command ends. A kill -9 or a power loss leaves them behind. Delete any leftover vaultik-* entry in the temporary directory yourself.

Trust the temporary directory as much as the directory you restore into.

Commands need different permissions

snapshot create needs only permission to write objects. Other commands need more:

• snapshot list, snapshot verify, snapshot restore and remote info list and read.
• snapshot purge, snapshot remove and remote nuke list and delete.
• prune lists, reads and deletes.

snapshot create --prune runs a prune on the backed-up machine. That machine then holds credentials that can delete its own backups. To keep it to write-only credentials, run prune from a separate, trusted machine.

New recipients cannot read old data

Changing age_recipients does not re-encrypt what is already stored. A chunk that is already stored is not written again, so it stays encrypted to the earlier recipients.

A new snapshot that uses such chunks cannot be restored with a new recipient's key alone. To make everything readable by a new key, run vaultik database delete and take a full backup to a new destination.

X25519 recipients only

vaultik accepts age X25519 public keys only. It rejects age SSH and plugin recipients.

The encrypted data is held by a third party for a long time. There is no second kind of recipient to fall back on if X25519 is broken, and no post-quantum option.

Order of writes

A backup writes in a fixed order:

• the marker file
• the blobs
• db.zst.age
• manifest.json.zst

A snapshot's metadata is written only after all its blobs are uploaded. The manifest is written last. A directory under metadata/ that has no manifest therefore belongs to a backup that did not finish.

The local index marks a snapshot complete only after both metadata files are uploaded. A backup that stops during that upload leaves the snapshot unfinished. The next backup removes it from the local index and stores its data again.

One short gap remains. A backup can stop after the upload and before the snapshot is marked complete. The destination then holds a snapshot that can be restored, and the local index does not list it. snapshot list shows it as remote only.

An upload that is cut off leaves no partial object under the final name on S3 or on a local directory. The same holds for an rclone remote that can move objects. An rclone remote that cannot move objects can keep a partial object.

Pruning

prune deletes two kinds of object at the destination:

• a blob that no manifest lists
• a directory under metadata/ that has no manifest

It reads the manifests, which are not encrypted, so it needs no private key.

These rules keep it from deleting data that is in use:

• It keeps every blob that any manifest lists.
• It deletes nothing when any manifest cannot be downloaded or decoded.
• It deletes a blob only once the blob is older than prune_grace_period.
• It deletes a directory only once every object in it is that old.
• It keeps a directory whose snapshot is in the local index.
• It leaves alone a directory whose name is not a remote key.

prune_grace_period defaults to 24h. It protects a backup that is still running. Such a backup has uploaded blobs and perhaps db.zst.age, but not yet its manifest. A prune from this machine or another keeps those objects, as long as no backup takes longer than the grace period.

The grace period does not protect one case. A backup does not upload a blob that is already at the destination, so that blob keeps the time of its first upload. The blob can be left over from an earlier backup that stopped before its manifest. A prune that runs before the new manifest is written can then delete that blob. The new snapshot cannot be restored afterwards.

To avoid this case, give each machine and each user its own destination.

remote nuke applies none of these rules.

What a restore needs

• the age private key
• the snapshot's db.zst.age
• the blobs that hold the files to restore

A restore does not read the manifest and does not need the local index.

It works in these steps:

• It downloads and decrypts db.zst.age.
• It checks that the database is the one for the snapshot that was asked for.
• It downloads and decrypts the blobs it needs. It checks each blob against its name.
• It writes each file from its chunks.
• It sets each file's permissions and times, and its owner when it runs as root.