vaultik/docs/REPOSTRUCTURE.md

Vaultik S3 Repository Structure

This document describes the structure and organization of data stored in the S3 bucket by Vaultik.

Overview

Vaultik stores all backup data in an S3-compatible object store. The repository consists of two main components:

1. Blobs - The actual backup data (content-addressed, encrypted)
2. Metadata - Snapshot information and manifests (partially encrypted)

Directory Structure

<bucket>/<prefix>/
├── blobs/
│   └── <hash[0:2]>/
│       └── <hash[2:4]>/
│           └── <full-hash>
└── metadata/
    └── <remote-key>/
        ├── db.zst.age
        └── manifest.json.zst

The metadata subdirectory is named with the remote key, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See Remote Key Derivation.

Blobs Directory (blobs/)

Structure

• Path format: blobs/<first-2-chars>/<next-2-chars>/<full-hash>
• Example: blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678
• Sharding: The two-level directory structure (using the first 4 characters of the hash) prevents any single directory from containing too many objects

Content

• What it contains: Packed collections of content-defined chunks from files
• Format: Zstandard compressed, then Age encrypted
• Encryption: Always encrypted with Age using the configured recipients
• Naming: Content-addressed. The blob's name is hex(SHA256(SHA256(uncompressed blob contents))) — the double SHA-256 of the concatenated chunk data, computed before compression and encryption, not a hash of the stored (compressed, encrypted) bytes. One consequence: only a holder of the age private key can check a stored blob's integrity, because matching a blob to its name means decrypting and decompressing it first — which is what restore and verify --deep do. Implemented in internal/blobgen (DoubleSHA256). This is the canonical description of blob naming; other documents and comments point here.

Why Encrypted

Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy.

Metadata Directory (metadata/)

Each snapshot has its own subdirectory. The directory is not named with the human-readable snapshot ID; it is named with the remote key — a one-way hash of that ID. The human ID is never written to the destination store as a directory name (see Remote Key Derivation).

Snapshot ID Format

The human-readable snapshot ID is used in CLI arguments, log lines, and the local database. It is not written to the destination store.

• Format: <hostname>_<snapshot-name>_<RFC3339> (or <hostname>_<RFC3339> if no name was specified)
• Example: laptop_home_2024-01-15T14:30:52Z
• Components:
  • Short hostname (everything before the first dot is stripped from the FQDN)
  • Snapshot name from the configured snapshots: map (optional)
  • RFC3339 UTC timestamp

This ID reveals the hostname, the configured snapshot name, and the backup time, so it is never used as the on-disk directory name — the remote key is used instead.

Remote Key Derivation

The remote key is hex(SHA256(SHA256("vaultik|" + snapshot-id))): a double SHA-256 over the snapshot ID, with a vaultik| domain-separation prefix. The result is a 64-character hex string. The hash is not reversible, but it uses no secret: an observer who guesses a candidate hostname and snapshot name can hash it the same way and confirm whether that snapshot is present. The remote key keeps names out of a plain listing; it does not hide them from a guess. Implemented in internal/snapshot/remotekey.go.

Worked example:

• Snapshot ID: server1_home_2025-06-01T12:00:00Z
• Remote key: 17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa
• Directory: metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/

A plain listing of the destination store therefore shows only these hashes, not the hostname or snapshot name of any backup — but because the hash uses no secret, a guessed hostname and snapshot name can be hashed and confirmed against the listing. The same remote key is stored in the manifest's snapshot_id field.

Files in Each Snapshot Directory

db.zst.age - Encrypted Database

• What it contains: Pruned binary SQLite database for this snapshot
• Format: Binary SQLite → Zstandard compressed → Age encrypted
• Encryption: Encrypted with Age
• Purpose: Contains full file metadata, chunk mappings, and all relationships
• Why encrypted: Contains sensitive metadata like file paths, permissions, and ownership

manifest.json.zst - Unencrypted Blob Manifest

• What it contains: JSON list of all blob hashes referenced by this snapshot
• Format: JSON → Zstandard compressed (NOT encrypted)
• Encryption: NOT encrypted
• Purpose: Enables pruning operations without requiring decryption keys
• Structure:

{
  "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
  "timestamp": "2025-06-01T12:00:00Z",
  "blob_count": 42,
  "total_compressed_size": 1048576,
  "blobs": [
    { "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 24576 },
    { "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 32768 }
  ]
}

snapshot_id is the remote key (a hash), not the human ID; timestamp is written in the clear.

Why Manifest is Unencrypted

The manifest must be readable without the private key to enable:

1. Pruning operations - Identifying unreferenced blobs for deletion
2. Storage analysis - Understanding space usage without decryption
3. Verification - Checking blob existence without decryption
4. Cross-snapshot deduplication analysis - Finding shared blobs between snapshots

The manifest contains the remote key, the backup timestamp, the blob count and total compressed size, and each blob's hash and compressed size. It contains no file names, paths, or other decrypted metadata.

Security Considerations

What's Encrypted

• All file content (in blobs)
• All file metadata (paths, permissions, timestamps, ownership in db.zst.age)
• File-to-chunk mappings (in db.zst.age)

What's Not Encrypted

• The remote key — directory names and the manifest snapshot_id, a one-way hash of the snapshot ID (see Remote Key Derivation)
• The backup timestamp (in manifest.json.zst)
• Blob hashes and their compressed sizes (in manifest.json.zst)
• Blob count and total compressed size per snapshot (in manifest.json.zst)

Privacy Implications

From the unencrypted data, an observer of the destination store can determine:

• When each backup was taken — not from the directory name, which is a one-way hash, but from the plaintext timestamp field in manifest.json.zst, which is published in the clear
• How many blobs each snapshot references, and the total compressed size
• The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)
• Whether a guessed hostname and snapshot name are present — the remote key is an unkeyed hash, so an observer holding candidate names can hash each one and match it against the directory listing. The human ID is never published, so it cannot be read off directly, but it can be confirmed by guessing.

Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both s3:// and file:// destinations regardless.

An observer cannot determine:

• The hostname or snapshot name of any backup by reading it off the store — the directory name and the manifest snapshot_id are unkeyed hashes of the human ID, so the text is never published (though a guessed name can be confirmed, as above)
• File names or paths
• File contents
• File permissions or ownership
• Directory structure
• Which chunks belong to which files

Accepted Risks

These are known, deliberate properties of the format and the tooling, recorded so an operator can weigh them rather than discover them.

1. No proof of authorship. Restore and verify --deep prove that data decrypts with the age private key and matches its unkeyed content hashes. They do not prove who wrote it: anyone who knows a recipient public key and can replace objects on the destination can substitute a snapshot they built. The recipient string is not stored at the destination, but a compromised backed-up host has it. Defences live on the destination side — bucket versioning or object lock, credentials for the source host that cannot delete or overwrite existing versions, and pruning from a trusted host. Note that S3 PutObject overwrites an existing key, so PUT permission alone is not append-only.
2. Compression reveals sizes. Blobs and db.zst.age are zstd-compressed then age-encrypted; the manifest is compressed only. age does not pad, so an object's size is the exact compressed length of its contents. All new chunks packed into one blob share a single zstd stream (8 MiB window, 4 MiB at compression levels 1-2), and a blob is closed at blob_size_limit and at the end of each configured path. Because a stored chunk is never packed again, someone who can write into a backed-up file and watch blob sizes learns something only when their controlled data and a secret land in the same chunk of a file that keeps changing. Advice: back up any outsider-writable directory as its own snapshot.
3. Chunking uses no secret. The FastCDC parameters are fixed and public. The default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of 2.5 MB or less is a single chunk. At the default 10 GB blob_size_limit a blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them.
4. Decrypted data on local disk. Several commands stage plaintext under $TMPDIR: snapshot restore writes decrypted blobs under vaultik-blobcache-*/ (no size cap) and the decrypted metadata database at vaultik-restore-*/snapshot.db; verify --deep writes that database at vaultik-verify-*/snapshot.db; snapshot create keeps a plaintext copy of the index at vaultik-snapshot-*/snapshot.db. These files are created 0600 and removed on success, but a kill -9 or a power loss leaves them behind — delete any leftover vaultik-* directory under $TMPDIR by hand. $TMPDIR should be trusted to the same degree as the restore target.
5. Store permissions differ per command. The backed-up host needs only PUT to run snapshot create: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — snapshot verify, snapshot restore, and prune list and read; prune, snapshot purge, snapshot remove, and remote nuke also delete. The recommended cron line uses --prune, which runs prune on the backed-up host, so granting that host --prune gives it credentials that can delete its own backups. To keep the source host to PUT only, prune from a separate trusted host instead.
6. Changing recipients does not re-encrypt existing data. Deduplicated chunks and same-named blobs already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run vaultik database delete and take a full backup to a fresh destination or prefix.
7. X25519 recipients only. vaultik rejects age ssh and plugin recipients. Long-lived ciphertext held by a third party (the destination operator) has no fallback if X25519 is ever broken: there is no second recipient type and no post-quantum option.

Consistency Guarantees

1. Blobs are immutable - Once written, a blob is never modified
2. Blobs are written before metadata - A snapshot's metadata is only written after all its blobs are successfully uploaded
3. Metadata is written atomically - Both db.zst.age and manifest.json.zst are written as complete files
4. A snapshot is marked complete in the local DB only after its metadata is uploaded - finalizeSnapshotMetadata runs ExportSnapshotMetadata first and records completion (MarkSnapshotComplete) only once the export succeeds (see the backup data flow in ARCHITECTURE.md). A crash during the export therefore leaves the snapshot incomplete, so the next backup's PruneDatabase drops it and re-backs-up its data, rather than leaving a completed-looking row in the local index with no matching metadata on the destination store. (A crash in the brief moment after the export succeeds but before completion is recorded leaves a fully-restorable snapshot on the destination that the local index drops as incomplete on the next run; snapshot list then reports it honestly as remote-only, which is the safe direction: the destination copy stays restorable.)

Pruning Safety

The prune operation is safe because:

1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references
2. Manifests are unencrypted and can be read without keys
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced
4. Prune requires exclusive access to the destination: running it during a concurrent backup can race a snapshot whose manifest is not yet written, so do not prune while a backup is in progress

Restoration Requirements

To restore from a backup, you need:

1. The Age private key - To decrypt blobs and database
2. The snapshot metadata - Both files from the snapshot's metadata directory
3. All referenced blobs - As listed in the manifest

The restoration process:

1. Download and decrypt the database dump to understand file structure
2. Download and decrypt the required blobs
3. Reconstruct files from their chunks
4. Restore file metadata (permissions, timestamps, etc.)