vaultik/docs/REPOSTRUCTURE.md

Vaultik S3 Repository Structure

This document describes the structure and organization of data stored in the S3 bucket by Vaultik.

Overview

Vaultik stores all backup data in an S3-compatible object store. The repository consists of two main components:

1. Blobs - The actual backup data (content-addressed, encrypted)
2. Metadata - Snapshot information and manifests (partially encrypted)

Directory Structure

<bucket>/<prefix>/
├── blobs/
│   └── <hash[0:2]>/
│       └── <hash[2:4]>/
│           └── <full-hash>
└── metadata/
    └── <remote-key>/
        ├── db.zst.age
        └── manifest.json.zst

The metadata subdirectory is named with the remote key, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See Remote Key Derivation.

Blobs Directory (blobs/)

Structure

• Path format: blobs/<first-2-chars>/<next-2-chars>/<full-hash>
• Example: blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678
• Sharding: The two-level directory structure (using the first 4 characters of the hash) prevents any single directory from containing too many objects

Content

• What it contains: Packed collections of content-defined chunks from files
• Format: Zstandard compressed, then Age encrypted
• Encryption: Always encrypted with Age using the configured recipients
• Naming: Content-addressed using SHA256 hash of the encrypted blob

Why Encrypted

Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy.

Metadata Directory (metadata/)

Each snapshot has its own subdirectory. The directory is not named with the human-readable snapshot ID; it is named with the remote key — a one-way hash of that ID. The human ID is never written to the destination store as a directory name (see Remote Key Derivation).

Snapshot ID Format

The human-readable snapshot ID is used in CLI arguments, log lines, and the local database. It is not written to the destination store.

• Format: <hostname>_<snapshot-name>_<RFC3339> (or <hostname>_<RFC3339> if no name was specified)
• Example: laptop_home_2024-01-15T14:30:52Z
• Components:
  • Short hostname (everything before the first dot is stripped from the FQDN)
  • Snapshot name from the configured snapshots: map (optional)
  • RFC3339 UTC timestamp

This ID reveals the hostname, the configured snapshot name, and the backup time, so it is never used as the on-disk directory name — the remote key is used instead.

Remote Key Derivation

The remote key is hex(SHA256(SHA256("vaultik|" + snapshot-id))): a double SHA-256 over the snapshot ID, with a vaultik| domain-separation prefix. The result is a 64-character hex string with no structure a remote observer can reverse. Implemented in internal/snapshot/remotekey.go.

Worked example:

• Snapshot ID: server1_home_2025-06-01T12:00:00Z
• Remote key: 17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa
• Directory: metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/

Because the hash is one-way, a listing of the destination store reveals neither the hostname nor the snapshot name of any backup. The same remote key is stored in the manifest's snapshot_id field.

Files in Each Snapshot Directory

db.zst.age - Encrypted Database

• What it contains: Pruned binary SQLite database for this snapshot
• Format: Binary SQLite → Zstandard compressed → Age encrypted
• Encryption: Encrypted with Age
• Purpose: Contains full file metadata, chunk mappings, and all relationships
• Why encrypted: Contains sensitive metadata like file paths, permissions, and ownership

manifest.json.zst - Unencrypted Blob Manifest

• What it contains: JSON list of all blob hashes referenced by this snapshot
• Format: JSON → Zstandard compressed (NOT encrypted)
• Encryption: NOT encrypted
• Purpose: Enables pruning operations without requiring decryption keys
• Structure:

{
  "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
  "timestamp": "2025-06-01T12:00:00Z",
  "blob_count": 42,
  "total_compressed_size": 1048576,
  "blobs": [
    { "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 24576 },
    { "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 32768 }
  ]
}

snapshot_id is the remote key (a hash), not the human ID; timestamp is written in the clear.

Why Manifest is Unencrypted

The manifest must be readable without the private key to enable:

1. Pruning operations - Identifying unreferenced blobs for deletion
2. Storage analysis - Understanding space usage without decryption
3. Verification - Checking blob existence without decryption
4. Cross-snapshot deduplication analysis - Finding shared blobs between snapshots

The manifest contains the remote key, the backup timestamp, the blob count and total compressed size, and each blob's hash and compressed size. It contains no file names, paths, or other decrypted metadata.

Security Considerations

What's Encrypted

• All file content (in blobs)
• All file metadata (paths, permissions, timestamps, ownership in db.zst.age)
• File-to-chunk mappings (in db.zst.age)

What's Not Encrypted

• The remote key — directory names and the manifest snapshot_id, a one-way hash of the snapshot ID (see Remote Key Derivation)
• The backup timestamp (in manifest.json.zst)
• Blob hashes and their compressed sizes (in manifest.json.zst)
• Blob count and total compressed size per snapshot (in manifest.json.zst)

Privacy Implications

From the unencrypted data, an observer of the destination store can determine:

• When each backup was taken — not from the directory name, which is a one-way hash, but from the plaintext timestamp field in manifest.json.zst, which is published in the clear
• How many blobs each snapshot references, and the total compressed size
• The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)

Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both s3:// and file:// destinations regardless.

An observer cannot determine:

• The hostname or snapshot name of any backup (the directory name and the manifest snapshot_id are one-way hashes of the human ID)
• File names or paths
• File contents
• File permissions or ownership
• Directory structure
• Which chunks belong to which files

Consistency Guarantees

1. Blobs are immutable - Once written, a blob is never modified
2. Blobs are written before metadata - A snapshot's metadata is only written after all its blobs are successfully uploaded
3. Metadata is written atomically - Both db.zst.age and manifest.json.zst are written as complete files
4. Snapshots are marked complete in local DB only after metadata upload - Ensures consistency between local and remote state

Pruning Safety

The prune operation is safe because:

1. It only deletes blobs not referenced in any manifest
2. Manifests are unencrypted and can be read without keys
3. The operation compares the latest local DB snapshot with the latest S3 snapshot to ensure consistency
4. Pruning will fail if these don't match, preventing accidental deletion of needed blobs

Restoration Requirements

To restore from a backup, you need:

1. The Age private key - To decrypt blobs and database
2. The snapshot metadata - Both files from the snapshot's metadata directory
3. All referenced blobs - As listed in the manifest

The restoration process:

1. Download and decrypt the database dump to understand file structure
2. Download and decrypt the required blobs
3. Reconstruct files from their chunks
4. Restore file metadata (permissions, timestamps, etc.)