vaultik/docs/REPOSTRUCTURE.md

# Vaultik S3 Repository Structure

This document describes the structure and organization of data stored in the S3
bucket by Vaultik.

## Overview

Vaultik stores all backup data in an S3-compatible object store. The repository
consists of two main components, plus a marker file:

1. **Blobs** - The actual backup data (content-addressed, encrypted)
2. **Metadata** - Snapshot information and manifests (partially encrypted)

## Directory Structure

```
<bucket>/<prefix>/
├── vaultik-destination.txt
├── blobs/
│   └── <hash[0:2]>/
│       └── <hash[2:4]>/
│           └── <full-hash>
└── metadata/
    └── <remote-key>/
        ├── db.zst.age
        └── manifest.json.zst
```

The metadata subdirectory is named with the **remote key**, a one-way hash of
the snapshot ID, not with the human-readable snapshot ID itself. See
[Remote Key Derivation](#remote-key-derivation).

## Marker File (`vaultik-destination.txt`)

Every backup writes this file at the root before it uploads anything. It holds
one fixed line of plain text and names no host, path or time. While the local
index holds snapshots stored there, a destination that answers a look-up of the
marker with not found is treated as not present: `snapshot create` refuses to
write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list`
and `remote nuke` behave as for a destination that cannot be listed. An
unplugged volume leaves an empty directory at its mount point, and without the
marker that directory would read as a destination whose snapshots are all gone.
`remote nuke` deletes the marker last.

When the look-up of the marker fails for any reason other than not found, the
check is skipped and the command goes on. S3 refuses it on every run to
credentials that allow only PUT, so a host with such credentials does not notice
a destination emptied since its last backup, and its next backup would leave out
the data its local index records as already stored. After the destination is
emptied, run `vaultik database delete` on such a host before it backs up again.

## Blobs Directory (`blobs/`)

### Structure

- **Path format**: `blobs/<first-2-chars>/<next-2-chars>/<full-hash>`
- **Example**:
  `blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`
- **Sharding**: The two-level directory structure (using the first 4 characters
  of the hash) prevents any single directory from containing too many objects

### Content

- **What it contains**: Packed collections of content-defined chunks from files
- **Format**: Zstandard compressed, then Age encrypted
- **Encryption**: Always encrypted with Age using the configured recipients
- **Naming**: Content-addressed. The blob's name is
  `hex(SHA256(SHA256(uncompressed blob contents)))` — the double SHA-256 of the
  concatenated chunk data, computed before compression and encryption, not a
  hash of the stored (compressed, encrypted) bytes. One consequence: only a
  holder of the age private key can check a stored blob's integrity, because
  matching a blob to its name means decrypting and decompressing it first —
  which is what `restore` and `verify --deep` do. Implemented in
  `internal/blobgen` (`DoubleSHA256`). This is the canonical description of blob
  naming; other documents and comments point here.

### Why Encrypted

Blobs contain the actual file data from backups and must be encrypted for
security. The content-addressing ensures deduplication while the encryption
ensures privacy.

## Metadata Directory (`metadata/`)

Each snapshot has its own subdirectory. The directory is **not** named with the
human-readable snapshot ID; it is named with the remote key — a one-way hash of
that ID. The human ID is never written to the destination store as a directory
name (see [Remote Key Derivation](#remote-key-derivation)).

### Snapshot ID Format

The human-readable snapshot ID is used in CLI arguments, log lines, and the
local database. It is not written to the destination store.

- **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>`
  if no name was specified)
- **Example**: `laptop_home_2024-01-15T14:30:52Z`
- **Components**:
    - Short hostname (everything before the first dot is stripped from the FQDN)
    - Snapshot name from the configured `snapshots:` map (optional)
    - RFC3339 UTC timestamp

This ID reveals the hostname, the configured snapshot name, and the backup time,
so it is never used as the on-disk directory name — the remote key is used
instead.

### Remote Key Derivation

The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double
SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The
result is a 64-character hex string. The hash is not reversible, but it uses no
secret: an observer who guesses a candidate hostname and snapshot name can hash
it the same way and confirm whether that snapshot is present. The remote key
keeps names out of a plain listing; it does not hide them from a guess.
Implemented in `internal/snapshot/remotekey.go`.

Worked example:

- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
- Directory:
  `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`

A plain listing of the destination store therefore shows only these hashes, not
the hostname or snapshot name of any backup — but because the hash uses no
secret, a guessed hostname and snapshot name can be hashed and confirmed against
the listing. The same remote key is stored in the manifest's `snapshot_id`
field.

### Files in Each Snapshot Directory

#### `db.zst.age` - Encrypted Database

- **What it contains**: Pruned binary SQLite database for this snapshot
- **Format**: Binary SQLite → Zstandard compressed → Age encrypted
- **Encryption**: Encrypted with Age
- **Purpose**: Contains full file metadata, chunk mappings, and all
  relationships
- **Why encrypted**: Contains sensitive metadata like file paths, permissions,
  and ownership

#### `manifest.json.zst` - Unencrypted Blob Manifest

- **What it contains**: JSON list of all blob hashes referenced by this snapshot
- **Format**: JSON → Zstandard compressed (NOT encrypted)
- **Encryption**: NOT encrypted
- **Purpose**: Enables pruning operations without requiring decryption keys
- **Structure**:

```json
{
    "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
    "timestamp": "2025-06-01T12:00:00Z",
    "blob_count": 42,
    "total_compressed_size": 1048576,
    "blobs": [
        {
            "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
            "compressed_size": 24576
        },
        {
            "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678",
            "compressed_size": 32768
        }
    ]
}
```

`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is
written in the clear.

### Why Manifest is Unencrypted

The manifest must be readable without the private key to enable:

1. **Pruning operations** - Identifying unreferenced blobs for deletion
2. **Storage analysis** - Understanding space usage without decryption
3. **Verification** - Checking blob existence without decryption
4. **Cross-snapshot deduplication analysis** - Finding shared blobs between
   snapshots

The manifest contains the remote key, the backup timestamp, the blob count and
total compressed size, and each blob's hash and compressed size. It contains no
file names, paths, or other decrypted metadata.

## Security Considerations

### What's Encrypted

- **All file content** (in blobs)
- **All file metadata** (paths, permissions, timestamps, ownership in
  db.zst.age)
- **File-to-chunk mappings** (in db.zst.age)

### What's Not Encrypted

- **The remote key** — directory names and the manifest `snapshot_id`, a one-way
  hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation))
- **The backup timestamp** (in manifest.json.zst)
- **Blob hashes and their compressed sizes** (in manifest.json.zst)
- **Blob count and total compressed size per snapshot** (in manifest.json.zst)

### Privacy Implications

From the unencrypted data, an observer of the destination store can determine:

- **When each backup was taken** — not from the directory name, which is a
  one-way hash, but from the plaintext `timestamp` field in manifest.json.zst,
  which is published in the clear
- How many blobs each snapshot references, and the total compressed size
- The compressed size of each blob, and which blobs are shared between snapshots
  (deduplication patterns)
- **Whether a guessed hostname and snapshot name are present** — the remote key
  is an unkeyed hash, so an observer holding candidate names can hash each one
  and match it against the directory listing. The human ID is never published,
  so it cannot be read off directly, but it can be confirmed by guessing.

Together these give an observer a timing-and-size profile of every snapshot.
This is an accepted, documented property of the format, not a defect: the
manifest is unencrypted so that pruning can run without the private key, and the
timing channel could not be closed by encrypting it anyway — object creation
times and per-object sizes stay visible at the storage layer on both `s3://` and
`file://` destinations regardless.

An observer cannot determine:

- The hostname or snapshot name of any backup by reading it off the store — the
  directory name and the manifest `snapshot_id` are unkeyed hashes of the human
  ID, so the text is never published (though a guessed name can be confirmed, as
  above)
- File names or paths
- File contents
- File permissions or ownership
- Directory structure
- Which chunks belong to which files

### Accepted Risks

These are known, deliberate properties of the format and the tooling, recorded
so an operator can weigh them rather than discover them.

1. **No proof of authorship.** Restore and `verify --deep` prove that data
   decrypts with the age private key and matches its unkeyed content hashes.
   They do not prove who wrote it: anyone who knows a recipient public key and
   can replace objects on the destination can substitute a snapshot they built.
   The recipient string is not stored at the destination, but a compromised
   backed-up host has it. Defences live on the destination side — bucket
   versioning or object lock, credentials for the source host that cannot delete
   or overwrite existing versions, and pruning from a trusted host. Note that S3
   `PutObject` overwrites an existing key, so PUT permission alone is not
   append-only.
2. **Compression reveals sizes.** Blobs and `db.zst.age` are zstd-compressed
   then age-encrypted; the manifest is compressed only. age does not pad, so an
   object's size is the exact compressed length of its contents. All new chunks
   packed into one blob share a single zstd stream (8 MiB window, 4 MiB at
   compression levels 1-2), and a blob is closed at `blob_size_limit` and at the
   end of each configured path. Because a stored chunk is never packed again,
   someone who can write into a backed-up file and watch blob sizes learns
   something only when their controlled data and a secret land in the same chunk
   of a file that keeps changing. Advice: back up any outsider-writable
   directory as its own snapshot.
3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The
   default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of
   2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a
   blob holds hundreds of chunks, so individual chunk lengths are not visible in
   the blob's size; lowering the limit toward the chunk size begins to expose
   them.
4. **Decrypted data on local disk.** Several commands stage plaintext under
   `$TMPDIR`: `snapshot restore` writes decrypted blobs under
   `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at
   `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at
   `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of
   the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600`
   and removed on success, but a `kill -9` or a power loss leaves them behind —
   delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR`
   should be trusted to the same degree as the restore target.
5. **Store permissions differ per command.** The backed-up host needs only PUT
   to run `snapshot create`: it writes blobs and metadata and neither reads nor
   deletes them. Other commands need more — `snapshot verify`,
   `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`,
   `snapshot remove`, and `remote nuke` also delete. The recommended cron line
   uses `--prune`, which runs `prune` on the backed-up host, so granting that
   host `--prune` gives it credentials that can delete its own backups. To keep
   the source host to PUT only, prune from a separate trusted host instead.
6. **Changing recipients does not re-encrypt existing data.** Deduplicated
   chunks and same-named blobs already on the destination stay encrypted to the
   recipients in force when they were written. A new snapshot that reuses them
   cannot be restored with a newly added recipient's key alone, because those
   reused objects were never encrypted to it. To make everything readable by a
   new key, run `vaultik database delete` and take a full backup to a fresh
   destination or prefix.
7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients.
   Long-lived ciphertext held by a third party (the destination operator) has no
   fallback if X25519 is ever broken: there is no second recipient type and no
   post-quantum option.

## Consistency Guarantees

1. **Blobs are immutable** - Once written, a blob is never modified
2. **Blobs are written before metadata** - A snapshot's metadata is only written
   after all its blobs are successfully uploaded
3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst
   are written as complete files
4. **A snapshot is marked complete in the local DB only after its metadata is
   uploaded** - `finalizeSnapshotMetadata` runs `ExportSnapshotMetadata` first
   and records completion (`MarkSnapshotComplete`) only once the export succeeds
   (see the backup data flow in [ARCHITECTURE.md](ARCHITECTURE.md)). A crash
   during the export therefore leaves the snapshot incomplete, so the next
   backup's `PruneDatabase` drops it and re-backs-up its data, rather than
   leaving a completed-looking row in the local index with no matching metadata
   on the destination store. (A crash in the brief moment after the export
   succeeds but before completion is recorded leaves a fully-restorable snapshot
   on the destination that the local index drops as incomplete on the next run;
   `snapshot list` then reports it honestly as remote-only, which is the safe
   direction: the destination copy stays restorable.)

## Pruning Safety

The prune operation is safe because:

1. It keeps every blob listed in any snapshot's manifest and deletes only blobs
   that no manifest references. Under `metadata/` it deletes only a directory
   that has no `manifest.json.zst`, whose name is a remote key, and that no
   local snapshot record claims
2. Manifests are unencrypted and can be read without keys
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and
   exits with an error, rather than treating that snapshot's blobs as
   unreferenced
4. It deletes an unreferenced blob only once the blob is older than
   `prune_grace_period` (default `24h`), and a directory under `metadata/` with
   no manifest only once every object in it is. A backup uploads its blobs and
   its `db.zst.age` before its manifest, so a prune running meanwhile, on this
   host or another sharing the destination, leaves them as long as no backup run
   takes longer than the grace period. The grace period does not protect a blob
   a backup reuses from an earlier run that stopped before its manifest: the
   backup skips the upload of a blob already on the destination, so that blob
   keeps the time of its first upload, and a prune before the backup's manifest
   is written can delete it; the new snapshot then cannot be restored.
   `remote nuke` has no such check

## Restoration Requirements

To restore from a backup, you need:

1. **The Age private key** - To decrypt blobs and database
2. **The snapshot metadata** - Both files from the snapshot's metadata directory
3. **All referenced blobs** - As listed in the manifest

The restoration process:

1. Download and decrypt the database dump to understand file structure
2. Download and decrypt the required blobs
3. Reconstruct files from their chunks
4. Restore file metadata (permissions, timestamps, etc.)