vaultik/docs/DATAMODEL.md

# vaultik data model

`vaultik` keeps a SQLite database on the backed-up machine, called the local
index. This document describes its tables and how each command uses them.

The local index has two jobs:

- It records which files, chunks and blobs are already stored. The next backup
  then uploads only what changed.
- It is the source of each snapshot's own database. A backup uploads a trimmed,
  encrypted copy of the index with every snapshot.

A restore does not need the local index. It reads the copy that was uploaded
with the snapshot.

## Terms

- **Snapshot**: one backup of the paths listed under one name in the config.
- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
  content.
- **Blob**: one object at the destination. It holds many chunks, compressed and
  then encrypted.
- **Destination**: the place the backups are stored.
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
  at the destination.

## Where the index is

The index is the file `index.sqlite` in the platform data directory. That is
`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/`
on macOS. The `index_path` setting and the `VAULTIK_INDEX_PATH` environment
variable change the path.

The file is readable only by its owner, because it lists every backed-up path.
`vaultik` sets that mode each time it opens the index.

## Schema migrations

`vaultik` supports schema migrations. They are the numbered SQL files in
`internal/database/schema/`, which are built into the binary.

- `000.sql` creates the `schema_migrations` table. That table records each
  migration that has run.
- `001.sql` creates every other table.

`database.New` opens the index. It then applies, in order, every migration that
the index has not recorded.

Before version 1.0, no migration is added, because nothing is installed anywhere
yet. A schema change edits `001.sql` directly, with the code that uses the
changed tables.

This has one consequence. An index created before such a change has already
recorded `001.sql` as run, so it keeps the old schema and can become unusable.
To rebuild it, run `vaultik database delete` and then a full backup.

After version 1.0, each schema change is a new numbered file. An existing index
is then migrated the first time an updated `vaultik` opens it. A file that was
part of a release is never edited.

Migrations apply to the local index only. A snapshot's database at the
destination is read as it was written. Each snapshot records the `vaultik`
version that wrote it.

## Tables

Times are whole seconds since the Unix epoch, in UTC. The two exceptions are
noted at their columns.

### `files`

One row for each backed-up path: a regular file, a symlink or a directory.

- `id`: a UUID. Primary key.
- `path`: the absolute path. Unique.
- `source_path`: the configured path that the scan found this entry under. A
  restore does not read it.
- `mtime`: the modification time.
- `mtime_nsec`: the nanoseconds within that second.
- `size`: the size in bytes.
- `mode`: the permission bits and the file type.
- `uid`: the owner's user ID.
- `gid`: the owner's group ID.
- `link_target`: the target of a symlink. It is `NULL` for other entries.

`idx_files_path` indexes `path`.

There is one row for each path, and it is updated when the file changes. The
row is shared by every snapshot in the local index that lists the path. An
older snapshot's own view of the file is in that snapshot's database at the
destination.

### `chunks`

One row for each distinct chunk.

- `chunk_hash`: the SHA-256 hash of the chunk's content, in hex. Primary key.
- `size`: the size in bytes.

### `file_chunks`

The chunks that make up each file, in order.

- `file_id`: the file. References `files.id`.
- `idx`: the position of the chunk within the file, counted from 0.
- `chunk_hash`: the chunk. References `chunks.chunk_hash`.

The primary key is (`file_id`, `idx`). `idx_file_chunks_chunk_hash` indexes
`chunk_hash`.

### `chunk_files`

The reverse of `file_chunks`: the files that contain each chunk.

- `chunk_hash`: the chunk. References `chunks.chunk_hash`.
- `file_id`: the file. References `files.id`.
- `file_offset`: the byte offset of the chunk within the file.
- `length`: the length of the chunk in bytes.

The primary key is (`chunk_hash`, `file_id`). `idx_chunk_files_file_id` indexes
`file_id`.

### `blobs`

One row for each blob.

- `id`: a UUID, given when packing of the blob starts. Primary key.
- `blob_hash`: the name of the blob at the destination. Unique. Until the blob
  is closed, it holds a placeholder that starts with `temp-placeholder-`.
- `created_ts`: when packing started.
- `finished_ts`: when the blob was closed. It is `NULL` before that.
- `uncompressed_size`: the total size of the blob's chunks.
- `compressed_size`: the size after compression and encryption.
- `uploaded_ts`: when the upload finished. It is `NULL` before that.

`blob_hash` is `hex(SHA256(SHA256(uncompressed contents)))`. See
[REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs).

Once a blob is uploaded, `compressed_size` is the size of the object at the
destination. A blob that was already there can have a different size from the
local copy. That happens when it was written with another compression level or
other recipients.

### `blob_chunks`

The chunks that each blob holds.

- `blob_id`: the blob. References `blobs.id`.
- `chunk_hash`: the chunk. References `chunks.chunk_hash`.
- `offset`: the byte offset of the chunk within the uncompressed blob.
- `length`: the length of the chunk in bytes.

The primary key is (`blob_id`, `chunk_hash`). `idx_blob_chunks_chunk_hash`
indexes `chunk_hash`.

A restore finds the blob for each chunk through this table.

### `snapshots`

One row for each snapshot.

- `id`: the snapshot ID. Primary key.
- `hostname`: the hostname of the machine that made the backup.
- `vaultik_version`: the `vaultik` version that made it.
- `vaultik_git_revision`: the git commit of that version.
- `started_at`: when the backup started.
- `completed_at`: when the backup finished. It is `NULL` until then.
- `file_count`: the number of files and symlinks in the snapshot. Directories
  are not counted.
- `chunk_count`: the number of chunks this snapshot stored that were not stored
  before.
- `new_chunk_size`: the total size of those chunks before compression.
- `blob_count`: the number of blobs this snapshot created.
- `total_size`: the total size of all files in the snapshot.
- `uncompressed_chunk_size`: the total size, before compression, of the
  distinct chunks the snapshot's files use.
- `blob_size`: the total compressed size of the blobs the snapshot uses.
- `blob_uncompressed_size`: the total uncompressed size of those blobs.
- `compression_ratio`: the compression ratio of those blobs.
- `compression_level`: the compression level the backup used.
- `upload_bytes`: the bytes uploaded during the backup.
- `upload_duration_ms`: the time spent uploading, in milliseconds.

The snapshot ID has the form `<hostname>_<snapshot-name>_<timestamp>`. An
example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:

- the hostname, up to its first `.`
- the snapshot name
- the start time, in UTC

`snapshot list` shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE` and
`new_chunk_size` as `NEW CHUNK SIZE`.

A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
does not show it. The next backup deletes it, and so does `prune`.

### `snapshot_files`

The files in each snapshot.

- `snapshot_id`: the snapshot. References `snapshots.id`.
- `file_id`: the file. References `files.id`.

The primary key is (`snapshot_id`, `file_id`). `idx_snapshot_files_file_id`
indexes `file_id`.

### `snapshot_blobs`

The blobs that each snapshot uses.

- `snapshot_id`: the snapshot. References `snapshots.id`.
- `blob_id`: the blob. References `blobs.id`.
- `blob_hash`: a copy of the blob's hash.

The primary key is (`snapshot_id`, `blob_id`). `idx_snapshot_blobs_blob_id`
indexes `blob_id`.

The manifest is built from this table.

### `snapshot_mount_points`

The mount points that a snapshot's backup found.

- `snapshot_id`: the snapshot. References `snapshots.id`.
- `path`: the absolute path of the directory.

The primary key is (`snapshot_id`, `path`).

A directory is recorded here when all of these hold:

- It is at or below a configured path.
- The backup found an entry in it that is not excluded.
- It is on a different filesystem from its parent directory.

The device number is read after the directory has been listed. Listing an
automount point mounts its volume, so the number is that of the volume. An
automount point whose volume cannot be mounted is not recorded, because the
backup finds no entry in it.

`snapshot create --prune` reads this table. It skips the prune when an earlier
snapshot of the same name recorded a mount point that the new backup did not
find. That happens when the volume is not mounted.

### `uploads`

One row for each blob that a backup uploaded.

- `blob_hash`: the blob. References `blobs.blob_hash`. Primary key.
- `snapshot_id`: the snapshot whose backup uploaded it. References
  `snapshots.id`.
- `uploaded_at`: when the upload started. It is written as a date and time, not
  as seconds.
- `size`: the size of the uploaded blob.
- `duration_ms`: the duration of the upload, in milliseconds.

`idx_uploads_snapshot_id` indexes `snapshot_id`.

A blob that was already at the destination gets no row here.

### `local_meta`

Settings that belong to this copy of the index.

- `key`: the name of the setting. Primary key.
- `value`: its value.

The only key is `storage_url`. It holds the destination that the index was
first used with. A command that changes data refuses to run when the config
names a different destination.

### `schema_migrations`

The migrations that have run.

- `version`: the number of the migration. Primary key.
- `applied_at`: when it ran. It is written as a date and time, not as seconds.

## How commands use the tables

### Backup

A backup works on the tables in this order:

- It deletes unfinished snapshots, and the rows that no snapshot uses.
- It creates a row in `snapshots`, with `completed_at` set to `NULL`.
- It deletes the `blobs` rows that were never uploaded.
- It walks each path and compares every entry with its row in `files`.
- It links each unchanged file to the snapshot in `snapshot_files`.
- It records each mount point in `snapshot_mount_points`.
- It reads each changed file and stores it. See below.
- It fills `snapshot_blobs` with every blob the snapshot uses.
- It stores the snapshot's totals in `snapshots`.
- It uploads the snapshot's metadata.
- It sets `completed_at`.

A file counts as changed when its row in `files` differs from the file on disk
in any of these:

- size
- modification time
- mode
- owner or group
- link target

A file is also read again when it lists a chunk that no uploaded blob holds.

Storing a changed file writes these rows:

- A row in `blobs`, when packing of a blob starts.
- When the blob is closed, in one transaction:
    - a row in `chunks` for each new chunk in the blob
    - the blob's rows in `blob_chunks`
    - the blob's `blob_hash`, sizes and `finished_ts`
- When the blob is stored at the destination, in one transaction:
    - `uploaded_ts` and `compressed_size` in `blobs`
    - a row in `snapshot_blobs`
    - a row in `uploads`, unless the blob was already at the destination
- The row in `files`, with its `file_chunks`, `chunk_files` and
  `snapshot_files` rows.

The row in `files` is written last. It is written only after every blob that
holds the file's new chunks is stored.

### Snapshot metadata

The metadata of a snapshot is made from a copy of the index:

- The index is copied into a temporary directory with `VACUUM INTO`.
- Every row that does not belong to this snapshot is deleted from the copy.
- `VACUUM` rewrites the copy, so that no deleted row is left in the file.
- The copy is compressed with zstd and encrypted with age.
- The manifest is built from the copy's `snapshot_blobs` table.
- The copy is uploaded as `metadata/<remote-key>/db.zst.age`.
- The manifest is uploaded as `metadata/<remote-key>/manifest.json.zst`.

The copy is made through SQLite, not by copying the file. The index runs in WAL
mode, so rows that are already committed can still be in the `-wal` file beside
it.

### Restore

A restore does not use the local index.

- It downloads and decrypts `db.zst.age` into a temporary directory.
- It opens that file read-only. It applies no migration to it.
- It reads the files to restore from `files`.
- It finds each file's chunks in `file_chunks`.
- It finds the blob and offset of each chunk in `blob_chunks`.

It does not read the manifest.

### Prune

`prune` changes the local index in two ways:

- It deletes each snapshot whose directory is gone from the destination.
- It deletes unfinished snapshots, and the rows that no snapshot uses.

The second step removes unused rows in this order:

- files that are in no snapshot
- blobs that are in no snapshot
- `blob_chunks` rows whose chunk is gone
- chunks that no file uses

`prune` then deletes unused data at the destination. That step reads the
manifests, not the local index. It reads the local index only to keep a
directory under `metadata/` whose snapshot is still recorded there.

## Code

Each table has a repository type in `internal/database`:

- `FileRepository`
- `ChunkRepository`
- `FileChunkRepository`
- `ChunkFileRepository`
- `BlobRepository`
- `BlobChunkRepository`
- `SnapshotRepository`
- `UploadRepository`
- `LocalMetaRepository`

`Repositories` holds one of each.

`Repositories.WithTx` runs a function in one transaction:

```go
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
    // several repository calls that use tx
})
```

The transaction commits when the function returns no error. Otherwise it is
rolled back.

## SQLite settings

- The index runs in WAL mode. A command that only reads, such as
  `snapshot list`, can read the index while a backup writes to it.
- A connection waits up to 10 seconds for a lock before it fails.
- Foreign keys are enforced.
- A process opens one connection, so all its access to the index is in turn.
- Queries pass their values as parameters.