vaultik/docs/DATAMODEL.md

vaultik data model

vaultik keeps a SQLite database on the backed-up machine, called the local index. This document describes its tables and how each command uses them.

The local index has two jobs:

• It records which files, chunks and blobs are already stored. The next backup then uploads only what changed.
• It is the source of each snapshot's own database. A backup uploads a trimmed, encrypted copy of the index with every snapshot.

A restore does not need the local index. It reads the copy that was uploaded with the snapshot.

Terms

• Snapshot: one backup of the paths listed under one name in the config.
• Chunk: a piece of a file's content, named by the SHA-256 hash of that content.
• Blob: one object at the destination. It holds many chunks, compressed and then encrypted.
• Destination: the place the backups are stored.
• Manifest: the unencrypted list of the blobs one snapshot uses.
• Remote key: a hash of the snapshot ID. It names the snapshot's directory at the destination.

Where the index is

The index is the file index.sqlite in the platform data directory. That is ~/.local/share/vaultik/ on Linux and ~/Library/Application Support/vaultik/ on macOS. The index_path setting and the VAULTIK_INDEX_PATH environment variable change the path.

The file is readable only by its owner, because it lists every backed-up path. vaultik sets that mode each time it opens the index.

Schema migrations

vaultik supports schema migrations. They are the numbered SQL files in internal/database/schema/, which are built into the binary.

• 000.sql creates the schema_migrations table. That table records each migration that has run.
• 001.sql creates every other table.

database.New opens the index. It then applies, in order, every migration that the index has not recorded.

Before version 1.0, no migration is added, because nothing is installed anywhere yet. A schema change edits 001.sql directly, with the code that uses the changed tables.

This has one consequence. An index created before such a change has already recorded 001.sql as run, so it keeps the old schema and can become unusable. To rebuild it, run vaultik database delete and then a full backup.

After version 1.0, each schema change is a new numbered file. An existing index is then migrated the first time an updated vaultik opens it. A file that was part of a release is never edited.

Migrations apply to the local index only. A snapshot's database at the destination is read as it was written. Each snapshot records the vaultik version that wrote it.

Tables

Times are whole seconds since the Unix epoch, in UTC. The two exceptions are noted at their columns.

files

One row for each backed-up path: a regular file, a symlink or a directory.

• id: a UUID. Primary key.
• path: the absolute path. Unique.
• source_path: the configured path that the scan found this entry under. A restore does not read it.
• mtime: the modification time.
• mtime_nsec: the nanoseconds within that second.
• size: the size in bytes.
• mode: the permission bits and the file type.
• uid: the owner's user ID.
• gid: the owner's group ID.
• link_target: the target of a symlink. It is NULL for other entries.

idx_files_path indexes path.

There is one row for each path, and it is updated when the file changes. The row is shared by every snapshot in the local index that lists the path. An older snapshot's own view of the file is in that snapshot's database at the destination.

chunks

One row for each distinct chunk.

• chunk_hash: the SHA-256 hash of the chunk's content, in hex. Primary key.
• size: the size in bytes.

file_chunks

The chunks that make up each file, in order.

• file_id: the file. References files.id.
• idx: the position of the chunk within the file, counted from 0.
• chunk_hash: the chunk. References chunks.chunk_hash.

The primary key is (file_id, idx). idx_file_chunks_chunk_hash indexes chunk_hash.

chunk_files

The reverse of file_chunks: the files that contain each chunk.

• chunk_hash: the chunk. References chunks.chunk_hash.
• file_id: the file. References files.id.
• file_offset: the byte offset of the chunk within the file.
• length: the length of the chunk in bytes.

The primary key is (chunk_hash, file_id). idx_chunk_files_file_id indexes file_id.

blobs

One row for each blob.

• id: a UUID, given when packing of the blob starts. Primary key.
• blob_hash: the name of the blob at the destination. Unique. Until the blob is closed, it holds a placeholder that starts with temp-placeholder-.
• created_ts: when packing started.
• finished_ts: when the blob was closed. It is NULL before that.
• uncompressed_size: the total size of the blob's chunks.
• compressed_size: the size after compression and encryption.
• uploaded_ts: when the upload finished. It is NULL before that.

blob_hash is hex(SHA256(SHA256(uncompressed contents))). See REPOSTRUCTURE.md.

Once a blob is uploaded, compressed_size is the size of the object at the destination. A blob that was already there can have a different size from the local copy. That happens when it was written with another compression level or other recipients.

blob_chunks

The chunks that each blob holds.

• blob_id: the blob. References blobs.id.
• chunk_hash: the chunk. References chunks.chunk_hash.
• offset: the byte offset of the chunk within the uncompressed blob.
• length: the length of the chunk in bytes.

The primary key is (blob_id, chunk_hash). idx_blob_chunks_chunk_hash indexes chunk_hash.

A restore finds the blob for each chunk through this table.

snapshots

One row for each snapshot.

• id: the snapshot ID. Primary key.
• hostname: the hostname of the machine that made the backup.
• vaultik_version: the vaultik version that made it.
• vaultik_git_revision: the git commit of that version.
• started_at: when the backup started.
• completed_at: when the backup finished. It is NULL until then.
• file_count: the number of files and symlinks in the snapshot. Directories are not counted.
• chunk_count: the number of chunks this snapshot stored that were not stored before.
• new_chunk_size: the total size of those chunks before compression.
• blob_count: the number of blobs this snapshot created.
• total_size: the total size of all files in the snapshot.
• uncompressed_chunk_size: the total size, before compression, of the distinct chunks the snapshot's files use.
• blob_size: the total compressed size of the blobs the snapshot uses.
• blob_uncompressed_size: the total uncompressed size of those blobs.
• compression_ratio: the compression ratio of those blobs.
• compression_level: the compression level the backup used.
• upload_bytes: the bytes uploaded during the backup.
• upload_duration_ms: the time spent uploading, in milliseconds.

The snapshot ID has the form <hostname>_<snapshot-name>_<timestamp>. An example is server1_home_2025-06-01T12:00:00Z. It has three parts:

• the hostname, up to its first .
• the snapshot name
• the start time, in UTC

snapshot list shows uncompressed_chunk_size as UNCOMPRESSED SIZE and new_chunk_size as NEW CHUNK SIZE.

A snapshot with completed_at set to NULL is unfinished. snapshot list does not show it. The next backup deletes it, and so does prune.

snapshot_files

The files in each snapshot.

• snapshot_id: the snapshot. References snapshots.id.
• file_id: the file. References files.id.

The primary key is (snapshot_id, file_id). idx_snapshot_files_file_id indexes file_id.

snapshot_blobs

The blobs that each snapshot uses.

• snapshot_id: the snapshot. References snapshots.id.
• blob_id: the blob. References blobs.id.
• blob_hash: a copy of the blob's hash.

The primary key is (snapshot_id, blob_id). idx_snapshot_blobs_blob_id indexes blob_id.

The manifest is built from this table.

snapshot_mount_points

The mount points that a snapshot's backup found.

• snapshot_id: the snapshot. References snapshots.id.
• path: the absolute path of the directory.

The primary key is (snapshot_id, path).

A directory is recorded here when all of these hold:

• It is at or below a configured path.
• The backup found an entry in it that is not excluded.
• It is on a different filesystem from its parent directory.

The device number is read after the directory has been listed. Listing an automount point mounts its volume, so the number is that of the volume. An automount point whose volume cannot be mounted is not recorded, because the backup finds no entry in it.

snapshot create --prune reads this table. It skips the prune when an earlier snapshot of the same name recorded a mount point that the new backup did not find. That happens when the volume is not mounted.

uploads

One row for each blob that a backup uploaded.

• blob_hash: the blob. References blobs.blob_hash. Primary key.
• snapshot_id: the snapshot whose backup uploaded it. References snapshots.id.
• uploaded_at: when the upload started. It is written as a date and time, not as seconds.
• size: the size of the uploaded blob.
• duration_ms: the duration of the upload, in milliseconds.

idx_uploads_snapshot_id indexes snapshot_id.

A blob that was already at the destination gets no row here.

local_meta

Settings that belong to this copy of the index.

• key: the name of the setting. Primary key.
• value: its value.

The only key is storage_url. It holds the destination that the index was first used with. A command that changes data refuses to run when the config names a different destination.

schema_migrations

The migrations that have run.

• version: the number of the migration. Primary key.
• applied_at: when it ran. It is written as a date and time, not as seconds.

How commands use the tables

Backup

A backup works on the tables in this order:

• It deletes unfinished snapshots, and the rows that no snapshot uses.
• It creates a row in snapshots, with completed_at set to NULL.
• It deletes the blobs rows that were never uploaded.
• It walks each path and compares every entry with its row in files.
• It links each unchanged file to the snapshot in snapshot_files.
• It records each mount point in snapshot_mount_points.
• It reads each changed file and stores it. See below.
• It fills snapshot_blobs with every blob the snapshot uses.
• It stores the snapshot's totals in snapshots.
• It uploads the snapshot's metadata.
• It sets completed_at.

A file counts as changed when its row in files differs from the file on disk in any of these:

• size
• modification time
• mode
• owner or group
• link target

A file is also read again when it lists a chunk that no uploaded blob holds.

Storing a changed file writes these rows:

• A row in blobs, when packing of a blob starts.
• When the blob is closed, in one transaction:
  • a row in chunks for each new chunk in the blob
  • the blob's rows in blob_chunks
  • the blob's blob_hash, sizes and finished_ts
• When the blob is stored at the destination, in one transaction:
  • uploaded_ts and compressed_size in blobs
  • a row in snapshot_blobs
  • a row in uploads, unless the blob was already at the destination
• The row in files, with its file_chunks, chunk_files and snapshot_files rows.

The row in files is written last. It is written only after every blob that holds the file's new chunks is stored.

Snapshot metadata

The metadata of a snapshot is made from a copy of the index:

• The index is copied into a temporary directory with VACUUM INTO.
• Every row that does not belong to this snapshot is deleted from the copy.
• VACUUM rewrites the copy, so that no deleted row is left in the file.
• The copy is compressed with zstd and encrypted with age.
• The manifest is built from the copy's snapshot_blobs table.
• The copy is uploaded as metadata/<remote-key>/db.zst.age.
• The manifest is uploaded as metadata/<remote-key>/manifest.json.zst.

The copy is made through SQLite, not by copying the file. The index runs in WAL mode, so rows that are already committed can still be in the -wal file beside it.

Restore

A restore does not use the local index.

• It downloads and decrypts db.zst.age into a temporary directory.
• It opens that file read-only. It applies no migration to it.
• It reads the files to restore from files.
• It finds each file's chunks in file_chunks.
• It finds the blob and offset of each chunk in blob_chunks.

It does not read the manifest.

Prune

prune changes the local index in two ways:

• It deletes each snapshot whose directory is gone from the destination.
• It deletes unfinished snapshots, and the rows that no snapshot uses.

The second step removes unused rows in this order:

• files that are in no snapshot
• blobs that are in no snapshot
• blob_chunks rows whose chunk is gone
• chunks that no file uses

prune then deletes unused data at the destination. That step reads the manifests, not the local index. It reads the local index only to keep a directory under metadata/ whose snapshot is still recorded there.

Code

Each table has a repository type in internal/database:

• FileRepository
• ChunkRepository
• FileChunkRepository
• ChunkFileRepository
• BlobRepository
• BlobChunkRepository
• SnapshotRepository
• UploadRepository
• LocalMetaRepository

Repositories holds one of each.

Repositories.WithTx runs a function in one transaction:

err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
    // several repository calls that use tx
})

The transaction commits when the function returns no error. Otherwise it is rolled back.

SQLite settings

• The index runs in WAL mode. A command that only reads, such as snapshot list, can read the index while a backup writes to it.
• A connection waits up to 10 seconds for a lock before it fails.
• Foreign keys are enforced.
• A process opens one connection, so all its access to the index is in turn.
• Queries pass their values as parameters.