Rewrite docs/DATAMODEL.md sentence by sentence
Rewritten from scratch against internal/database on next. Every table and column in schema/001.sql is listed, with the indexes. Added what the old text lacked: files.source_path, the local_meta and schema_migrations tables, and five indexes. Corrected what it had wrong: blobs.blob_hash holds a placeholder, not NULL, until the blob is closed; blob_chunks rows are written when the blob is closed; queries are parameterised, not prepared. Model: fable-5-1
This commit is contained in:
+369
-272
@@ -1,308 +1,405 @@
|
|||||||
# Vaultik Data Model
|
# vaultik data model
|
||||||
|
|
||||||
## Overview
|
`vaultik` keeps a SQLite database on the backed-up machine, called the local
|
||||||
|
index. This document describes its tables and how each command uses them.
|
||||||
|
|
||||||
Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication.
|
The local index has two jobs:
|
||||||
|
|
||||||
**Important Notes:**
|
- It records which files, chunks and blobs are already stored. The next backup
|
||||||
|
then uploads only what changed.
|
||||||
|
- It is the source of each snapshot's own database. A backup uploads a trimmed,
|
||||||
|
encrypted copy of the index with every snapshot.
|
||||||
|
|
||||||
- **Version Compatibility**: In rare cases, you may need to use the same version
|
A restore does not need the local index. It reads the copy that was uploaded
|
||||||
of Vaultik to restore a backup as was used to create it. This ensures
|
with the snapshot.
|
||||||
compatibility with the metadata format stored in S3.
|
|
||||||
|
|
||||||
## Schema Migrations
|
## Terms
|
||||||
|
|
||||||
Vaultik supports schema migrations. They are the numbered SQL files in
|
- **Snapshot**: one backup of the paths listed under one name in the config.
|
||||||
`internal/database/schema/`, embedded in the binary: `000.sql` creates the
|
- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
|
||||||
`schema_migrations` table, which records each migration that has run, and
|
content.
|
||||||
`001.sql` creates the application tables. `database.New` opens a database and
|
- **Blob**: one object at the destination. It holds many chunks, compressed and
|
||||||
applies, in order, every migration that database has not yet recorded.
|
then encrypted.
|
||||||
|
- **Destination**: the place the backups are stored.
|
||||||
|
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
|
||||||
|
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
|
||||||
|
at the destination.
|
||||||
|
|
||||||
**Before 1.0** no migrations are added, because nothing is installed anywhere
|
## Where the index is
|
||||||
yet. A schema change edits `001.sql` (and the code that uses the affected
|
|
||||||
tables) directly. A local database created before the change has already
|
|
||||||
recorded `001.sql` as run, so it keeps the old schema and can become unusable;
|
|
||||||
`vaultik database delete` followed by a full backup rebuilds it.
|
|
||||||
|
|
||||||
**After 1.0** each schema change is a new numbered file, so an existing local
|
|
||||||
database is migrated the first time an updated vaultik opens it. A file that
|
|
||||||
has shipped in a release is never edited.
|
|
||||||
|
|
||||||
## Database Tables
|
|
||||||
|
|
||||||
### 1. `files`
|
|
||||||
Stores metadata about files in the filesystem being backed up.
|
|
||||||
|
|
||||||
**Columns:**
|
|
||||||
- `id` (TEXT PRIMARY KEY) - UUID for the file record
|
|
||||||
- `path` (TEXT NOT NULL UNIQUE) - Absolute file path
|
|
||||||
- `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix epoch
|
|
||||||
- `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to 999999999
|
|
||||||
- `size` (INTEGER NOT NULL) - File size in bytes
|
|
||||||
- `mode` (INTEGER NOT NULL) - Unix file permissions and type
|
|
||||||
- `uid` (INTEGER NOT NULL) - User ID of file owner
|
|
||||||
- `gid` (INTEGER NOT NULL) - Group ID of file owner
|
|
||||||
- `link_target` (TEXT) - Symlink target path (NULL for regular files)
|
|
||||||
|
|
||||||
**Indexes:**
|
|
||||||
- `idx_files_path` on `path` for efficient lookups
|
|
||||||
|
|
||||||
**Purpose:** Tracks file metadata to detect changes between backup runs. Used for incremental backup decisions. The UUID primary key provides stable references that don't change if files are moved.
|
|
||||||
|
|
||||||
### 2. `chunks`
|
|
||||||
Stores information about content-defined chunks created from files.
|
|
||||||
|
|
||||||
**Columns:**
|
|
||||||
- `chunk_hash` (TEXT PRIMARY KEY) - SHA256 hash of chunk content
|
|
||||||
- `size` (INTEGER NOT NULL) - Chunk size in bytes
|
|
||||||
|
|
||||||
**Purpose:** Enables deduplication by tracking unique chunks across all files.
|
The index is the file `index.sqlite` in the platform data directory. That is
|
||||||
|
`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/`
|
||||||
### 3. `file_chunks`
|
on macOS. The `index_path` setting and the `VAULTIK_INDEX_PATH` environment
|
||||||
Maps files to their constituent chunks in order.
|
variable change the path.
|
||||||
|
|
||||||
**Columns:**
|
The file is readable only by its owner, because it lists every backed-up path.
|
||||||
- `file_id` (TEXT) - File ID (FK to files.id)
|
`vaultik` sets that mode each time it opens the index.
|
||||||
- `idx` (INTEGER) - Chunk index within file (0-based)
|
|
||||||
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
|
|
||||||
- PRIMARY KEY (`file_id`, `idx`)
|
|
||||||
|
|
||||||
**Purpose:** Allows reconstruction of files from chunks during restore.
|
## Schema migrations
|
||||||
|
|
||||||
### 4. `chunk_files`
|
`vaultik` supports schema migrations. They are the numbered SQL files in
|
||||||
Reverse mapping showing which files contain each chunk.
|
`internal/database/schema/`, which are built into the binary.
|
||||||
|
|
||||||
**Columns:**
|
|
||||||
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
|
|
||||||
- `file_id` (TEXT) - File ID (FK to files.id)
|
|
||||||
- `file_offset` (INTEGER) - Byte offset of chunk within file
|
|
||||||
- `length` (INTEGER) - Length of chunk in bytes
|
|
||||||
- PRIMARY KEY (`chunk_hash`, `file_id`)
|
|
||||||
|
|
||||||
**Purpose:** Supports efficient queries for chunk usage and deduplication statistics.
|
- `000.sql` creates the `schema_migrations` table. That table records each
|
||||||
|
migration that has run.
|
||||||
|
- `001.sql` creates every other table.
|
||||||
|
|
||||||
### 5. `blobs`
|
`database.New` opens the index. It then applies, in order, every migration that
|
||||||
Stores information about packed, compressed, and encrypted blob files.
|
the index has not recorded.
|
||||||
|
|
||||||
**Columns:**
|
Before version 1.0, no migration is added, because nothing is installed anywhere
|
||||||
- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts
|
yet. A schema change edits `001.sql` directly, with the code that uses the
|
||||||
- `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (NULL until finalized); see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs)
|
changed tables.
|
||||||
- `created_ts` (INTEGER NOT NULL) - Creation timestamp
|
|
||||||
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
|
|
||||||
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
|
|
||||||
- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage: when the blob was already stored, under another compression level or other recipients, this can differ from the size of the local copy
|
|
||||||
- `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded)
|
|
||||||
|
|
||||||
**Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs.
|
This has one consequence. An index created before such a change has already
|
||||||
|
recorded `001.sql` as run, so it keeps the old schema and can become unusable.
|
||||||
|
To rebuild it, run `vaultik database delete` and then a full backup.
|
||||||
|
|
||||||
### 6. `blob_chunks`
|
After version 1.0, each schema change is a new numbered file. An existing index
|
||||||
Maps chunks to the blobs that contain them.
|
is then migrated the first time an updated `vaultik` opens it. A file that was
|
||||||
|
part of a release is never edited.
|
||||||
**Columns:**
|
|
||||||
- `blob_id` (TEXT) - Blob ID (FK to blobs.id)
|
|
||||||
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
|
|
||||||
- `offset` (INTEGER) - Byte offset of chunk within blob (before compression)
|
|
||||||
- `length` (INTEGER) - Length of chunk in bytes
|
|
||||||
- PRIMARY KEY (`blob_id`, `chunk_hash`)
|
|
||||||
|
|
||||||
**Purpose:** Enables chunk retrieval from blobs during restore operations.
|
|
||||||
|
|
||||||
### 7. `snapshots`
|
|
||||||
Tracks backup snapshots.
|
|
||||||
|
|
||||||
**Columns:**
|
|
||||||
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g. `server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the snapshot name, and an RFC 3339 UTC timestamp)
|
|
||||||
- `hostname` (TEXT) - Hostname where backup was created
|
|
||||||
- `vaultik_version` (TEXT) - Version of Vaultik used
|
|
||||||
- `vaultik_git_revision` (TEXT) - Git revision of Vaultik used
|
|
||||||
- `started_at` (INTEGER) - Start timestamp
|
|
||||||
- `completed_at` (INTEGER) - Completion timestamp (NULL if in progress)
|
|
||||||
- `file_count` (INTEGER) - Number of files in snapshot
|
|
||||||
- `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not stored before
|
|
||||||
- `new_chunk_size` (INTEGER) - Total size of those chunks before compression; `snapshot list` shows it as NEW CHUNK SIZE
|
|
||||||
- `blob_count` (INTEGER) - Number of blobs this snapshot created
|
|
||||||
- `total_size` (INTEGER) - Total size of all files
|
|
||||||
- `uncompressed_chunk_size` (INTEGER) - Total size before compression of the distinct chunks the snapshot's files reference, recorded when its backup ran; `snapshot list` shows it as UNCOMPRESSED SIZE
|
|
||||||
- `blob_size` (INTEGER) - Total compressed size of all referenced blobs
|
|
||||||
- `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced blobs
|
|
||||||
- `compression_ratio` (REAL) - Compression ratio achieved
|
|
||||||
- `compression_level` (INTEGER) - Compression level used for this snapshot
|
|
||||||
- `upload_bytes` (INTEGER) - Total bytes uploaded during this snapshot
|
|
||||||
- `upload_duration_ms` (INTEGER) - Total milliseconds spent uploading to S3
|
|
||||||
|
|
||||||
**Purpose:** Provides snapshot metadata and statistics including version tracking for compatibility.
|
|
||||||
|
|
||||||
### 8. `snapshot_files`
|
|
||||||
Maps snapshots to the files they contain.
|
|
||||||
|
|
||||||
**Columns:**
|
|
||||||
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
|
|
||||||
- `file_id` (TEXT) - File ID (FK to files.id)
|
|
||||||
- PRIMARY KEY (`snapshot_id`, `file_id`)
|
|
||||||
|
|
||||||
**Purpose:** Records which files are included in each snapshot.
|
|
||||||
|
|
||||||
### 9. `snapshot_blobs`
|
|
||||||
Maps snapshots to the blobs they reference.
|
|
||||||
|
|
||||||
**Columns:**
|
|
||||||
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
|
|
||||||
- `blob_id` (TEXT) - Blob ID (FK to blobs.id)
|
|
||||||
- `blob_hash` (TEXT) - Denormalized blob hash for manifest generation
|
|
||||||
- PRIMARY KEY (`snapshot_id`, `blob_id`)
|
|
||||||
|
|
||||||
**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation.
|
|
||||||
|
|
||||||
### 10. `snapshot_mount_points`
|
|
||||||
The directories at or below a configured path, the configured path itself included, that a snapshot's backup found an entry in and, once it had listed them, found on a different filesystem from their parent directory (a different device number), as a mount point is. An automount point whose volume cannot be mounted is not recorded, because the backup finds no entry in it.
|
|
||||||
|
|
||||||
**Columns:**
|
|
||||||
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
|
|
||||||
- `path` (TEXT) - Absolute path of the directory
|
|
||||||
- PRIMARY KEY (`snapshot_id`, `path`)
|
|
||||||
|
|
||||||
**Purpose:** Lets `snapshot create --prune` skip the prune when a backup does not find a mount point that an earlier snapshot of the same name found, as when the volume is not mounted.
|
|
||||||
|
|
||||||
### 11. `uploads`
|
|
||||||
Tracks blob upload metrics.
|
|
||||||
|
|
||||||
**Columns:**
|
|
||||||
- `blob_hash` (TEXT PRIMARY KEY) - Hash of uploaded blob
|
|
||||||
- `snapshot_id` (TEXT NOT NULL) - The snapshot that triggered this upload (FK to snapshots.id)
|
|
||||||
- `uploaded_at` (INTEGER) - Upload timestamp
|
|
||||||
- `size` (INTEGER) - Size of uploaded blob
|
|
||||||
- `duration_ms` (INTEGER) - Upload duration in milliseconds
|
|
||||||
|
|
||||||
**Purpose:** Performance monitoring and tracking which blobs were newly created (uploaded) during each snapshot.
|
|
||||||
|
|
||||||
## Data Flow and Operations
|
|
||||||
|
|
||||||
### 1. Backup Process
|
|
||||||
|
|
||||||
1. **File Scanning**
|
|
||||||
- `INSERT OR REPLACE INTO files` - Update file metadata
|
|
||||||
- `SELECT * FROM files WHERE path = ?` - Check if file has changed
|
|
||||||
- `INSERT INTO snapshot_files` - Add file to current snapshot
|
|
||||||
- `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a different filesystem from its parent that the walk found an entry in
|
|
||||||
|
|
||||||
2. **Chunking** (for changed files)
|
|
||||||
- `INSERT OR IGNORE INTO chunks` - Store new chunks
|
|
||||||
- `INSERT INTO file_chunks` - Map chunks to file
|
|
||||||
- `INSERT INTO chunk_files` - Create reverse mapping
|
|
||||||
|
|
||||||
3. **Blob Packing**
|
|
||||||
- `INSERT INTO blobs` - Create blob record with UUID (blob_hash NULL)
|
|
||||||
- `INSERT INTO blob_chunks` - Associate chunks with blob immediately
|
|
||||||
- `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing
|
|
||||||
|
|
||||||
4. **Upload**
|
|
||||||
- `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (the upload is skipped when the blob is already stored)
|
|
||||||
- `INSERT INTO uploads` - Record upload metrics with snapshot_id
|
|
||||||
- `INSERT INTO snapshot_blobs` - Associate blob with snapshot
|
|
||||||
|
|
||||||
5. **Snapshot Completion**
|
|
||||||
- `UPDATE snapshots SET completed_at = ?, stats...` - Finalize snapshot
|
|
||||||
- Generate and upload blob manifest from `snapshot_blobs`
|
|
||||||
|
|
||||||
### 2. Incremental Backup
|
|
||||||
|
|
||||||
1. **Change Detection**
|
|
||||||
- `SELECT * FROM files WHERE path = ?` - Get previous file metadata
|
|
||||||
- Compare mtime, size, mode to detect changes
|
|
||||||
- Re-chunk a file that lists a chunk no uploaded blob holds, even when its metadata is unchanged
|
|
||||||
- Skip unchanged files but still add to `snapshot_files`
|
|
||||||
|
|
||||||
2. **Chunk Reuse**
|
|
||||||
- `SELECT * FROM blob_chunks WHERE chunk_hash = ?` - Find existing chunks
|
|
||||||
- `INSERT INTO snapshot_blobs` - Reference existing blobs for unchanged files
|
|
||||||
|
|
||||||
### 3. Snapshot Metadata Export
|
|
||||||
|
|
||||||
After a snapshot is completed:
|
|
||||||
1. Copy database to temporary file
|
|
||||||
2. Clean temporary database to contain only current snapshot data
|
|
||||||
3. VACUUM the trimmed database so deleted rows leave no pages behind
|
|
||||||
4. Compress with zstd and encrypt with age
|
|
||||||
5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
|
|
||||||
6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`
|
|
||||||
|
|
||||||
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext. The hash uses no secret, so a guessed hostname and snapshot name can still be confirmed against a listing; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its [Accepted Risks](REPOSTRUCTURE.md#accepted-risks).
|
Migrations apply to the local index only. A snapshot's database at the
|
||||||
|
destination is read as it was written. Each snapshot records the `vaultik`
|
||||||
|
version that wrote it.
|
||||||
|
|
||||||
### 4. Restore Process
|
## Tables
|
||||||
|
|
||||||
The restore process doesn't use the local database. Instead:
|
Times are whole seconds since the Unix epoch, in UTC. The two exceptions are
|
||||||
1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from S3
|
noted at their columns.
|
||||||
2. Downloads the blobs holding the chunks of the files being restored, found through that database's `blob_chunks` table; the manifest is not read
|
|
||||||
3. Reconstructs files from decrypted and decompressed chunks
|
|
||||||
|
|
||||||
### 5. Pruning
|
### `files`
|
||||||
|
|
||||||
1. **Identify Unreferenced Blobs**
|
One row for each backed-up path: a regular file, a symlink or a directory.
|
||||||
- Query blobs not referenced by any remaining snapshot
|
|
||||||
- Delete from S3, once older than `prune_grace_period`, and from the local
|
|
||||||
database
|
|
||||||
2. **Delete Metadata Directories With No Manifest**
|
|
||||||
- Delete each directory under `metadata/` that has no `manifest.json.zst`
|
|
||||||
and no local snapshot record, once every object in it is older than
|
|
||||||
`prune_grace_period`
|
|
||||||
|
|
||||||
### 6. Incomplete Snapshot Cleanup
|
- `id`: a UUID. Primary key.
|
||||||
|
- `path`: the absolute path. Unique.
|
||||||
|
- `source_path`: the configured path that the scan found this entry under. A
|
||||||
|
restore does not read it.
|
||||||
|
- `mtime`: the modification time.
|
||||||
|
- `mtime_nsec`: the nanoseconds within that second.
|
||||||
|
- `size`: the size in bytes.
|
||||||
|
- `mode`: the permission bits and the file type.
|
||||||
|
- `uid`: the owner's user ID.
|
||||||
|
- `gid`: the owner's group ID.
|
||||||
|
- `link_target`: the target of a symlink. It is `NULL` for other entries.
|
||||||
|
|
||||||
Before each backup:
|
`idx_files_path` indexes `path`.
|
||||||
1. Query incomplete snapshots (where `completed_at IS NULL`)
|
|
||||||
2. Delete each one and all its associations, without checking S3 for its metadata
|
|
||||||
3. Clean up orphaned files, chunks, and blobs
|
|
||||||
|
|
||||||
## Repository Pattern
|
There is one row for each path, and it is updated when the file changes. The
|
||||||
|
row is shared by every snapshot in the local index that lists the path. An
|
||||||
|
older snapshot's own view of the file is in that snapshot's database at the
|
||||||
|
destination.
|
||||||
|
|
||||||
Vaultik uses a repository pattern for database access:
|
### `chunks`
|
||||||
|
|
||||||
- `FileRepository` - CRUD operations for files and file metadata
|
|
||||||
- `ChunkRepository` - CRUD operations for content chunks
|
|
||||||
- `FileChunkRepository` - Manage file-to-chunk mappings
|
|
||||||
- `ChunkFileRepository` - Manage chunk-to-file reverse mappings
|
|
||||||
- `BlobRepository` - Manage blob lifecycle (creation, finalization, upload)
|
|
||||||
- `BlobChunkRepository` - Manage blob-to-chunk associations
|
|
||||||
- `SnapshotRepository` - Manage snapshots and their relationships
|
|
||||||
- `UploadRepository` - Track blob upload metrics
|
|
||||||
|
|
||||||
Each repository provides methods like:
|
One row for each distinct chunk.
|
||||||
- `Create()` - Insert new record
|
|
||||||
- `GetByID()` / `GetByPath()` / `GetByHash()` - Retrieve records
|
|
||||||
- `Update()` - Update existing records
|
|
||||||
- `Delete()` - Remove records
|
|
||||||
- Specialized queries for each entity type (e.g., `DeleteOrphaned()`, `GetIncompleteByHostname()`)
|
|
||||||
|
|
||||||
## Transaction Management
|
- `chunk_hash`: the SHA-256 hash of the chunk's content, in hex. Primary key.
|
||||||
|
- `size`: the size in bytes.
|
||||||
|
|
||||||
All database operations that modify multiple tables are wrapped in transactions:
|
### `file_chunks`
|
||||||
|
|
||||||
|
The chunks that make up each file, in order.
|
||||||
|
|
||||||
|
- `file_id`: the file. References `files.id`.
|
||||||
|
- `idx`: the position of the chunk within the file, counted from 0.
|
||||||
|
- `chunk_hash`: the chunk. References `chunks.chunk_hash`.
|
||||||
|
|
||||||
|
The primary key is (`file_id`, `idx`). `idx_file_chunks_chunk_hash` indexes
|
||||||
|
`chunk_hash`.
|
||||||
|
|
||||||
|
### `chunk_files`
|
||||||
|
|
||||||
|
The reverse of `file_chunks`: the files that contain each chunk.
|
||||||
|
|
||||||
|
- `chunk_hash`: the chunk. References `chunks.chunk_hash`.
|
||||||
|
- `file_id`: the file. References `files.id`.
|
||||||
|
- `file_offset`: the byte offset of the chunk within the file.
|
||||||
|
- `length`: the length of the chunk in bytes.
|
||||||
|
|
||||||
|
The primary key is (`chunk_hash`, `file_id`). `idx_chunk_files_file_id` indexes
|
||||||
|
`file_id`.
|
||||||
|
|
||||||
|
### `blobs`
|
||||||
|
|
||||||
|
One row for each blob.
|
||||||
|
|
||||||
|
- `id`: a UUID, given when packing of the blob starts. Primary key.
|
||||||
|
- `blob_hash`: the name of the blob at the destination. Unique. Until the blob
|
||||||
|
is closed, it holds a placeholder that starts with `temp-placeholder-`.
|
||||||
|
- `created_ts`: when packing started.
|
||||||
|
- `finished_ts`: when the blob was closed. It is `NULL` before that.
|
||||||
|
- `uncompressed_size`: the total size of the blob's chunks.
|
||||||
|
- `compressed_size`: the size after compression and encryption.
|
||||||
|
- `uploaded_ts`: when the upload finished. It is `NULL` before that.
|
||||||
|
|
||||||
|
`blob_hash` is `hex(SHA256(SHA256(uncompressed contents)))`. See
|
||||||
|
[REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs).
|
||||||
|
|
||||||
|
Once a blob is uploaded, `compressed_size` is the size of the object at the
|
||||||
|
destination. A blob that was already there can have a different size from the
|
||||||
|
local copy. That happens when it was written with another compression level or
|
||||||
|
other recipients.
|
||||||
|
|
||||||
|
### `blob_chunks`
|
||||||
|
|
||||||
|
The chunks that each blob holds.
|
||||||
|
|
||||||
|
- `blob_id`: the blob. References `blobs.id`.
|
||||||
|
- `chunk_hash`: the chunk. References `chunks.chunk_hash`.
|
||||||
|
- `offset`: the byte offset of the chunk within the uncompressed blob.
|
||||||
|
- `length`: the length of the chunk in bytes.
|
||||||
|
|
||||||
|
The primary key is (`blob_id`, `chunk_hash`). `idx_blob_chunks_chunk_hash`
|
||||||
|
indexes `chunk_hash`.
|
||||||
|
|
||||||
|
A restore finds the blob for each chunk through this table.
|
||||||
|
|
||||||
|
### `snapshots`
|
||||||
|
|
||||||
|
One row for each snapshot.
|
||||||
|
|
||||||
|
- `id`: the snapshot ID. Primary key.
|
||||||
|
- `hostname`: the hostname of the machine that made the backup.
|
||||||
|
- `vaultik_version`: the `vaultik` version that made it.
|
||||||
|
- `vaultik_git_revision`: the git commit of that version.
|
||||||
|
- `started_at`: when the backup started.
|
||||||
|
- `completed_at`: when the backup finished. It is `NULL` until then.
|
||||||
|
- `file_count`: the number of files and symlinks in the snapshot. Directories
|
||||||
|
are not counted.
|
||||||
|
- `chunk_count`: the number of chunks this snapshot stored that were not stored
|
||||||
|
before.
|
||||||
|
- `new_chunk_size`: the total size of those chunks before compression.
|
||||||
|
- `blob_count`: the number of blobs this snapshot created.
|
||||||
|
- `total_size`: the total size of all files in the snapshot.
|
||||||
|
- `uncompressed_chunk_size`: the total size, before compression, of the
|
||||||
|
distinct chunks the snapshot's files use.
|
||||||
|
- `blob_size`: the total compressed size of the blobs the snapshot uses.
|
||||||
|
- `blob_uncompressed_size`: the total uncompressed size of those blobs.
|
||||||
|
- `compression_ratio`: the compression ratio of those blobs.
|
||||||
|
- `compression_level`: the compression level the backup used.
|
||||||
|
- `upload_bytes`: the bytes uploaded during the backup.
|
||||||
|
- `upload_duration_ms`: the time spent uploading, in milliseconds.
|
||||||
|
|
||||||
|
The snapshot ID has the form `<hostname>_<snapshot-name>_<timestamp>`. An
|
||||||
|
example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
|
||||||
|
|
||||||
|
- the hostname, up to its first `.`
|
||||||
|
- the snapshot name
|
||||||
|
- the start time, in UTC
|
||||||
|
|
||||||
|
`snapshot list` shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE` and
|
||||||
|
`new_chunk_size` as `NEW CHUNK SIZE`.
|
||||||
|
|
||||||
|
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
|
||||||
|
does not show it. The next backup deletes it, and so does `prune`.
|
||||||
|
|
||||||
|
### `snapshot_files`
|
||||||
|
|
||||||
|
The files in each snapshot.
|
||||||
|
|
||||||
|
- `snapshot_id`: the snapshot. References `snapshots.id`.
|
||||||
|
- `file_id`: the file. References `files.id`.
|
||||||
|
|
||||||
|
The primary key is (`snapshot_id`, `file_id`). `idx_snapshot_files_file_id`
|
||||||
|
indexes `file_id`.
|
||||||
|
|
||||||
|
### `snapshot_blobs`
|
||||||
|
|
||||||
|
The blobs that each snapshot uses.
|
||||||
|
|
||||||
|
- `snapshot_id`: the snapshot. References `snapshots.id`.
|
||||||
|
- `blob_id`: the blob. References `blobs.id`.
|
||||||
|
- `blob_hash`: a copy of the blob's hash.
|
||||||
|
|
||||||
|
The primary key is (`snapshot_id`, `blob_id`). `idx_snapshot_blobs_blob_id`
|
||||||
|
indexes `blob_id`.
|
||||||
|
|
||||||
|
The manifest is built from this table.
|
||||||
|
|
||||||
|
### `snapshot_mount_points`
|
||||||
|
|
||||||
|
The mount points that a snapshot's backup found.
|
||||||
|
|
||||||
|
- `snapshot_id`: the snapshot. References `snapshots.id`.
|
||||||
|
- `path`: the absolute path of the directory.
|
||||||
|
|
||||||
|
The primary key is (`snapshot_id`, `path`).
|
||||||
|
|
||||||
|
A directory is recorded here when all of these hold:
|
||||||
|
|
||||||
|
- It is at or below a configured path.
|
||||||
|
- The backup found an entry in it that is not excluded.
|
||||||
|
- It is on a different filesystem from its parent directory.
|
||||||
|
|
||||||
|
The device number is read after the directory has been listed. Listing an
|
||||||
|
automount point mounts its volume, so the number is that of the volume. An
|
||||||
|
automount point whose volume cannot be mounted is not recorded, because the
|
||||||
|
backup finds no entry in it.
|
||||||
|
|
||||||
|
`snapshot create --prune` reads this table. It skips the prune when an earlier
|
||||||
|
snapshot of the same name recorded a mount point that the new backup did not
|
||||||
|
find. That happens when the volume is not mounted.
|
||||||
|
|
||||||
|
### `uploads`
|
||||||
|
|
||||||
|
One row for each blob that a backup uploaded.
|
||||||
|
|
||||||
|
- `blob_hash`: the blob. References `blobs.blob_hash`. Primary key.
|
||||||
|
- `snapshot_id`: the snapshot whose backup uploaded it. References
|
||||||
|
`snapshots.id`.
|
||||||
|
- `uploaded_at`: when the upload started. It is written as a date and time, not
|
||||||
|
as seconds.
|
||||||
|
- `size`: the size of the uploaded blob.
|
||||||
|
- `duration_ms`: the duration of the upload, in milliseconds.
|
||||||
|
|
||||||
|
`idx_uploads_snapshot_id` indexes `snapshot_id`.
|
||||||
|
|
||||||
|
A blob that was already at the destination gets no row here.
|
||||||
|
|
||||||
|
### `local_meta`
|
||||||
|
|
||||||
|
Settings that belong to this copy of the index.
|
||||||
|
|
||||||
|
- `key`: the name of the setting. Primary key.
|
||||||
|
- `value`: its value.
|
||||||
|
|
||||||
|
The only key is `storage_url`. It holds the destination that the index was
|
||||||
|
first used with. A command that changes data refuses to run when the config
|
||||||
|
names a different destination.
|
||||||
|
|
||||||
|
### `schema_migrations`
|
||||||
|
|
||||||
|
The migrations that have run.
|
||||||
|
|
||||||
|
- `version`: the number of the migration. Primary key.
|
||||||
|
- `applied_at`: when it ran. It is written as a date and time, not as seconds.
|
||||||
|
|
||||||
|
## How commands use the tables
|
||||||
|
|
||||||
|
### Backup
|
||||||
|
|
||||||
|
A backup works on the tables in this order:
|
||||||
|
|
||||||
|
- It deletes unfinished snapshots, and the rows that no snapshot uses.
|
||||||
|
- It creates a row in `snapshots`, with `completed_at` set to `NULL`.
|
||||||
|
- It deletes the `blobs` rows that were never uploaded.
|
||||||
|
- It walks each path and compares every entry with its row in `files`.
|
||||||
|
- It links each unchanged file to the snapshot in `snapshot_files`.
|
||||||
|
- It records each mount point in `snapshot_mount_points`.
|
||||||
|
- It reads each changed file and stores it. See below.
|
||||||
|
- It fills `snapshot_blobs` with every blob the snapshot uses.
|
||||||
|
- It stores the snapshot's totals in `snapshots`.
|
||||||
|
- It uploads the snapshot's metadata.
|
||||||
|
- It sets `completed_at`.
|
||||||
|
|
||||||
|
A file counts as changed when its row in `files` differs from the file on disk
|
||||||
|
in any of these:
|
||||||
|
|
||||||
|
- size
|
||||||
|
- modification time
|
||||||
|
- mode
|
||||||
|
- owner or group
|
||||||
|
- link target
|
||||||
|
|
||||||
|
A file is also read again when it lists a chunk that no uploaded blob holds.
|
||||||
|
|
||||||
|
Storing a changed file writes these rows:
|
||||||
|
|
||||||
|
- A row in `blobs`, when packing of a blob starts.
|
||||||
|
- When the blob is closed, in one transaction:
|
||||||
|
- a row in `chunks` for each new chunk in the blob
|
||||||
|
- the blob's rows in `blob_chunks`
|
||||||
|
- the blob's `blob_hash`, sizes and `finished_ts`
|
||||||
|
- When the blob is stored at the destination, in one transaction:
|
||||||
|
- `uploaded_ts` and `compressed_size` in `blobs`
|
||||||
|
- a row in `snapshot_blobs`
|
||||||
|
- a row in `uploads`, unless the blob was already at the destination
|
||||||
|
- The row in `files`, with its `file_chunks`, `chunk_files` and
|
||||||
|
`snapshot_files` rows.
|
||||||
|
|
||||||
|
The row in `files` is written last. It is written only after every blob that
|
||||||
|
holds the file's new chunks is stored.
|
||||||
|
|
||||||
|
### Snapshot metadata
|
||||||
|
|
||||||
|
The metadata of a snapshot is made from a copy of the index:
|
||||||
|
|
||||||
|
- The index is copied into a temporary directory with `VACUUM INTO`.
|
||||||
|
- Every row that does not belong to this snapshot is deleted from the copy.
|
||||||
|
- `VACUUM` rewrites the copy, so that no deleted row is left in the file.
|
||||||
|
- The copy is compressed with zstd and encrypted with age.
|
||||||
|
- The manifest is built from the copy's `snapshot_blobs` table.
|
||||||
|
- The copy is uploaded as `metadata/<remote-key>/db.zst.age`.
|
||||||
|
- The manifest is uploaded as `metadata/<remote-key>/manifest.json.zst`.
|
||||||
|
|
||||||
|
The copy is made through SQLite, not by copying the file. The index runs in WAL
|
||||||
|
mode, so rows that are already committed can still be in the `-wal` file beside
|
||||||
|
it.
|
||||||
|
|
||||||
|
### Restore
|
||||||
|
|
||||||
|
A restore does not use the local index.
|
||||||
|
|
||||||
|
- It downloads and decrypts `db.zst.age` into a temporary directory.
|
||||||
|
- It opens that file read-only. It applies no migration to it.
|
||||||
|
- It reads the files to restore from `files`.
|
||||||
|
- It finds each file's chunks in `file_chunks`.
|
||||||
|
- It finds the blob and offset of each chunk in `blob_chunks`.
|
||||||
|
|
||||||
|
It does not read the manifest.
|
||||||
|
|
||||||
|
### Prune
|
||||||
|
|
||||||
|
`prune` changes the local index in two ways:
|
||||||
|
|
||||||
|
- It deletes each snapshot whose directory is gone from the destination.
|
||||||
|
- It deletes unfinished snapshots, and the rows that no snapshot uses.
|
||||||
|
|
||||||
|
The second step removes unused rows in this order:
|
||||||
|
|
||||||
|
- files that are in no snapshot
|
||||||
|
- blobs that are in no snapshot
|
||||||
|
- `blob_chunks` rows whose chunk is gone
|
||||||
|
- chunks that no file uses
|
||||||
|
|
||||||
|
`prune` then deletes unused data at the destination. That step reads the
|
||||||
|
manifests, not the local index. It reads the local index only to keep a
|
||||||
|
directory under `metadata/` whose snapshot is still recorded there.
|
||||||
|
|
||||||
|
## Code
|
||||||
|
|
||||||
|
Each table has a repository type in `internal/database`:
|
||||||
|
|
||||||
|
- `FileRepository`
|
||||||
|
- `ChunkRepository`
|
||||||
|
- `FileChunkRepository`
|
||||||
|
- `ChunkFileRepository`
|
||||||
|
- `BlobRepository`
|
||||||
|
- `BlobChunkRepository`
|
||||||
|
- `SnapshotRepository`
|
||||||
|
- `UploadRepository`
|
||||||
|
- `LocalMetaRepository`
|
||||||
|
|
||||||
|
`Repositories` holds one of each.
|
||||||
|
|
||||||
|
`Repositories.WithTx` runs a function in one transaction:
|
||||||
|
|
||||||
```go
|
```go
|
||||||
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
|
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
|
||||||
// Multiple repository operations using tx
|
// several repository calls that use tx
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
This ensures consistency, especially important for operations like:
|
The transaction commits when the function returns no error. Otherwise it is
|
||||||
- Creating file-chunk mappings
|
rolled back.
|
||||||
- Associating chunks with blobs
|
|
||||||
- Updating snapshot statistics
|
|
||||||
|
|
||||||
## Performance Considerations
|
## SQLite settings
|
||||||
|
|
||||||
1. **Indexes**:
|
- The index runs in WAL mode. A command that only reads, such as
|
||||||
- Primary keys are automatically indexed
|
`snapshot list`, can read the index while a backup writes to it.
|
||||||
- `idx_files_path` on `files(path)` for efficient file lookups
|
- A connection waits up to 10 seconds for a lock before it fails.
|
||||||
|
- Foreign keys are enforced.
|
||||||
2. **Prepared Statements**: All queries use prepared statements for performance and security
|
- A process opens one connection, so all its access to the index is in turn.
|
||||||
|
- Queries pass their values as parameters.
|
||||||
3. **Batch Operations**: Where possible, operations are batched within transactions
|
|
||||||
|
|
||||||
4. **Write-Ahead Logging**: The local index runs in SQLite WAL mode with a 10-second busy timeout, so a read-only command such as `snapshot list` can read it while a backup writes to it. Committed rows can sit in the `-wal` file beside the index until a checkpoint, so the metadata export copies the index through SQLite (`VACUUM INTO`), not as a file
|
|
||||||
|
|
||||||
## Data Integrity
|
|
||||||
|
|
||||||
1. **Foreign Keys**: Enforced through CASCADE DELETE and application-level repository methods
|
|
||||||
2. **Unique Constraints**: Chunk hashes, file paths, and blob hashes are unique
|
|
||||||
3. **Null Handling**: Nullable fields clearly indicate in-progress operations
|
|
||||||
4. **Timestamp Tracking**: All major operations record timestamps for auditing
|
|
||||||
|
|||||||
Reference in New Issue
Block a user