sneak/vaultik
1
0 Fork 0
Code Issues 13 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

Rewrite docs/DATAMODEL.md sentence by sentence

Browse Source 
Rewritten from scratch against internal/database on next. Every table
and column in schema/001.sql is listed, with the indexes.

Added what the old text lacked: files.source_path, the local_meta and
schema_migrations tables, and five indexes. Corrected what it had
wrong: blobs.blob_hash holds a placeholder, not NULL, until the blob is
closed; blob_chunks rows are written when the blob is closed; queries
are parameterised, not prepared.

Model: fable-5-1
This commit is contained in:
sneak authored and clawbot committed 2026-11-11 07:01:54 +00:00
1 parent 5afcd4f6b5
commit 137ef59cae
1 file changed
+317 -301
Download Patch File Download Diff File Expand all files Collapse all files
docs/DATAMODEL.md
+317 -301
View File
@@ -1,389 +1,405 @@
# Vaultik Data Model # vaultik data model
## Overview `vaultik` keeps a SQLite database on the backed-up machine, called the local
index. This document describes its tables and how each command uses them.
Vaultik uses a local SQLite database to track file metadata, chunk mappings, and The local index has two jobs:
blob associations during the backup process. This database serves as an index
for incremental backups and enables efficient deduplication.
**Important Notes:** - It records which files, chunks and blobs are already stored. The next backup
then uploads only what changed.
- It is the source of each snapshot's own database. A backup uploads a trimmed,
encrypted copy of the index with every snapshot.
- **Version Compatibility**: In rare cases, you may need to use the same version A restore does not need the local index. It reads the copy that was uploaded
of Vaultik to restore a backup as was used to create it. This ensures with the snapshot.
compatibility with the metadata format stored in S3.
## Schema Migrations ## Terms
Vaultik supports schema migrations. They are the numbered SQL files in - **Snapshot**: one backup of the paths listed under one name in the config.
`internal/database/schema/`, embedded in the binary: `000.sql` creates the - **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
`schema_migrations` table, which records each migration that has run, and content.
`001.sql` creates the application tables. `database.New` opens a database and - **Blob**: one object at the destination. It holds many chunks, compressed and
applies, in order, every migration that database has not yet recorded. then encrypted.
- **Destination**: the place the backups are stored.
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
at the destination.
**Before 1.0** no migrations are added, because nothing is installed anywhere ## Where the index is
yet. A schema change edits `001.sql` (and the code that uses the affected
tables) directly. A local database created before the change has already
recorded `001.sql` as run, so it keeps the old schema and can become unusable;
`vaultik database delete` followed by a full backup rebuilds it.
**After 1.0** each schema change is a new numbered file, so an existing local The index is the file `index.sqlite` in the platform data directory. That is
database is migrated the first time an updated vaultik opens it. A file that has `~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/`
shipped in a release is never edited. on macOS. The `index_path` setting and the `VAULTIK_INDEX_PATH` environment
variable change the path.
## Database Tables The file is readable only by its owner, because it lists every backed-up path.
`vaultik` sets that mode each time it opens the index.
### 1. `files` ## Schema migrations
Stores metadata about files in the filesystem being backed up. `vaultik` supports schema migrations. They are the numbered SQL files in
`internal/database/schema/`, which are built into the binary.
**Columns:** - `000.sql` creates the `schema_migrations` table. That table records each
migration that has run.
- `001.sql` creates every other table.
- `id` (TEXT PRIMARY KEY) - UUID for the file record `database.New` opens the index. It then applies, in order, every migration that
- `path` (TEXT NOT NULL UNIQUE) - Absolute file path the index has not recorded.
- `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix
epoch
- `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to
999999999
- `size` (INTEGER NOT NULL) - File size in bytes
- `mode` (INTEGER NOT NULL) - Unix file permissions and type
- `uid` (INTEGER NOT NULL) - User ID of file owner
- `gid` (INTEGER NOT NULL) - Group ID of file owner
- `link_target` (TEXT) - Symlink target path (NULL for regular files)
**Indexes:** Before version 1.0, no migration is added, because nothing is installed anywhere
yet. A schema change edits `001.sql` directly, with the code that uses the
changed tables.
- `idx_files_path` on `path` for efficient lookups This has one consequence. An index created before such a change has already
recorded `001.sql` as run, so it keeps the old schema and can become unusable.
To rebuild it, run `vaultik database delete` and then a full backup.
**Purpose:** Tracks file metadata to detect changes between backup runs. Used After version 1.0, each schema change is a new numbered file. An existing index
for incremental backup decisions. The UUID primary key provides stable is then migrated the first time an updated `vaultik` opens it. A file that was
references that don't change if files are moved. part of a release is never edited.
### 2. `chunks` Migrations apply to the local index only. A snapshot's database at the
destination is read as it was written. Each snapshot records the `vaultik`
version that wrote it.
Stores information about content-defined chunks created from files. ## Tables
**Columns:** Times are whole seconds since the Unix epoch, in UTC. The two exceptions are
noted at their columns.
- `chunk_hash` (TEXT PRIMARY KEY) - SHA256 hash of chunk content ### `files`
- `size` (INTEGER NOT NULL) - Chunk size in bytes
**Purpose:** Enables deduplication by tracking unique chunks across all files. One row for each backed-up path: a regular file, a symlink or a directory.
### 3. `file_chunks` - `id`: a UUID. Primary key.
- `path`: the absolute path. Unique.
- `source_path`: the configured path that the scan found this entry under. A
restore does not read it.
- `mtime`: the modification time.
- `mtime_nsec`: the nanoseconds within that second.
- `size`: the size in bytes.
- `mode`: the permission bits and the file type.
- `uid`: the owner's user ID.
- `gid`: the owner's group ID.
- `link_target`: the target of a symlink. It is `NULL` for other entries.
Maps files to their constituent chunks in order. `idx_files_path` indexes `path`.
**Columns:** There is one row for each path, and it is updated when the file changes. The
row is shared by every snapshot in the local index that lists the path. An
older snapshot's own view of the file is in that snapshot's database at the
destination.
- `file_id` (TEXT) - File ID (FK to files.id) ### `chunks`
- `idx` (INTEGER) - Chunk index within file (0-based)
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
- PRIMARY KEY (`file_id`, `idx`)
**Purpose:** Allows reconstruction of files from chunks during restore. One row for each distinct chunk.
### 4. `chunk_files` - `chunk_hash`: the SHA-256 hash of the chunk's content, in hex. Primary key.
- `size`: the size in bytes.
Reverse mapping showing which files contain each chunk. ### `file_chunks`
**Columns:** The chunks that make up each file, in order.
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash) - `file_id`: the file. References `files.id`.
- `file_id` (TEXT) - File ID (FK to files.id) - `idx`: the position of the chunk within the file, counted from 0.
- `file_offset` (INTEGER) - Byte offset of chunk within file - `chunk_hash`: the chunk. References `chunks.chunk_hash`.
- `length` (INTEGER) - Length of chunk in bytes
- PRIMARY KEY (`chunk_hash`, `file_id`)
**Purpose:** Supports efficient queries for chunk usage and deduplication The primary key is (`file_id`, `idx`). `idx_file_chunks_chunk_hash` indexes
statistics. `chunk_hash`.
### 5. `blobs` ### `chunk_files`
Stores information about packed, compressed, and encrypted blob files. The reverse of `file_chunks`: the files that contain each chunk.
**Columns:** - `chunk_hash`: the chunk. References `chunks.chunk_hash`.
- `file_id`: the file. References `files.id`.
- `file_offset`: the byte offset of the chunk within the file.
- `length`: the length of the chunk in bytes.
- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts The primary key is (`chunk_hash`, `file_id`). `idx_chunk_files_file_id` indexes
- `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`, `file_id`.
computed before compression and encryption (NULL until finalized); see
[REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs)
- `created_ts` (INTEGER NOT NULL) - Creation timestamp
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before
compression
- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and
encryption. Once uploaded, the size of the object in storage: when the blob
was already stored, under another compression level or other recipients, this
can differ from the size of the local copy
- `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded)
**Purpose:** Tracks blob lifecycle from creation through upload. The UUID ### `blobs`
primary key allows immediate association of chunks with blobs.
### 6. `blob_chunks` One row for each blob.
Maps chunks to the blobs that contain them. - `id`: a UUID, given when packing of the blob starts. Primary key.
- `blob_hash`: the name of the blob at the destination. Unique. Until the blob
is closed, it holds a placeholder that starts with `temp-placeholder-`.
- `created_ts`: when packing started.
- `finished_ts`: when the blob was closed. It is `NULL` before that.
- `uncompressed_size`: the total size of the blob's chunks.
- `compressed_size`: the size after compression and encryption.
- `uploaded_ts`: when the upload finished. It is `NULL` before that.
**Columns:** `blob_hash` is `hex(SHA256(SHA256(uncompressed contents)))`. See
[REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs).
- `blob_id` (TEXT) - Blob ID (FK to blobs.id) Once a blob is uploaded, `compressed_size` is the size of the object at the
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash) destination. A blob that was already there can have a different size from the
- `offset` (INTEGER) - Byte offset of chunk within blob (before compression) local copy. That happens when it was written with another compression level or
- `length` (INTEGER) - Length of chunk in bytes other recipients.
- PRIMARY KEY (`blob_id`, `chunk_hash`)
**Purpose:** Enables chunk retrieval from blobs during restore operations. ### `blob_chunks`
### 7. `snapshots` The chunks that each blob holds.
Tracks backup snapshots. - `blob_id`: the blob. References `blobs.id`.
- `chunk_hash`: the chunk. References `chunks.chunk_hash`.
- `offset`: the byte offset of the chunk within the uncompressed blob.
- `length`: the length of the chunk in bytes.
**Columns:** The primary key is (`blob_id`, `chunk_hash`). `idx_blob_chunks_chunk_hash`
indexes `chunk_hash`.
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g. A restore finds the blob for each chunk through this table.
`server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the
snapshot name, and an RFC 3339 UTC timestamp)
- `hostname` (TEXT) - Hostname where backup was created
- `vaultik_version` (TEXT) - Version of Vaultik used
- `vaultik_git_revision` (TEXT) - Git revision of Vaultik used
- `started_at` (INTEGER) - Start timestamp
- `completed_at` (INTEGER) - Completion timestamp (NULL if in progress)
- `file_count` (INTEGER) - Number of files in snapshot
- `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not
stored before
- `new_chunk_size` (INTEGER) - Total size of those chunks before compression;
`snapshot list` shows it as NEW CHUNK SIZE
- `blob_count` (INTEGER) - Number of blobs this snapshot created
- `total_size` (INTEGER) - Total size of all files
- `uncompressed_chunk_size` (INTEGER) - Total size before compression of the
distinct chunks the snapshot's files reference, recorded when its backup ran;
`snapshot list` shows it as UNCOMPRESSED SIZE
- `blob_size` (INTEGER) - Total compressed size of all referenced blobs
- `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced
blobs
- `compression_ratio` (REAL) - Compression ratio achieved
- `compression_level` (INTEGER) - Compression level used for this snapshot
- `upload_bytes` (INTEGER) - Total bytes uploaded during this snapshot
- `upload_duration_ms` (INTEGER) - Total milliseconds spent uploading to S3
**Purpose:** Provides snapshot metadata and statistics including version ### `snapshots`
tracking for compatibility.
### 8. `snapshot_files` One row for each snapshot.
Maps snapshots to the files they contain. - `id`: the snapshot ID. Primary key.
- `hostname`: the hostname of the machine that made the backup.
- `vaultik_version`: the `vaultik` version that made it.
- `vaultik_git_revision`: the git commit of that version.
- `started_at`: when the backup started.
- `completed_at`: when the backup finished. It is `NULL` until then.
- `file_count`: the number of files and symlinks in the snapshot. Directories
are not counted.
- `chunk_count`: the number of chunks this snapshot stored that were not stored
before.
- `new_chunk_size`: the total size of those chunks before compression.
- `blob_count`: the number of blobs this snapshot created.
- `total_size`: the total size of all files in the snapshot.
- `uncompressed_chunk_size`: the total size, before compression, of the
distinct chunks the snapshot's files use.
- `blob_size`: the total compressed size of the blobs the snapshot uses.
- `blob_uncompressed_size`: the total uncompressed size of those blobs.
- `compression_ratio`: the compression ratio of those blobs.
- `compression_level`: the compression level the backup used.
- `upload_bytes`: the bytes uploaded during the backup.
- `upload_duration_ms`: the time spent uploading, in milliseconds.
**Columns:** The snapshot ID has the form `<hostname>_<snapshot-name>_<timestamp>`. An
example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) - the hostname, up to its first `.`
- `file_id` (TEXT) - File ID (FK to files.id) - the snapshot name
- PRIMARY KEY (`snapshot_id`, `file_id`) - the start time, in UTC
**Purpose:** Records which files are included in each snapshot. `snapshot list` shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE` and
`new_chunk_size` as `NEW CHUNK SIZE`.
### 9. `snapshot_blobs` A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
does not show it. The next backup deletes it, and so does `prune`.
Maps snapshots to the blobs they reference. ### `snapshot_files`
**Columns:** The files in each snapshot.
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) - `snapshot_id`: the snapshot. References `snapshots.id`.
- `blob_id` (TEXT) - Blob ID (FK to blobs.id) - `file_id`: the file. References `files.id`.
- `blob_hash` (TEXT) - Denormalized blob hash for manifest generation
- PRIMARY KEY (`snapshot_id`, `blob_id`)
**Purpose:** Tracks blob dependencies for snapshots and enables manifest The primary key is (`snapshot_id`, `file_id`). `idx_snapshot_files_file_id`
generation. indexes `file_id`.
### 10. `snapshot_mount_points` ### `snapshot_blobs`
The directories at or below a configured path, the configured path itself The blobs that each snapshot uses.
included, that a snapshot's backup found an entry in and, once it had listed
them, found on a different filesystem from their parent directory (a different
device number), as a mount point is. An automount point whose volume cannot be
mounted is not recorded, because the backup finds no entry in it.
**Columns:**
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
- `path` (TEXT) - Absolute path of the directory
- PRIMARY KEY (`snapshot_id`, `path`)
**Purpose:** Lets `snapshot create --prune` skip the prune when a backup does - `snapshot_id`: the snapshot. References `snapshots.id`.
not find a mount point that an earlier snapshot of the same name found, as when - `blob_id`: the blob. References `blobs.id`.
the volume is not mounted. - `blob_hash`: a copy of the blob's hash.
### 11. `uploads` The primary key is (`snapshot_id`, `blob_id`). `idx_snapshot_blobs_blob_id`
indexes `blob_id`.
Tracks blob upload metrics. The manifest is built from this table.
**Columns:** ### `snapshot_mount_points`
- `blob_hash` (TEXT PRIMARY KEY) - Hash of uploaded blob The mount points that a snapshot's backup found.
- `snapshot_id` (TEXT NOT NULL) - The snapshot that triggered this upload (FK to
snapshots.id)
- `uploaded_at` (INTEGER) - Upload timestamp
- `size` (INTEGER) - Size of uploaded blob
- `duration_ms` (INTEGER) - Upload duration in milliseconds
**Purpose:** Performance monitoring and tracking which blobs were newly created - `snapshot_id`: the snapshot. References `snapshots.id`.
(uploaded) during each snapshot. - `path`: the absolute path of the directory.
## Data Flow and Operations The primary key is (`snapshot_id`, `path`).
### 1. Backup Process A directory is recorded here when all of these hold:
1. **File Scanning** - It is at or below a configured path.
- `INSERT OR REPLACE INTO files` - Update file metadata - The backup found an entry in it that is not excluded.
- `SELECT * FROM files WHERE path = ?` - Check if file has changed - It is on a different filesystem from its parent directory.
- `INSERT INTO snapshot_files` - Add file to current snapshot
- `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a
different filesystem from its parent that the walk found an entry in
2. **Chunking** (for changed files)
- `INSERT OR IGNORE INTO chunks` - Store new chunks
- `INSERT INTO file_chunks` - Map chunks to file
- `INSERT INTO chunk_files` - Create reverse mapping
3. **Blob Packing**
- `INSERT INTO blobs` - Create blob record with UUID (blob_hash NULL)
- `INSERT INTO blob_chunks` - Associate chunks with blob immediately
- `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after
packing
4. **Upload** The device number is read after the directory has been listed. Listing an
- `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as automount point mounts its volume, so the number is that of the volume. An
uploaded, with the size of the object in storage (the upload is skipped automount point whose volume cannot be mounted is not recorded, because the
when the blob is already stored) backup finds no entry in it.
- `INSERT INTO uploads` - Record upload metrics with snapshot_id
- `INSERT INTO snapshot_blobs` - Associate blob with snapshot
5. **Snapshot Completion** `snapshot create --prune` reads this table. It skips the prune when an earlier
- `UPDATE snapshots SET completed_at = ?, stats...` - Finalize snapshot snapshot of the same name recorded a mount point that the new backup did not
- Generate and upload blob manifest from `snapshot_blobs` find. That happens when the volume is not mounted.
### 2. Incremental Backup ### `uploads`
1. **Change Detection** One row for each blob that a backup uploaded.
- `SELECT * FROM files WHERE path = ?` - Get previous file metadata
- Compare mtime, size, mode to detect changes
- Re-chunk a file that lists a chunk no uploaded blob holds, even when its
metadata is unchanged
- Skip unchanged files but still add to `snapshot_files`
2. **Chunk Reuse** - `blob_hash`: the blob. References `blobs.blob_hash`. Primary key.
- `SELECT * FROM blob_chunks WHERE chunk_hash = ?` - Find existing chunks - `snapshot_id`: the snapshot whose backup uploaded it. References
- `INSERT INTO snapshot_blobs` - Reference existing blobs for unchanged `snapshots.id`.
files - `uploaded_at`: when the upload started. It is written as a date and time, not
as seconds.
- `size`: the size of the uploaded blob.
- `duration_ms`: the duration of the upload, in milliseconds.
### 3. Snapshot Metadata Export `idx_uploads_snapshot_id` indexes `snapshot_id`.
After a snapshot is completed: A blob that was already at the destination gets no row here.
1. Copy database to temporary file ### `local_meta`
2. Clean temporary database to contain only current snapshot data
3. VACUUM the trimmed database so deleted rows leave no pages behind
4. Compress with zstd and encrypt with age
5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
6. Generate blob manifest and upload as
`metadata/{remote-key}/manifest.json.zst`
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so Settings that belong to this copy of the index.
the ID is never written to the store in plaintext. The hash uses no secret, so a
guessed hostname and snapshot name can still be confirmed against a listing; see
[REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its
[Accepted Risks](REPOSTRUCTURE.md#accepted-risks).
### 4. Restore Process - `key`: the name of the setting. Primary key.
- `value`: its value.
The restore process doesn't use the local database. Instead: The only key is `storage_url`. It holds the destination that the index was
first used with. A command that changes data refuses to run when the config
1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from names a different destination.
S3
2. Downloads the blobs holding the chunks of the files being restored, found
through that database's `blob_chunks` table; the manifest is not read
3. Reconstructs files from decrypted and decompressed chunks
### 5. Pruning ### `schema_migrations`
1. **Identify Unreferenced Blobs**
- Query blobs not referenced by any remaining snapshot
- Delete from S3, once older than `prune_grace_period`, and from the local
database
2. **Delete Metadata Directories With No Manifest**
- Delete each directory under `metadata/` that has no `manifest.json.zst`
and no local snapshot record, once every object in it is older than
`prune_grace_period`
### 6. Incomplete Snapshot Cleanup The migrations that have run.
Before each backup: - `version`: the number of the migration. Primary key.
- `applied_at`: when it ran. It is written as a date and time, not as seconds.
1. Query incomplete snapshots (where `completed_at IS NULL`)
2. Delete each one and all its associations, without checking S3 for its ## How commands use the tables
metadata
3. Clean up orphaned files, chunks, and blobs ### Backup
## Repository Pattern A backup works on the tables in this order:
Vaultik uses a repository pattern for database access: - It deletes unfinished snapshots, and the rows that no snapshot uses.
- It creates a row in `snapshots`, with `completed_at` set to `NULL`.
- `FileRepository` - CRUD operations for files and file metadata - It deletes the `blobs` rows that were never uploaded.
- `ChunkRepository` - CRUD operations for content chunks - It walks each path and compares every entry with its row in `files`.
- `FileChunkRepository` - Manage file-to-chunk mappings - It links each unchanged file to the snapshot in `snapshot_files`.
- `ChunkFileRepository` - Manage chunk-to-file reverse mappings - It records each mount point in `snapshot_mount_points`.
- `BlobRepository` - Manage blob lifecycle (creation, finalization, upload) - It reads each changed file and stores it. See below.
- `BlobChunkRepository` - Manage blob-to-chunk associations - It fills `snapshot_blobs` with every blob the snapshot uses.
- `SnapshotRepository` - Manage snapshots and their relationships - It stores the snapshot's totals in `snapshots`.
- `UploadRepository` - Track blob upload metrics - It uploads the snapshot's metadata.
- It sets `completed_at`.
Each repository provides methods like:
A file counts as changed when its row in `files` differs from the file on disk
- `Create()` - Insert new record in any of these:
- `GetByID()` / `GetByPath()` / `GetByHash()` - Retrieve records
- `Update()` - Update existing records - size
- `Delete()` - Remove records - modification time
- Specialized queries for each entity type (e.g., `DeleteOrphaned()`, - mode
`GetIncompleteByHostname()`) - owner or group
- link target
## Transaction Management
A file is also read again when it lists a chunk that no uploaded blob holds.
All database operations that modify multiple tables are wrapped in transactions:
Storing a changed file writes these rows:
- A row in `blobs`, when packing of a blob starts.
- When the blob is closed, in one transaction:
- a row in `chunks` for each new chunk in the blob
- the blob's rows in `blob_chunks`
- the blob's `blob_hash`, sizes and `finished_ts`
- When the blob is stored at the destination, in one transaction:
- `uploaded_ts` and `compressed_size` in `blobs`
- a row in `snapshot_blobs`
- a row in `uploads`, unless the blob was already at the destination
- The row in `files`, with its `file_chunks`, `chunk_files` and
`snapshot_files` rows.
The row in `files` is written last. It is written only after every blob that
holds the file's new chunks is stored.
### Snapshot metadata
The metadata of a snapshot is made from a copy of the index:
- The index is copied into a temporary directory with `VACUUM INTO`.
- Every row that does not belong to this snapshot is deleted from the copy.
- `VACUUM` rewrites the copy, so that no deleted row is left in the file.
- The copy is compressed with zstd and encrypted with age.
- The manifest is built from the copy's `snapshot_blobs` table.
- The copy is uploaded as `metadata/<remote-key>/db.zst.age`.
- The manifest is uploaded as `metadata/<remote-key>/manifest.json.zst`.
The copy is made through SQLite, not by copying the file. The index runs in WAL
mode, so rows that are already committed can still be in the `-wal` file beside
it.
### Restore
A restore does not use the local index.
- It downloads and decrypts `db.zst.age` into a temporary directory.
- It opens that file read-only. It applies no migration to it.
- It reads the files to restore from `files`.
- It finds each file's chunks in `file_chunks`.
- It finds the blob and offset of each chunk in `blob_chunks`.
It does not read the manifest.
### Prune
`prune` changes the local index in two ways:
- It deletes each snapshot whose directory is gone from the destination.
- It deletes unfinished snapshots, and the rows that no snapshot uses.
The second step removes unused rows in this order:
- files that are in no snapshot
- blobs that are in no snapshot
- `blob_chunks` rows whose chunk is gone
- chunks that no file uses
`prune` then deletes unused data at the destination. That step reads the
manifests, not the local index. It reads the local index only to keep a
directory under `metadata/` whose snapshot is still recorded there.
## Code
Each table has a repository type in `internal/database`:
- `FileRepository`
- `ChunkRepository`
- `FileChunkRepository`
- `ChunkFileRepository`
- `BlobRepository`
- `BlobChunkRepository`
- `SnapshotRepository`
- `UploadRepository`
- `LocalMetaRepository`
`Repositories` holds one of each.
`Repositories.WithTx` runs a function in one transaction:
```go ```go
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error { err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
// Multiple repository operations using tx // several repository calls that use tx
}) })
``` ```
This ensures consistency, especially important for operations like: The transaction commits when the function returns no error. Otherwise it is
rolled back.
- Creating file-chunk mappings ## SQLite settings
- Associating chunks with blobs
- Updating snapshot statistics
## Performance Considerations - The index runs in WAL mode. A command that only reads, such as
`snapshot list`, can read the index while a backup writes to it.
1. **Indexes**: - A connection waits up to 10 seconds for a lock before it fails.
- Primary keys are automatically indexed - Foreign keys are enforced.
- `idx_files_path` on `files(path)` for efficient file lookups - A process opens one connection, so all its access to the index is in turn.
- Queries pass their values as parameters.
2. **Prepared Statements**: All queries use prepared statements for performance
and security
3. **Batch Operations**: Where possible, operations are batched within
transactions
4. **Write-Ahead Logging**: The local index runs in SQLite WAL mode with a
10-second busy timeout, so a read-only command such as `snapshot list` can
read it while a backup writes to it. Committed rows can sit in the `-wal`
file beside the index until a checkpoint, so the metadata export copies the
index through SQLite (`VACUUM INTO`), not as a file
## Data Integrity
1. **Foreign Keys**: Enforced through CASCADE DELETE and application-level
repository methods
2. **Unique Constraints**: Chunk hashes, file paths, and blob hashes are unique
3. **Null Handling**: Nullable fields clearly indicate in-progress operations
4. **Timestamp Tracking**: All major operations record timestamps for auditing