diff --git a/docs/DATAMODEL.md b/docs/DATAMODEL.md index 3b197e3..f011a7a 100644 --- a/docs/DATAMODEL.md +++ b/docs/DATAMODEL.md @@ -1,308 +1,405 @@ -# Vaultik Data Model +# vaultik data model -## Overview +`vaultik` keeps a SQLite database on the backed-up machine, called the local +index. This document describes its tables and how each command uses them. -Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication. +The local index has two jobs: -**Important Notes:** +- It records which files, chunks and blobs are already stored. The next backup + then uploads only what changed. +- It is the source of each snapshot's own database. A backup uploads a trimmed, + encrypted copy of the index with every snapshot. -- **Version Compatibility**: In rare cases, you may need to use the same version - of Vaultik to restore a backup as was used to create it. This ensures - compatibility with the metadata format stored in S3. +A restore does not need the local index. It reads the copy that was uploaded +with the snapshot. -## Schema Migrations +## Terms -Vaultik supports schema migrations. They are the numbered SQL files in -`internal/database/schema/`, embedded in the binary: `000.sql` creates the -`schema_migrations` table, which records each migration that has run, and -`001.sql` creates the application tables. `database.New` opens a database and -applies, in order, every migration that database has not yet recorded. +- **Snapshot**: one backup of the paths listed under one name in the config. +- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that + content. +- **Blob**: one object at the destination. It holds many chunks, compressed and + then encrypted. +- **Destination**: the place the backups are stored. +- **Manifest**: the unencrypted list of the blobs one snapshot uses. +- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory + at the destination. -**Before 1.0** no migrations are added, because nothing is installed anywhere -yet. A schema change edits `001.sql` (and the code that uses the affected -tables) directly. A local database created before the change has already -recorded `001.sql` as run, so it keeps the old schema and can become unusable; -`vaultik database delete` followed by a full backup rebuilds it. - -**After 1.0** each schema change is a new numbered file, so an existing local -database is migrated the first time an updated vaultik opens it. A file that -has shipped in a release is never edited. - -## Database Tables - -### 1. `files` -Stores metadata about files in the filesystem being backed up. - -**Columns:** -- `id` (TEXT PRIMARY KEY) - UUID for the file record -- `path` (TEXT NOT NULL UNIQUE) - Absolute file path -- `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix epoch -- `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to 999999999 -- `size` (INTEGER NOT NULL) - File size in bytes -- `mode` (INTEGER NOT NULL) - Unix file permissions and type -- `uid` (INTEGER NOT NULL) - User ID of file owner -- `gid` (INTEGER NOT NULL) - Group ID of file owner -- `link_target` (TEXT) - Symlink target path (NULL for regular files) - -**Indexes:** -- `idx_files_path` on `path` for efficient lookups - -**Purpose:** Tracks file metadata to detect changes between backup runs. Used for incremental backup decisions. The UUID primary key provides stable references that don't change if files are moved. - -### 2. `chunks` -Stores information about content-defined chunks created from files. - -**Columns:** -- `chunk_hash` (TEXT PRIMARY KEY) - SHA256 hash of chunk content -- `size` (INTEGER NOT NULL) - Chunk size in bytes +## Where the index is -**Purpose:** Enables deduplication by tracking unique chunks across all files. - -### 3. `file_chunks` -Maps files to their constituent chunks in order. +The index is the file `index.sqlite` in the platform data directory. That is +`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/` +on macOS. The `index_path` setting and the `VAULTIK_INDEX_PATH` environment +variable change the path. -**Columns:** -- `file_id` (TEXT) - File ID (FK to files.id) -- `idx` (INTEGER) - Chunk index within file (0-based) -- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash) -- PRIMARY KEY (`file_id`, `idx`) +The file is readable only by its owner, because it lists every backed-up path. +`vaultik` sets that mode each time it opens the index. -**Purpose:** Allows reconstruction of files from chunks during restore. +## Schema migrations -### 4. `chunk_files` -Reverse mapping showing which files contain each chunk. - -**Columns:** -- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash) -- `file_id` (TEXT) - File ID (FK to files.id) -- `file_offset` (INTEGER) - Byte offset of chunk within file -- `length` (INTEGER) - Length of chunk in bytes -- PRIMARY KEY (`chunk_hash`, `file_id`) +`vaultik` supports schema migrations. They are the numbered SQL files in +`internal/database/schema/`, which are built into the binary. -**Purpose:** Supports efficient queries for chunk usage and deduplication statistics. +- `000.sql` creates the `schema_migrations` table. That table records each + migration that has run. +- `001.sql` creates every other table. -### 5. `blobs` -Stores information about packed, compressed, and encrypted blob files. +`database.New` opens the index. It then applies, in order, every migration that +the index has not recorded. -**Columns:** -- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts -- `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (NULL until finalized); see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs) -- `created_ts` (INTEGER NOT NULL) - Creation timestamp -- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress) -- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression -- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage: when the blob was already stored, under another compression level or other recipients, this can differ from the size of the local copy -- `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded) +Before version 1.0, no migration is added, because nothing is installed anywhere +yet. A schema change edits `001.sql` directly, with the code that uses the +changed tables. -**Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs. +This has one consequence. An index created before such a change has already +recorded `001.sql` as run, so it keeps the old schema and can become unusable. +To rebuild it, run `vaultik database delete` and then a full backup. -### 6. `blob_chunks` -Maps chunks to the blobs that contain them. - -**Columns:** -- `blob_id` (TEXT) - Blob ID (FK to blobs.id) -- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash) -- `offset` (INTEGER) - Byte offset of chunk within blob (before compression) -- `length` (INTEGER) - Length of chunk in bytes -- PRIMARY KEY (`blob_id`, `chunk_hash`) - -**Purpose:** Enables chunk retrieval from blobs during restore operations. - -### 7. `snapshots` -Tracks backup snapshots. - -**Columns:** -- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g. `server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the snapshot name, and an RFC 3339 UTC timestamp) -- `hostname` (TEXT) - Hostname where backup was created -- `vaultik_version` (TEXT) - Version of Vaultik used -- `vaultik_git_revision` (TEXT) - Git revision of Vaultik used -- `started_at` (INTEGER) - Start timestamp -- `completed_at` (INTEGER) - Completion timestamp (NULL if in progress) -- `file_count` (INTEGER) - Number of files in snapshot -- `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not stored before -- `new_chunk_size` (INTEGER) - Total size of those chunks before compression; `snapshot list` shows it as NEW CHUNK SIZE -- `blob_count` (INTEGER) - Number of blobs this snapshot created -- `total_size` (INTEGER) - Total size of all files -- `uncompressed_chunk_size` (INTEGER) - Total size before compression of the distinct chunks the snapshot's files reference, recorded when its backup ran; `snapshot list` shows it as UNCOMPRESSED SIZE -- `blob_size` (INTEGER) - Total compressed size of all referenced blobs -- `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced blobs -- `compression_ratio` (REAL) - Compression ratio achieved -- `compression_level` (INTEGER) - Compression level used for this snapshot -- `upload_bytes` (INTEGER) - Total bytes uploaded during this snapshot -- `upload_duration_ms` (INTEGER) - Total milliseconds spent uploading to S3 - -**Purpose:** Provides snapshot metadata and statistics including version tracking for compatibility. - -### 8. `snapshot_files` -Maps snapshots to the files they contain. - -**Columns:** -- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) -- `file_id` (TEXT) - File ID (FK to files.id) -- PRIMARY KEY (`snapshot_id`, `file_id`) - -**Purpose:** Records which files are included in each snapshot. - -### 9. `snapshot_blobs` -Maps snapshots to the blobs they reference. - -**Columns:** -- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) -- `blob_id` (TEXT) - Blob ID (FK to blobs.id) -- `blob_hash` (TEXT) - Denormalized blob hash for manifest generation -- PRIMARY KEY (`snapshot_id`, `blob_id`) - -**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation. - -### 10. `snapshot_mount_points` -The directories at or below a configured path, the configured path itself included, that a snapshot's backup found an entry in and, once it had listed them, found on a different filesystem from their parent directory (a different device number), as a mount point is. An automount point whose volume cannot be mounted is not recorded, because the backup finds no entry in it. - -**Columns:** -- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) -- `path` (TEXT) - Absolute path of the directory -- PRIMARY KEY (`snapshot_id`, `path`) - -**Purpose:** Lets `snapshot create --prune` skip the prune when a backup does not find a mount point that an earlier snapshot of the same name found, as when the volume is not mounted. - -### 11. `uploads` -Tracks blob upload metrics. - -**Columns:** -- `blob_hash` (TEXT PRIMARY KEY) - Hash of uploaded blob -- `snapshot_id` (TEXT NOT NULL) - The snapshot that triggered this upload (FK to snapshots.id) -- `uploaded_at` (INTEGER) - Upload timestamp -- `size` (INTEGER) - Size of uploaded blob -- `duration_ms` (INTEGER) - Upload duration in milliseconds - -**Purpose:** Performance monitoring and tracking which blobs were newly created (uploaded) during each snapshot. - -## Data Flow and Operations - -### 1. Backup Process - -1. **File Scanning** - - `INSERT OR REPLACE INTO files` - Update file metadata - - `SELECT * FROM files WHERE path = ?` - Check if file has changed - - `INSERT INTO snapshot_files` - Add file to current snapshot - - `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a different filesystem from its parent that the walk found an entry in - -2. **Chunking** (for changed files) - - `INSERT OR IGNORE INTO chunks` - Store new chunks - - `INSERT INTO file_chunks` - Map chunks to file - - `INSERT INTO chunk_files` - Create reverse mapping - -3. **Blob Packing** - - `INSERT INTO blobs` - Create blob record with UUID (blob_hash NULL) - - `INSERT INTO blob_chunks` - Associate chunks with blob immediately - - `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing - -4. **Upload** - - `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (the upload is skipped when the blob is already stored) - - `INSERT INTO uploads` - Record upload metrics with snapshot_id - - `INSERT INTO snapshot_blobs` - Associate blob with snapshot - -5. **Snapshot Completion** - - `UPDATE snapshots SET completed_at = ?, stats...` - Finalize snapshot - - Generate and upload blob manifest from `snapshot_blobs` - -### 2. Incremental Backup - -1. **Change Detection** - - `SELECT * FROM files WHERE path = ?` - Get previous file metadata - - Compare mtime, size, mode to detect changes - - Re-chunk a file that lists a chunk no uploaded blob holds, even when its metadata is unchanged - - Skip unchanged files but still add to `snapshot_files` - -2. **Chunk Reuse** - - `SELECT * FROM blob_chunks WHERE chunk_hash = ?` - Find existing chunks - - `INSERT INTO snapshot_blobs` - Reference existing blobs for unchanged files - -### 3. Snapshot Metadata Export - -After a snapshot is completed: -1. Copy database to temporary file -2. Clean temporary database to contain only current snapshot data -3. VACUUM the trimmed database so deleted rows leave no pages behind -4. Compress with zstd and encrypt with age -5. Upload to S3 as `metadata/{remote-key}/db.zst.age` -6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst` +After version 1.0, each schema change is a new numbered file. An existing index +is then migrated the first time an updated `vaultik` opens it. A file that was +part of a release is never edited. -The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext. The hash uses no secret, so a guessed hostname and snapshot name can still be confirmed against a listing; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its [Accepted Risks](REPOSTRUCTURE.md#accepted-risks). +Migrations apply to the local index only. A snapshot's database at the +destination is read as it was written. Each snapshot records the `vaultik` +version that wrote it. -### 4. Restore Process +## Tables -The restore process doesn't use the local database. Instead: -1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from S3 -2. Downloads the blobs holding the chunks of the files being restored, found through that database's `blob_chunks` table; the manifest is not read -3. Reconstructs files from decrypted and decompressed chunks +Times are whole seconds since the Unix epoch, in UTC. The two exceptions are +noted at their columns. -### 5. Pruning +### `files` -1. **Identify Unreferenced Blobs** - - Query blobs not referenced by any remaining snapshot - - Delete from S3, once older than `prune_grace_period`, and from the local - database -2. **Delete Metadata Directories With No Manifest** - - Delete each directory under `metadata/` that has no `manifest.json.zst` - and no local snapshot record, once every object in it is older than - `prune_grace_period` +One row for each backed-up path: a regular file, a symlink or a directory. -### 6. Incomplete Snapshot Cleanup +- `id`: a UUID. Primary key. +- `path`: the absolute path. Unique. +- `source_path`: the configured path that the scan found this entry under. A + restore does not read it. +- `mtime`: the modification time. +- `mtime_nsec`: the nanoseconds within that second. +- `size`: the size in bytes. +- `mode`: the permission bits and the file type. +- `uid`: the owner's user ID. +- `gid`: the owner's group ID. +- `link_target`: the target of a symlink. It is `NULL` for other entries. -Before each backup: -1. Query incomplete snapshots (where `completed_at IS NULL`) -2. Delete each one and all its associations, without checking S3 for its metadata -3. Clean up orphaned files, chunks, and blobs +`idx_files_path` indexes `path`. -## Repository Pattern +There is one row for each path, and it is updated when the file changes. The +row is shared by every snapshot in the local index that lists the path. An +older snapshot's own view of the file is in that snapshot's database at the +destination. -Vaultik uses a repository pattern for database access: - -- `FileRepository` - CRUD operations for files and file metadata -- `ChunkRepository` - CRUD operations for content chunks -- `FileChunkRepository` - Manage file-to-chunk mappings -- `ChunkFileRepository` - Manage chunk-to-file reverse mappings -- `BlobRepository` - Manage blob lifecycle (creation, finalization, upload) -- `BlobChunkRepository` - Manage blob-to-chunk associations -- `SnapshotRepository` - Manage snapshots and their relationships -- `UploadRepository` - Track blob upload metrics +### `chunks` -Each repository provides methods like: -- `Create()` - Insert new record -- `GetByID()` / `GetByPath()` / `GetByHash()` - Retrieve records -- `Update()` - Update existing records -- `Delete()` - Remove records -- Specialized queries for each entity type (e.g., `DeleteOrphaned()`, `GetIncompleteByHostname()`) +One row for each distinct chunk. -## Transaction Management +- `chunk_hash`: the SHA-256 hash of the chunk's content, in hex. Primary key. +- `size`: the size in bytes. -All database operations that modify multiple tables are wrapped in transactions: +### `file_chunks` + +The chunks that make up each file, in order. + +- `file_id`: the file. References `files.id`. +- `idx`: the position of the chunk within the file, counted from 0. +- `chunk_hash`: the chunk. References `chunks.chunk_hash`. + +The primary key is (`file_id`, `idx`). `idx_file_chunks_chunk_hash` indexes +`chunk_hash`. + +### `chunk_files` + +The reverse of `file_chunks`: the files that contain each chunk. + +- `chunk_hash`: the chunk. References `chunks.chunk_hash`. +- `file_id`: the file. References `files.id`. +- `file_offset`: the byte offset of the chunk within the file. +- `length`: the length of the chunk in bytes. + +The primary key is (`chunk_hash`, `file_id`). `idx_chunk_files_file_id` indexes +`file_id`. + +### `blobs` + +One row for each blob. + +- `id`: a UUID, given when packing of the blob starts. Primary key. +- `blob_hash`: the name of the blob at the destination. Unique. Until the blob + is closed, it holds a placeholder that starts with `temp-placeholder-`. +- `created_ts`: when packing started. +- `finished_ts`: when the blob was closed. It is `NULL` before that. +- `uncompressed_size`: the total size of the blob's chunks. +- `compressed_size`: the size after compression and encryption. +- `uploaded_ts`: when the upload finished. It is `NULL` before that. + +`blob_hash` is `hex(SHA256(SHA256(uncompressed contents)))`. See +[REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs). + +Once a blob is uploaded, `compressed_size` is the size of the object at the +destination. A blob that was already there can have a different size from the +local copy. That happens when it was written with another compression level or +other recipients. + +### `blob_chunks` + +The chunks that each blob holds. + +- `blob_id`: the blob. References `blobs.id`. +- `chunk_hash`: the chunk. References `chunks.chunk_hash`. +- `offset`: the byte offset of the chunk within the uncompressed blob. +- `length`: the length of the chunk in bytes. + +The primary key is (`blob_id`, `chunk_hash`). `idx_blob_chunks_chunk_hash` +indexes `chunk_hash`. + +A restore finds the blob for each chunk through this table. + +### `snapshots` + +One row for each snapshot. + +- `id`: the snapshot ID. Primary key. +- `hostname`: the hostname of the machine that made the backup. +- `vaultik_version`: the `vaultik` version that made it. +- `vaultik_git_revision`: the git commit of that version. +- `started_at`: when the backup started. +- `completed_at`: when the backup finished. It is `NULL` until then. +- `file_count`: the number of files and symlinks in the snapshot. Directories + are not counted. +- `chunk_count`: the number of chunks this snapshot stored that were not stored + before. +- `new_chunk_size`: the total size of those chunks before compression. +- `blob_count`: the number of blobs this snapshot created. +- `total_size`: the total size of all files in the snapshot. +- `uncompressed_chunk_size`: the total size, before compression, of the + distinct chunks the snapshot's files use. +- `blob_size`: the total compressed size of the blobs the snapshot uses. +- `blob_uncompressed_size`: the total uncompressed size of those blobs. +- `compression_ratio`: the compression ratio of those blobs. +- `compression_level`: the compression level the backup used. +- `upload_bytes`: the bytes uploaded during the backup. +- `upload_duration_ms`: the time spent uploading, in milliseconds. + +The snapshot ID has the form `__`. An +example is `server1_home_2025-06-01T12:00:00Z`. It has three parts: + +- the hostname, up to its first `.` +- the snapshot name +- the start time, in UTC + +`snapshot list` shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE` and +`new_chunk_size` as `NEW CHUNK SIZE`. + +A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list` +does not show it. The next backup deletes it, and so does `prune`. + +### `snapshot_files` + +The files in each snapshot. + +- `snapshot_id`: the snapshot. References `snapshots.id`. +- `file_id`: the file. References `files.id`. + +The primary key is (`snapshot_id`, `file_id`). `idx_snapshot_files_file_id` +indexes `file_id`. + +### `snapshot_blobs` + +The blobs that each snapshot uses. + +- `snapshot_id`: the snapshot. References `snapshots.id`. +- `blob_id`: the blob. References `blobs.id`. +- `blob_hash`: a copy of the blob's hash. + +The primary key is (`snapshot_id`, `blob_id`). `idx_snapshot_blobs_blob_id` +indexes `blob_id`. + +The manifest is built from this table. + +### `snapshot_mount_points` + +The mount points that a snapshot's backup found. + +- `snapshot_id`: the snapshot. References `snapshots.id`. +- `path`: the absolute path of the directory. + +The primary key is (`snapshot_id`, `path`). + +A directory is recorded here when all of these hold: + +- It is at or below a configured path. +- The backup found an entry in it that is not excluded. +- It is on a different filesystem from its parent directory. + +The device number is read after the directory has been listed. Listing an +automount point mounts its volume, so the number is that of the volume. An +automount point whose volume cannot be mounted is not recorded, because the +backup finds no entry in it. + +`snapshot create --prune` reads this table. It skips the prune when an earlier +snapshot of the same name recorded a mount point that the new backup did not +find. That happens when the volume is not mounted. + +### `uploads` + +One row for each blob that a backup uploaded. + +- `blob_hash`: the blob. References `blobs.blob_hash`. Primary key. +- `snapshot_id`: the snapshot whose backup uploaded it. References + `snapshots.id`. +- `uploaded_at`: when the upload started. It is written as a date and time, not + as seconds. +- `size`: the size of the uploaded blob. +- `duration_ms`: the duration of the upload, in milliseconds. + +`idx_uploads_snapshot_id` indexes `snapshot_id`. + +A blob that was already at the destination gets no row here. + +### `local_meta` + +Settings that belong to this copy of the index. + +- `key`: the name of the setting. Primary key. +- `value`: its value. + +The only key is `storage_url`. It holds the destination that the index was +first used with. A command that changes data refuses to run when the config +names a different destination. + +### `schema_migrations` + +The migrations that have run. + +- `version`: the number of the migration. Primary key. +- `applied_at`: when it ran. It is written as a date and time, not as seconds. + +## How commands use the tables + +### Backup + +A backup works on the tables in this order: + +- It deletes unfinished snapshots, and the rows that no snapshot uses. +- It creates a row in `snapshots`, with `completed_at` set to `NULL`. +- It deletes the `blobs` rows that were never uploaded. +- It walks each path and compares every entry with its row in `files`. +- It links each unchanged file to the snapshot in `snapshot_files`. +- It records each mount point in `snapshot_mount_points`. +- It reads each changed file and stores it. See below. +- It fills `snapshot_blobs` with every blob the snapshot uses. +- It stores the snapshot's totals in `snapshots`. +- It uploads the snapshot's metadata. +- It sets `completed_at`. + +A file counts as changed when its row in `files` differs from the file on disk +in any of these: + +- size +- modification time +- mode +- owner or group +- link target + +A file is also read again when it lists a chunk that no uploaded blob holds. + +Storing a changed file writes these rows: + +- A row in `blobs`, when packing of a blob starts. +- When the blob is closed, in one transaction: + - a row in `chunks` for each new chunk in the blob + - the blob's rows in `blob_chunks` + - the blob's `blob_hash`, sizes and `finished_ts` +- When the blob is stored at the destination, in one transaction: + - `uploaded_ts` and `compressed_size` in `blobs` + - a row in `snapshot_blobs` + - a row in `uploads`, unless the blob was already at the destination +- The row in `files`, with its `file_chunks`, `chunk_files` and + `snapshot_files` rows. + +The row in `files` is written last. It is written only after every blob that +holds the file's new chunks is stored. + +### Snapshot metadata + +The metadata of a snapshot is made from a copy of the index: + +- The index is copied into a temporary directory with `VACUUM INTO`. +- Every row that does not belong to this snapshot is deleted from the copy. +- `VACUUM` rewrites the copy, so that no deleted row is left in the file. +- The copy is compressed with zstd and encrypted with age. +- The manifest is built from the copy's `snapshot_blobs` table. +- The copy is uploaded as `metadata//db.zst.age`. +- The manifest is uploaded as `metadata//manifest.json.zst`. + +The copy is made through SQLite, not by copying the file. The index runs in WAL +mode, so rows that are already committed can still be in the `-wal` file beside +it. + +### Restore + +A restore does not use the local index. + +- It downloads and decrypts `db.zst.age` into a temporary directory. +- It opens that file read-only. It applies no migration to it. +- It reads the files to restore from `files`. +- It finds each file's chunks in `file_chunks`. +- It finds the blob and offset of each chunk in `blob_chunks`. + +It does not read the manifest. + +### Prune + +`prune` changes the local index in two ways: + +- It deletes each snapshot whose directory is gone from the destination. +- It deletes unfinished snapshots, and the rows that no snapshot uses. + +The second step removes unused rows in this order: + +- files that are in no snapshot +- blobs that are in no snapshot +- `blob_chunks` rows whose chunk is gone +- chunks that no file uses + +`prune` then deletes unused data at the destination. That step reads the +manifests, not the local index. It reads the local index only to keep a +directory under `metadata/` whose snapshot is still recorded there. + +## Code + +Each table has a repository type in `internal/database`: + +- `FileRepository` +- `ChunkRepository` +- `FileChunkRepository` +- `ChunkFileRepository` +- `BlobRepository` +- `BlobChunkRepository` +- `SnapshotRepository` +- `UploadRepository` +- `LocalMetaRepository` + +`Repositories` holds one of each. + +`Repositories.WithTx` runs a function in one transaction: ```go err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error { - // Multiple repository operations using tx + // several repository calls that use tx }) ``` -This ensures consistency, especially important for operations like: -- Creating file-chunk mappings -- Associating chunks with blobs -- Updating snapshot statistics +The transaction commits when the function returns no error. Otherwise it is +rolled back. -## Performance Considerations +## SQLite settings -1. **Indexes**: - - Primary keys are automatically indexed - - `idx_files_path` on `files(path)` for efficient file lookups - -2. **Prepared Statements**: All queries use prepared statements for performance and security - -3. **Batch Operations**: Where possible, operations are batched within transactions - -4. **Write-Ahead Logging**: The local index runs in SQLite WAL mode with a 10-second busy timeout, so a read-only command such as `snapshot list` can read it while a backup writes to it. Committed rows can sit in the `-wal` file beside the index until a checkpoint, so the metadata export copies the index through SQLite (`VACUUM INTO`), not as a file - -## Data Integrity - -1. **Foreign Keys**: Enforced through CASCADE DELETE and application-level repository methods -2. **Unique Constraints**: Chunk hashes, file paths, and blob hashes are unique -3. **Null Handling**: Nullable fields clearly indicate in-progress operations -4. **Timestamp Tracking**: All major operations record timestamps for auditing \ No newline at end of file +- The index runs in WAL mode. A command that only reads, such as + `snapshot list`, can read the index while a backup writes to it. +- A connection waits up to 10 seconds for a lock before it fails. +- Foreign keys are enforced. +- A process opens one connection, so all its access to the index is in turn. +- Queries pass their values as parameters.