vaultik/docs/DATAMODEL.md

# Vaultik Data Model

## Overview

Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication.

**Important Notes:**

This section is the authoritative explanation of the schema/migration story;
other documents (the README and `AGENTS.md`) link here.

- **No upgrade path between versions (pre-1.0)**: Vaultik has no supported way to
  carry an existing local index across a schema change. The index is disposable
  — if the on-disk schema changes between versions, delete the local SQLite
  database (`vaultik database delete`) and run a full backup. Remote storage is
  unaffected; the new index re-deduplicates against existing remote blobs. This
  is the standing project policy, and it is separate from the schema bootstrap
  described next.
- **Schema bootstrap**: a fresh database is populated from numbered SQL files
  embedded in the binary under `internal/database/schema/`. `000.sql` creates the
  `schema_migrations` table; `001.sql` creates the application tables. On opening
  a database the code applies each numbered file that has not yet run and records
  its version in `schema_migrations`. This bootstraps a new database; it does not
  upgrade an existing one between released versions.
- **Changing the schema (pre-1.0)**: edit `internal/database/schema/001.sql` (and
  the code that touches the affected tables) directly. Do not add new numbered
  files — there is no installed base to migrate.
- **Disposability expires at 1.0**: the index is treated as disposable only until
  1.0 ships and is tagged. Once 1.0 is tagged that clause expires and the
  question of upgrading existing indexes returns. It is deliberately left open
  here.
- **Version Compatibility**: In rare cases, you may need to use the same version
  of Vaultik to restore a backup as was used to create it. This ensures
  compatibility with the metadata format stored in S3.

## Database Tables

### 1. `files`
Stores metadata about files in the filesystem being backed up.

**Columns:**
- `id` (TEXT PRIMARY KEY) - UUID for the file record
- `path` (TEXT NOT NULL UNIQUE) - Absolute file path
- `mtime` (INTEGER NOT NULL) - Modification time as Unix timestamp
- `size` (INTEGER NOT NULL) - File size in bytes
- `mode` (INTEGER NOT NULL) - Unix file permissions and type
- `uid` (INTEGER NOT NULL) - User ID of file owner
- `gid` (INTEGER NOT NULL) - Group ID of file owner
- `link_target` (TEXT) - Symlink target path (NULL for regular files)

**Indexes:**
- `idx_files_path` on `path` for efficient lookups

**Purpose:** Tracks file metadata to detect changes between backup runs. Used for incremental backup decisions. The UUID primary key provides stable references that don't change if files are moved.

### 2. `chunks`
Stores information about content-defined chunks created from files.

**Columns:**
- `chunk_hash` (TEXT PRIMARY KEY) - SHA256 hash of chunk content
- `size` (INTEGER NOT NULL) - Chunk size in bytes

**Purpose:** Enables deduplication by tracking unique chunks across all files.

### 3. `file_chunks`
Maps files to their constituent chunks in order.

**Columns:**
- `file_id` (TEXT) - File ID (FK to files.id)
- `idx` (INTEGER) - Chunk index within file (0-based)
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
- PRIMARY KEY (`file_id`, `idx`)

**Purpose:** Allows reconstruction of files from chunks during restore.

### 4. `chunk_files`
Reverse mapping showing which files contain each chunk.

**Columns:**
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
- `file_id` (TEXT) - File ID (FK to files.id)
- `file_offset` (INTEGER) - Byte offset of chunk within file
- `length` (INTEGER) - Length of chunk in bytes
- PRIMARY KEY (`chunk_hash`, `file_id`)

**Purpose:** Supports efficient queries for chunk usage and deduplication statistics.

### 5. `blobs`
Stores information about packed, compressed, and encrypted blob files.

**Columns:**
- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts
- `blob_hash` (TEXT UNIQUE) - SHA256 hash of final blob (NULL until finalized)
- `created_ts` (INTEGER NOT NULL) - Creation timestamp
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption
- `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded)

**Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs.

### 6. `blob_chunks`
Maps chunks to the blobs that contain them.

**Columns:**
- `blob_id` (TEXT) - Blob ID (FK to blobs.id)
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
- `offset` (INTEGER) - Byte offset of chunk within blob (before compression)
- `length` (INTEGER) - Length of chunk in bytes
- PRIMARY KEY (`blob_id`, `chunk_hash`)

**Purpose:** Enables chunk retrieval from blobs during restore operations.

### 7. `snapshots`
Tracks backup snapshots.

**Columns:**
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: hostname-YYYYMMDD-HHMMSSZ)
- `hostname` (TEXT) - Hostname where backup was created
- `vaultik_version` (TEXT) - Version of Vaultik used
- `vaultik_git_revision` (TEXT) - Git revision of Vaultik used
- `started_at` (INTEGER) - Start timestamp
- `completed_at` (INTEGER) - Completion timestamp (NULL if in progress)
- `file_count` (INTEGER) - Number of files in snapshot
- `chunk_count` (INTEGER) - Number of unique chunks
- `blob_count` (INTEGER) - Number of blobs referenced
- `total_size` (INTEGER) - Total size of all files
- `blob_size` (INTEGER) - Total size of all blobs (compressed)
- `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced blobs
- `compression_ratio` (REAL) - Compression ratio achieved
- `compression_level` (INTEGER) - Compression level used for this snapshot
- `upload_bytes` (INTEGER) - Total bytes uploaded during this snapshot
- `upload_duration_ms` (INTEGER) - Total milliseconds spent uploading to S3

**Purpose:** Provides snapshot metadata and statistics including version tracking for compatibility.

### 8. `snapshot_files`
Maps snapshots to the files they contain.

**Columns:**
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
- `file_id` (TEXT) - File ID (FK to files.id)
- PRIMARY KEY (`snapshot_id`, `file_id`)

**Purpose:** Records which files are included in each snapshot.

### 9. `snapshot_blobs`
Maps snapshots to the blobs they reference.

**Columns:**
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
- `blob_id` (TEXT) - Blob ID (FK to blobs.id)
- `blob_hash` (TEXT) - Denormalized blob hash for manifest generation
- PRIMARY KEY (`snapshot_id`, `blob_id`)

**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation.

### 10. `uploads`
Tracks blob upload metrics.

**Columns:**
- `blob_hash` (TEXT PRIMARY KEY) - Hash of uploaded blob
- `snapshot_id` (TEXT NOT NULL) - The snapshot that triggered this upload (FK to snapshots.id)
- `uploaded_at` (INTEGER) - Upload timestamp
- `size` (INTEGER) - Size of uploaded blob
- `duration_ms` (INTEGER) - Upload duration in milliseconds

**Purpose:** Performance monitoring and tracking which blobs were newly created (uploaded) during each snapshot.

## Data Flow and Operations

### 1. Backup Process

1. **File Scanning**
   - `INSERT OR REPLACE INTO files` - Update file metadata
   - `SELECT * FROM files WHERE path = ?` - Check if file has changed
   - `INSERT INTO snapshot_files` - Add file to current snapshot

2. **Chunking** (for changed files)
   - `INSERT OR IGNORE INTO chunks` - Store new chunks
   - `INSERT INTO file_chunks` - Map chunks to file
   - `INSERT INTO chunk_files` - Create reverse mapping

3. **Blob Packing**
   - `INSERT INTO blobs` - Create blob record with UUID (blob_hash NULL)
   - `INSERT INTO blob_chunks` - Associate chunks with blob immediately
   - `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing

4. **Upload**
   - `UPDATE blobs SET uploaded_ts = ?` - Mark blob as uploaded
   - `INSERT INTO uploads` - Record upload metrics with snapshot_id
   - `INSERT INTO snapshot_blobs` - Associate blob with snapshot

5. **Snapshot Completion**
   - `UPDATE snapshots SET completed_at = ?, stats...` - Finalize snapshot
   - Generate and upload blob manifest from `snapshot_blobs`

### 2. Incremental Backup

1. **Change Detection**
   - `SELECT * FROM files WHERE path = ?` - Get previous file metadata
   - Compare mtime, size, mode to detect changes
   - Skip unchanged files but still add to `snapshot_files`

2. **Chunk Reuse**
   - `SELECT * FROM blob_chunks WHERE chunk_hash = ?` - Find existing chunks
   - `INSERT INTO snapshot_blobs` - Reference existing blobs for unchanged files

### 3. Snapshot Metadata Export

After a snapshot is completed:
1. Copy database to temporary file
2. Clean temporary database to contain only current snapshot data
3. VACUUM the trimmed database so deleted rows leave no pages behind
4. Compress with zstd and encrypt with age
5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`

The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation).

### 4. Restore Process

The restore process doesn't use the local database. Instead:
1. Downloads snapshot metadata from S3
2. Downloads required blobs based on manifest
3. Reconstructs files from decrypted and decompressed chunks

### 5. Pruning

1. **Identify Unreferenced Blobs**
   - Query blobs not referenced by any remaining snapshot
   - Delete from S3 and local database

### 6. Incomplete Snapshot Cleanup

Before each backup:
1. Query incomplete snapshots (where `completed_at IS NULL`)
2. Check if metadata exists in S3
3. If no metadata, delete snapshot and all associations
4. Clean up orphaned files, chunks, and blobs

## Repository Pattern

Vaultik uses a repository pattern for database access:

- `FileRepository` - CRUD operations for files and file metadata
- `ChunkRepository` - CRUD operations for content chunks
- `FileChunkRepository` - Manage file-to-chunk mappings
- `ChunkFileRepository` - Manage chunk-to-file reverse mappings  
- `BlobRepository` - Manage blob lifecycle (creation, finalization, upload)
- `BlobChunkRepository` - Manage blob-to-chunk associations
- `SnapshotRepository` - Manage snapshots and their relationships
- `UploadRepository` - Track blob upload metrics

Each repository provides methods like:
- `Create()` - Insert new record
- `GetByID()` / `GetByPath()` / `GetByHash()` - Retrieve records
- `Update()` - Update existing records
- `Delete()` - Remove records
- Specialized queries for each entity type (e.g., `DeleteOrphaned()`, `GetIncompleteByHostname()`)

## Transaction Management

All database operations that modify multiple tables are wrapped in transactions:

```go
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
    // Multiple repository operations using tx
})
```

This ensures consistency, especially important for operations like:
- Creating file-chunk mappings
- Associating chunks with blobs
- Updating snapshot statistics

## Performance Considerations

1. **Indexes**: 
   - Primary keys are automatically indexed
   - `idx_files_path` on `files(path)` for efficient file lookups

2. **Prepared Statements**: All queries use prepared statements for performance and security

3. **Batch Operations**: Where possible, operations are batched within transactions

4. **Write-Ahead Logging**: SQLite WAL mode is enabled for better concurrency

## Data Integrity

1. **Foreign Keys**: Enforced through CASCADE DELETE and application-level repository methods
2. **Unique Constraints**: Chunk hashes, file paths, and blob hashes are unique
3. **Null Handling**: Nullable fields clearly indicate in-progress operations
4. **Timestamp Tracking**: All major operations record timestamps for auditing