Run make fmt over the Markdown

Only prettier's output; no text change.

Model: opus-5-5
This commit is contained in:
sneak committed 2026-11-10 20:02:10 +00:00
1 parent 7a3cd581b2
commit 514e4f9f8b
6 files changed
+2236 -2047

No files matched your search

+114 -39
View File
@@ -1,10 +1,14 @@
# Vaultik Architecture
This document describes the internal architecture of Vaultik, focusing on the data model, type instantiation, and the relationships between core modules.
This document describes the internal architecture of Vaultik, focusing on the
data model, type instantiation, and the relationships between core modules.
## Overview
Vaultik is a backup system that uses content-defined chunking for deduplication and packs chunks into large, compressed, encrypted blobs for efficient cloud storage. The system is built around dependency injection using [uber-go/fx](https://github.com/uber-go/fx).
Vaultik is a backup system that uses content-defined chunking for deduplication
and packs chunks into large, compressed, encrypted blobs for efficient cloud
storage. The system is built around dependency injection using
[uber-go/fx](https://github.com/uber-go/fx).
## Data Flow
@@ -53,57 +57,88 @@ The database tracks five primary entities and their relationships:
### Entity Descriptions
#### File (`database.File`)
Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:
Represents a file, directory, or symlink in the backup system. Stores metadata
needed for restoration:
- Path, mtime
- Size, mode, ownership (uid, gid)
- Symlink target (if applicable)
It also stores `source_path`, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it.
It also stores `source_path`, the source directory the scan found it under, made
absolute and with symlinks resolved. Restore does not read it.
#### Chunk (`database.Chunk`)
A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm:
A content-addressed unit of data. Files are split into variable-size chunks
using the FastCDC algorithm:
- `ChunkHash`: SHA256 hash of chunk content (primary key)
- `Size`: Chunk size in bytes
Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for the 10MB default average).
Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for
the 10MB default average).
#### FileChunk (`database.FileChunk`)
Maps files to their constituent chunks:
- `FileID`: Reference to the file
- `Idx`: Position of this chunk within the file (0-indexed)
- `ChunkHash`: Reference to the chunk
#### Blob (`database.Blob`)
The final storage unit uploaded to S3. Contains many compressed and encrypted chunks:
The final storage unit uploaded to S3. Contains many compressed and encrypted
chunks:
- `ID`: UUID assigned at creation
- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs))
- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before
compression and encryption (see
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs))
- `UncompressedSize`: Total raw chunk data before compression
- `CompressedSize`: Size after zstd compression and age encryption
- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
Blob creation process:
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted
3. Concurrently, the same bytes are compressed with zstd, then encrypted with
age (recipients configured in config), and written to a temporary file
4. On finalize, the blob's name is the double SHA-256 of the uncompressed
contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed,
encrypted bytes
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and
then deleted
A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`.
A backup needs free temporary space, because each blob is written whole to a
temporary file before it is uploaded (up to about `blob_size_limit`; an rclone
destination that cannot stream uploads needs about twice that) and the metadata
export writes copies of the local index. Temporary files go to `$TMPDIR`
(default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to
`/var/tmp`.
#### BlobChunk (`database.BlobChunk`)
Maps chunks to their position within blobs:
- `BlobID`: Reference to the blob
- `ChunkHash`: Reference to the chunk
- `Offset`: Byte offset within the uncompressed blob
- `Length`: Chunk size
#### Snapshot (`database.Snapshot`)
Represents a point-in-time backup:
- `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g. `server1_home_2025-06-01T12:00:00Z`)
- `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g.
`server1_home_2025-06-01T12:00:00Z`)
- Tracks file count, chunk count, blob count, sizes, compression ratio
- `CompletedAt`: Null until snapshot finishes successfully
#### SnapshotFile / SnapshotBlob
Join tables linking snapshots to their files and blobs.
### Relationship Summary
@@ -140,21 +175,27 @@ fx.New(
### Key Type Instantiation Points
#### 1. Config (`config.Config`)
- **Created by**: `config.Module` via `config.LoadConfig()`
- **When**: Application startup (fx DI)
- **Contains**: All configuration from YAML file (S3 credentials, encryption keys, paths, etc.)
- **Contains**: All configuration from YAML file (S3 credentials, encryption
keys, paths, etc.)
#### 2. Database (`database.DB`)
- **Created by**: `database.Module` via `database.New()`
- **When**: Application startup (fx DI)
- **Contains**: SQLite connection, path reference
#### 3. Repositories (`database.Repositories`)
- **Created by**: `database.Module` via `database.NewRepositories()`
- **When**: Application startup (fx DI)
- **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots, etc.)
- **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots,
etc.)
#### 4. Vaultik (`vaultik.Vaultik`)
- **Created by**: `vaultik.New(VaultikParams)`
- **When**: Application startup (fx DI)
- **Contains**: All dependencies for backup operations
@@ -176,11 +217,13 @@ type Vaultik struct {
```
#### 5. SnapshotManager (`snapshot.SnapshotManager`)
- **Created by**: `snapshot.Module` via `snapshot.NewSnapshotManager()`
- **When**: Application startup (fx DI)
- **Responsibility**: Creates/completes snapshots, exports metadata to S3
#### 6. Scanner (`snapshot.Scanner`)
- **Created by**: `ScannerFactory(ScannerParams)`
- **When**: Each `CreateSnapshot()` call
- **Contains**: Chunker, Packer, progress reporter
@@ -194,20 +237,22 @@ scanner := v.ScannerFactory(snapshot.ScannerParams{
```
#### 7. Chunker (`chunker.Chunker`)
- **Created by**: `chunker.NewChunker(avgChunkSize)`
- **When**: Inside `snapshot.NewScanner()`
- **Configuration**:
- `avgChunkSize`: From config (default 10MB)
- `minChunkSize`: avgChunkSize / 4
- `maxChunkSize`: avgChunkSize * 4
- `avgChunkSize`: From config (default 10MB)
- `minChunkSize`: avgChunkSize / 4
- `maxChunkSize`: avgChunkSize \* 4
#### 8. Packer (`blob.Packer`)
- **Created by**: `blob.NewPacker(PackerConfig)`
- **When**: Inside `snapshot.NewScanner()`
- **Configuration**:
- `MaxBlobSize`: Maximum blob size before finalization (typically 10GB)
- `CompressionLevel`: zstd level (1-19)
- `Recipients`: age public keys for encryption
- `MaxBlobSize`: Maximum blob size before finalization (typically 10GB)
- `CompressionLevel`: zstd level (1-19)
- `Recipients`: age public keys for encryption
```go
// snapshot/scanner.go: NewScanner()
@@ -224,17 +269,22 @@ packer, err := blob.NewPacker(packerCfg)
## Module Responsibilities
### `internal/cli`
Entry point for fx application. Combines all modules and handles signal interrupts.
Entry point for fx application. Combines all modules and handles signal
interrupts.
Key functions:
- `NewApp(AppOptions)` → Creates fx.App with all modules
- `RunApp(ctx, app)` → Starts app, handles graceful shutdown
- `RunWithApp(ctx, opts)` → Convenience wrapper
### `internal/vaultik`
Main orchestrator containing all dependencies and command implementations.
Key methods:
- `New(VaultikParams)` → Constructor (fx DI)
- `CreateSnapshot(opts)` → Main backup operation
- `ListSnapshots(jsonOutput)` → List available snapshots
@@ -242,27 +292,34 @@ Key methods:
- `PurgeSnapshots(...)` → Remove old snapshots
### `internal/chunker`
Content-defined chunking using FastCDC algorithm.
Key types:
- `Chunk` → Hash, Data, Offset, Size
- `Chunker` → avgChunkSize, minChunkSize, maxChunkSize
Key methods:
- `NewChunker(avgChunkSize)` → Constructor
- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback (preferred)
- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback
(preferred)
- `ChunkReader(reader)` → Return all chunks at once (memory-intensive)
### `internal/blob`
Blob packing: accumulates chunks, compresses, encrypts, tracks metadata.
Key types:
- `Packer` → Thread-safe blob accumulator
- `ChunkRef` → Hash + Data for adding to packer
- `FinishedBlob` → Completed blob ready for upload
- `BlobWithReader` → FinishedBlob + io.Reader for streaming upload
Key methods:
- `NewPacker(PackerConfig)` → Constructor
- `AddChunk(ChunkRef)` → Add chunk to current blob
- `FinalizeBlob()` → Compress, encrypt, hash current blob
@@ -272,35 +329,48 @@ Key methods:
### `internal/snapshot`
#### Scanner
Orchestrates the backup process for a directory.
Key methods:
- `NewScanner(ScannerConfig)` → Constructor (creates Chunker + Packer)
- `Scan(ctx, path, snapshotID)` → Main scan operation
Scan phases:
1. **Phase 1**: Walk directory, identify files needing processing
2. **Phase 1b**: Count as deleted the files that the previous snapshot of the same name lists, the walk did not find, and are gone from disk; a file that is excluded, or inside an excluded directory, is not counted
2. **Phase 1b**: Count as deleted the files that the previous snapshot of the
same name lists, the walk did not find, and are gone from disk; a file that
is excluded, or inside an excluded directory, is not counted
3. **Phase 2**: Process files (chunk → pack → upload)
#### SnapshotManager
Manages snapshot lifecycle and metadata export.
Key methods:
- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record
- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot references
- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot
references
- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3
- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a successful export
- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark complete (no export between)
- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a
successful export
- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark
complete (no export between)
### `internal/database`
SQLite database for local index. Single-writer mode for thread safety.
Key types:
- `DB` → Database connection wrapper
- `Repositories` → Collection of all repository interfaces
Repository interfaces:
- `FilesRepository` → CRUD for File records
- `ChunksRepository` → CRUD for Chunk records
- `BlobsRepository` → CRUD for Blob records
@@ -359,11 +429,16 @@ CreateSnapshot(opts)
## Deduplication Strategy
1. **File-level**: Files and symlinks unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked
1. **File-level**: Files and symlinks unchanged since last backup are skipped
(metadata comparison: size, mtime, mode, uid, gid, and a symlink's target),
unless the file lists a chunk that no uploaded blob holds; such a file is
re-chunked
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded.
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash
already exists in the database, the chunk data is not re-uploaded.
3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a blob are skipped.
3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a
blob are skipped.
## Storage Layout in S3
@@ -382,17 +457,16 @@ bucket/
└── manifest.json.zst # Blob list (for pruning/verification)
```
A destination that answers a look-up of `vaultik-destination.txt` with not
found is treated as not present while the local index holds snapshots stored
there (`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
A destination that answers a look-up of `vaultik-destination.txt` with not found
is treated as not present while the local index holds snapshots stored there
(`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
`snapshot create` refuses to write to it, `prune`, `snapshot purge` and
`remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a
destination that cannot be listed. This keeps those commands from taking an
unplugged volume's empty mount point for a destination whose snapshots are
all gone. When the look-up fails for any reason other than not found, for
example when S3 refuses it to credentials that allow only PUT, the check is
skipped, so a host with such credentials does not notice an emptied
destination; see
unplugged volume's empty mount point for a destination whose snapshots are all
gone. When the look-up fails for any reason other than not found, for example
when S3 refuses it to credentials that allow only PUT, the check is skipped, so
a host with such credentials does not notice an emptied destination; see
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt).
The `file://` backend never creates its destination directory, so a write to a
missing one fails.
@@ -407,5 +481,6 @@ derivation and a worked example.
- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
- `Scanner`: Uses `packerMu` mutex to coordinate blob finalization.
- `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread safety.
- `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread
safety.
- `Repositories.WithTx()`: Handles transaction lifecycle automatically.