Format Markdown with prettier and move documents out of the root (closes #411)
check / check (push) Successful in 13m11s
check / check (push) Successful in 13m11s
REPO_POLICIES.md names prettier as the Markdown formatter and keeps documents out of the root. script/fmt and script/fmt-check now run prettier over all Markdown (4-space indents, proseWrap always) besides gofmt, with node and yarn installed by script/bootstrap as in sneak/prompts. All Markdown is reformatted with no text change. ARCHITECTURE.md and TODO.md move to docs/, config.example.yml to configs/, and every reference follows. Model: opus-5-5
This commit was merged in pull request #415.
This commit is contained in:
clawbot committed
1 parent
9236c0727c
commit
aae7fa17ae
17 files changed
+2592
-2257
No files matched your search
@@ -0,0 +1,486 @@
|
||||
# Vaultik Architecture
|
||||
|
||||
This document describes the internal architecture of Vaultik, focusing on the
|
||||
data model, type instantiation, and the relationships between core modules.
|
||||
|
||||
## Overview
|
||||
|
||||
Vaultik is a backup system that uses content-defined chunking for deduplication
|
||||
and packs chunks into large, compressed, encrypted blobs for efficient cloud
|
||||
storage. The system is built around dependency injection using
|
||||
[uber-go/fx](https://github.com/uber-go/fx).
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
Source Files
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Scanner │ Walks directories, detects changed files
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Chunker │ Splits files into variable-size chunks (FastCDC)
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Packer │ Accumulates chunks, compresses (zstd), encrypts (age)
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ S3 Client │ Uploads blobs to remote storage
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
## Data Model
|
||||
|
||||
### Core Entities
|
||||
|
||||
The database tracks five primary entities and their relationships:
|
||||
|
||||
```
|
||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ Snapshot │────▶│ File │────▶│ Chunk │
|
||||
└──────────────┘ └──────────────┘ └──────────────┘
|
||||
│ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────┐ ┌──────────────┐
|
||||
│ Blob │◀─────────────────────────│ BlobChunk │
|
||||
└──────────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
### Entity Descriptions
|
||||
|
||||
#### File (`database.File`)
|
||||
|
||||
Represents a file, directory, or symlink in the backup system. Stores metadata
|
||||
needed for restoration:
|
||||
|
||||
- Path, mtime
|
||||
- Size, mode, ownership (uid, gid)
|
||||
- Symlink target (if applicable)
|
||||
|
||||
It also stores `source_path`, the source directory the scan found it under, made
|
||||
absolute and with symlinks resolved. Restore does not read it.
|
||||
|
||||
#### Chunk (`database.Chunk`)
|
||||
|
||||
A content-addressed unit of data. Files are split into variable-size chunks
|
||||
using the FastCDC algorithm:
|
||||
|
||||
- `ChunkHash`: SHA256 hash of chunk content (primary key)
|
||||
- `Size`: Chunk size in bytes
|
||||
|
||||
Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for
|
||||
the 10MB default average).
|
||||
|
||||
#### FileChunk (`database.FileChunk`)
|
||||
|
||||
Maps files to their constituent chunks:
|
||||
|
||||
- `FileID`: Reference to the file
|
||||
- `Idx`: Position of this chunk within the file (0-indexed)
|
||||
- `ChunkHash`: Reference to the chunk
|
||||
|
||||
#### Blob (`database.Blob`)
|
||||
|
||||
The final storage unit uploaded to S3. Contains many compressed and encrypted
|
||||
chunks:
|
||||
|
||||
- `ID`: UUID assigned at creation
|
||||
- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before
|
||||
compression and encryption (see
|
||||
[docs/REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs))
|
||||
- `UncompressedSize`: Total raw chunk data before compression
|
||||
- `CompressedSize`: Size after zstd compression and age encryption
|
||||
- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
|
||||
|
||||
Blob creation process:
|
||||
|
||||
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
|
||||
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
|
||||
3. Concurrently, the same bytes are compressed with zstd, then encrypted with
|
||||
age (recipients configured in config), and written to a temporary file
|
||||
4. On finalize, the blob's name is the double SHA-256 of the uncompressed
|
||||
contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed,
|
||||
encrypted bytes
|
||||
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and
|
||||
then deleted
|
||||
|
||||
A backup needs free temporary space, because each blob is written whole to a
|
||||
temporary file before it is uploaded (up to about `blob_size_limit`; an rclone
|
||||
destination that cannot stream uploads needs about twice that) and the metadata
|
||||
export writes copies of the local index. Temporary files go to `$TMPDIR`
|
||||
(default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to
|
||||
`/var/tmp`.
|
||||
|
||||
#### BlobChunk (`database.BlobChunk`)
|
||||
|
||||
Maps chunks to their position within blobs:
|
||||
|
||||
- `BlobID`: Reference to the blob
|
||||
- `ChunkHash`: Reference to the chunk
|
||||
- `Offset`: Byte offset within the uncompressed blob
|
||||
- `Length`: Chunk size
|
||||
|
||||
#### Snapshot (`database.Snapshot`)
|
||||
|
||||
Represents a point-in-time backup:
|
||||
|
||||
- `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g.
|
||||
`server1_home_2025-06-01T12:00:00Z`)
|
||||
- Tracks file count, chunk count, blob count, sizes, compression ratio
|
||||
- `CompletedAt`: Null until snapshot finishes successfully
|
||||
|
||||
#### SnapshotFile / SnapshotBlob
|
||||
|
||||
Join tables linking snapshots to their files and blobs.
|
||||
|
||||
### Relationship Summary
|
||||
|
||||
```
|
||||
Snapshot 1──────────▶ N SnapshotFile N ◀────────── 1 File
|
||||
Snapshot 1──────────▶ N SnapshotBlob N ◀────────── 1 Blob
|
||||
File 1──────────▶ N FileChunk N ◀────────── 1 Chunk
|
||||
Blob 1──────────▶ N BlobChunk N ◀────────── 1 Chunk
|
||||
```
|
||||
|
||||
## Type Instantiation
|
||||
|
||||
### Application Startup
|
||||
|
||||
The CLI uses fx for dependency injection. Here's the instantiation order:
|
||||
|
||||
```go
|
||||
// cli/app.go: NewApp()
|
||||
fx.New(
|
||||
fx.Supply(config.Path(opts.ConfigPath)), // 1. Config path
|
||||
fx.Supply(opts.LogOptions), // 2. Log options
|
||||
fx.Provide(globals.New), // 3. Globals
|
||||
fx.Provide(log.New), // 4. Logger config
|
||||
config.Module, // 5. Config
|
||||
database.Module, // 6. Database + Repositories
|
||||
log.Module, // 7. Logger initialization
|
||||
storage.Module, // 8. Storage backend (S3/file/rclone)
|
||||
snapshot.Module, // 9. SnapshotManager + ScannerFactory
|
||||
fx.Provide(vaultik.New), // 10. Vaultik orchestrator
|
||||
)
|
||||
```
|
||||
|
||||
### Key Type Instantiation Points
|
||||
|
||||
#### 1. Config (`config.Config`)
|
||||
|
||||
- **Created by**: `config.Module` via `config.LoadConfig()`
|
||||
- **When**: Application startup (fx DI)
|
||||
- **Contains**: All configuration from YAML file (S3 credentials, encryption
|
||||
keys, paths, etc.)
|
||||
|
||||
#### 2. Database (`database.DB`)
|
||||
|
||||
- **Created by**: `database.Module` via `database.New()`
|
||||
- **When**: Application startup (fx DI)
|
||||
- **Contains**: SQLite connection, path reference
|
||||
|
||||
#### 3. Repositories (`database.Repositories`)
|
||||
|
||||
- **Created by**: `database.Module` via `database.NewRepositories()`
|
||||
- **When**: Application startup (fx DI)
|
||||
- **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots,
|
||||
etc.)
|
||||
|
||||
#### 4. Vaultik (`vaultik.Vaultik`)
|
||||
|
||||
- **Created by**: `vaultik.New(VaultikParams)`
|
||||
- **When**: Application startup (fx DI)
|
||||
- **Contains**: All dependencies for backup operations
|
||||
|
||||
```go
|
||||
type Vaultik struct {
|
||||
Globals *globals.Globals
|
||||
Config *config.Config
|
||||
DB *database.DB
|
||||
Repositories *database.Repositories
|
||||
Storage storage.Storer
|
||||
ScannerFactory snapshot.ScannerFactory
|
||||
SnapshotManager *snapshot.SnapshotManager
|
||||
Shutdowner fx.Shutdowner
|
||||
Fs afero.Fs
|
||||
ctx context.Context
|
||||
cancel context.CancelFunc
|
||||
}
|
||||
```
|
||||
|
||||
#### 5. SnapshotManager (`snapshot.SnapshotManager`)
|
||||
|
||||
- **Created by**: `snapshot.Module` via `snapshot.NewSnapshotManager()`
|
||||
- **When**: Application startup (fx DI)
|
||||
- **Responsibility**: Creates/completes snapshots, exports metadata to S3
|
||||
|
||||
#### 6. Scanner (`snapshot.Scanner`)
|
||||
|
||||
- **Created by**: `ScannerFactory(ScannerParams)`
|
||||
- **When**: Each `CreateSnapshot()` call
|
||||
- **Contains**: Chunker, Packer, progress reporter
|
||||
|
||||
```go
|
||||
// vaultik/snapshot.go: CreateSnapshot()
|
||||
scanner := v.ScannerFactory(snapshot.ScannerParams{
|
||||
EnableProgress: !opts.Cron,
|
||||
Fs: v.Fs,
|
||||
})
|
||||
```
|
||||
|
||||
#### 7. Chunker (`chunker.Chunker`)
|
||||
|
||||
- **Created by**: `chunker.NewChunker(avgChunkSize)`
|
||||
- **When**: Inside `snapshot.NewScanner()`
|
||||
- **Configuration**:
|
||||
- `avgChunkSize`: From config (default 10MB)
|
||||
- `minChunkSize`: avgChunkSize / 4
|
||||
- `maxChunkSize`: avgChunkSize \* 4
|
||||
|
||||
#### 8. Packer (`blob.Packer`)
|
||||
|
||||
- **Created by**: `blob.NewPacker(PackerConfig)`
|
||||
- **When**: Inside `snapshot.NewScanner()`
|
||||
- **Configuration**:
|
||||
- `MaxBlobSize`: Maximum blob size before finalization (typically 10GB)
|
||||
- `CompressionLevel`: zstd level (1-19)
|
||||
- `Recipients`: age public keys for encryption
|
||||
|
||||
```go
|
||||
// snapshot/scanner.go: NewScanner()
|
||||
packerCfg := blob.PackerConfig{
|
||||
MaxBlobSize: cfg.MaxBlobSize,
|
||||
CompressionLevel: cfg.CompressionLevel,
|
||||
Recipients: cfg.AgeRecipients,
|
||||
Repositories: cfg.Repositories,
|
||||
Fs: cfg.FS,
|
||||
}
|
||||
packer, err := blob.NewPacker(packerCfg)
|
||||
```
|
||||
|
||||
## Module Responsibilities
|
||||
|
||||
### `internal/cli`
|
||||
|
||||
Entry point for fx application. Combines all modules and handles signal
|
||||
interrupts.
|
||||
|
||||
Key functions:
|
||||
|
||||
- `NewApp(AppOptions)` → Creates fx.App with all modules
|
||||
- `RunApp(ctx, app)` → Starts app, handles graceful shutdown
|
||||
- `RunWithApp(ctx, opts)` → Convenience wrapper
|
||||
|
||||
### `internal/vaultik`
|
||||
|
||||
Main orchestrator containing all dependencies and command implementations.
|
||||
|
||||
Key methods:
|
||||
|
||||
- `New(VaultikParams)` → Constructor (fx DI)
|
||||
- `CreateSnapshot(opts)` → Main backup operation
|
||||
- `ListSnapshots(jsonOutput)` → List available snapshots
|
||||
- `VerifySnapshot(id, deep)` → Verify snapshot integrity
|
||||
- `PurgeSnapshots(...)` → Remove old snapshots
|
||||
|
||||
### `internal/chunker`
|
||||
|
||||
Content-defined chunking using FastCDC algorithm.
|
||||
|
||||
Key types:
|
||||
|
||||
- `Chunk` → Hash, Data, Offset, Size
|
||||
- `Chunker` → avgChunkSize, minChunkSize, maxChunkSize
|
||||
|
||||
Key methods:
|
||||
|
||||
- `NewChunker(avgChunkSize)` → Constructor
|
||||
- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback
|
||||
(preferred)
|
||||
- `ChunkReader(reader)` → Return all chunks at once (memory-intensive)
|
||||
|
||||
### `internal/blob`
|
||||
|
||||
Blob packing: accumulates chunks, compresses, encrypts, tracks metadata.
|
||||
|
||||
Key types:
|
||||
|
||||
- `Packer` → Thread-safe blob accumulator
|
||||
- `ChunkRef` → Hash + Data for adding to packer
|
||||
- `FinishedBlob` → Completed blob ready for upload
|
||||
- `BlobWithReader` → FinishedBlob + io.Reader for streaming upload
|
||||
|
||||
Key methods:
|
||||
|
||||
- `NewPacker(PackerConfig)` → Constructor
|
||||
- `AddChunk(ChunkRef)` → Add chunk to current blob
|
||||
- `FinalizeBlob()` → Compress, encrypt, hash current blob
|
||||
- `Flush()` → Finalize any in-progress blob
|
||||
- `SetBlobHandler(func)` → Set callback for upload
|
||||
|
||||
### `internal/snapshot`
|
||||
|
||||
#### Scanner
|
||||
|
||||
Orchestrates the backup process for a directory.
|
||||
|
||||
Key methods:
|
||||
|
||||
- `NewScanner(ScannerConfig)` → Constructor (creates Chunker + Packer)
|
||||
- `Scan(ctx, path, snapshotID)` → Main scan operation
|
||||
|
||||
Scan phases:
|
||||
|
||||
1. **Phase 1**: Walk directory, identify files needing processing
|
||||
2. **Phase 1b**: Count as deleted the files that the previous snapshot of the
|
||||
same name lists, the walk did not find, and are gone from disk; a file that
|
||||
is excluded, or inside an excluded directory, is not counted
|
||||
3. **Phase 2**: Process files (chunk → pack → upload)
|
||||
|
||||
#### SnapshotManager
|
||||
|
||||
Manages snapshot lifecycle and metadata export.
|
||||
|
||||
Key methods:
|
||||
|
||||
- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record
|
||||
- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot
|
||||
references
|
||||
- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3
|
||||
- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a
|
||||
successful export
|
||||
- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark
|
||||
complete (no export between)
|
||||
|
||||
### `internal/database`
|
||||
|
||||
SQLite database for local index. Single-writer mode for thread safety.
|
||||
|
||||
Key types:
|
||||
|
||||
- `DB` → Database connection wrapper
|
||||
- `Repositories` → Collection of all repository interfaces
|
||||
|
||||
Repository interfaces:
|
||||
|
||||
- `FilesRepository` → CRUD for File records
|
||||
- `ChunksRepository` → CRUD for Chunk records
|
||||
- `BlobsRepository` → CRUD for Blob records
|
||||
- `SnapshotsRepository` → CRUD for Snapshot records
|
||||
- Plus join table repositories (FileChunks, BlobChunks, etc.)
|
||||
|
||||
## Snapshot Creation Flow
|
||||
|
||||
```
|
||||
CreateSnapshot(opts)
|
||||
│
|
||||
├─► markDestination() // Check the marker (see Storage Layout in S3), then write it
|
||||
│
|
||||
├─► PruneDatabase() // Critical: avoid dedup errors
|
||||
│
|
||||
├─► SnapshotManager.CreateSnapshot() // Create DB record
|
||||
│
|
||||
├─► For each source directory:
|
||||
│ │
|
||||
│ ├─► scanner.Scan(ctx, path, snapshotID)
|
||||
│ │ │
|
||||
│ │ ├─► Phase 1: scanPhase()
|
||||
│ │ │ Walk directory
|
||||
│ │ │ Check file metadata changes
|
||||
│ │ │ Build list of files to process
|
||||
│ │ │
|
||||
│ │ ├─► Phase 1b: detectDeletedFiles()
|
||||
│ │ │
|
||||
│ │ └─► Phase 2: processPhase()
|
||||
│ │ For each file:
|
||||
│ │ chunker.ChunkReaderStreaming()
|
||||
│ │ For each chunk:
|
||||
│ │ packer.AddChunk()
|
||||
│ │ If blob full → FinalizeBlob()
|
||||
│ │ → handleBlobReady()
|
||||
│ │ → s3Client.PutObjectWithProgress()
|
||||
│ │ packer.Flush() // Final blob
|
||||
│ │
|
||||
│ └─► Accumulate statistics
|
||||
│
|
||||
├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs
|
||||
│
|
||||
├─► SnapshotManager.UpdateSnapshotStatsExtended()
|
||||
│
|
||||
├─► SnapshotManager.ExportSnapshotMetadata()
|
||||
│ │
|
||||
│ ├─► Copy database to temp file
|
||||
│ ├─► Clean to only current snapshot data (VACUUM)
|
||||
│ ├─► Compress binary SQLite with zstd
|
||||
│ ├─► Encrypt with age
|
||||
│ ├─► Upload db.zst.age to storage
|
||||
│ └─► Upload manifest.json.zst to storage
|
||||
│
|
||||
└─► SnapshotManager.MarkSnapshotComplete() // only after the export succeeds
|
||||
```
|
||||
|
||||
## Deduplication Strategy
|
||||
|
||||
1. **File-level**: Files and symlinks unchanged since last backup are skipped
|
||||
(metadata comparison: size, mtime, mode, uid, gid, and a symlink's target),
|
||||
unless the file lists a chunk that no uploaded blob holds; such a file is
|
||||
re-chunked
|
||||
|
||||
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash
|
||||
already exists in the database, the chunk data is not re-uploaded.
|
||||
|
||||
3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a
|
||||
blob are skipped.
|
||||
|
||||
## Storage Layout in S3
|
||||
|
||||
```
|
||||
bucket/
|
||||
├── vaultik-destination.txt # Marker written by every backup
|
||||
│
|
||||
├── blobs/
|
||||
│ └── {hash[0:2]}/
|
||||
│ └── {hash[2:4]}/
|
||||
│ └── {full-hash} # Compressed+encrypted blob
|
||||
│
|
||||
└── metadata/
|
||||
└── {remote-key}/
|
||||
├── db.zst.age # Encrypted binary SQLite database
|
||||
└── manifest.json.zst # Blob list (for pruning/verification)
|
||||
```
|
||||
|
||||
A destination that answers a look-up of `vaultik-destination.txt` with not found
|
||||
is treated as not present while the local index holds snapshots stored there
|
||||
(`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
|
||||
`snapshot create` refuses to write to it, `prune`, `snapshot purge` and
|
||||
`remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a
|
||||
destination that cannot be listed. This keeps those commands from taking an
|
||||
unplugged volume's empty mount point for a destination whose snapshots are all
|
||||
gone. When the look-up fails for any reason other than not found, for example
|
||||
when S3 refuses it to credentials that allow only PUT, the check is skipped, so
|
||||
a host with such credentials does not notice an emptied destination; see
|
||||
[docs/REPOSTRUCTURE.md](REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt).
|
||||
The `file://` backend never creates its destination directory, so a write to a
|
||||
missing one fails.
|
||||
|
||||
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
|
||||
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
|
||||
written to the store as a directory name. See
|
||||
[docs/REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) for the
|
||||
derivation and a worked example.
|
||||
|
||||
## Thread Safety
|
||||
|
||||
- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
|
||||
- `Scanner`: Uses `packerMu` mutex to coordinate blob finalization.
|
||||
- `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread
|
||||
safety.
|
||||
- `Repositories.WithTx()`: Handles transaction lifecycle automatically.
|
||||
Reference in new issue
Block a user