vaultik/ARCHITECTURE.md

Vaultik Architecture

This document describes the internal architecture of Vaultik, focusing on the data model, type instantiation, and the relationships between core modules.

Overview

Vaultik is a backup system that uses content-defined chunking for deduplication and packs chunks into large, compressed, encrypted blobs for efficient cloud storage. The system is built around dependency injection using uber-go/fx.

Data Flow

Source Files
     │
     ▼
┌─────────────────┐
│    Scanner      │  Walks directories, detects changed files
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│    Chunker      │  Splits files into variable-size chunks (FastCDC)
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│    Packer       │  Accumulates chunks, compresses (zstd), encrypts (age)
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   S3 Client     │  Uploads blobs to remote storage
└─────────────────┘

Data Model

Core Entities

The database tracks five primary entities and their relationships:

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   Snapshot   │────▶│     File     │────▶│    Chunk     │
└──────────────┘     └──────────────┘     └──────────────┘
       │                                         │
       │                                         │
       ▼                                         ▼
┌──────────────┐                          ┌──────────────┐
│     Blob     │◀─────────────────────────│  BlobChunk   │
└──────────────┘                          └──────────────┘

Entity Descriptions

File (database.File)

Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:

• Path, mtime
• Size, mode, ownership (uid, gid)
• Symlink target (if applicable)

It also stores source_path, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it.

Chunk (database.Chunk)

A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm:

• ChunkHash: SHA256 hash of chunk content (primary key)
• Size: Chunk size in bytes

Chunk sizes vary between avgChunkSize/4 and avgChunkSize*4 (2.5MB-40MB for the 10MB default average).

FileChunk (database.FileChunk)

Maps files to their constituent chunks:

• FileID: Reference to the file
• Idx: Position of this chunk within the file (0-indexed)
• ChunkHash: Reference to the chunk

Blob (database.Blob)

The final storage unit uploaded to S3. Contains many compressed and encrypted chunks:

• ID: UUID assigned at creation
• Hash: hex(SHA256(SHA256(uncompressed blob contents))), computed before compression and encryption (see docs/REPOSTRUCTURE.md)
• UncompressedSize: Total raw chunk data before compression
• CompressedSize: Size after zstd compression and age encryption
• CreatedTS, FinishedTS, UploadedTS: Lifecycle timestamps

Blob creation process:

1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — hex(SHA256(SHA256(...))) — not a hash of the compressed, encrypted bytes
5. The finished file is uploaded to blobs/{hash[0:2]}/{hash[2:4]}/{hash} and then deleted

A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about blob_size_limit; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to $TMPDIR (default /tmp); with TMPDIR unset, SQLite writes one of those copies to /var/tmp.

BlobChunk (database.BlobChunk)

Maps chunks to their position within blobs:

• BlobID: Reference to the blob
• ChunkHash: Reference to the chunk
• Offset: Byte offset within the uncompressed blob
• Length: Chunk size

Snapshot (database.Snapshot)

Represents a point-in-time backup:

• ID: Format is {hostname}_{snapshot-name}_{RFC3339} (e.g. server1_home_2025-06-01T12:00:00Z)
• Tracks file count, chunk count, blob count, sizes, compression ratio
• CompletedAt: Null until snapshot finishes successfully

SnapshotFile / SnapshotBlob

Join tables linking snapshots to their files and blobs.

Relationship Summary

Snapshot 1──────────▶ N SnapshotFile N ◀────────── 1 File
Snapshot 1──────────▶ N SnapshotBlob N ◀────────── 1 Blob
File     1──────────▶ N FileChunk    N ◀────────── 1 Chunk
Blob     1──────────▶ N BlobChunk    N ◀────────── 1 Chunk

Type Instantiation

Application Startup

The CLI uses fx for dependency injection. Here's the instantiation order:

// cli/app.go: NewApp()
fx.New(
    fx.Supply(config.Path(opts.ConfigPath)),         // 1. Config path
    fx.Supply(opts.LogOptions),                      // 2. Log options
    fx.Provide(globals.New),                         // 3. Globals
    fx.Provide(log.New),                             // 4. Logger config
    config.Module,                                   // 5. Config
    database.Module,                                 // 6. Database + Repositories
    log.Module,                                      // 7. Logger initialization
    storage.Module,                                  // 8. Storage backend (S3/file/rclone)
    snapshot.Module,                                 // 9. SnapshotManager + ScannerFactory
    fx.Provide(vaultik.New),                         // 10. Vaultik orchestrator
)

Key Type Instantiation Points

1. Config (config.Config)

• Created by: config.Module via config.LoadConfig()
• When: Application startup (fx DI)
• Contains: All configuration from YAML file (S3 credentials, encryption keys, paths, etc.)

2. Database (database.DB)

• Created by: database.Module via database.New()
• When: Application startup (fx DI)
• Contains: SQLite connection, path reference

3. Repositories (database.Repositories)

• Created by: database.Module via database.NewRepositories()
• When: Application startup (fx DI)
• Contains: All repository interfaces (Files, Chunks, Blobs, Snapshots, etc.)

4. Vaultik (vaultik.Vaultik)

• Created by: vaultik.New(VaultikParams)
• When: Application startup (fx DI)
• Contains: All dependencies for backup operations

type Vaultik struct {
    Globals         *globals.Globals
    Config          *config.Config
    DB              *database.DB
    Repositories    *database.Repositories
    Storage         storage.Storer
    ScannerFactory  snapshot.ScannerFactory
    SnapshotManager *snapshot.SnapshotManager
    Shutdowner      fx.Shutdowner
    Fs              afero.Fs
    ctx             context.Context
    cancel          context.CancelFunc
}

5. SnapshotManager (snapshot.SnapshotManager)

• Created by: snapshot.Module via snapshot.NewSnapshotManager()
• When: Application startup (fx DI)
• Responsibility: Creates/completes snapshots, exports metadata to S3

6. Scanner (snapshot.Scanner)

• Created by: ScannerFactory(ScannerParams)
• When: Each CreateSnapshot() call
• Contains: Chunker, Packer, progress reporter

// vaultik/snapshot.go: CreateSnapshot()
scanner := v.ScannerFactory(snapshot.ScannerParams{
    EnableProgress: !opts.Cron,
    Fs:             v.Fs,
})

7. Chunker (chunker.Chunker)

• Created by: chunker.NewChunker(avgChunkSize)
• When: Inside snapshot.NewScanner()
• Configuration:
  • avgChunkSize: From config (default 10MB)
  • minChunkSize: avgChunkSize / 4
  • maxChunkSize: avgChunkSize * 4

8. Packer (blob.Packer)

• Created by: blob.NewPacker(PackerConfig)
• When: Inside snapshot.NewScanner()
• Configuration:
  • MaxBlobSize: Maximum blob size before finalization (typically 10GB)
  • CompressionLevel: zstd level (1-19)
  • Recipients: age public keys for encryption

// snapshot/scanner.go: NewScanner()
packerCfg := blob.PackerConfig{
    MaxBlobSize:      cfg.MaxBlobSize,
    CompressionLevel: cfg.CompressionLevel,
    Recipients:       cfg.AgeRecipients,
    Repositories:     cfg.Repositories,
    Fs:               cfg.FS,
}
packer, err := blob.NewPacker(packerCfg)

Module Responsibilities

internal/cli

Entry point for fx application. Combines all modules and handles signal interrupts.

Key functions:

• NewApp(AppOptions) → Creates fx.App with all modules
• RunApp(ctx, app) → Starts app, handles graceful shutdown
• RunWithApp(ctx, opts) → Convenience wrapper

internal/vaultik

Main orchestrator containing all dependencies and command implementations.

Key methods:

• New(VaultikParams) → Constructor (fx DI)
• CreateSnapshot(opts) → Main backup operation
• ListSnapshots(jsonOutput) → List available snapshots
• VerifySnapshot(id, deep) → Verify snapshot integrity
• PurgeSnapshots(...) → Remove old snapshots

internal/chunker

Content-defined chunking using FastCDC algorithm.

Key types:

• Chunk → Hash, Data, Offset, Size
• Chunker → avgChunkSize, minChunkSize, maxChunkSize

Key methods:

• NewChunker(avgChunkSize) → Constructor
• ChunkReaderStreaming(reader, callback) → Stream chunks with callback (preferred)
• ChunkReader(reader) → Return all chunks at once (memory-intensive)

internal/blob

Blob packing: accumulates chunks, compresses, encrypts, tracks metadata.

Key types:

• Packer → Thread-safe blob accumulator
• ChunkRef → Hash + Data for adding to packer
• FinishedBlob → Completed blob ready for upload
• BlobWithReader → FinishedBlob + io.Reader for streaming upload

Key methods:

• NewPacker(PackerConfig) → Constructor
• AddChunk(ChunkRef) → Add chunk to current blob
• FinalizeBlob() → Compress, encrypt, hash current blob
• Flush() → Finalize any in-progress blob
• SetBlobHandler(func) → Set callback for upload

internal/snapshot

Scanner

Orchestrates the backup process for a directory.

Key methods:

• NewScanner(ScannerConfig) → Constructor (creates Chunker + Packer)
• Scan(ctx, path, snapshotID) → Main scan operation

Scan phases:

1. Phase 1: Walk directory, identify files needing processing
2. Phase 1b: Count as deleted the files that the previous snapshot of the same name lists and the walk did not find; a file that is excluded, or inside an excluded directory, is not counted
3. Phase 2: Process files (chunk → pack → upload)

SnapshotManager

Manages snapshot lifecycle and metadata export.

Key methods:

• CreateSnapshot(ctx, hostname, version, commit) → Create snapshot record
• PopulateSnapshotBlobs(ctx, snapshotID) → Record every blob the snapshot references
• ExportSnapshotMetadata(ctx, dbPath, snapshotID) → Export to S3
• MarkSnapshotComplete(ctx, snapshotID) → Record completion, only after a successful export
• CompleteSnapshot(ctx, snapshotID) → Convenience: populate blobs, then mark complete (no export between)

internal/database

SQLite database for local index. Single-writer mode for thread safety.

Key types:

• DB → Database connection wrapper
• Repositories → Collection of all repository interfaces

Repository interfaces:

• FilesRepository → CRUD for File records
• ChunksRepository → CRUD for Chunk records
• BlobsRepository → CRUD for Blob records
• SnapshotsRepository → CRUD for Snapshot records
• Plus join table repositories (FileChunks, BlobChunks, etc.)

Snapshot Creation Flow

CreateSnapshot(opts)
    │
    ├─► PruneDatabase()   // Critical: avoid dedup errors
    │
    ├─► SnapshotManager.CreateSnapshot()   // Create DB record
    │
    ├─► For each source directory:
    │       │
    │       ├─► scanner.Scan(ctx, path, snapshotID)
    │       │       │
    │       │       ├─► Phase 1: scanPhase()
    │       │       │       Walk directory
    │       │       │       Check file metadata changes
    │       │       │       Build list of files to process
    │       │       │
    │       │       ├─► Phase 1b: detectDeletedFiles()
    │       │       │
    │       │       └─► Phase 2: processPhase()
    │       │               For each file:
    │       │                   chunker.ChunkReaderStreaming()
    │       │                   For each chunk:
    │       │                       packer.AddChunk()
    │       │                       If blob full → FinalizeBlob()
    │       │                           → handleBlobReady()
    │       │                           → s3Client.PutObjectWithProgress()
    │       │               packer.Flush()  // Final blob
    │       │
    │       └─► Accumulate statistics
    │
    ├─► SnapshotManager.PopulateSnapshotBlobs()   // record referenced blobs
    │
    ├─► SnapshotManager.UpdateSnapshotStatsExtended()
    │
    ├─► SnapshotManager.ExportSnapshotMetadata()
    │       │
    │       ├─► Copy database to temp file
    │       ├─► Clean to only current snapshot data (VACUUM)
    │       ├─► Compress binary SQLite with zstd
    │       ├─► Encrypt with age
    │       ├─► Upload db.zst.age to storage
    │       └─► Upload manifest.json.zst to storage
    │
    └─► SnapshotManager.MarkSnapshotComplete()   // only after the export succeeds

Deduplication Strategy

1. File-level: Files unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked

2. Chunk-level: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded.

3. Blob-level: Blobs contain only unique chunks. Duplicate chunks within a blob are skipped.

Storage Layout in S3

bucket/
├── blobs/
│   └── {hash[0:2]}/
│       └── {hash[2:4]}/
│           └── {full-hash}          # Compressed+encrypted blob
│
└── metadata/
    └── {remote-key}/
        ├── db.zst.age               # Encrypted binary SQLite database
        └── manifest.json.zst        # Blob list (for pruning/verification)

The {remote-key} directory name is a one-way double SHA-256 hash of the human snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never written to the store as a directory name. See docs/REPOSTRUCTURE.md for the derivation and a worked example.

Thread Safety

• Packer: Thread-safe via mutex. Multiple goroutines can call AddChunk().
• Scanner: Uses packerMu mutex to coordinate blob finalization.
• Database: Single-writer mode (MaxOpenConns=1) ensures SQLite thread safety.
• Repositories.WithTx(): Handles transaction lifecycle automatically.