sneak/vaultik
1
0
Fork 0
Code Issues 2 Pull Requests 2 Actions 6 Packages Projects Releases Wiki Activity

Rewrite ARCHITECTURE.md sentence by sentence
check / check (push) Waiting to run
Details

Browse Source 
Rewritten from scratch against the code on next: the packages, the fx
wiring, and the steps of a backup, a restore, a verify and a prune.

Names that no longer exist in the code are gone: the S3 client as the
upload step, blob.BlobWithReader, PurgeSnapshots and the repository
interfaces. The storage layer is described through storage.Storer.

Model: fable-5-1
This commit is contained in:
sneak
2026-11-06 18:23:17 +00:00
parent de9ef62073
commit 43c9b2c39b
1 changed files with 316 additions and 351 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ARCHITECTURE.md
+316 -351
View File
@@ -1,411 +1,376 @@
# Vaultik Architecture
# vaultik architecture
This document describes the internal architecture of Vaultik, focusing on the data model, type instantiation, and the relationships between core modules.
This document describes how the `vaultik` code is organised. It covers the
packages, how they are wired together, and what happens during a backup, a
restore, a verify and a prune.
## Overview
Two other documents cover the stored data:
Vaultik is a backup system that uses content-defined chunking for deduplication and packs chunks into large, compressed, encrypted blobs for efficient cloud storage. The system is built around dependency injection using [uber-go/fx](https://github.com/uber-go/fx).
- [docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index.
- [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the
destination.
## Data Flow
## Terms
- **Snapshot**: one backup of the paths listed under one name in the config.
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
of its content.
- **Blob**: one object at the destination. It holds many chunks, compressed and
then encrypted.
- **Local index**: the SQLite database on the backed-up machine.
- **Destination**: the place the backups are stored.
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
at the destination.
## How a backup moves data
```
Source Files
source files
│
▼
┌─────────────────┐
│ Scanner │ Walks directories, detects changed files
└────────┬────────┘
│
▼
┌─────────────────┐
│ Chunker │ Splits files into variable-size chunks (FastCDC)
└────────┬────────┘
│
▼
┌─────────────────┐
│ Packer │ Accumulates chunks, compresses (zstd), encrypts (age)
└────────┬────────┘
│
▼
┌─────────────────┐
│ S3 Client │ Uploads blobs to remote storage
└─────────────────┘
Scanner walks the paths and finds the files that changed
│
▼
Chunker cuts each changed file into chunks
│
▼
Packer packs new chunks into a blob, compressed and encrypted
│
▼
Storer uploads the blob to the destination
```
## Data Model
## Packages
### Core Entities
- `cmd/vaultik`: the `main` function. It starts optional profiling and calls
`cli.Entry`.
- `internal/cli`: the commands, their flags and their help text. It builds the
application, takes the lock and handles interrupts.
- `internal/vaultik`: the code behind each command. `Vaultik` holds every
dependency a command uses.
- `internal/snapshot`: the `Scanner`, which backs up one path, and the
`SnapshotManager`, which creates snapshot records and uploads their metadata.
- `internal/chunker`: cuts a stream into chunks with the FastCDC algorithm.
- `internal/blob`: the `Packer`, which collects chunks into blobs.
- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
and the reader that reverses it.
- `internal/database`: the local index. It holds the schema and one repository
type for each table.
- `internal/storage`: the `Storer` interface and its three implementations, for
S3, a local directory and rclone.
- `internal/s3`: the S3 client that the S3 `Storer` uses.
- `internal/config`: loads and checks the config file.
- `internal/ui`: formats the status lines, warnings and errors a user sees.
- `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that keeps two data-changing commands from
running at once.
- `internal/globals`: the version, commit and build date, set when the binary
is built.
- `internal/types`: distinct types for IDs, hashes and paths.
The database tracks five primary entities and their relationships:
## Wiring
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Snapshot │────▶│ File │────▶│ Chunk │
└──────────────┘ └──────────────┘ └──────────────┘
│ │
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Blob │◀─────────────────────────│ BlobChunk │
└──────────────┘ └──────────────┘
```
### Entity Descriptions
#### File (`database.File`)
Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:
- Path, mtime
- Size, mode, ownership (uid, gid)
- Symlink target (if applicable)
It also stores `source_path`, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it.
#### Chunk (`database.Chunk`)
A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm:
- `ChunkHash`: SHA256 hash of chunk content (primary key)
- `Size`: Chunk size in bytes
Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for the 10MB default average).
#### FileChunk (`database.FileChunk`)
Maps files to their constituent chunks:
- `FileID`: Reference to the file
- `Idx`: Position of this chunk within the file (0-indexed)
- `ChunkHash`: Reference to the chunk
#### Blob (`database.Blob`)
The final storage unit uploaded to S3. Contains many compressed and encrypted chunks:
- `ID`: UUID assigned at creation
- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs))
- `UncompressedSize`: Total raw chunk data before compression
- `CompressedSize`: Size after zstd compression and age encryption
- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
Blob creation process:
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted
A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`.
#### BlobChunk (`database.BlobChunk`)
Maps chunks to their position within blobs:
- `BlobID`: Reference to the blob
- `ChunkHash`: Reference to the chunk
- `Offset`: Byte offset within the uncompressed blob
- `Length`: Chunk size
#### Snapshot (`database.Snapshot`)
Represents a point-in-time backup:
- `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g. `server1_home_2025-06-01T12:00:00Z`)
- Tracks file count, chunk count, blob count, sizes, compression ratio
- `CompletedAt`: Null until snapshot finishes successfully
#### SnapshotFile / SnapshotBlob
Join tables linking snapshots to their files and blobs.
### Relationship Summary
```
Snapshot 1──────────▶ N SnapshotFile N ◀────────── 1 File
Snapshot 1──────────▶ N SnapshotBlob N ◀────────── 1 Blob
File 1──────────▶ N FileChunk N ◀────────── 1 Chunk
Blob 1──────────▶ N BlobChunk N ◀────────── 1 Chunk
```
## Type Instantiation
### Application Startup
The CLI uses fx for dependency injection. Here's the instantiation order:
The application is assembled with [uber-go/fx](https://github.com/uber-go/fx), a
dependency injection library. `cli.NewApp` lists the providers:
```go
// cli/app.go: NewApp()
fx.New(
fx.Supply(config.Path(opts.ConfigPath)), // 1. Config path
fx.Supply(opts.LogOptions), // 2. Log options
fx.Provide(globals.New), // 3. Globals
fx.Provide(log.New), // 4. Logger config
config.Module, // 5. Config
database.Module, // 6. Database + Repositories
log.Module, // 7. Logger initialization
storage.Module, // 8. Storage backend (S3/file/rclone)
snapshot.Module, // 9. SnapshotManager + ScannerFactory
fx.Provide(vaultik.New), // 10. Vaultik orchestrator
)
fx.Supply(config.Path(opts.ConfigPath)),
fx.Supply(opts.LogOptions),
fx.Provide(globals.New),
fx.Provide(log.New),
config.Module,
database.Module,
log.Module,
storage.Module,
snapshot.Module,
fx.Provide(vaultik.New),
fx.Invoke(setupGlobals),
fx.NopLogger,
```
### Key Type Instantiation Points
Each module provides these:
#### 1. Config (`config.Config`)
- **Created by**: `config.Module` via `config.LoadConfig()`
- **When**: Application startup (fx DI)
- **Contains**: All configuration from YAML file (S3 credentials, encryption keys, paths, etc.)
- `config.Module` provides `*config.Config`, from `config.Load`.
- `database.Module` provides `*database.DB` and `*database.Repositories`. It
creates the index file with mode `0600` when it does not exist.
- `storage.Module` provides the `storage.Storer` that the config selects.
- `snapshot.Module` provides the `snapshot.ScannerFactory` and the
`*snapshot.SnapshotManager`.
- `vaultik.New` provides `*vaultik.Vaultik`.
#### 2. Database (`database.DB`)
- **Created by**: `database.Module` via `database.New()`
- **When**: Application startup (fx DI)
- **Contains**: SQLite connection, path reference
A command runs through `cli.RunOperation`. It works like this:
#### 3. Repositories (`database.Repositories`)
- **Created by**: `database.Module` via `database.NewRepositories()`
- **When**: Application startup (fx DI)
- **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots, etc.)
- `RunWithApp` takes the lock when the command changes data, and then starts the
application.
- The command's function runs in its own goroutine.
- When the function returns, it asks the application to shut down.
- On `SIGINT` or `SIGTERM`, the stop hook cancels the `Vaultik` context. It then
waits for the function to return, so that its temporary files are removed.
- The wait ends after 30 seconds at most.
#### 4. Vaultik (`vaultik.Vaultik`)
- **Created by**: `vaultik.New(VaultikParams)`
- **When**: Application startup (fx DI)
- **Contains**: All dependencies for backup operations
The `config` commands, `database delete`, `completion` and `version` do not
build the application.
```go
type Vaultik struct {
Globals *globals.Globals
Config *config.Config
DB *database.DB
Repositories *database.Repositories
Storage storage.Storer
ScannerFactory snapshot.ScannerFactory
SnapshotManager *snapshot.SnapshotManager
Shutdowner fx.Shutdowner
Fs afero.Fs
ctx context.Context
cancel context.CancelFunc
}
```
## Backup
#### 5. SnapshotManager (`snapshot.SnapshotManager`)
- **Created by**: `snapshot.Module` via `snapshot.NewSnapshotManager()`
- **When**: Application startup (fx DI)
- **Responsibility**: Creates/completes snapshots, exports metadata to S3
`Vaultik.CreateSnapshot` runs a backup. It does these steps in order:
#### 6. Scanner (`snapshot.Scanner`)
- **Created by**: `ScannerFactory(ScannerParams)`
- **When**: Each `CreateSnapshot()` call
- **Contains**: Chunker, Packer, progress reporter
- `EnsureStorageBinding` checks that the local index belongs to the configured
destination.
- `markDestination` checks the marker file and then writes it.
- `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `createNamedSnapshot` runs once for each snapshot name.
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
```go
// vaultik/snapshot.go: CreateSnapshot()
scanner := v.ScannerFactory(snapshot.ScannerParams{
EnableProgress: !opts.Cron,
Fs: v.Fs,
})
```
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
left in the index would make the backup skip files that were never fully
stored.
#### 7. Chunker (`chunker.Chunker`)
- **Created by**: `chunker.NewChunker(avgChunkSize)`
- **When**: Inside `snapshot.NewScanner()`
- **Configuration**:
- `avgChunkSize`: From config (default 10MB)
- `minChunkSize`: avgChunkSize / 4
- `maxChunkSize`: avgChunkSize * 4
### One snapshot
#### 8. Packer (`blob.Packer`)
- **Created by**: `blob.NewPacker(PackerConfig)`
- **When**: Inside `snapshot.NewScanner()`
- **Configuration**:
- `MaxBlobSize`: Maximum blob size before finalization (typically 10GB)
- `CompressionLevel`: zstd level (1-19)
- `Recipients`: age public keys for encryption
`createNamedSnapshot` does these steps:
```go
// snapshot/scanner.go: NewScanner()
packerCfg := blob.PackerConfig{
MaxBlobSize: cfg.MaxBlobSize,
CompressionLevel: cfg.CompressionLevel,
Recipients: cfg.AgeRecipients,
Repositories: cfg.Repositories,
Fs: cfg.FS,
}
packer, err := blob.NewPacker(packerCfg)
```
- `resolveSnapshotPaths` makes each configured path absolute and resolves
symlinks.
- `SnapshotManager.CreateSnapshotWithName` creates the snapshot record. The
record is not yet marked complete.
- `Scanner.Scan` backs up each path in turn.
- `finalizeSnapshotMetadata` uploads the snapshot's metadata and marks the
snapshot complete.
## Module Responsibilities
A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a
`Chunker` and a `Packer`.
### `internal/cli`
Entry point for fx application. Combines all modules and handles signal interrupts.
### Scanning one path
Key functions:
- `NewApp(AppOptions)` → Creates fx.App with all modules
- `RunApp(ctx, app)` → Starts app, handles graceful shutdown
- `RunWithApp(ctx, opts)` → Convenience wrapper
`Scanner.Scan` works in these steps:
### `internal/vaultik`
Main orchestrator containing all dependencies and command implementations.
- `repairInterruptedBlobs` deletes the records of blobs whose upload never
finished. Their data is then stored again.
- `loadDatabaseState` loads the known files and chunks into memory.
- `scanPhase` walks the path. It sorts each entry into changed or unchanged.
- `detectDeletedFiles` counts the files that the previous snapshot of this name
had and that are now gone.
- `batchAddFilesToSnapshot` records the unchanged files in the new snapshot.
- `recordDirectories` records every directory, and the mount points found.
- `processPhase` reads each changed file and stores it.
Key methods:
- `New(VaultikParams)` → Constructor (fx DI)
- `CreateSnapshot(opts)` → Main backup operation
- `ListSnapshots(jsonOutput)` → List available snapshots
- `VerifySnapshot(id, deep)` → Verify snapshot integrity
- `PurgeSnapshots(...)` → Remove old snapshots
The walk handles each kind of entry differently:
### `internal/chunker`
Content-defined chunking using FastCDC algorithm.
- A regular file or a symlink is compared with its record in the index.
- A directory is recorded with its metadata. It has no content to store.
- A device node, a named pipe or a socket is skipped.
- An entry that matches an exclude pattern is skipped. For a directory, so is
everything in it.
Key types:
- `Chunk` → Hash, Data, Offset, Size
- `Chunker` → avgChunkSize, minChunkSize, maxChunkSize
### Deduplication
Key methods:
- `NewChunker(avgChunkSize)` → Constructor
- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback (preferred)
- `ChunkReader(reader)` → Return all chunks at once (memory-intensive)
Data is stored once at two levels.
### `internal/blob`
Blob packing: accumulates chunks, compresses, encrypts, tracks metadata.
- **Files.** A file is not read again when its record in the index still matches
it. The record matches when the size, modification time, mode, owner, group
and link target are all the same.
- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk whose
hash is already in the index is not stored again.
Key types:
- `Packer` → Thread-safe blob accumulator
- `ChunkRef` → Hash + Data for adding to packer
- `FinishedBlob` → Completed blob ready for upload
- `BlobWithReader` → FinishedBlob + io.Reader for streaming upload
One case overrides the file check. A file is read again when it lists a chunk
that no uploaded blob holds. This happens after the only snapshot that used
such a blob is removed. `loadFilesToRechunk` finds these files.
Key methods:
- `NewPacker(PackerConfig)` → Constructor
- `AddChunk(ChunkRef)` → Add chunk to current blob
- `FinalizeBlob()` → Compress, encrypt, hash current blob
- `Flush()` → Finalize any in-progress blob
- `SetBlobHandler(func)` → Set callback for upload
### Chunks
### `internal/snapshot`
`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
on the content, so an insertion early in a file moves only nearby chunks.
#### Scanner
Orchestrates the backup process for a directory.
`chunk_size` sets the average chunk size. A chunk is between a quarter of that
size and four times that size. With the default of 10 MB, a chunk is between 2.5
MB and 40 MB.
Key methods:
- `NewScanner(ScannerConfig)` → Constructor (creates Chunker + Packer)
- `Scan(ctx, path, snapshotID)` → Main scan operation
`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole
file is never held in memory.
Scan phases:
1. **Phase 1**: Walk directory, identify files needing processing
2. **Phase 1b**: Count as deleted the files that the previous snapshot of the same name lists, the walk did not find, and are gone from disk; a file that is excluded, or inside an excluded directory, is not counted
3. **Phase 2**: Process files (chunk → pack → upload)
### Blobs
#### SnapshotManager
Manages snapshot lifecycle and metadata export.
`blob.Packer` collects new chunks into a blob. For each blob it does this:
Key methods:
- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record
- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot references
- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3
- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a successful export
- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark complete (no export between)
- It gives the blob a UUID and creates a temporary file.
- It adds each chunk's bytes to a running SHA-256 hash.
- In the same pass it compresses the bytes with zstd and encrypts them with
age. The result goes to the temporary file.
- It records each chunk's offset and length within the blob.
- It closes the blob when the next chunk would pass `blob_size_limit`, and at
the end of each path.
### `internal/database`
SQLite database for local index. Single-writer mode for thread safety.
The blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. It is computed
from the chunk data before compression and encryption.
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs) says what follows from
that.
Key types:
- `DB` → Database connection wrapper
- `Repositories` → Collection of all repository interfaces
When a blob is closed, `Scanner.handleBlobReady` stores it:
Repository interfaces:
- `FilesRepository` → CRUD for File records
- `ChunksRepository` → CRUD for Chunk records
- `BlobsRepository` → CRUD for Blob records
- `SnapshotsRepository` → CRUD for Snapshot records
- Plus join table repositories (FileChunks, BlobChunks, etc.)
- It looks the blob up at the destination. A blob that is already there is not
uploaded again.
- Otherwise it uploads the temporary file to
`blobs/<hash[0:2]>/<hash[2:4]>/<hash>`.
- It records the upload in the index and links the blob to the snapshot.
- It deletes the temporary file.
## Snapshot Creation Flow
A file's record is written to the index only after every blob that holds its
chunks is uploaded.
```
CreateSnapshot(opts)
│
├─► markDestination() // Check the marker (see Storage Layout in S3), then write it
│
├─► PruneDatabase() // Critical: avoid dedup errors
│
├─► SnapshotManager.CreateSnapshot() // Create DB record
│
├─► For each source directory:
│ │
│ ├─► scanner.Scan(ctx, path, snapshotID)
│ │ │
│ │ ├─► Phase 1: scanPhase()
│ │ │ Walk directory
│ │ │ Check file metadata changes
│ │ │ Build list of files to process
│ │ │
│ │ ├─► Phase 1b: detectDeletedFiles()
│ │ │
│ │ └─► Phase 2: processPhase()
│ │ For each file:
│ │ chunker.ChunkReaderStreaming()
│ │ For each chunk:
│ │ packer.AddChunk()
│ │ If blob full → FinalizeBlob()
│ │ → handleBlobReady()
│ │ → s3Client.PutObjectWithProgress()
│ │ packer.Flush() // Final blob
│ │
│ └─► Accumulate statistics
│
├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs
│
├─► SnapshotManager.UpdateSnapshotStatsExtended()
│
├─► SnapshotManager.ExportSnapshotMetadata()
│ │
│ ├─► Copy database to temp file
│ ├─► Clean to only current snapshot data (VACUUM)
│ ├─► Compress binary SQLite with zstd
│ ├─► Encrypt with age
│ ├─► Upload db.zst.age to storage
│ └─► Upload manifest.json.zst to storage
│
└─► SnapshotManager.MarkSnapshotComplete() // only after the export succeeds
```
### Snapshot metadata
## Deduplication Strategy
`finalizeSnapshotMetadata` does these steps in order:
1. **File-level**: Files and symlinks unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked
- `PopulateSnapshotBlobs` records every blob the snapshot uses.
- `UpdateSnapshotStatsExtended` stores the snapshot's totals.
- `ExportSnapshotMetadata` uploads the metadata.
- `MarkSnapshotComplete` marks the snapshot complete.
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded.
Completion is recorded last on purpose. A backup that stops during the upload
leaves the snapshot unfinished, and the next backup removes it and stores its
data again.
3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a blob are skipped.
`ExportSnapshotMetadata` works on a copy of the index, never on the index
itself:
## Storage Layout in S3
- It copies the index into a temporary directory with `VACUUM INTO`.
- It deletes from the copy everything that does not belong to this snapshot.
- It runs `VACUUM`, so that no deleted row is left in the file.
- It compresses the copy with zstd and encrypts it with age.
- It builds the manifest from the copy.
- It uploads `metadata/<remote-key>/db.zst.age`, and then
`metadata/<remote-key>/manifest.json.zst`.
```
bucket/
├── vaultik-destination.txt # Marker written by every backup
│
├── blobs/
│ └── {hash[0:2]}/
│ └── {hash[2:4]}/
│ └── {full-hash} # Compressed+encrypted blob
│
└── metadata/
└── {remote-key}/
├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Blob list (for pruning/verification)
```
The manifest is uploaded last. A directory under `metadata/` with no manifest
therefore marks a backup that did not finish.
A destination that answers a look-up of `vaultik-destination.txt` with not
found is treated as not present while the local index holds snapshots stored
there (`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
`snapshot create` refuses to write to it, `prune`, `snapshot purge` and
`remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a
destination that cannot be listed. This keeps those commands from taking an
unplugged volume's empty mount point for a destination whose snapshots are
all gone. When the look-up fails for any reason other than not found, for
example when S3 refuses it to credentials that allow only PUT, the check is
skipped, so a host with such credentials does not notice an emptied
destination; see
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt).
The `file://` backend never creates its destination directory, so a write to a
missing one fails.
## Restore
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
written to the store as a directory name. See
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation and a worked example.
`Vaultik.Restore` does these steps:
## Thread Safety
- `restoreIdentities` parses the age private key. A missing or malformed key
stops the restore before anything is downloaded.
- `downloadSnapshotDB` downloads and decrypts the snapshot's database into a
temporary directory.
- `getFilesToRestore` selects the files, from the paths given.
- `restoreAllFiles` downloads blobs and writes the files.
- With `--verify`, `verifyRestoredFiles` reads each restored file and checks its
chunk hashes.
- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
- `Scanner`: Uses `packerMu` mutex to coordinate blob finalization.
- `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread safety.
- `Repositories.WithTx()`: Handles transaction lifecycle automatically.
The restore does not read the local index or the manifest. It finds each file's
chunks and blobs through the downloaded database.
### Checks on the downloaded database
Decryption shows that a database is readable. It does not show that the
database is honest. The restore therefore checks it:
- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
trigger, a view or a virtual table, or that lacks an expected table.
- `verifySnapshotDBIdentity` checks that the snapshot in the database hashes to
the remote key that was asked for.
- `containedRestorePath` refuses a stored path that would leave the target
directory. It also refuses to write through a symlink below the target.
- A blob is not decompressed beyond the size the database records for it.
- Each downloaded blob is hashed. The hash must match the blob's name.
### Order of work
`restorePlan` orders the files so that few blobs are on disk at once:
- Downloaded blobs are kept decrypted in a cache directory.
- A file is restored once every blob it needs is in the cache.
- When no file is ready, the restore downloads the blobs for the file that
needs the fewest.
- `restoreSweeper` deletes a cached blob once no remaining file needs it.
A regular file is created with mode `0600` and gets its stored mode after its
content is written. A directory gets its stored mode, owner and modification
time after the whole restore.
## Verify
`Vaultik.VerifySnapshotWithOptions` checks a snapshot without the private key:
- It downloads the manifest.
- It checks that the snapshot's encrypted database is present.
- It checks that each listed blob is present with the listed size.
`Vaultik.RunDeepVerify` needs the private key. It does these steps:
- It downloads the manifest and the database. It checks the database as a
restore does.
- `verifyManifestAgainstDatabase` checks that both list the same blobs with the
same sizes.
- `verifyBlobExistenceFromDB` checks that every blob is present with the
recorded size.
- `verifyBlob` downloads each blob, checks the hash of each chunk in it, and
checks the hash that names the blob.
## Prune
`Vaultik.Prune` does three steps:
- `CleanupLocalSnapshots` drops each snapshot from the local index whose
directory is gone from the destination.
- `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `PruneBlobs` deletes unused data at the destination.
`PruneBlobs` reads every manifest at the destination. It then deletes two kinds
of object, once they are older than `prune_grace_period`:
- a blob that no manifest lists
- a directory under `metadata/` that has no manifest, and whose snapshot is not
in the local index
It deletes nothing when any manifest cannot be read.
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#pruning) explains why this is
safe, and where it is not.
## The destination
`storage.Storer` is the only way the code reaches the destination. It has these
methods:
- `Put` and `PutWithProgress` store an object.
- `Get` reads an object.
- `Stat` reports an object's size.
- `Delete` removes an object.
- `List` and `ListStream` list the objects under a prefix.
- `DeletePartialUploads` removes what an interrupted upload left behind.
- `Info` describes the destination.
`storage.NewStorer` picks the implementation from `storage_url`:
- `s3://` gives the S3 implementation.
- `file://` gives the local directory implementation.
- `rclone://` gives the rclone implementation.
With no `storage_url`, it builds the S3 implementation from the `s3` settings.
## The marker file and the destination binding
Two checks keep the local index and the destination in step.
- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when
the file is reported missing while the local index holds snapshots.
- `EnsureStorageBinding` records the configured destination in the local index
on first use. Later it fails when the config names a different destination.
Every command that changes data calls `EnsureStorageBinding` first.
## Concurrency
- The local index is opened with one connection, so all access to it is in
turn.
- The index runs in SQLite's WAL mode. A second process can therefore read it
while a backup writes to it.
- `Packer` guards its state with a mutex.
- `Scanner` holds `packerMu` while a chunk is added to the packer and while a
blob is closed.
- `Repositories.WithTx` runs a function in one transaction. It commits when the
function succeeds and rolls back when it fails.