sneak/vaultik
1
0
Fork 0
Code Issues 2 Pull Requests 2 Actions 6 Packages Projects Releases Wiki Activity

Rewrite ARCHITECTURE.md sentence by sentence
check / check (push) Waiting to run
Details

Browse Source 
Rewritten from scratch against the code on next: the packages, the fx
wiring, and the steps of a backup, a restore, a verify and a prune.

Names that no longer exist in the code are gone: the S3 client as the
upload step, blob.BlobWithReader, PurgeSnapshots and the repository
interfaces. The storage layer is described through storage.Storer.

Model: fable-5-1
This commit is contained in:
sneak
2026-11-06 18:23:17 +00:00
parent de9ef62073
commit 43c9b2c39b
1 changed files with 316 additions and 351 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ARCHITECTURE.md
+316 -351
View File
@@ -1,411 +1,376 @@
# Vaultik Architecture # vaultik architecture
This document describes the internal architecture of Vaultik, focusing on the data model, type instantiation, and the relationships between core modules. This document describes how the `vaultik` code is organised. It covers the
packages, how they are wired together, and what happens during a backup, a
restore, a verify and a prune.
## Overview Two other documents cover the stored data:
Vaultik is a backup system that uses content-defined chunking for deduplication and packs chunks into large, compressed, encrypted blobs for efficient cloud storage. The system is built around dependency injection using [uber-go/fx](https://github.com/uber-go/fx). - [docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index.
- [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the
destination.
## Data Flow ## Terms
- **Snapshot**: one backup of the paths listed under one name in the config.
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
of its content.
- **Blob**: one object at the destination. It holds many chunks, compressed and
then encrypted.
- **Local index**: the SQLite database on the backed-up machine.
- **Destination**: the place the backups are stored.
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
at the destination.
## How a backup moves data
``` ```
Source Files source files
│ │
▼ ▼
┌─────────────────┐ Scanner walks the paths and finds the files that changed
│ Scanner │ Walks directories, detects changed files │
└────────┬────────┘ ▼
│ Chunker cuts each changed file into chunks
▼ │
┌─────────────────┐ ▼
│ Chunker │ Splits files into variable-size chunks (FastCDC) Packer packs new chunks into a blob, compressed and encrypted
└────────┬────────┘ │
│ ▼
▼ Storer uploads the blob to the destination
┌─────────────────┐
│ Packer │ Accumulates chunks, compresses (zstd), encrypts (age)
└────────┬────────┘
│
▼
┌─────────────────┐
│ S3 Client │ Uploads blobs to remote storage
└─────────────────┘
``` ```
## Data Model ## Packages
### Core Entities - `cmd/vaultik`: the `main` function. It starts optional profiling and calls
`cli.Entry`.
- `internal/cli`: the commands, their flags and their help text. It builds the
application, takes the lock and handles interrupts.
- `internal/vaultik`: the code behind each command. `Vaultik` holds every
dependency a command uses.
- `internal/snapshot`: the `Scanner`, which backs up one path, and the
`SnapshotManager`, which creates snapshot records and uploads their metadata.
- `internal/chunker`: cuts a stream into chunks with the FastCDC algorithm.
- `internal/blob`: the `Packer`, which collects chunks into blobs.
- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
and the reader that reverses it.
- `internal/database`: the local index. It holds the schema and one repository
type for each table.
- `internal/storage`: the `Storer` interface and its three implementations, for
S3, a local directory and rclone.
- `internal/s3`: the S3 client that the S3 `Storer` uses.
- `internal/config`: loads and checks the config file.
- `internal/ui`: formats the status lines, warnings and errors a user sees.
- `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that keeps two data-changing commands from
running at once.
- `internal/globals`: the version, commit and build date, set when the binary
is built.
- `internal/types`: distinct types for IDs, hashes and paths.
The database tracks five primary entities and their relationships: ## Wiring
``` The application is assembled with [uber-go/fx](https://github.com/uber-go/fx), a
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ dependency injection library. `cli.NewApp` lists the providers:
│ Snapshot │────▶│ File │────▶│ Chunk │
└──────────────┘ └──────────────┘ └──────────────┘
│ │
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Blob │◀─────────────────────────│ BlobChunk │
└──────────────┘ └──────────────┘
```
### Entity Descriptions
#### File (`database.File`)
Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:
- Path, mtime
- Size, mode, ownership (uid, gid)
- Symlink target (if applicable)
It also stores `source_path`, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it.
#### Chunk (`database.Chunk`)
A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm:
- `ChunkHash`: SHA256 hash of chunk content (primary key)
- `Size`: Chunk size in bytes
Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for the 10MB default average).
#### FileChunk (`database.FileChunk`)
Maps files to their constituent chunks:
- `FileID`: Reference to the file
- `Idx`: Position of this chunk within the file (0-indexed)
- `ChunkHash`: Reference to the chunk
#### Blob (`database.Blob`)
The final storage unit uploaded to S3. Contains many compressed and encrypted chunks:
- `ID`: UUID assigned at creation
- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs))
- `UncompressedSize`: Total raw chunk data before compression
- `CompressedSize`: Size after zstd compression and age encryption
- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
Blob creation process:
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted
A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`.
#### BlobChunk (`database.BlobChunk`)
Maps chunks to their position within blobs:
- `BlobID`: Reference to the blob
- `ChunkHash`: Reference to the chunk
- `Offset`: Byte offset within the uncompressed blob
- `Length`: Chunk size
#### Snapshot (`database.Snapshot`)
Represents a point-in-time backup:
- `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g. `server1_home_2025-06-01T12:00:00Z`)
- Tracks file count, chunk count, blob count, sizes, compression ratio
- `CompletedAt`: Null until snapshot finishes successfully
#### SnapshotFile / SnapshotBlob
Join tables linking snapshots to their files and blobs.
### Relationship Summary
```
Snapshot 1──────────▶ N SnapshotFile N ◀────────── 1 File
Snapshot 1──────────▶ N SnapshotBlob N ◀────────── 1 Blob
File 1──────────▶ N FileChunk N ◀────────── 1 Chunk
Blob 1──────────▶ N BlobChunk N ◀────────── 1 Chunk
```
## Type Instantiation
### Application Startup
The CLI uses fx for dependency injection. Here's the instantiation order:
```go ```go
// cli/app.go: NewApp() fx.Supply(config.Path(opts.ConfigPath)),
fx.New( fx.Supply(opts.LogOptions),
fx.Supply(config.Path(opts.ConfigPath)), // 1. Config path fx.Provide(globals.New),
fx.Supply(opts.LogOptions), // 2. Log options fx.Provide(log.New),
fx.Provide(globals.New), // 3. Globals config.Module,
fx.Provide(log.New), // 4. Logger config database.Module,
config.Module, // 5. Config log.Module,
database.Module, // 6. Database + Repositories storage.Module,
log.Module, // 7. Logger initialization snapshot.Module,
storage.Module, // 8. Storage backend (S3/file/rclone) fx.Provide(vaultik.New),
snapshot.Module, // 9. SnapshotManager + ScannerFactory fx.Invoke(setupGlobals),
fx.Provide(vaultik.New), // 10. Vaultik orchestrator fx.NopLogger,
)
``` ```
### Key Type Instantiation Points Each module provides these:
#### 1. Config (`config.Config`) - `config.Module` provides `*config.Config`, from `config.Load`.
- **Created by**: `config.Module` via `config.LoadConfig()` - `database.Module` provides `*database.DB` and `*database.Repositories`. It
- **When**: Application startup (fx DI) creates the index file with mode `0600` when it does not exist.
- **Contains**: All configuration from YAML file (S3 credentials, encryption keys, paths, etc.) - `storage.Module` provides the `storage.Storer` that the config selects.
- `snapshot.Module` provides the `snapshot.ScannerFactory` and the
`*snapshot.SnapshotManager`.
- `vaultik.New` provides `*vaultik.Vaultik`.
#### 2. Database (`database.DB`) A command runs through `cli.RunOperation`. It works like this:
- **Created by**: `database.Module` via `database.New()`
- **When**: Application startup (fx DI)
- **Contains**: SQLite connection, path reference
#### 3. Repositories (`database.Repositories`) - `RunWithApp` takes the lock when the command changes data, and then starts the
- **Created by**: `database.Module` via `database.NewRepositories()` application.
- **When**: Application startup (fx DI) - The command's function runs in its own goroutine.
- **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots, etc.) - When the function returns, it asks the application to shut down.
- On `SIGINT` or `SIGTERM`, the stop hook cancels the `Vaultik` context. It then
waits for the function to return, so that its temporary files are removed.
- The wait ends after 30 seconds at most.
#### 4. Vaultik (`vaultik.Vaultik`) The `config` commands, `database delete`, `completion` and `version` do not
- **Created by**: `vaultik.New(VaultikParams)` build the application.
- **When**: Application startup (fx DI)
- **Contains**: All dependencies for backup operations
```go ## Backup
type Vaultik struct {
Globals *globals.Globals
Config *config.Config
DB *database.DB
Repositories *database.Repositories
Storage storage.Storer
ScannerFactory snapshot.ScannerFactory
SnapshotManager *snapshot.SnapshotManager
Shutdowner fx.Shutdowner
Fs afero.Fs
ctx context.Context
cancel context.CancelFunc
}
```
#### 5. SnapshotManager (`snapshot.SnapshotManager`) `Vaultik.CreateSnapshot` runs a backup. It does these steps in order:
- **Created by**: `snapshot.Module` via `snapshot.NewSnapshotManager()`
- **When**: Application startup (fx DI)
- **Responsibility**: Creates/completes snapshots, exports metadata to S3
#### 6. Scanner (`snapshot.Scanner`) - `EnsureStorageBinding` checks that the local index belongs to the configured
- **Created by**: `ScannerFactory(ScannerParams)` destination.
- **When**: Each `CreateSnapshot()` call - `markDestination` checks the marker file and then writes it.
- **Contains**: Chunker, Packer, progress reporter - `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `createNamedSnapshot` runs once for each snapshot name.
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
```go `PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
// vaultik/snapshot.go: CreateSnapshot() left in the index would make the backup skip files that were never fully
scanner := v.ScannerFactory(snapshot.ScannerParams{ stored.
EnableProgress: !opts.Cron,
Fs: v.Fs,
})
```
#### 7. Chunker (`chunker.Chunker`) ### One snapshot
- **Created by**: `chunker.NewChunker(avgChunkSize)`
- **When**: Inside `snapshot.NewScanner()`
- **Configuration**:
- `avgChunkSize`: From config (default 10MB)
- `minChunkSize`: avgChunkSize / 4
- `maxChunkSize`: avgChunkSize * 4
#### 8. Packer (`blob.Packer`) `createNamedSnapshot` does these steps:
- **Created by**: `blob.NewPacker(PackerConfig)`
- **When**: Inside `snapshot.NewScanner()`
- **Configuration**:
- `MaxBlobSize`: Maximum blob size before finalization (typically 10GB)
- `CompressionLevel`: zstd level (1-19)
- `Recipients`: age public keys for encryption
```go - `resolveSnapshotPaths` makes each configured path absolute and resolves
// snapshot/scanner.go: NewScanner() symlinks.
packerCfg := blob.PackerConfig{ - `SnapshotManager.CreateSnapshotWithName` creates the snapshot record. The
MaxBlobSize: cfg.MaxBlobSize, record is not yet marked complete.
CompressionLevel: cfg.CompressionLevel, - `Scanner.Scan` backs up each path in turn.
Recipients: cfg.AgeRecipients, - `finalizeSnapshotMetadata` uploads the snapshot's metadata and marks the
Repositories: cfg.Repositories, snapshot complete.
Fs: cfg.FS,
}
packer, err := blob.NewPacker(packerCfg)
```
## Module Responsibilities A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a
`Chunker` and a `Packer`.
### `internal/cli` ### Scanning one path
Entry point for fx application. Combines all modules and handles signal interrupts.
Key functions: `Scanner.Scan` works in these steps:
- `NewApp(AppOptions)` → Creates fx.App with all modules
- `RunApp(ctx, app)` → Starts app, handles graceful shutdown
- `RunWithApp(ctx, opts)` → Convenience wrapper
### `internal/vaultik` - `repairInterruptedBlobs` deletes the records of blobs whose upload never
Main orchestrator containing all dependencies and command implementations. finished. Their data is then stored again.
- `loadDatabaseState` loads the known files and chunks into memory.
- `scanPhase` walks the path. It sorts each entry into changed or unchanged.
- `detectDeletedFiles` counts the files that the previous snapshot of this name
had and that are now gone.
- `batchAddFilesToSnapshot` records the unchanged files in the new snapshot.
- `recordDirectories` records every directory, and the mount points found.
- `processPhase` reads each changed file and stores it.
Key methods: The walk handles each kind of entry differently:
- `New(VaultikParams)` → Constructor (fx DI)
- `CreateSnapshot(opts)` → Main backup operation
- `ListSnapshots(jsonOutput)` → List available snapshots
- `VerifySnapshot(id, deep)` → Verify snapshot integrity
- `PurgeSnapshots(...)` → Remove old snapshots
### `internal/chunker` - A regular file or a symlink is compared with its record in the index.
Content-defined chunking using FastCDC algorithm. - A directory is recorded with its metadata. It has no content to store.
- A device node, a named pipe or a socket is skipped.
- An entry that matches an exclude pattern is skipped. For a directory, so is
everything in it.
Key types: ### Deduplication
- `Chunk` → Hash, Data, Offset, Size
- `Chunker` → avgChunkSize, minChunkSize, maxChunkSize
Key methods: Data is stored once at two levels.
- `NewChunker(avgChunkSize)` → Constructor
- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback (preferred)
- `ChunkReader(reader)` → Return all chunks at once (memory-intensive)
### `internal/blob` - **Files.** A file is not read again when its record in the index still matches
Blob packing: accumulates chunks, compresses, encrypts, tracks metadata. it. The record matches when the size, modification time, mode, owner, group
and link target are all the same.
- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk whose
hash is already in the index is not stored again.
Key types: One case overrides the file check. A file is read again when it lists a chunk
- `Packer` → Thread-safe blob accumulator that no uploaded blob holds. This happens after the only snapshot that used
- `ChunkRef` → Hash + Data for adding to packer such a blob is removed. `loadFilesToRechunk` finds these files.
- `FinishedBlob` → Completed blob ready for upload
- `BlobWithReader` → FinishedBlob + io.Reader for streaming upload
Key methods: ### Chunks
- `NewPacker(PackerConfig)` → Constructor
- `AddChunk(ChunkRef)` → Add chunk to current blob
- `FinalizeBlob()` → Compress, encrypt, hash current blob
- `Flush()` → Finalize any in-progress blob
- `SetBlobHandler(func)` → Set callback for upload
### `internal/snapshot` `chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
on the content, so an insertion early in a file moves only nearby chunks.
#### Scanner `chunk_size` sets the average chunk size. A chunk is between a quarter of that
Orchestrates the backup process for a directory. size and four times that size. With the default of 10 MB, a chunk is between 2.5
MB and 40 MB.
Key methods: `ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole
- `NewScanner(ScannerConfig)` → Constructor (creates Chunker + Packer) file is never held in memory.
- `Scan(ctx, path, snapshotID)` → Main scan operation
Scan phases: ### Blobs
1. **Phase 1**: Walk directory, identify files needing processing
2. **Phase 1b**: Count as deleted the files that the previous snapshot of the same name lists, the walk did not find, and are gone from disk; a file that is excluded, or inside an excluded directory, is not counted
3. **Phase 2**: Process files (chunk → pack → upload)
#### SnapshotManager `blob.Packer` collects new chunks into a blob. For each blob it does this:
Manages snapshot lifecycle and metadata export.
Key methods: - It gives the blob a UUID and creates a temporary file.
- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record - It adds each chunk's bytes to a running SHA-256 hash.
- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot references - In the same pass it compresses the bytes with zstd and encrypts them with
- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3 age. The result goes to the temporary file.
- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a successful export - It records each chunk's offset and length within the blob.
- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark complete (no export between) - It closes the blob when the next chunk would pass `blob_size_limit`, and at
the end of each path.
### `internal/database` The blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. It is computed
SQLite database for local index. Single-writer mode for thread safety. from the chunk data before compression and encryption.
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs) says what follows from
that.
Key types: When a blob is closed, `Scanner.handleBlobReady` stores it:
- `DB` → Database connection wrapper
- `Repositories` → Collection of all repository interfaces
Repository interfaces: - It looks the blob up at the destination. A blob that is already there is not
- `FilesRepository` → CRUD for File records uploaded again.
- `ChunksRepository` → CRUD for Chunk records - Otherwise it uploads the temporary file to
- `BlobsRepository` → CRUD for Blob records `blobs/<hash[0:2]>/<hash[2:4]>/<hash>`.
- `SnapshotsRepository` → CRUD for Snapshot records - It records the upload in the index and links the blob to the snapshot.
- Plus join table repositories (FileChunks, BlobChunks, etc.) - It deletes the temporary file.
## Snapshot Creation Flow A file's record is written to the index only after every blob that holds its
chunks is uploaded.
``` ### Snapshot metadata
CreateSnapshot(opts)
│
├─► markDestination() // Check the marker (see Storage Layout in S3), then write it
│
├─► PruneDatabase() // Critical: avoid dedup errors
│
├─► SnapshotManager.CreateSnapshot() // Create DB record
│
├─► For each source directory:
│ │
│ ├─► scanner.Scan(ctx, path, snapshotID)
│ │ │
│ │ ├─► Phase 1: scanPhase()
│ │ │ Walk directory
│ │ │ Check file metadata changes
│ │ │ Build list of files to process
│ │ │
│ │ ├─► Phase 1b: detectDeletedFiles()
│ │ │
│ │ └─► Phase 2: processPhase()
│ │ For each file:
│ │ chunker.ChunkReaderStreaming()
│ │ For each chunk:
│ │ packer.AddChunk()
│ │ If blob full → FinalizeBlob()
│ │ → handleBlobReady()
│ │ → s3Client.PutObjectWithProgress()
│ │ packer.Flush() // Final blob
│ │
│ └─► Accumulate statistics
│
├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs
│
├─► SnapshotManager.UpdateSnapshotStatsExtended()
│
├─► SnapshotManager.ExportSnapshotMetadata()
│ │
│ ├─► Copy database to temp file
│ ├─► Clean to only current snapshot data (VACUUM)
│ ├─► Compress binary SQLite with zstd
│ ├─► Encrypt with age
│ ├─► Upload db.zst.age to storage
│ └─► Upload manifest.json.zst to storage
│
└─► SnapshotManager.MarkSnapshotComplete() // only after the export succeeds
```
## Deduplication Strategy `finalizeSnapshotMetadata` does these steps in order:
1. **File-level**: Files and symlinks unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked - `PopulateSnapshotBlobs` records every blob the snapshot uses.
- `UpdateSnapshotStatsExtended` stores the snapshot's totals.
- `ExportSnapshotMetadata` uploads the metadata.
- `MarkSnapshotComplete` marks the snapshot complete.
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded. Completion is recorded last on purpose. A backup that stops during the upload
leaves the snapshot unfinished, and the next backup removes it and stores its
data again.
3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a blob are skipped. `ExportSnapshotMetadata` works on a copy of the index, never on the index
itself:
## Storage Layout in S3 - It copies the index into a temporary directory with `VACUUM INTO`.
- It deletes from the copy everything that does not belong to this snapshot.
- It runs `VACUUM`, so that no deleted row is left in the file.
- It compresses the copy with zstd and encrypts it with age.
- It builds the manifest from the copy.
- It uploads `metadata/<remote-key>/db.zst.age`, and then
`metadata/<remote-key>/manifest.json.zst`.
``` The manifest is uploaded last. A directory under `metadata/` with no manifest
bucket/ therefore marks a backup that did not finish.
├── vaultik-destination.txt # Marker written by every backup
│
├── blobs/
│ └── {hash[0:2]}/
│ └── {hash[2:4]}/
│ └── {full-hash} # Compressed+encrypted blob
│
└── metadata/
└── {remote-key}/
├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Blob list (for pruning/verification)
```
A destination that answers a look-up of `vaultik-destination.txt` with not ## Restore
found is treated as not present while the local index holds snapshots stored
there (`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
`snapshot create` refuses to write to it, `prune`, `snapshot purge` and
`remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a
destination that cannot be listed. This keeps those commands from taking an
unplugged volume's empty mount point for a destination whose snapshots are
all gone. When the look-up fails for any reason other than not found, for
example when S3 refuses it to credentials that allow only PUT, the check is
skipped, so a host with such credentials does not notice an emptied
destination; see
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt).
The `file://` backend never creates its destination directory, so a write to a
missing one fails.
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human `Vaultik.Restore` does these steps:
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
written to the store as a directory name. See
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation and a worked example.
## Thread Safety - `restoreIdentities` parses the age private key. A missing or malformed key
stops the restore before anything is downloaded.
- `downloadSnapshotDB` downloads and decrypts the snapshot's database into a
temporary directory.
- `getFilesToRestore` selects the files, from the paths given.
- `restoreAllFiles` downloads blobs and writes the files.
- With `--verify`, `verifyRestoredFiles` reads each restored file and checks its
chunk hashes.
- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`. The restore does not read the local index or the manifest. It finds each file's
- `Scanner`: Uses `packerMu` mutex to coordinate blob finalization. chunks and blobs through the downloaded database.
- `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread safety.
- `Repositories.WithTx()`: Handles transaction lifecycle automatically. ### Checks on the downloaded database
Decryption shows that a database is readable. It does not show that the
database is honest. The restore therefore checks it:
- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
trigger, a view or a virtual table, or that lacks an expected table.
- `verifySnapshotDBIdentity` checks that the snapshot in the database hashes to
the remote key that was asked for.
- `containedRestorePath` refuses a stored path that would leave the target
directory. It also refuses to write through a symlink below the target.
- A blob is not decompressed beyond the size the database records for it.
- Each downloaded blob is hashed. The hash must match the blob's name.
### Order of work
`restorePlan` orders the files so that few blobs are on disk at once:
- Downloaded blobs are kept decrypted in a cache directory.
- A file is restored once every blob it needs is in the cache.
- When no file is ready, the restore downloads the blobs for the file that
needs the fewest.
- `restoreSweeper` deletes a cached blob once no remaining file needs it.
A regular file is created with mode `0600` and gets its stored mode after its
content is written. A directory gets its stored mode, owner and modification
time after the whole restore.
## Verify
`Vaultik.VerifySnapshotWithOptions` checks a snapshot without the private key:
- It downloads the manifest.
- It checks that the snapshot's encrypted database is present.
- It checks that each listed blob is present with the listed size.
`Vaultik.RunDeepVerify` needs the private key. It does these steps:
- It downloads the manifest and the database. It checks the database as a
restore does.
- `verifyManifestAgainstDatabase` checks that both list the same blobs with the
same sizes.
- `verifyBlobExistenceFromDB` checks that every blob is present with the
recorded size.
- `verifyBlob` downloads each blob, checks the hash of each chunk in it, and
checks the hash that names the blob.
## Prune
`Vaultik.Prune` does three steps:
- `CleanupLocalSnapshots` drops each snapshot from the local index whose
directory is gone from the destination.
- `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `PruneBlobs` deletes unused data at the destination.
`PruneBlobs` reads every manifest at the destination. It then deletes two kinds
of object, once they are older than `prune_grace_period`:
- a blob that no manifest lists
- a directory under `metadata/` that has no manifest, and whose snapshot is not
in the local index
It deletes nothing when any manifest cannot be read.
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#pruning) explains why this is
safe, and where it is not.
## The destination
`storage.Storer` is the only way the code reaches the destination. It has these
methods:
- `Put` and `PutWithProgress` store an object.
- `Get` reads an object.
- `Stat` reports an object's size.
- `Delete` removes an object.
- `List` and `ListStream` list the objects under a prefix.
- `DeletePartialUploads` removes what an interrupted upload left behind.
- `Info` describes the destination.
`storage.NewStorer` picks the implementation from `storage_url`:
- `s3://` gives the S3 implementation.
- `file://` gives the local directory implementation.
- `rclone://` gives the rclone implementation.
With no `storage_url`, it builds the S3 implementation from the `s3` settings.
## The marker file and the destination binding
Two checks keep the local index and the destination in step.
- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when
the file is reported missing while the local index holds snapshots.
- `EnsureStorageBinding` records the configured destination in the local index
on first use. Later it fails when the config names a different destination.
Every command that changes data calls `EnsureStorageBinding` first.
## Concurrency
- The local index is opened with one connection, so all access to it is in
turn.
- The index runs in SQLite's WAL mode. A second process can therefore read it
while a backup writes to it.
- `Packer` guards its state with a mutex.
- `Scanner` holds `packerMu` while a chunk is added to the packer and while a
blob is closed.
- `Repositories.WithTx` runs a function in one transaction. It commits when the
function succeeds and rolls back when it fails.