sneak/vaultik
1
0 Fork 0
Code Issues 13 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

Rewrite ARCHITECTURE.md sentence by sentence

Browse Source 
Rewritten from scratch against the code on next: the packages, the fx
wiring, and the steps of a backup, a restore, a verify and a prune.

Names that no longer exist in the code are gone: the S3 client as the
upload step, blob.BlobWithReader, PurgeSnapshots and the repository
interfaces. The storage layer is described through storage.Storer.

Model: fable-5-1
This commit is contained in:
sneak authored and clawbot committed 2026-11-11 07:01:54 +00:00
1 parent 8058b775d6
commit 5afcd4f6b5
1 file changed
+325 -435
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+325 -435
View File
@@ -1,486 +1,376 @@
# Vaultik Architecture
# vaultik architecture
This document describes the internal architecture of Vaultik, focusing on the
data model, type instantiation, and the relationships between core modules.
This document describes how the `vaultik` code is organised. It covers the
packages, how they are wired together, and what happens during a backup, a
restore, a verify and a prune.
## Overview
Two other documents cover the stored data:
Vaultik is a backup system that uses content-defined chunking for deduplication
and packs chunks into large, compressed, encrypted blobs for efficient cloud
storage. The system is built around dependency injection using
[uber-go/fx](https://github.com/uber-go/fx).
- [docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index.
- [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the
destination.
## Data Flow
## Terms
- **Snapshot**: one backup of the paths listed under one name in the config.
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
of its content.
- **Blob**: one object at the destination. It holds many chunks, compressed and
then encrypted.
- **Local index**: the SQLite database on the backed-up machine.
- **Destination**: the place the backups are stored.
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
at the destination.
## How a backup moves data
```
Source Files
source files
│
▼
┌─────────────────┐
│ Scanner │ Walks directories, detects changed files
└────────┬────────┘
│
▼
┌─────────────────┐
│ Chunker │ Splits files into variable-size chunks (FastCDC)
└────────┬────────┘
│
▼
┌─────────────────┐
│ Packer │ Accumulates chunks, compresses (zstd), encrypts (age)
└────────┬────────┘
│
▼
┌─────────────────┐
│ S3 Client │ Uploads blobs to remote storage
└─────────────────┘
Scanner walks the paths and finds the files that changed
│
▼
Chunker cuts each changed file into chunks
│
▼
Packer packs new chunks into a blob, compressed and encrypted
│
▼
Storer uploads the blob to the destination
```
## Data Model
## Packages
### Core Entities
- `cmd/vaultik`: the `main` function. It starts optional profiling and calls
`cli.Entry`.
- `internal/cli`: the commands, their flags and their help text. It builds the
application, takes the lock and handles interrupts.
- `internal/vaultik`: the code behind each command. `Vaultik` holds every
dependency a command uses.
- `internal/snapshot`: the `Scanner`, which backs up one path, and the
`SnapshotManager`, which creates snapshot records and uploads their metadata.
- `internal/chunker`: cuts a stream into chunks with the FastCDC algorithm.
- `internal/blob`: the `Packer`, which collects chunks into blobs.
- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
and the reader that reverses it.
- `internal/database`: the local index. It holds the schema and one repository
type for each table.
- `internal/storage`: the `Storer` interface and its three implementations, for
S3, a local directory and rclone.
- `internal/s3`: the S3 client that the S3 `Storer` uses.
- `internal/config`: loads and checks the config file.
- `internal/ui`: formats the status lines, warnings and errors a user sees.
- `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that keeps two data-changing commands from
running at once.
- `internal/globals`: the version, commit and build date, set when the binary
is built.
- `internal/types`: distinct types for IDs, hashes and paths.
The database tracks five primary entities and their relationships:
## Wiring
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Snapshot │────▶│ File │────▶│ Chunk │
└──────────────┘ └──────────────┘ └──────────────┘
│ │
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Blob │◀─────────────────────────│ BlobChunk │
└──────────────┘ └──────────────┘
```
### Entity Descriptions
#### File (`database.File`)
Represents a file, directory, or symlink in the backup system. Stores metadata
needed for restoration:
- Path, mtime
- Size, mode, ownership (uid, gid)
- Symlink target (if applicable)
It also stores `source_path`, the source directory the scan found it under, made
absolute and with symlinks resolved. Restore does not read it.
#### Chunk (`database.Chunk`)
A content-addressed unit of data. Files are split into variable-size chunks
using the FastCDC algorithm:
- `ChunkHash`: SHA256 hash of chunk content (primary key)
- `Size`: Chunk size in bytes
Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for
the 10MB default average).
#### FileChunk (`database.FileChunk`)
Maps files to their constituent chunks:
- `FileID`: Reference to the file
- `Idx`: Position of this chunk within the file (0-indexed)
- `ChunkHash`: Reference to the chunk
#### Blob (`database.Blob`)
The final storage unit uploaded to S3. Contains many compressed and encrypted
chunks:
- `ID`: UUID assigned at creation
- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before
compression and encryption (see
[docs/REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs))
- `UncompressedSize`: Total raw chunk data before compression
- `CompressedSize`: Size after zstd compression and age encryption
- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
Blob creation process:
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
3. Concurrently, the same bytes are compressed with zstd, then encrypted with
age (recipients configured in config), and written to a temporary file
4. On finalize, the blob's name is the double SHA-256 of the uncompressed
contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed,
encrypted bytes
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and
then deleted
A backup needs free temporary space, because each blob is written whole to a
temporary file before it is uploaded (up to about `blob_size_limit`; an rclone
destination that cannot stream uploads needs about twice that) and the metadata
export writes copies of the local index. Temporary files go to `$TMPDIR`
(default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to
`/var/tmp`.
#### BlobChunk (`database.BlobChunk`)
Maps chunks to their position within blobs:
- `BlobID`: Reference to the blob
- `ChunkHash`: Reference to the chunk
- `Offset`: Byte offset within the uncompressed blob
- `Length`: Chunk size
#### Snapshot (`database.Snapshot`)
Represents a point-in-time backup:
- `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g.
`server1_home_2025-06-01T12:00:00Z`)
- Tracks file count, chunk count, blob count, sizes, compression ratio
- `CompletedAt`: Null until snapshot finishes successfully
#### SnapshotFile / SnapshotBlob
Join tables linking snapshots to their files and blobs.
### Relationship Summary
```
Snapshot 1──────────▶ N SnapshotFile N ◀────────── 1 File
Snapshot 1──────────▶ N SnapshotBlob N ◀────────── 1 Blob
File 1──────────▶ N FileChunk N ◀────────── 1 Chunk
Blob 1──────────▶ N BlobChunk N ◀────────── 1 Chunk
```
## Type Instantiation
### Application Startup
The CLI uses fx for dependency injection. Here's the instantiation order:
The application is assembled with [uber-go/fx](https://github.com/uber-go/fx), a
dependency injection library. `cli.NewApp` lists the providers:
```go
// cli/app.go: NewApp()
fx.New(
fx.Supply(config.Path(opts.ConfigPath)), // 1. Config path
fx.Supply(opts.LogOptions), // 2. Log options
fx.Provide(globals.New), // 3. Globals
fx.Provide(log.New), // 4. Logger config
config.Module, // 5. Config
database.Module, // 6. Database + Repositories
log.Module, // 7. Logger initialization
storage.Module, // 8. Storage backend (S3/file/rclone)
snapshot.Module, // 9. SnapshotManager + ScannerFactory
fx.Provide(vaultik.New), // 10. Vaultik orchestrator
)
fx.Supply(config.Path(opts.ConfigPath)),
fx.Supply(opts.LogOptions),
fx.Provide(globals.New),
fx.Provide(log.New),
config.Module,
database.Module,
log.Module,
storage.Module,
snapshot.Module,
fx.Provide(vaultik.New),
fx.Invoke(setupGlobals),
fx.NopLogger,
```
### Key Type Instantiation Points
#### 1. Config (`config.Config`)
- **Created by**: `config.Module` via `config.LoadConfig()`
- **When**: Application startup (fx DI)
- **Contains**: All configuration from YAML file (S3 credentials, encryption
keys, paths, etc.)
#### 2. Database (`database.DB`)
Each module provides these:
- **Created by**: `database.Module` via `database.New()`
- **When**: Application startup (fx DI)
- **Contains**: SQLite connection, path reference
- `config.Module` provides `*config.Config`, from `config.Load`.
- `database.Module` provides `*database.DB` and `*database.Repositories`. It
creates the index file with mode `0600` when it does not exist.
- `storage.Module` provides the `storage.Storer` that the config selects.
- `snapshot.Module` provides the `snapshot.ScannerFactory` and the
`*snapshot.SnapshotManager`.
- `vaultik.New` provides `*vaultik.Vaultik`.
#### 3. Repositories (`database.Repositories`)
- **Created by**: `database.Module` via `database.NewRepositories()`
- **When**: Application startup (fx DI)
- **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots,
etc.)
A command runs through `cli.RunOperation`. It works like this:
#### 4. Vaultik (`vaultik.Vaultik`)
- `RunWithApp` takes the lock when the command changes data, and then starts the
application.
- The command's function runs in its own goroutine.
- When the function returns, it asks the application to shut down.
- On `SIGINT` or `SIGTERM`, the stop hook cancels the `Vaultik` context. It then
waits for the function to return, so that its temporary files are removed.
- The wait ends after 30 seconds at most.
- **Created by**: `vaultik.New(VaultikParams)`
- **When**: Application startup (fx DI)
- **Contains**: All dependencies for backup operations
The `config` commands, `database delete`, `completion` and `version` do not
build the application.
```go
type Vaultik struct {
Globals *globals.Globals
Config *config.Config
DB *database.DB
Repositories *database.Repositories
Storage storage.Storer
ScannerFactory snapshot.ScannerFactory
SnapshotManager *snapshot.SnapshotManager
Shutdowner fx.Shutdowner
Fs afero.Fs
ctx context.Context
cancel context.CancelFunc
}
```
## Backup
#### 5. SnapshotManager (`snapshot.SnapshotManager`)
`Vaultik.CreateSnapshot` runs a backup. It does these steps in order:
- **Created by**: `snapshot.Module` via `snapshot.NewSnapshotManager()`
- **When**: Application startup (fx DI)
- **Responsibility**: Creates/completes snapshots, exports metadata to S3
- `EnsureStorageBinding` checks that the local index belongs to the configured
destination.
- `markDestination` checks the marker file and then writes it.
- `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `createNamedSnapshot` runs once for each snapshot name.
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
#### 6. Scanner (`snapshot.Scanner`)
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
left in the index would make the backup skip files that were never fully
stored.
- **Created by**: `ScannerFactory(ScannerParams)`
- **When**: Each `CreateSnapshot()` call
- **Contains**: Chunker, Packer, progress reporter
### One snapshot
```go
// vaultik/snapshot.go: CreateSnapshot()
scanner := v.ScannerFactory(snapshot.ScannerParams{
EnableProgress: !opts.Cron,
Fs: v.Fs,
})
```
`createNamedSnapshot` does these steps:
#### 7. Chunker (`chunker.Chunker`)
- `resolveSnapshotPaths` makes each configured path absolute and resolves
symlinks.
- `SnapshotManager.CreateSnapshotWithName` creates the snapshot record. The
record is not yet marked complete.
- `Scanner.Scan` backs up each path in turn.
- `finalizeSnapshotMetadata` uploads the snapshot's metadata and marks the
snapshot complete.
- **Created by**: `chunker.NewChunker(avgChunkSize)`
- **When**: Inside `snapshot.NewScanner()`
- **Configuration**:
- `avgChunkSize`: From config (default 10MB)
- `minChunkSize`: avgChunkSize / 4
- `maxChunkSize`: avgChunkSize \* 4
A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a
`Chunker` and a `Packer`.
#### 8. Packer (`blob.Packer`)
### Scanning one path
- **Created by**: `blob.NewPacker(PackerConfig)`
- **When**: Inside `snapshot.NewScanner()`
- **Configuration**:
- `MaxBlobSize`: Maximum blob size before finalization (typically 10GB)
- `CompressionLevel`: zstd level (1-19)
- `Recipients`: age public keys for encryption
`Scanner.Scan` works in these steps:
```go
// snapshot/scanner.go: NewScanner()
packerCfg := blob.PackerConfig{
MaxBlobSize: cfg.MaxBlobSize,
CompressionLevel: cfg.CompressionLevel,
Recipients: cfg.AgeRecipients,
Repositories: cfg.Repositories,
Fs: cfg.FS,
}
packer, err := blob.NewPacker(packerCfg)
```
- `repairInterruptedBlobs` deletes the records of blobs whose upload never
finished. Their data is then stored again.
- `loadDatabaseState` loads the known files and chunks into memory.
- `scanPhase` walks the path. It sorts each entry into changed or unchanged.
- `detectDeletedFiles` counts the files that the previous snapshot of this name
had and that are now gone.
- `batchAddFilesToSnapshot` records the unchanged files in the new snapshot.
- `recordDirectories` records every directory, and the mount points found.
- `processPhase` reads each changed file and stores it.
## Module Responsibilities
The walk handles each kind of entry differently:
### `internal/cli`
- A regular file or a symlink is compared with its record in the index.
- A directory is recorded with its metadata. It has no content to store.
- A device node, a named pipe or a socket is skipped.
- An entry that matches an exclude pattern is skipped. For a directory, so is
everything in it.
Entry point for fx application. Combines all modules and handles signal
interrupts.
### Deduplication
Key functions:
Data is stored once at two levels.
- `NewApp(AppOptions)` → Creates fx.App with all modules
- `RunApp(ctx, app)` → Starts app, handles graceful shutdown
- `RunWithApp(ctx, opts)` → Convenience wrapper
- **Files.** A file is not read again when its record in the index still matches
it. The record matches when the size, modification time, mode, owner, group
and link target are all the same.
- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk whose
hash is already in the index is not stored again.
### `internal/vaultik`
One case overrides the file check. A file is read again when it lists a chunk
that no uploaded blob holds. This happens after the only snapshot that used
such a blob is removed. `loadFilesToRechunk` finds these files.
Main orchestrator containing all dependencies and command implementations.
### Chunks
Key methods:
`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
on the content, so an insertion early in a file moves only nearby chunks.
- `New(VaultikParams)` → Constructor (fx DI)
- `CreateSnapshot(opts)` → Main backup operation
- `ListSnapshots(jsonOutput)` → List available snapshots
- `VerifySnapshot(id, deep)` → Verify snapshot integrity
- `PurgeSnapshots(...)` → Remove old snapshots
`chunk_size` sets the average chunk size. A chunk is between a quarter of that
size and four times that size. With the default of 10 MB, a chunk is between 2.5
MB and 40 MB.
### `internal/chunker`
`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole
file is never held in memory.
Content-defined chunking using FastCDC algorithm.
### Blobs
Key types:
`blob.Packer` collects new chunks into a blob. For each blob it does this:
- `Chunk` → Hash, Data, Offset, Size
- `Chunker` → avgChunkSize, minChunkSize, maxChunkSize
- It gives the blob a UUID and creates a temporary file.
- It adds each chunk's bytes to a running SHA-256 hash.
- In the same pass it compresses the bytes with zstd and encrypts them with
age. The result goes to the temporary file.
- It records each chunk's offset and length within the blob.
- It closes the blob when the next chunk would pass `blob_size_limit`, and at
the end of each path.
Key methods:
The blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. It is computed
from the chunk data before compression and encryption.
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs) says what follows from
that.
- `NewChunker(avgChunkSize)` → Constructor
- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback
(preferred)
- `ChunkReader(reader)` → Return all chunks at once (memory-intensive)
When a blob is closed, `Scanner.handleBlobReady` stores it:
### `internal/blob`
- It looks the blob up at the destination. A blob that is already there is not
uploaded again.
- Otherwise it uploads the temporary file to
`blobs/<hash[0:2]>/<hash[2:4]>/<hash>`.
- It records the upload in the index and links the blob to the snapshot.
- It deletes the temporary file.
Blob packing: accumulates chunks, compresses, encrypts, tracks metadata.
A file's record is written to the index only after every blob that holds its
chunks is uploaded.
Key types:
### Snapshot metadata
- `Packer` → Thread-safe blob accumulator
- `ChunkRef` → Hash + Data for adding to packer
- `FinishedBlob` → Completed blob ready for upload
- `BlobWithReader` → FinishedBlob + io.Reader for streaming upload
Key methods:
- `NewPacker(PackerConfig)` → Constructor
- `AddChunk(ChunkRef)` → Add chunk to current blob
- `FinalizeBlob()` → Compress, encrypt, hash current blob
- `Flush()` → Finalize any in-progress blob
- `SetBlobHandler(func)` → Set callback for upload
### `internal/snapshot`
#### Scanner
Orchestrates the backup process for a directory.
Key methods:
- `NewScanner(ScannerConfig)` → Constructor (creates Chunker + Packer)
- `Scan(ctx, path, snapshotID)` → Main scan operation
Scan phases:
1. **Phase 1**: Walk directory, identify files needing processing
2. **Phase 1b**: Count as deleted the files that the previous snapshot of the
same name lists, the walk did not find, and are gone from disk; a file that
is excluded, or inside an excluded directory, is not counted
3. **Phase 2**: Process files (chunk → pack → upload)
#### SnapshotManager
Manages snapshot lifecycle and metadata export.
Key methods:
- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record
- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot
references
- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3
- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a
successful export
- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark
complete (no export between)
### `internal/database`
SQLite database for local index. Single-writer mode for thread safety.
Key types:
- `DB` → Database connection wrapper
- `Repositories` → Collection of all repository interfaces
Repository interfaces:
- `FilesRepository` → CRUD for File records
- `ChunksRepository` → CRUD for Chunk records
- `BlobsRepository` → CRUD for Blob records
- `SnapshotsRepository` → CRUD for Snapshot records
- Plus join table repositories (FileChunks, BlobChunks, etc.)
## Snapshot Creation Flow
```
CreateSnapshot(opts)
│
├─► markDestination() // Check the marker (see Storage Layout in S3), then write it
│
├─► PruneDatabase() // Critical: avoid dedup errors
│
├─► SnapshotManager.CreateSnapshot() // Create DB record
│
├─► For each source directory:
│ │
│ ├─► scanner.Scan(ctx, path, snapshotID)
│ │ │
│ │ ├─► Phase 1: scanPhase()
│ │ │ Walk directory
│ │ │ Check file metadata changes
│ │ │ Build list of files to process
│ │ │
│ │ ├─► Phase 1b: detectDeletedFiles()
│ │ │
│ │ └─► Phase 2: processPhase()
│ │ For each file:
│ │ chunker.ChunkReaderStreaming()
│ │ For each chunk:
│ │ packer.AddChunk()
│ │ If blob full → FinalizeBlob()
│ │ → handleBlobReady()
│ │ → s3Client.PutObjectWithProgress()
│ │ packer.Flush() // Final blob
│ │
│ └─► Accumulate statistics
│
├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs
│
├─► SnapshotManager.UpdateSnapshotStatsExtended()
│
├─► SnapshotManager.ExportSnapshotMetadata()
│ │
│ ├─► Copy database to temp file
│ ├─► Clean to only current snapshot data (VACUUM)
│ ├─► Compress binary SQLite with zstd
│ ├─► Encrypt with age
│ ├─► Upload db.zst.age to storage
│ └─► Upload manifest.json.zst to storage
│
└─► SnapshotManager.MarkSnapshotComplete() // only after the export succeeds
```
## Deduplication Strategy
1. **File-level**: Files and symlinks unchanged since last backup are skipped
(metadata comparison: size, mtime, mode, uid, gid, and a symlink's target),
unless the file lists a chunk that no uploaded blob holds; such a file is
re-chunked
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash
already exists in the database, the chunk data is not re-uploaded.
3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a
blob are skipped.
## Storage Layout in S3
```
bucket/
├── vaultik-destination.txt # Marker written by every backup
│
├── blobs/
│ └── {hash[0:2]}/
│ └── {hash[2:4]}/
│ └── {full-hash} # Compressed+encrypted blob
│
└── metadata/
└── {remote-key}/
├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Blob list (for pruning/verification)
```
A destination that answers a look-up of `vaultik-destination.txt` with not found
is treated as not present while the local index holds snapshots stored there
(`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
`snapshot create` refuses to write to it, `prune`, `snapshot purge` and
`remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a
destination that cannot be listed. This keeps those commands from taking an
unplugged volume's empty mount point for a destination whose snapshots are all
gone. When the look-up fails for any reason other than not found, for example
when S3 refuses it to credentials that allow only PUT, the check is skipped, so
a host with such credentials does not notice an emptied destination; see
[docs/REPOSTRUCTURE.md](REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt).
The `file://` backend never creates its destination directory, so a write to a
missing one fails.
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
written to the store as a directory name. See
[docs/REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) for the
derivation and a worked example.
## Thread Safety
- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
- `Scanner`: Uses `packerMu` mutex to coordinate blob finalization.
- `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread
safety.
- `Repositories.WithTx()`: Handles transaction lifecycle automatically.
`finalizeSnapshotMetadata` does these steps in order:
- `PopulateSnapshotBlobs` records every blob the snapshot uses.
- `UpdateSnapshotStatsExtended` stores the snapshot's totals.
- `ExportSnapshotMetadata` uploads the metadata.
- `MarkSnapshotComplete` marks the snapshot complete.
Completion is recorded last on purpose. A backup that stops during the upload
leaves the snapshot unfinished, and the next backup removes it and stores its
data again.
`ExportSnapshotMetadata` works on a copy of the index, never on the index
itself:
- It copies the index into a temporary directory with `VACUUM INTO`.
- It deletes from the copy everything that does not belong to this snapshot.
- It runs `VACUUM`, so that no deleted row is left in the file.
- It compresses the copy with zstd and encrypts it with age.
- It builds the manifest from the copy.
- It uploads `metadata/<remote-key>/db.zst.age`, and then
`metadata/<remote-key>/manifest.json.zst`.
The manifest is uploaded last. A directory under `metadata/` with no manifest
therefore marks a backup that did not finish.
## Restore
`Vaultik.Restore` does these steps:
- `restoreIdentities` parses the age private key. A missing or malformed key
stops the restore before anything is downloaded.
- `downloadSnapshotDB` downloads and decrypts the snapshot's database into a
temporary directory.
- `getFilesToRestore` selects the files, from the paths given.
- `restoreAllFiles` downloads blobs and writes the files.
- With `--verify`, `verifyRestoredFiles` reads each restored file and checks its
chunk hashes.
The restore does not read the local index or the manifest. It finds each file's
chunks and blobs through the downloaded database.
### Checks on the downloaded database
Decryption shows that a database is readable. It does not show that the
database is honest. The restore therefore checks it:
- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
trigger, a view or a virtual table, or that lacks an expected table.
- `verifySnapshotDBIdentity` checks that the snapshot in the database hashes to
the remote key that was asked for.
- `containedRestorePath` refuses a stored path that would leave the target
directory. It also refuses to write through a symlink below the target.
- A blob is not decompressed beyond the size the database records for it.
- Each downloaded blob is hashed. The hash must match the blob's name.
### Order of work
`restorePlan` orders the files so that few blobs are on disk at once:
- Downloaded blobs are kept decrypted in a cache directory.
- A file is restored once every blob it needs is in the cache.
- When no file is ready, the restore downloads the blobs for the file that
needs the fewest.
- `restoreSweeper` deletes a cached blob once no remaining file needs it.
A regular file is created with mode `0600` and gets its stored mode after its
content is written. A directory gets its stored mode, owner and modification
time after the whole restore.
## Verify
`Vaultik.VerifySnapshotWithOptions` checks a snapshot without the private key:
- It downloads the manifest.
- It checks that the snapshot's encrypted database is present.
- It checks that each listed blob is present with the listed size.
`Vaultik.RunDeepVerify` needs the private key. It does these steps:
- It downloads the manifest and the database. It checks the database as a
restore does.
- `verifyManifestAgainstDatabase` checks that both list the same blobs with the
same sizes.
- `verifyBlobExistenceFromDB` checks that every blob is present with the
recorded size.
- `verifyBlob` downloads each blob, checks the hash of each chunk in it, and
checks the hash that names the blob.
## Prune
`Vaultik.Prune` does three steps:
- `CleanupLocalSnapshots` drops each snapshot from the local index whose
directory is gone from the destination.
- `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `PruneBlobs` deletes unused data at the destination.
`PruneBlobs` reads every manifest at the destination. It then deletes two kinds
of object, once they are older than `prune_grace_period`:
- a blob that no manifest lists
- a directory under `metadata/` that has no manifest, and whose snapshot is not
in the local index
It deletes nothing when any manifest cannot be read.
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#pruning) explains why this is
safe, and where it is not.
## The destination
`storage.Storer` is the only way the code reaches the destination. It has these
methods:
- `Put` and `PutWithProgress` store an object.
- `Get` reads an object.
- `Stat` reports an object's size.
- `Delete` removes an object.
- `List` and `ListStream` list the objects under a prefix.
- `DeletePartialUploads` removes what an interrupted upload left behind.
- `Info` describes the destination.
`storage.NewStorer` picks the implementation from `storage_url`:
- `s3://` gives the S3 implementation.
- `file://` gives the local directory implementation.
- `rclone://` gives the rclone implementation.
With no `storage_url`, it builds the S3 implementation from the `s3` settings.
## The marker file and the destination binding
Two checks keep the local index and the destination in step.
- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when
the file is reported missing while the local index holds snapshots.
- `EnsureStorageBinding` records the configured destination in the local index
on first use. Later it fails when the config names a different destination.
Every command that changes data calls `EnsureStorageBinding` first.
## Concurrency
- The local index is opened with one connection, so all access to it is in
turn.
- The index runs in SQLite's WAL mode. A second process can therefore read it
while a backup writes to it.
- `Packer` guards its state with a mutex.
- `Scanner` holds `packerMu` while a chunk is added to the packer and while a
blob is closed.
- `Repositories.WithTx` runs a function in one transaction. It commits when the
function succeeds and rolls back when it fails.