From 43c9b2c39bc8ac462f4d2b93219407bca75ba9b6 Mon Sep 17 00:00:00 2001 From: sneak Date: Fri, 6 Nov 2026 18:23:17 +0000 Subject: [PATCH] Rewrite ARCHITECTURE.md sentence by sentence Rewritten from scratch against the code on next: the packages, the fx wiring, and the steps of a backup, a restore, a verify and a prune. Names that no longer exist in the code are gone: the S3 client as the upload step, blob.BlobWithReader, PurgeSnapshots and the repository interfaces. The storage layer is described through storage.Storer. Model: fable-5-1 --- ARCHITECTURE.md | 667 +++++++++++++++++++++++------------------------- 1 file changed, 316 insertions(+), 351 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b41dc22..48a44b2 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,411 +1,376 @@ -# Vaultik Architecture +# vaultik architecture -This document describes the internal architecture of Vaultik, focusing on the data model, type instantiation, and the relationships between core modules. +This document describes how the `vaultik` code is organised. It covers the +packages, how they are wired together, and what happens during a backup, a +restore, a verify and a prune. -## Overview +Two other documents cover the stored data: -Vaultik is a backup system that uses content-defined chunking for deduplication and packs chunks into large, compressed, encrypted blobs for efficient cloud storage. The system is built around dependency injection using [uber-go/fx](https://github.com/uber-go/fx). +- [docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index. +- [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the + destination. -## Data Flow +## Terms + +- **Snapshot**: one backup of the paths listed under one name in the config. +- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash + of its content. +- **Blob**: one object at the destination. It holds many chunks, compressed and + then encrypted. +- **Local index**: the SQLite database on the backed-up machine. +- **Destination**: the place the backups are stored. +- **Manifest**: the unencrypted list of the blobs one snapshot uses. +- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory + at the destination. + +## How a backup moves data ``` -Source Files +source files │ ▼ -┌─────────────────┐ -│ Scanner │ Walks directories, detects changed files -└────────┬────────┘ - │ - ▼ -┌─────────────────┐ -│ Chunker │ Splits files into variable-size chunks (FastCDC) -└────────┬────────┘ - │ - ▼ -┌─────────────────┐ -│ Packer │ Accumulates chunks, compresses (zstd), encrypts (age) -└────────┬────────┘ - │ - ▼ -┌─────────────────┐ -│ S3 Client │ Uploads blobs to remote storage -└─────────────────┘ + Scanner walks the paths and finds the files that changed + │ + ▼ + Chunker cuts each changed file into chunks + │ + ▼ + Packer packs new chunks into a blob, compressed and encrypted + │ + ▼ + Storer uploads the blob to the destination ``` -## Data Model +## Packages -### Core Entities +- `cmd/vaultik`: the `main` function. It starts optional profiling and calls + `cli.Entry`. +- `internal/cli`: the commands, their flags and their help text. It builds the + application, takes the lock and handles interrupts. +- `internal/vaultik`: the code behind each command. `Vaultik` holds every + dependency a command uses. +- `internal/snapshot`: the `Scanner`, which backs up one path, and the + `SnapshotManager`, which creates snapshot records and uploads their metadata. +- `internal/chunker`: cuts a stream into chunks with the FastCDC algorithm. +- `internal/blob`: the `Packer`, which collects chunks into blobs. +- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob, + and the reader that reverses it. +- `internal/database`: the local index. It holds the schema and one repository + type for each table. +- `internal/storage`: the `Storer` interface and its three implementations, for + S3, a local directory and rclone. +- `internal/s3`: the S3 client that the S3 `Storer` uses. +- `internal/config`: loads and checks the config file. +- `internal/ui`: formats the status lines, warnings and errors a user sees. +- `internal/log`: the logger. It writes to stderr. +- `internal/pidlock`: the lock that keeps two data-changing commands from + running at once. +- `internal/globals`: the version, commit and build date, set when the binary + is built. +- `internal/types`: distinct types for IDs, hashes and paths. -The database tracks five primary entities and their relationships: +## Wiring -``` -┌──────────────┐ ┌──────────────┐ ┌──────────────┐ -│ Snapshot │────▶│ File │────▶│ Chunk │ -└──────────────┘ └──────────────┘ └──────────────┘ - │ │ - │ │ - ▼ ▼ -┌──────────────┐ ┌──────────────┐ -│ Blob │◀─────────────────────────│ BlobChunk │ -└──────────────┘ └──────────────┘ -``` - -### Entity Descriptions - -#### File (`database.File`) -Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration: -- Path, mtime -- Size, mode, ownership (uid, gid) -- Symlink target (if applicable) - -It also stores `source_path`, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it. - -#### Chunk (`database.Chunk`) -A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm: -- `ChunkHash`: SHA256 hash of chunk content (primary key) -- `Size`: Chunk size in bytes - -Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for the 10MB default average). - -#### FileChunk (`database.FileChunk`) -Maps files to their constituent chunks: -- `FileID`: Reference to the file -- `Idx`: Position of this chunk within the file (0-indexed) -- `ChunkHash`: Reference to the chunk - -#### Blob (`database.Blob`) -The final storage unit uploaded to S3. Contains many compressed and encrypted chunks: -- `ID`: UUID assigned at creation -- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs)) -- `UncompressedSize`: Total raw chunk data before compression -- `CompressedSize`: Size after zstd compression and age encryption -- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps - -Blob creation process: -1. Chunks are accumulated (up to MaxBlobSize, typically 10GB) -2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256 -3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file -4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes -5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted - -A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`. - -#### BlobChunk (`database.BlobChunk`) -Maps chunks to their position within blobs: -- `BlobID`: Reference to the blob -- `ChunkHash`: Reference to the chunk -- `Offset`: Byte offset within the uncompressed blob -- `Length`: Chunk size - -#### Snapshot (`database.Snapshot`) -Represents a point-in-time backup: -- `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g. `server1_home_2025-06-01T12:00:00Z`) -- Tracks file count, chunk count, blob count, sizes, compression ratio -- `CompletedAt`: Null until snapshot finishes successfully - -#### SnapshotFile / SnapshotBlob -Join tables linking snapshots to their files and blobs. - -### Relationship Summary - -``` -Snapshot 1──────────▶ N SnapshotFile N ◀────────── 1 File -Snapshot 1──────────▶ N SnapshotBlob N ◀────────── 1 Blob -File 1──────────▶ N FileChunk N ◀────────── 1 Chunk -Blob 1──────────▶ N BlobChunk N ◀────────── 1 Chunk -``` - -## Type Instantiation - -### Application Startup - -The CLI uses fx for dependency injection. Here's the instantiation order: +The application is assembled with [uber-go/fx](https://github.com/uber-go/fx), a +dependency injection library. `cli.NewApp` lists the providers: ```go -// cli/app.go: NewApp() -fx.New( - fx.Supply(config.Path(opts.ConfigPath)), // 1. Config path - fx.Supply(opts.LogOptions), // 2. Log options - fx.Provide(globals.New), // 3. Globals - fx.Provide(log.New), // 4. Logger config - config.Module, // 5. Config - database.Module, // 6. Database + Repositories - log.Module, // 7. Logger initialization - storage.Module, // 8. Storage backend (S3/file/rclone) - snapshot.Module, // 9. SnapshotManager + ScannerFactory - fx.Provide(vaultik.New), // 10. Vaultik orchestrator -) +fx.Supply(config.Path(opts.ConfigPath)), +fx.Supply(opts.LogOptions), +fx.Provide(globals.New), +fx.Provide(log.New), +config.Module, +database.Module, +log.Module, +storage.Module, +snapshot.Module, +fx.Provide(vaultik.New), +fx.Invoke(setupGlobals), +fx.NopLogger, ``` -### Key Type Instantiation Points +Each module provides these: -#### 1. Config (`config.Config`) -- **Created by**: `config.Module` via `config.LoadConfig()` -- **When**: Application startup (fx DI) -- **Contains**: All configuration from YAML file (S3 credentials, encryption keys, paths, etc.) +- `config.Module` provides `*config.Config`, from `config.Load`. +- `database.Module` provides `*database.DB` and `*database.Repositories`. It + creates the index file with mode `0600` when it does not exist. +- `storage.Module` provides the `storage.Storer` that the config selects. +- `snapshot.Module` provides the `snapshot.ScannerFactory` and the + `*snapshot.SnapshotManager`. +- `vaultik.New` provides `*vaultik.Vaultik`. -#### 2. Database (`database.DB`) -- **Created by**: `database.Module` via `database.New()` -- **When**: Application startup (fx DI) -- **Contains**: SQLite connection, path reference +A command runs through `cli.RunOperation`. It works like this: -#### 3. Repositories (`database.Repositories`) -- **Created by**: `database.Module` via `database.NewRepositories()` -- **When**: Application startup (fx DI) -- **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots, etc.) +- `RunWithApp` takes the lock when the command changes data, and then starts the + application. +- The command's function runs in its own goroutine. +- When the function returns, it asks the application to shut down. +- On `SIGINT` or `SIGTERM`, the stop hook cancels the `Vaultik` context. It then + waits for the function to return, so that its temporary files are removed. +- The wait ends after 30 seconds at most. -#### 4. Vaultik (`vaultik.Vaultik`) -- **Created by**: `vaultik.New(VaultikParams)` -- **When**: Application startup (fx DI) -- **Contains**: All dependencies for backup operations +The `config` commands, `database delete`, `completion` and `version` do not +build the application. -```go -type Vaultik struct { - Globals *globals.Globals - Config *config.Config - DB *database.DB - Repositories *database.Repositories - Storage storage.Storer - ScannerFactory snapshot.ScannerFactory - SnapshotManager *snapshot.SnapshotManager - Shutdowner fx.Shutdowner - Fs afero.Fs - ctx context.Context - cancel context.CancelFunc -} -``` +## Backup -#### 5. SnapshotManager (`snapshot.SnapshotManager`) -- **Created by**: `snapshot.Module` via `snapshot.NewSnapshotManager()` -- **When**: Application startup (fx DI) -- **Responsibility**: Creates/completes snapshots, exports metadata to S3 +`Vaultik.CreateSnapshot` runs a backup. It does these steps in order: -#### 6. Scanner (`snapshot.Scanner`) -- **Created by**: `ScannerFactory(ScannerParams)` -- **When**: Each `CreateSnapshot()` call -- **Contains**: Chunker, Packer, progress reporter +- `EnsureStorageBinding` checks that the local index belongs to the configured + destination. +- `markDestination` checks the marker file and then writes it. +- `PruneDatabase` removes unfinished snapshots and unused records from the + local index. +- `createNamedSnapshot` runs once for each snapshot name. +- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data. -```go -// vaultik/snapshot.go: CreateSnapshot() -scanner := v.ScannerFactory(snapshot.ScannerParams{ - EnableProgress: !opts.Cron, - Fs: v.Fs, -}) -``` +`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot +left in the index would make the backup skip files that were never fully +stored. -#### 7. Chunker (`chunker.Chunker`) -- **Created by**: `chunker.NewChunker(avgChunkSize)` -- **When**: Inside `snapshot.NewScanner()` -- **Configuration**: - - `avgChunkSize`: From config (default 10MB) - - `minChunkSize`: avgChunkSize / 4 - - `maxChunkSize`: avgChunkSize * 4 +### One snapshot -#### 8. Packer (`blob.Packer`) -- **Created by**: `blob.NewPacker(PackerConfig)` -- **When**: Inside `snapshot.NewScanner()` -- **Configuration**: - - `MaxBlobSize`: Maximum blob size before finalization (typically 10GB) - - `CompressionLevel`: zstd level (1-19) - - `Recipients`: age public keys for encryption +`createNamedSnapshot` does these steps: -```go -// snapshot/scanner.go: NewScanner() -packerCfg := blob.PackerConfig{ - MaxBlobSize: cfg.MaxBlobSize, - CompressionLevel: cfg.CompressionLevel, - Recipients: cfg.AgeRecipients, - Repositories: cfg.Repositories, - Fs: cfg.FS, -} -packer, err := blob.NewPacker(packerCfg) -``` +- `resolveSnapshotPaths` makes each configured path absolute and resolves + symlinks. +- `SnapshotManager.CreateSnapshotWithName` creates the snapshot record. The + record is not yet marked complete. +- `Scanner.Scan` backs up each path in turn. +- `finalizeSnapshotMetadata` uploads the snapshot's metadata and marks the + snapshot complete. -## Module Responsibilities +A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a +`Chunker` and a `Packer`. -### `internal/cli` -Entry point for fx application. Combines all modules and handles signal interrupts. +### Scanning one path -Key functions: -- `NewApp(AppOptions)` → Creates fx.App with all modules -- `RunApp(ctx, app)` → Starts app, handles graceful shutdown -- `RunWithApp(ctx, opts)` → Convenience wrapper +`Scanner.Scan` works in these steps: -### `internal/vaultik` -Main orchestrator containing all dependencies and command implementations. +- `repairInterruptedBlobs` deletes the records of blobs whose upload never + finished. Their data is then stored again. +- `loadDatabaseState` loads the known files and chunks into memory. +- `scanPhase` walks the path. It sorts each entry into changed or unchanged. +- `detectDeletedFiles` counts the files that the previous snapshot of this name + had and that are now gone. +- `batchAddFilesToSnapshot` records the unchanged files in the new snapshot. +- `recordDirectories` records every directory, and the mount points found. +- `processPhase` reads each changed file and stores it. -Key methods: -- `New(VaultikParams)` → Constructor (fx DI) -- `CreateSnapshot(opts)` → Main backup operation -- `ListSnapshots(jsonOutput)` → List available snapshots -- `VerifySnapshot(id, deep)` → Verify snapshot integrity -- `PurgeSnapshots(...)` → Remove old snapshots +The walk handles each kind of entry differently: -### `internal/chunker` -Content-defined chunking using FastCDC algorithm. +- A regular file or a symlink is compared with its record in the index. +- A directory is recorded with its metadata. It has no content to store. +- A device node, a named pipe or a socket is skipped. +- An entry that matches an exclude pattern is skipped. For a directory, so is + everything in it. -Key types: -- `Chunk` → Hash, Data, Offset, Size -- `Chunker` → avgChunkSize, minChunkSize, maxChunkSize +### Deduplication -Key methods: -- `NewChunker(avgChunkSize)` → Constructor -- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback (preferred) -- `ChunkReader(reader)` → Return all chunks at once (memory-intensive) +Data is stored once at two levels. -### `internal/blob` -Blob packing: accumulates chunks, compresses, encrypts, tracks metadata. +- **Files.** A file is not read again when its record in the index still matches + it. The record matches when the size, modification time, mode, owner, group + and link target are all the same. +- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk whose + hash is already in the index is not stored again. -Key types: -- `Packer` → Thread-safe blob accumulator -- `ChunkRef` → Hash + Data for adding to packer -- `FinishedBlob` → Completed blob ready for upload -- `BlobWithReader` → FinishedBlob + io.Reader for streaming upload +One case overrides the file check. A file is read again when it lists a chunk +that no uploaded blob holds. This happens after the only snapshot that used +such a blob is removed. `loadFilesToRechunk` finds these files. -Key methods: -- `NewPacker(PackerConfig)` → Constructor -- `AddChunk(ChunkRef)` → Add chunk to current blob -- `FinalizeBlob()` → Compress, encrypt, hash current blob -- `Flush()` → Finalize any in-progress blob -- `SetBlobHandler(func)` → Set callback for upload +### Chunks -### `internal/snapshot` +`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend +on the content, so an insertion early in a file moves only nearby chunks. -#### Scanner -Orchestrates the backup process for a directory. +`chunk_size` sets the average chunk size. A chunk is between a quarter of that +size and four times that size. With the default of 10 MB, a chunk is between 2.5 +MB and 40 MB. -Key methods: -- `NewScanner(ScannerConfig)` → Constructor (creates Chunker + Packer) -- `Scan(ctx, path, snapshotID)` → Main scan operation +`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole +file is never held in memory. -Scan phases: -1. **Phase 1**: Walk directory, identify files needing processing -2. **Phase 1b**: Count as deleted the files that the previous snapshot of the same name lists, the walk did not find, and are gone from disk; a file that is excluded, or inside an excluded directory, is not counted -3. **Phase 2**: Process files (chunk → pack → upload) +### Blobs -#### SnapshotManager -Manages snapshot lifecycle and metadata export. +`blob.Packer` collects new chunks into a blob. For each blob it does this: -Key methods: -- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record -- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot references -- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3 -- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a successful export -- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark complete (no export between) +- It gives the blob a UUID and creates a temporary file. +- It adds each chunk's bytes to a running SHA-256 hash. +- In the same pass it compresses the bytes with zstd and encrypts them with + age. The result goes to the temporary file. +- It records each chunk's offset and length within the blob. +- It closes the blob when the next chunk would pass `blob_size_limit`, and at + the end of each path. -### `internal/database` -SQLite database for local index. Single-writer mode for thread safety. +The blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. It is computed +from the chunk data before compression and encryption. +[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs) says what follows from +that. -Key types: -- `DB` → Database connection wrapper -- `Repositories` → Collection of all repository interfaces +When a blob is closed, `Scanner.handleBlobReady` stores it: -Repository interfaces: -- `FilesRepository` → CRUD for File records -- `ChunksRepository` → CRUD for Chunk records -- `BlobsRepository` → CRUD for Blob records -- `SnapshotsRepository` → CRUD for Snapshot records -- Plus join table repositories (FileChunks, BlobChunks, etc.) +- It looks the blob up at the destination. A blob that is already there is not + uploaded again. +- Otherwise it uploads the temporary file to + `blobs///`. +- It records the upload in the index and links the blob to the snapshot. +- It deletes the temporary file. -## Snapshot Creation Flow +A file's record is written to the index only after every blob that holds its +chunks is uploaded. -``` -CreateSnapshot(opts) - │ - ├─► markDestination() // Check the marker (see Storage Layout in S3), then write it - │ - ├─► PruneDatabase() // Critical: avoid dedup errors - │ - ├─► SnapshotManager.CreateSnapshot() // Create DB record - │ - ├─► For each source directory: - │ │ - │ ├─► scanner.Scan(ctx, path, snapshotID) - │ │ │ - │ │ ├─► Phase 1: scanPhase() - │ │ │ Walk directory - │ │ │ Check file metadata changes - │ │ │ Build list of files to process - │ │ │ - │ │ ├─► Phase 1b: detectDeletedFiles() - │ │ │ - │ │ └─► Phase 2: processPhase() - │ │ For each file: - │ │ chunker.ChunkReaderStreaming() - │ │ For each chunk: - │ │ packer.AddChunk() - │ │ If blob full → FinalizeBlob() - │ │ → handleBlobReady() - │ │ → s3Client.PutObjectWithProgress() - │ │ packer.Flush() // Final blob - │ │ - │ └─► Accumulate statistics - │ - ├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs - │ - ├─► SnapshotManager.UpdateSnapshotStatsExtended() - │ - ├─► SnapshotManager.ExportSnapshotMetadata() - │ │ - │ ├─► Copy database to temp file - │ ├─► Clean to only current snapshot data (VACUUM) - │ ├─► Compress binary SQLite with zstd - │ ├─► Encrypt with age - │ ├─► Upload db.zst.age to storage - │ └─► Upload manifest.json.zst to storage - │ - └─► SnapshotManager.MarkSnapshotComplete() // only after the export succeeds -``` +### Snapshot metadata -## Deduplication Strategy +`finalizeSnapshotMetadata` does these steps in order: -1. **File-level**: Files and symlinks unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked +- `PopulateSnapshotBlobs` records every blob the snapshot uses. +- `UpdateSnapshotStatsExtended` stores the snapshot's totals. +- `ExportSnapshotMetadata` uploads the metadata. +- `MarkSnapshotComplete` marks the snapshot complete. -2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded. +Completion is recorded last on purpose. A backup that stops during the upload +leaves the snapshot unfinished, and the next backup removes it and stores its +data again. -3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a blob are skipped. +`ExportSnapshotMetadata` works on a copy of the index, never on the index +itself: -## Storage Layout in S3 +- It copies the index into a temporary directory with `VACUUM INTO`. +- It deletes from the copy everything that does not belong to this snapshot. +- It runs `VACUUM`, so that no deleted row is left in the file. +- It compresses the copy with zstd and encrypts it with age. +- It builds the manifest from the copy. +- It uploads `metadata//db.zst.age`, and then + `metadata//manifest.json.zst`. -``` -bucket/ -├── vaultik-destination.txt # Marker written by every backup -│ -├── blobs/ -│ └── {hash[0:2]}/ -│ └── {hash[2:4]}/ -│ └── {full-hash} # Compressed+encrypted blob -│ -└── metadata/ - └── {remote-key}/ - ├── db.zst.age # Encrypted binary SQLite database - └── manifest.json.zst # Blob list (for pruning/verification) -``` +The manifest is uploaded last. A directory under `metadata/` with no manifest +therefore marks a backup that did not finish. -A destination that answers a look-up of `vaultik-destination.txt` with not -found is treated as not present while the local index holds snapshots stored -there (`checkDestinationMarker` in `internal/vaultik/destination_marker.go`): -`snapshot create` refuses to write to it, `prune`, `snapshot purge` and -`remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a -destination that cannot be listed. This keeps those commands from taking an -unplugged volume's empty mount point for a destination whose snapshots are -all gone. When the look-up fails for any reason other than not found, for -example when S3 refuses it to credentials that allow only PUT, the check is -skipped, so a host with such credentials does not notice an emptied -destination; see -[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt). -The `file://` backend never creates its destination directory, so a write to a -missing one fails. +## Restore -The `{remote-key}` directory name is a one-way double SHA-256 hash of the human -snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never -written to the store as a directory name. See -[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the -derivation and a worked example. +`Vaultik.Restore` does these steps: -## Thread Safety +- `restoreIdentities` parses the age private key. A missing or malformed key + stops the restore before anything is downloaded. +- `downloadSnapshotDB` downloads and decrypts the snapshot's database into a + temporary directory. +- `getFilesToRestore` selects the files, from the paths given. +- `restoreAllFiles` downloads blobs and writes the files. +- With `--verify`, `verifyRestoredFiles` reads each restored file and checks its + chunk hashes. -- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`. -- `Scanner`: Uses `packerMu` mutex to coordinate blob finalization. -- `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread safety. -- `Repositories.WithTx()`: Handles transaction lifecycle automatically. +The restore does not read the local index or the manifest. It finds each file's +chunks and blobs through the downloaded database. + +### Checks on the downloaded database + +Decryption shows that a database is readable. It does not show that the +database is honest. The restore therefore checks it: + +- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a + trigger, a view or a virtual table, or that lacks an expected table. +- `verifySnapshotDBIdentity` checks that the snapshot in the database hashes to + the remote key that was asked for. +- `containedRestorePath` refuses a stored path that would leave the target + directory. It also refuses to write through a symlink below the target. +- A blob is not decompressed beyond the size the database records for it. +- Each downloaded blob is hashed. The hash must match the blob's name. + +### Order of work + +`restorePlan` orders the files so that few blobs are on disk at once: + +- Downloaded blobs are kept decrypted in a cache directory. +- A file is restored once every blob it needs is in the cache. +- When no file is ready, the restore downloads the blobs for the file that + needs the fewest. +- `restoreSweeper` deletes a cached blob once no remaining file needs it. + +A regular file is created with mode `0600` and gets its stored mode after its +content is written. A directory gets its stored mode, owner and modification +time after the whole restore. + +## Verify + +`Vaultik.VerifySnapshotWithOptions` checks a snapshot without the private key: + +- It downloads the manifest. +- It checks that the snapshot's encrypted database is present. +- It checks that each listed blob is present with the listed size. + +`Vaultik.RunDeepVerify` needs the private key. It does these steps: + +- It downloads the manifest and the database. It checks the database as a + restore does. +- `verifyManifestAgainstDatabase` checks that both list the same blobs with the + same sizes. +- `verifyBlobExistenceFromDB` checks that every blob is present with the + recorded size. +- `verifyBlob` downloads each blob, checks the hash of each chunk in it, and + checks the hash that names the blob. + +## Prune + +`Vaultik.Prune` does three steps: + +- `CleanupLocalSnapshots` drops each snapshot from the local index whose + directory is gone from the destination. +- `PruneDatabase` removes unfinished snapshots and unused records from the + local index. +- `PruneBlobs` deletes unused data at the destination. + +`PruneBlobs` reads every manifest at the destination. It then deletes two kinds +of object, once they are older than `prune_grace_period`: + +- a blob that no manifest lists +- a directory under `metadata/` that has no manifest, and whose snapshot is not + in the local index + +It deletes nothing when any manifest cannot be read. +[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#pruning) explains why this is +safe, and where it is not. + +## The destination + +`storage.Storer` is the only way the code reaches the destination. It has these +methods: + +- `Put` and `PutWithProgress` store an object. +- `Get` reads an object. +- `Stat` reports an object's size. +- `Delete` removes an object. +- `List` and `ListStream` list the objects under a prefix. +- `DeletePartialUploads` removes what an interrupted upload left behind. +- `Info` describes the destination. + +`storage.NewStorer` picks the implementation from `storage_url`: + +- `s3://` gives the S3 implementation. +- `file://` gives the local directory implementation. +- `rclone://` gives the rclone implementation. + +With no `storage_url`, it builds the S3 implementation from the `s3` settings. + +## The marker file and the destination binding + +Two checks keep the local index and the destination in step. + +- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when + the file is reported missing while the local index holds snapshots. +- `EnsureStorageBinding` records the configured destination in the local index + on first use. Later it fails when the config names a different destination. + +Every command that changes data calls `EnsureStorageBinding` first. + +## Concurrency + +- The local index is opened with one connection, so all access to it is in + turn. +- The index runs in SQLite's WAL mode. A second process can therefore read it + while a backup writes to it. +- `Packer` guards its state with a mutex. +- `Scanner` holds `packerMu` while a chunk is added to the packer and while a + blob is closed. +- `Repositories.WithTx` runs a function in one transaction. It commits when the + function succeeds and rolls back when it fails.