diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b41dc22..a58cacd 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,411 +1,401 @@ -# Vaultik Architecture +# vaultik architecture -This document describes the internal architecture of Vaultik, focusing on the data model, type instantiation, and the relationships between core modules. +This document describes how the `vaultik` code is organised. It covers three +things: -## Overview +- the packages +- how they are wired together +- what happens during a backup, a restore, a verify and a prune -Vaultik is a backup system that uses content-defined chunking for deduplication and packs chunks into large, compressed, encrypted blobs for efficient cloud storage. The system is built around dependency injection using [uber-go/fx](https://github.com/uber-go/fx). +Two other documents cover the stored data: -## Data Flow +- [docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index. +- [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the + destination. + +## Terms + +- **Snapshot**: one backup of the paths listed under one name in the config. +- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash + of its content. +- **Blob**: one object at the destination. It holds many chunks, compressed and + then encrypted. +- **Local index**: the SQLite database on the backed-up machine. +- **Destination**: the place the backups are stored. +- **Manifest**: the unencrypted list of the blobs one snapshot uses. +- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory + at the destination. + +## How a backup moves data ``` -Source Files +source files │ ▼ -┌─────────────────┐ -│ Scanner │ Walks directories, detects changed files -└────────┬────────┘ - │ - ▼ -┌─────────────────┐ -│ Chunker │ Splits files into variable-size chunks (FastCDC) -└────────┬────────┘ - │ - ▼ -┌─────────────────┐ -│ Packer │ Accumulates chunks, compresses (zstd), encrypts (age) -└────────┬────────┘ - │ - ▼ -┌─────────────────┐ -│ S3 Client │ Uploads blobs to remote storage -└─────────────────┘ + Scanner walks the paths and finds the files that changed + │ + ▼ + Chunker cuts each changed file into chunks + │ + ▼ + Packer packs new chunks into a blob, compressed and encrypted + │ + ▼ + Storer uploads the blob to the destination ``` -## Data Model +## Packages -### Core Entities +- `cmd/vaultik`: the `main` function. It starts optional profiling and calls + `cli.Entry`. +- `internal/cli`: the commands, their flags and their help text. It builds the + application, takes the lock and handles interrupts. +- `internal/vaultik`: the code behind each command. `Vaultik` holds every + dependency a command uses. +- `internal/snapshot`: the `Scanner`, which backs up one path, and the + `SnapshotManager`, which creates snapshot records and uploads their metadata. +- `internal/chunker`: cuts a stream into chunks with the FastCDC algorithm. +- `internal/blob`: the `Packer`, which collects chunks into blobs. +- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob, + and the reader that reverses it. +- `internal/database`: the local index. It holds the schema and the repository + types that read and write the tables. + [docs/DATAMODEL.md](docs/DATAMODEL.md#code) says which type serves which + table. +- `internal/storage`: the `Storer` interface and its three implementations, for + S3, a local directory and rclone. +- `internal/s3`: the S3 client that the S3 `Storer` uses. +- `internal/config`: loads and checks the config file. +- `internal/ui`: formats the status lines, warnings and errors a user sees. +- `internal/log`: the logger. It writes to stderr. +- `internal/pidlock`: the lock that keeps two data-changing commands from + running at once. +- `internal/globals`: the version, commit and commit date, set when the binary + is built. +- `internal/types`: distinct types for IDs, hashes and paths. -The database tracks five primary entities and their relationships: +## Wiring -``` -┌──────────────┐ ┌──────────────┐ ┌──────────────┐ -│ Snapshot │────▶│ File │────▶│ Chunk │ -└──────────────┘ └──────────────┘ └──────────────┘ - │ │ - │ │ - ▼ ▼ -┌──────────────┐ ┌──────────────┐ -│ Blob │◀─────────────────────────│ BlobChunk │ -└──────────────┘ └──────────────┘ -``` - -### Entity Descriptions - -#### File (`database.File`) -Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration: -- Path, mtime -- Size, mode, ownership (uid, gid) -- Symlink target (if applicable) - -It also stores `source_path`, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it. - -#### Chunk (`database.Chunk`) -A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm: -- `ChunkHash`: SHA256 hash of chunk content (primary key) -- `Size`: Chunk size in bytes - -Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for the 10MB default average). - -#### FileChunk (`database.FileChunk`) -Maps files to their constituent chunks: -- `FileID`: Reference to the file -- `Idx`: Position of this chunk within the file (0-indexed) -- `ChunkHash`: Reference to the chunk - -#### Blob (`database.Blob`) -The final storage unit uploaded to S3. Contains many compressed and encrypted chunks: -- `ID`: UUID assigned at creation -- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs)) -- `UncompressedSize`: Total raw chunk data before compression -- `CompressedSize`: Size after zstd compression and age encryption -- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps - -Blob creation process: -1. Chunks are accumulated (up to MaxBlobSize, typically 10GB) -2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256 -3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file -4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes -5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted - -A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`. - -#### BlobChunk (`database.BlobChunk`) -Maps chunks to their position within blobs: -- `BlobID`: Reference to the blob -- `ChunkHash`: Reference to the chunk -- `Offset`: Byte offset within the uncompressed blob -- `Length`: Chunk size - -#### Snapshot (`database.Snapshot`) -Represents a point-in-time backup: -- `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g. `server1_home_2025-06-01T12:00:00Z`) -- Tracks file count, chunk count, blob count, sizes, compression ratio -- `CompletedAt`: Null until snapshot finishes successfully - -#### SnapshotFile / SnapshotBlob -Join tables linking snapshots to their files and blobs. - -### Relationship Summary - -``` -Snapshot 1──────────▶ N SnapshotFile N ◀────────── 1 File -Snapshot 1──────────▶ N SnapshotBlob N ◀────────── 1 Blob -File 1──────────▶ N FileChunk N ◀────────── 1 Chunk -Blob 1──────────▶ N BlobChunk N ◀────────── 1 Chunk -``` - -## Type Instantiation - -### Application Startup - -The CLI uses fx for dependency injection. Here's the instantiation order: +The application is assembled with [uber-go/fx](https://github.com/uber-go/fx), a +dependency injection library. `cli.NewApp` lists the providers: ```go -// cli/app.go: NewApp() -fx.New( - fx.Supply(config.Path(opts.ConfigPath)), // 1. Config path - fx.Supply(opts.LogOptions), // 2. Log options - fx.Provide(globals.New), // 3. Globals - fx.Provide(log.New), // 4. Logger config - config.Module, // 5. Config - database.Module, // 6. Database + Repositories - log.Module, // 7. Logger initialization - storage.Module, // 8. Storage backend (S3/file/rclone) - snapshot.Module, // 9. SnapshotManager + ScannerFactory - fx.Provide(vaultik.New), // 10. Vaultik orchestrator -) +fx.Supply(config.Path(opts.ConfigPath)), +fx.Supply(opts.LogOptions), +fx.Provide(globals.New), +fx.Provide(log.New), +config.Module, +database.Module, +log.Module, +storage.Module, +snapshot.Module, +fx.Provide(vaultik.New), +fx.Invoke(setupGlobals), +fx.NopLogger, ``` -### Key Type Instantiation Points +Each module provides these: -#### 1. Config (`config.Config`) -- **Created by**: `config.Module` via `config.LoadConfig()` -- **When**: Application startup (fx DI) -- **Contains**: All configuration from YAML file (S3 credentials, encryption keys, paths, etc.) +- `config.Module` provides `*config.Config`, from `config.Load`. +- `database.Module` provides `*database.DB` and `*database.Repositories`. It + creates the index file with mode `0600` when it does not exist. +- `storage.Module` provides the `storage.Storer` that the config selects. +- `snapshot.Module` provides the `snapshot.ScannerFactory` and the + `*snapshot.SnapshotManager`. +- `vaultik.New` provides `*vaultik.Vaultik`. -#### 2. Database (`database.DB`) -- **Created by**: `database.Module` via `database.New()` -- **When**: Application startup (fx DI) -- **Contains**: SQLite connection, path reference +A command runs through `cli.RunOperation`. It works like this: -#### 3. Repositories (`database.Repositories`) -- **Created by**: `database.Module` via `database.NewRepositories()` -- **When**: Application startup (fx DI) -- **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots, etc.) +- `RunWithApp` takes the lock when the command changes data, and then starts the + application. +- The command's function runs in its own goroutine. +- When the function returns, it asks the application to shut down. +- On `SIGINT` or `SIGTERM`, the stop hook cancels the `Vaultik` context. It then + waits for the function to return, so that its temporary files are removed. +- The wait ends after 30 seconds at most. -#### 4. Vaultik (`vaultik.Vaultik`) -- **Created by**: `vaultik.New(VaultikParams)` -- **When**: Application startup (fx DI) -- **Contains**: All dependencies for backup operations +The `config` commands, `database delete`, `completion` and `version` do not +build the application. -```go -type Vaultik struct { - Globals *globals.Globals - Config *config.Config - DB *database.DB - Repositories *database.Repositories - Storage storage.Storer - ScannerFactory snapshot.ScannerFactory - SnapshotManager *snapshot.SnapshotManager - Shutdowner fx.Shutdowner - Fs afero.Fs - ctx context.Context - cancel context.CancelFunc -} -``` +## Backup -#### 5. SnapshotManager (`snapshot.SnapshotManager`) -- **Created by**: `snapshot.Module` via `snapshot.NewSnapshotManager()` -- **When**: Application startup (fx DI) -- **Responsibility**: Creates/completes snapshots, exports metadata to S3 +`Vaultik.CreateSnapshot` runs a backup. It does these steps in order: -#### 6. Scanner (`snapshot.Scanner`) -- **Created by**: `ScannerFactory(ScannerParams)` -- **When**: Each `CreateSnapshot()` call -- **Contains**: Chunker, Packer, progress reporter +- `EnsureStorageBinding` checks that the local index belongs to the configured + destination. +- `markDestination` checks the marker file and then writes it. +- `PruneDatabase` removes unfinished snapshots and unused records from the + local index. +- `createNamedSnapshot` runs once for each snapshot name. +- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data. -```go -// vaultik/snapshot.go: CreateSnapshot() -scanner := v.ScannerFactory(snapshot.ScannerParams{ - EnableProgress: !opts.Cron, - Fs: v.Fs, -}) -``` +`CreateSnapshot` calls `skipPostBackupPrune` instead of `runPostBackupPrune` in +these cases: -#### 7. Chunker (`chunker.Chunker`) -- **Created by**: `chunker.NewChunker(avgChunkSize)` -- **When**: Inside `snapshot.NewScanner()` -- **Configuration**: - - `avgChunkSize`: From config (default 10MB) - - `minChunkSize`: avgChunkSize / 4 - - `maxChunkSize`: avgChunkSize * 4 +- `--skip-errors` skipped a file or directory that could not be read. +- A configured path held nothing. +- An earlier snapshot of the same name found a mount point that this backup did + not find. -#### 8. Packer (`blob.Packer`) -- **Created by**: `blob.NewPacker(PackerConfig)` -- **When**: Inside `snapshot.NewScanner()` -- **Configuration**: - - `MaxBlobSize`: Maximum blob size before finalization (typically 10GB) - - `CompressionLevel`: zstd level (1-19) - - `Recipients`: age public keys for encryption +`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot +left in the index would make the backup skip files that were never fully +stored. -```go -// snapshot/scanner.go: NewScanner() -packerCfg := blob.PackerConfig{ - MaxBlobSize: cfg.MaxBlobSize, - CompressionLevel: cfg.CompressionLevel, - Recipients: cfg.AgeRecipients, - Repositories: cfg.Repositories, - Fs: cfg.FS, -} -packer, err := blob.NewPacker(packerCfg) -``` +### One snapshot -## Module Responsibilities +`createNamedSnapshot` does these steps: -### `internal/cli` -Entry point for fx application. Combines all modules and handles signal interrupts. +- `resolveSnapshotPaths` makes each configured path absolute and resolves + symlinks. +- `SnapshotManager.CreateSnapshotWithName` creates the snapshot record. The + record is not yet marked complete. +- `Scanner.Scan` backs up each path in turn. +- `finalizeSnapshotMetadata` uploads the snapshot's metadata and marks the + snapshot complete. -Key functions: -- `NewApp(AppOptions)` → Creates fx.App with all modules -- `RunApp(ctx, app)` → Starts app, handles graceful shutdown -- `RunWithApp(ctx, opts)` → Convenience wrapper +A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a +`Chunker` and a `Packer`. -### `internal/vaultik` -Main orchestrator containing all dependencies and command implementations. +### Scanning one path -Key methods: -- `New(VaultikParams)` → Constructor (fx DI) -- `CreateSnapshot(opts)` → Main backup operation -- `ListSnapshots(jsonOutput)` → List available snapshots -- `VerifySnapshot(id, deep)` → Verify snapshot integrity -- `PurgeSnapshots(...)` → Remove old snapshots +`Scanner.Scan` works in these steps: -### `internal/chunker` -Content-defined chunking using FastCDC algorithm. +- `repairInterruptedBlobs` deletes the records of blobs whose upload never + finished. Their data is then stored again. +- `loadDatabaseState` loads the known files and chunks into memory. +- `scanPhase` walks the path. It sorts each entry into changed or unchanged. +- `detectDeletedFiles` counts the files that the previous snapshot of this name + had and that are now gone. +- `batchAddFilesToSnapshot` records the unchanged files in the new snapshot. +- `recordDirectories` records every directory, and the mount points found. +- `processPhase` reads each changed file and stores it. -Key types: -- `Chunk` → Hash, Data, Offset, Size -- `Chunker` → avgChunkSize, minChunkSize, maxChunkSize +The walk handles each kind of entry differently: -Key methods: -- `NewChunker(avgChunkSize)` → Constructor -- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback (preferred) -- `ChunkReader(reader)` → Return all chunks at once (memory-intensive) +- A regular file or a symlink is compared with its record in the index. +- A directory is recorded with its metadata. It has no content to store. +- A device node, a named pipe or a socket is skipped. +- An entry that matches an exclude pattern is skipped. For a directory, so is + everything in it. -### `internal/blob` -Blob packing: accumulates chunks, compresses, encrypts, tracks metadata. +### Deduplication -Key types: -- `Packer` → Thread-safe blob accumulator -- `ChunkRef` → Hash + Data for adding to packer -- `FinishedBlob` → Completed blob ready for upload -- `BlobWithReader` → FinishedBlob + io.Reader for streaming upload +Data is stored once at two levels. -Key methods: -- `NewPacker(PackerConfig)` → Constructor -- `AddChunk(ChunkRef)` → Add chunk to current blob -- `FinalizeBlob()` → Compress, encrypt, hash current blob -- `Flush()` → Finalize any in-progress blob -- `SetBlobHandler(func)` → Set callback for upload +- **Files.** A file is not read again when its record in the index still matches + it. +- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk whose + hash is already in the index is not stored again. -### `internal/snapshot` +A file's record matches when all of these are unchanged: -#### Scanner -Orchestrates the backup process for a directory. +- the size +- the modification time +- the mode +- the owner and group +- the link target -Key methods: -- `NewScanner(ScannerConfig)` → Constructor (creates Chunker + Packer) -- `Scan(ctx, path, snapshotID)` → Main scan operation +One case overrides the file check. A file is read again when it lists a chunk +that no uploaded blob holds. This happens when the only snapshot that used the +blob holding that chunk is removed. `loadFilesToRechunk` finds these files. -Scan phases: -1. **Phase 1**: Walk directory, identify files needing processing -2. **Phase 1b**: Count as deleted the files that the previous snapshot of the same name lists, the walk did not find, and are gone from disk; a file that is excluded, or inside an excluded directory, is not counted -3. **Phase 2**: Process files (chunk → pack → upload) +### Chunks -#### SnapshotManager -Manages snapshot lifecycle and metadata export. +`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend +on the content, so an insertion early in a file moves only nearby chunks. -Key methods: -- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record -- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot references -- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3 -- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a successful export -- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark complete (no export between) +`chunk_size` sets the average chunk size. No chunk is larger than four times +this size. Only a file's last chunk can be smaller than a quarter of it. With +the default of 10 MB, those limits are 40 MB and 2.5 MB. -### `internal/database` -SQLite database for local index. Single-writer mode for thread safety. +`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole +file is never held in memory. -Key types: -- `DB` → Database connection wrapper -- `Repositories` → Collection of all repository interfaces +### Blobs -Repository interfaces: -- `FilesRepository` → CRUD for File records -- `ChunksRepository` → CRUD for Chunk records -- `BlobsRepository` → CRUD for Blob records -- `SnapshotsRepository` → CRUD for Snapshot records -- Plus join table repositories (FileChunks, BlobChunks, etc.) +`blob.Packer` collects new chunks into a blob. For each blob it does this: -## Snapshot Creation Flow +- It gives the blob a UUID and creates a temporary file. +- It adds each chunk's bytes to a running SHA-256 hash. +- In the same pass it compresses the bytes with zstd and encrypts them with + age. The result goes to the temporary file. +- It records each chunk's offset and length within the blob. +- It closes the blob when the next chunk would pass `blob_size_limit`, and at + the end of each path. -``` -CreateSnapshot(opts) - │ - ├─► markDestination() // Check the marker (see Storage Layout in S3), then write it - │ - ├─► PruneDatabase() // Critical: avoid dedup errors - │ - ├─► SnapshotManager.CreateSnapshot() // Create DB record - │ - ├─► For each source directory: - │ │ - │ ├─► scanner.Scan(ctx, path, snapshotID) - │ │ │ - │ │ ├─► Phase 1: scanPhase() - │ │ │ Walk directory - │ │ │ Check file metadata changes - │ │ │ Build list of files to process - │ │ │ - │ │ ├─► Phase 1b: detectDeletedFiles() - │ │ │ - │ │ └─► Phase 2: processPhase() - │ │ For each file: - │ │ chunker.ChunkReaderStreaming() - │ │ For each chunk: - │ │ packer.AddChunk() - │ │ If blob full → FinalizeBlob() - │ │ → handleBlobReady() - │ │ → s3Client.PutObjectWithProgress() - │ │ packer.Flush() // Final blob - │ │ - │ └─► Accumulate statistics - │ - ├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs - │ - ├─► SnapshotManager.UpdateSnapshotStatsExtended() - │ - ├─► SnapshotManager.ExportSnapshotMetadata() - │ │ - │ ├─► Copy database to temp file - │ ├─► Clean to only current snapshot data (VACUUM) - │ ├─► Compress binary SQLite with zstd - │ ├─► Encrypt with age - │ ├─► Upload db.zst.age to storage - │ └─► Upload manifest.json.zst to storage - │ - └─► SnapshotManager.MarkSnapshotComplete() // only after the export succeeds -``` +The blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. It is computed +from the chunk data before compression and encryption. +[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs) says what follows from +that. -## Deduplication Strategy +When a blob is closed, `Scanner.handleBlobReady` stores it: -1. **File-level**: Files and symlinks unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked +- It looks the blob up at the destination. A blob that is already there is not + uploaded again. +- Otherwise it uploads the temporary file to + `blobs///`. +- It records the upload in the index and links the blob to the snapshot. +- It deletes the temporary file. -2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded. +A file's record is written to the index only after every blob that holds its +chunks is uploaded. -3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a blob are skipped. +### Snapshot metadata -## Storage Layout in S3 +`finalizeSnapshotMetadata` does these steps in order: -``` -bucket/ -├── vaultik-destination.txt # Marker written by every backup -│ -├── blobs/ -│ └── {hash[0:2]}/ -│ └── {hash[2:4]}/ -│ └── {full-hash} # Compressed+encrypted blob -│ -└── metadata/ - └── {remote-key}/ - ├── db.zst.age # Encrypted binary SQLite database - └── manifest.json.zst # Blob list (for pruning/verification) -``` +- `PopulateSnapshotBlobs` records every blob the snapshot uses. +- `UpdateSnapshotStatsExtended` stores the snapshot's totals. +- `ExportSnapshotMetadata` uploads the metadata. +- `MarkSnapshotComplete` marks the snapshot complete. -A destination that answers a look-up of `vaultik-destination.txt` with not -found is treated as not present while the local index holds snapshots stored -there (`checkDestinationMarker` in `internal/vaultik/destination_marker.go`): -`snapshot create` refuses to write to it, `prune`, `snapshot purge` and -`remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a -destination that cannot be listed. This keeps those commands from taking an -unplugged volume's empty mount point for a destination whose snapshots are -all gone. When the look-up fails for any reason other than not found, for -example when S3 refuses it to credentials that allow only PUT, the check is -skipped, so a host with such credentials does not notice an emptied -destination; see -[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt). -The `file://` backend never creates its destination directory, so a write to a -missing one fails. +Completion is recorded last on purpose. A backup that stops during the upload +leaves the snapshot unfinished, and the next backup removes it and stores its +data again. -The `{remote-key}` directory name is a one-way double SHA-256 hash of the human -snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never -written to the store as a directory name. See -[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the -derivation and a worked example. +`ExportSnapshotMetadata` works on a copy of the index, never on the index +itself: -## Thread Safety +- It copies the index into a temporary directory with `VACUUM INTO`. +- It deletes the rows of every other snapshot from the copy. It also deletes the + files and blobs that this snapshot does not use. It then deletes each chunk + that no remaining file and no remaining blob uses. +- It keeps the `local_meta` and `schema_migrations` tables as they are. The + uploaded database therefore names the destination, and lists the migrations + that have run. +- It runs `VACUUM`, so that no deleted row is left in the file. +- It compresses the copy with zstd and encrypts it with age. +- It builds the manifest from the copy. +- It uploads `metadata//db.zst.age`, and then + `metadata//manifest.json.zst`. -- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`. -- `Scanner`: Uses `packerMu` mutex to coordinate blob finalization. -- `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread safety. -- `Repositories.WithTx()`: Handles transaction lifecycle automatically. +The manifest is uploaded last. A directory under `metadata/` with no manifest +therefore marks a backup that did not finish. + +## Restore + +`Vaultik.Restore` does these steps: + +- `restoreIdentities` parses the age private key. A missing or malformed key + stops the restore before anything is downloaded. +- `downloadSnapshotDB` downloads and decrypts the snapshot's database into a + temporary directory. +- `getFilesToRestore` selects the files, from the paths given. +- `restoreAllFiles` downloads blobs and writes the files. +- With `--verify`, `verifyRestoredFiles` reads each restored file and checks its + chunk hashes. + +The restore does not read the local index or the manifest. It finds each file's +chunks and blobs through the downloaded database. + +### Checks on the downloaded database + +Decryption shows that a database is readable. It does not show that the +database is honest. The restore therefore checks it: + +- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a + trigger, a view or a virtual table, or that lacks an expected table. +- `verifySnapshotDBIdentity` checks that the snapshot in the database hashes to + the remote key that was asked for. +- `containedRestorePath` refuses a stored path that would leave the target + directory. It also refuses to write through a symlink below the target. +- A blob is not decompressed beyond the size that the database gives for it. +- Each downloaded blob is hashed. The hash must match the blob's name. + +### Order of work + +`restorePlan` orders the files so that few blobs are on disk at once: + +- Downloaded blobs are kept decrypted in a cache directory. +- A file is restored once every blob it needs is in the cache. +- When no file is ready, the restore downloads the blobs for the file that + needs the fewest. +- `restoreSweeper` deletes a cached blob once no remaining file needs it. + +A regular file is created with mode `0600` and gets its stored mode after its +content is written. A directory gets its stored mode, owner and modification +time after the whole restore. + +## Verify + +`Vaultik.VerifySnapshotWithOptions` checks a snapshot without the private key: + +- It downloads the manifest. +- It checks that the snapshot's encrypted database is present. +- It checks that each listed blob is present with the listed size. + +`Vaultik.RunDeepVerify` needs the private key. It does these steps: + +- It downloads the manifest and the database. It checks the database as a + restore does. +- `verifyManifestAgainstDatabase` checks that both list the same blobs with the + same sizes. +- `verifyBlobExistenceFromDB` checks that every blob is present with the + recorded size. +- `verifyBlob` downloads each blob, checks the hash of each chunk in it, and + checks the hash that names the blob. + +## Prune + +`Vaultik.Prune` does three steps: + +- `CleanupLocalSnapshots` drops from the local index each snapshot whose + manifest is gone from the destination. +- `PruneDatabase` removes unfinished snapshots and unused records from the + local index. +- `PruneBlobs` deletes unused data at the destination. + +`PruneBlobs` reads every manifest at the destination. It then deletes two kinds +of object, once they are older than `prune_grace_period`: + +- a blob that no manifest lists +- a directory under `metadata/` that has no manifest, and whose snapshot is not + in the local index + +It deletes nothing when any manifest cannot be read. +[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#pruning) explains why this is +safe, and where it is not. + +## The destination + +`storage.Storer` is the only way the code reaches the destination. It has these +methods: + +- `Put` and `PutWithProgress` store an object. +- `Get` reads an object. +- `Stat` reports an object's size. +- `Delete` removes an object. +- `List` and `ListStream` list the objects under a prefix. +- `DeletePartialUploads` removes what an interrupted upload left behind. +- `Info` describes the destination. + +`storage.NewStorer` picks the implementation from `storage_url`: + +- `s3://` gives the S3 implementation. +- `file://` gives the local directory implementation. +- `rclone://` gives the rclone implementation. + +With no `storage_url`, it builds the S3 implementation from the `s3` settings. + +## The marker file and the destination binding + +Two checks keep the local index and the destination in step. + +- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when + the file is reported missing while the local index holds snapshots. +- `EnsureStorageBinding` records the configured destination in the local index + on first use. Later it fails when the config names a different destination. + +Every command that changes data calls `EnsureStorageBinding` first. + +## Concurrency + +- The local index is opened with one connection, so its queries run one at a + time. +- The index runs in SQLite's WAL mode. A second process can therefore read it + while a backup writes to it. +- `Packer` guards its state with a mutex. +- `Scanner` holds `packerMu` while a chunk is added to the packer and while a + blob is closed. +- `Repositories.WithTx` runs a function in one transaction. It commits when the + function succeeds and rolls back when it fails. diff --git a/README.md b/README.md index f83d065..7064798 100644 --- a/README.md +++ b/README.md @@ -1,1123 +1,1379 @@ # vaultik (ваултик) -`vaultik` is an incremental backup tool written in Go. It encrypts data -using an `age` public key and uploads each encrypted blob directly to a -remote S3-compatible object store. It requires no private keys, secrets, or -credentials (other than those required to PUT to encrypted object storage, -such as S3 API keys) stored on the backed-up system. +`vaultik` is an MIT-licensed command-line tool for incremental, encrypted +backups, written in Go by [@sneak](https://sneak.berlin). It encrypts your files +with a public key and uploads them to an S3-compatible object store, a local +directory or an rclone remote. -## quickstart +The machine being backed up holds only the public key. It cannot decrypt its own +backups. You keep the private key somewhere else and need it only to restore. + +## Rationale + +Other backup tools, such as `restic`, `borg` and `duplicity`, are designed for +machines that can hold a decryption key. `vaultik` is for machines that should +not hold one. + +It was written to meet these requirements together: + +- open source +- no passphrase or private key on the backed-up machine +- incremental +- compressed +- encrypted +- direct upload to S3-compatible storage, with no intermediate step or tool + +## Features + +- Encryption with [age](https://age-encryption.org/) (X25519 and + ChaCha20-Poly1305), to one or more public keys. +- Incremental backups. A file is read again only when it has changed, or when + part of its content has to be stored again. +- Deduplication. Identical content is stored once, across files and across + snapshots. +- zstd compression at a level you choose. +- Small files are packed together into large objects. This keeps the number of + S3 requests low. +- File contents, names, paths and metadata are encrypted at the destination. +- Three kinds of destination: S3, a local directory and any rclone remote. +- Regular files, symlinks and directories are backed up, with their permissions, + owners and modification times. +- Pure Go, with no C code. It builds for Linux and macOS on `amd64` and `arm64`. + +## Terms + +- **Snapshot**: one backup of the paths listed under one name in the config, + taken at one time. +- **Snapshot ID**: the name of a snapshot, in the form + `__`. An example is + `server1_home_2025-06-01T12:00:00Z`. +- **Destination**: the place the backups are stored. +- **Chunk**: a piece of a file's content. `vaultik` cuts each file into chunks + and stores each distinct chunk once. +- **Blob**: one object at the destination. It holds many chunks, compressed and + then encrypted. +- **Local index**: a SQLite database on the backed-up machine. It records what + is already stored, so the next backup does not upload it again. +- **Manifest**: the list of blobs one snapshot uses. It is stored at the + destination without encryption. +- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory + at the destination, so the destination never shows the snapshot ID. + +## Getting started + +Installing needs Go 1.26.1 or later. Generating a key pair needs `age-keygen`, +which comes with [age](https://age-encryption.org/). ```sh -# install +# Install vaultik. go install sneak.berlin/go/vaultik/cmd/vaultik@latest -# create a default config file (prints the path it wrote to) +# Write a default config file. The command prints the path it wrote. vaultik config init -# generate an age keypair; keep the private key file somewhere safe and -# offline — you need it to restore, and the backed-up machine does not need it +# Generate a key pair. Keep this file somewhere safe, away from this machine. age-keygen -o vaultik_backup_private_key.txt grep 'public key' vaultik_backup_private_key.txt -# configure the encryption key and backup destination +# Set the public key and the destination. vaultik config set age_recipients.0 age1YOUR_PUBLIC_KEY_HERE vaultik config set storage_url "file:///Volumes/usbstick/mybackup" -# create the destination directory; vaultik never creates it, so a backup -# to a volume that is not plugged in fails instead of filling the disk +# Create the destination directory. vaultik never creates it. mkdir /Volumes/usbstick/mybackup -# macOS only: grant your terminal app Full Disk Access first -# (System Settings → Privacy & Security → Full Disk Access), otherwise -# the backup will abort with a permission error on protected directories - -# run your first backup (the default config backs up ~ and /Applications -# with sensible excludes) +# Run the first backup. vaultik snapshot create -# see what you have +# See what is stored. vaultik snapshot list ``` -Features: +The default config is written for macOS. It backs up your home directory and +`/Applications`, with a list of excludes. A configured path that does not exist +stops the backup. On another system, run `vaultik config edit` first and change +the `snapshots` section. -* modern encryption ([age](https://age-encryption.org/), X25519 + ChaCha20-Poly1305) -* content-defined chunking with deduplication (FastCDC) -* incremental backups (a file is re-chunked only when it changed or a - chunk it lists is held by no uploaded blob) -* multithreaded zstd compression at configurable levels -* content-addressed immutable storage -* local state tracking in SQLite (enables write-only incremental backups) -* no mutable remote metadata -* no plaintext file paths or metadata in remote storage -* packs small files into large blobs (keeps S3 operation counts down) -* backs up regular files, symlinks, empty directories, and file permissions -* pluggable storage backends: S3, local filesystem, rclone (70+ providers) -* pure Go (no CGO), cross-compiles to linux/darwin × amd64/arm64 +On macOS, grant your terminal app Full Disk Access before the first backup. The +setting is under System Settings, Privacy & Security, Full Disk Access. Without +it the backup stops with a permission error on a protected directory. -## why - -Other backup tools like `restic`, `borg`, and `duplicity` are designed for -environments where the source host can store secrets and has access to -decryption keys. `vaultik` is for environments where you don't want to -store backup decryption keys on your hosts — only public keys for -encryption. - -Requirements that no existing tool meets: - -* open source -* no passphrases or private keys on the source host -* incremental -* compressed -* encrypted -* s3 compatible without an intermediate step or tool - -## daily use +## Everyday use ```sh -# verify a snapshot (shallow: checks all blobs are present with the listed size) +# Check that a snapshot's blobs are all present, with the listed sizes. vaultik snapshot verify -# put the private key file in the environment (reading it from the file -# keeps the key out of your shell history); the whole age-keygen file, -# with one or more identities, is accepted +# Put the private key in the environment. Reading it from the file keeps it +# out of your shell history. export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" -# deep verify (downloads every blob, decrypts it, and re-hashes it to -# detect corruption — this checks integrity, not who wrote the blob) +# Download, decrypt and re-hash every blob of a snapshot. vaultik snapshot verify --deep -# restore (requires the private key). Restore into a new directory you own, -# writable only by you — not a shared location like /tmp +# Restore a snapshot into a new directory that only you can write. vaultik snapshot restore ~/vaultik-restore -# daily cron job: back up, keep a 4-week rolling window of snapshots +# A daily cron line: back up, then keep four weeks of snapshots. # 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w ``` -On a destination shared between hosts or users, the prune in that cron line -relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup -another one is still running, and its directory under `metadata/` that has no -manifest yet, so it must be longer than the longest backup run. It does not -protect a blob that backup reuses from an earlier run that stopped before its -manifest: the prune can delete such a blob, and the new snapshot then cannot -be restored. To not depend on the grace period, give each host its own prefix -or directory. See [locking](#locking). +`--prune` deletes data at the destination. Read +[Sharing a destination](#sharing-a-destination) before you use it on a +destination that more than one machine or user writes to. -## restoring on another machine +## Restoring on another machine -Restoring on a host that never ran the backup — a replacement machine -after the original is gone — is the case vaultik is built for. That host -needs only three things: the `vaultik` binary, the age **private** key, -and the storage credentials for the destination. It does **not** need the -local index, the original config file, or the original hostname. +You can restore on a machine that never ran the backup, such as a replacement +for a lost one. That machine needs three things: + +- the `vaultik` binary +- the age private key +- the destination, and credentials that can list and read it + +It does not need the original local index, config file or hostname. ```sh -# install -go install sneak.berlin/go/vaultik/cmd/vaultik@latest - -# create a config and point it at the ORIGINAL backup destination +# Create a config and point it at the original destination. vaultik config init vaultik config set storage_url "s3://bucket/prefix?endpoint=https://s3.example.com" vaultik config set s3.access_key_id "..." vaultik config set s3.secret_access_key "..." -# see what is on the destination store +# See what the destination holds. This needs no private key. vaultik snapshot list ``` -`snapshot list` reads the destination store without the private key. A -snapshot that is not in this host's (empty) local index is shown as -remote-only: its row is identified by `` rather than by -a `hostname_name_timestamp` name, because the name lives only in the -local index and the encrypted database and cannot be recovered from the -store. Its timestamp and compressed size are real. (See the `snapshot -list` description under [command details](#command-details) for the full -explanation.) +On this machine the local index is empty, so the list cannot show snapshot IDs. +Each snapshot appears as ``, with the first 12 characters of +its remote key. The snapshot ID is stored only in the local index and in the +snapshot's encrypted database. -Use that remote key — the hex printed inside ``, or the -full `remote_key` from `snapshot list --json` — to restore and verify: +Use that remote key in place of the snapshot ID: ```sh -# put the private key file in the environment (reading it from the file -# keeps the key out of your shell history) export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" -# restore everything to a new directory you own (writable only by you, not a -# shared location like /tmp), then check every restored file's chunk hashes +# Restore everything, then check the chunk hashes of every restored file. vaultik snapshot restore --verify ~/vaultik-restore -# optionally, deep-verify the snapshot against the store (downloads every -# blob, decrypts it, and re-hashes it to detect corruption — this checks -# integrity, not who wrote the blob) +# Optionally, also check every blob of the snapshot at the destination. vaultik snapshot verify --deep ``` -`age_recipients` (the public key) is not needed to restore — only the -private key in `VAULTIK_AGE_SECRET_KEY`. Both the abbreviated key printed -in the table and the full 64-character key from `--json` are accepted; a -leading part of the key is enough as long as it is unambiguous. +Any leading part of the remote key works, as long as it matches one snapshot. +`snapshot list --json` gives the full 64-character key as `remote_key`. ---- +A machine that only restores does not need `age_recipients`. -## cli - -### commands +## Commands ```sh -vaultik [--config ] config init -vaultik [--config ] config edit -vaultik [--config ] config get -vaultik [--config ] config set -vaultik [--config ] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than ] -vaultik [--config ] snapshot list [--json] # alias: ls -vaultik [--config ] snapshot verify [--deep] [--json] -vaultik [--config ] snapshot purge [--keep-latest | --older-than ] [--snapshot ...] [--force] -vaultik [--config ] snapshot remove [--dry-run] [--force] [--local-only] [--json] # alias: rm -vaultik [--config ] snapshot restore [paths...] [--verify] -vaultik [--config ] prune [--force] [--json] -vaultik [--config ] info -vaultik [--config ] remote info [--json] -vaultik [--config ] remote nuke --force -vaultik [--config ] database delete [--force] +vaultik config init +vaultik config edit +vaultik config get +vaultik config set +vaultik snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than ] +vaultik snapshot list [--json] +vaultik snapshot verify [--deep] [--json] +vaultik snapshot purge (--keep-latest | --older-than ) [--snapshot ...] [--force] +vaultik snapshot remove [--dry-run] [--force] [--local-only] [--json] +vaultik snapshot restore [paths...] [--verify] +vaultik prune [--force] [--json] +vaultik info +vaultik remote info [--json] +vaultik remote nuke --force +vaultik database delete [--force] vaultik completion vaultik version ``` -### global flags +`snapshot list` has the alias `snapshot ls`. `snapshot remove` has the alias +`snapshot rm`. -* `--config `: Path to config file (default: `$VAULTIK_CONFIG`, then platform config dir, then `/etc/vaultik/config.yml`) -* `--verbose`, `-v`: Enable verbose output (on stderr — see below) -* `--debug`: Enable debug output (on stderr — see below) -* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner) -* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run. A run that skipped anything still finishes but exits non-zero: `snapshot create` keeps the snapshot and counts each file or directory it skipped as failed. A file removed during the backup is also counted as failed but does not change the exit status. +### What every command needs -### locking +`config init`, `completion` and `version` need nothing. -Commands that write persistent state — `snapshot create`, `snapshot -purge`, `snapshot remove`, `prune`, and `remote nuke` — take a -process-wide lock at `$XDG_DATA_HOME/vaultik/vaultik.pid` -(`~/.local/share/vaultik/vaultik.pid` on Linux) for the whole run. Only -one of them runs at a time: a second one exits immediately with an -"already running" error rather than waiting, so two writers can never -corrupt the local index. +`config edit`, `config get` and `config set` need a config file that exists. It +does not have to be complete. -The lock covers one user on one machine. It does not stop another user, -or another host, from writing to the same destination store at the same -time. A backup uploads its blobs, and then its `db.zst.age`, before its -manifest, so a prune from elsewhere at that moment finds blobs no snapshot -lists and a directory under `metadata/` with no manifest. It -deletes such a blob only once it is older than `prune_grace_period` (default -`24h`), and such a directory only once every object in it is, so the grace -period must be longer than the longest backup run of any host or user sharing -the destination. The grace period does not protect a blob a backup reuses -from an earlier run that stopped before its manifest: a backup skips the -upload of a blob already on the destination, so that blob keeps the time of -its first upload, and a prune from elsewhere during the backup can delete it; -the new snapshot then cannot be restored. `remote nuke` deletes everything -regardless. To not depend on the grace period, give each host or user its own -prefix (or directory) on the destination. +Every other command needs a config file that loads. See +[Configuration](#configuration) for the checks. -Read-only commands — `info`, `snapshot list`, `snapshot verify`, and -`remote info` — do not take the lock and are never blocked, so they run -even while a backup is in progress. `snapshot restore` does not take the -lock either: it writes only to the target directory you name, not the -local index or the destination store. `config`, `database delete`, -`completion`, and `version` do not take the lock. +Every other command except `database delete` also needs these: -### stdout and stderr +- Write access to the local index. The command creates the file, and its + directory, when they do not exist. +- For an `rclone://` destination, an rclone config file that defines the remote. -Log output — everything from `--verbose` and `--debug`, and every -warning and error the logger emits — goes to **stderr**, and so does the -startup banner. stdout carries the output you asked for: tables, the -documents produced by `--json`, `config get` values, and completion -scripts. +Each command below lists what it needs beyond this. -This means `vaultik snapshot list --verbose > out.txt` captures the -listing and leaves the diagnostics on your terminal. To capture both, -redirect stderr as well (`> out.txt 2> log.txt`, or `> out.txt 2>&1` to -interleave them). +No command needs root. A backup can read only what its user can read. A restore +sets file owners only when it runs as root. -The split is what makes `--json` usable from a script. Warnings and -errors are never suppressed — not by `--quiet`, not by `--cron` — so a -logger on stdout would eventually land a log line inside a JSON -document and break the parse. A config file with group- or -world-readable permissions is enough to trigger it. +### Global flags -Format follows the stream: when stderr is a terminal the records are -colorized one-liners, and when it is redirected or piped they are -JSON, one object per line. +- `--config `: the config file to use. See + [Where the config file is](#where-the-config-file-is). +- `--verbose`, `-v`: write informational log records to stderr. +- `--debug`: write debug log records to stderr. +- `--quiet`, `-q`: print only warnings, errors and the output a script reads. + See [Output](#output). +- `--skip-errors`: skip a file that cannot be read during a backup, or cannot be + restored during a restore. Without it the command stops at the first such + file. -Under `--json`, stdout holds the document and nothing else. The startup -banner is suppressed, as `--quiet` and `--cron` suppress it, and the -progress narration a command would otherwise print — such as the stale -local records `prune` reconciles away — is suppressed too, so it cannot -land ahead of the document. Every `--json` command therefore pipes on -its own, with no additional flag: `vaultik snapshot list --json | jq .` -and `vaultik prune --json | jq .` both work as written. +`--skip-errors` applies to `snapshot create` and `snapshot restore` only. A +command that skipped anything still finishes, and then exits non-zero. -### environment variables +### Durations -* `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `snapshot restore` and `snapshot verify --deep`). May hold the whole `age-keygen` file — comments and every identity in it are accepted. Set it from the file, e.g. `export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"`, so the key is not typed into your shell history. -* `VAULTIK_CONFIG`: Path to config file (overridden by `--config`) -* `VAULTIK_INDEX_PATH`: Override local SQLite index path -* `VAULTIK_CPUPROFILE`: Write a CPU profile to this path for the duration of the run (development/debugging) -* `VAULTIK_MEMPROFILE`: Write a heap profile to this path when the run exits (development/debugging) +`--keep-newer-than`, `--older-than` and `prune_grace_period` take a duration. -### shell completion +- `d`: days. +- `w`: weeks. +- `mo`: months of 30 days. +- `y`: years of 365 days. +- `h`, `m` and `s`: hours, minutes and seconds, as in `36h` or `90m`. + +`m` means minutes and `mo` means months. Days, weeks, months and years take +whole numbers and can be combined, as in `2w3d`. + +`prune_grace_period` accepts only hours, minutes and seconds. + +### `config init` + +Writes a default config file, with a comment that explains each setting it +lists. + +- It writes to the path from `--config`, or else from `VAULTIK_CONFIG`. +- Otherwise it writes to `config.yml` in the platform config directory. That is + `~/.config/vaultik/` on Linux and `~/Library/Application Support/vaultik/` on + macOS. +- As root it writes to `/etc/vaultik/config.yml` instead. +- It creates the file with mode `0600`, because the file can hold credentials. +- It refuses to overwrite a file that exists. + +### `config edit` + +Opens the config file in `$EDITOR`. It uses `vi` when `EDITOR` is not set. + +### `config get ` + +Prints one value from the config file. The key is a dotted path, as in +`vaultik config get storage_url` or `vaultik config get age_recipients.0`. A +list or a map prints as YAML. + +### `config set ` + +Sets one value in the config file. The key is a dotted path, as in +`vaultik config set compression_level 9`. + +- Comments and formatting in the file are kept. +- A key that does not exist yet is created. A missing parent becomes a list when + the next part of the path is a number, and a map otherwise. +- In a list, the index one past the end appends an entry. +- The command prints the key it set, and never the value. +- It sets the file's mode to `0600` when group or others could read it. + +The command refuses a change that would stop the config from loading. An example +is `vaultik config set compression_level 99`. The file is then left unchanged. + +A config that does not load yet is checked less strictly. An example is the file +`config init` writes, which has no `storage_url`. There the command refuses only +a value of the wrong type, such as a word where a number belongs. + +### `snapshot create` + +Backs up the configured snapshots. With no names it backs up all of them. Each +name gets its own snapshot. + +It needs: + +- At least one public key in `age_recipients`. +- Read access to the configured paths. +- Permission to write objects at the destination. See + [Destination permissions](#destination-permissions). +- Free temporary space. See [Temporary files](#temporary-files). +- The [lock](#locking). + +It does not need the private key. + +A file is read again when any of these holds: + +- Its size has changed since the last backup. +- Its modification time has changed. +- Its mode has changed. +- Its owner or group has changed. +- Its link target has changed. +- No uploaded blob holds one of its chunks. This happens when the only snapshot + that used the blob holding that chunk is removed. + +Every other file is recorded in the new snapshot without being read. + +Flags: + +- `--cron`: print nothing when the backup succeeds. Warnings and errors are + still printed. +- `--prune`: delete old snapshots and unused data after the backup. See below. +- `--keep-newer-than `: with `--prune`, keep every snapshot newer than + this. + +The backup stops before it writes anything in these cases: + +- A `file://` destination directory does not exist. +- The destination has no marker file, although the local index holds snapshots. + See [The marker file](#the-marker-file). +- The destination in the config is not the one the local index was used with. + See [Changing the destination](#changing-the-destination). + +The backup also stops when a configured path is a symlink whose target does not +exist. It stops when a configured path does not exist, unless `--skip-errors` is +given. + +With `--skip-errors`, a file or directory that cannot be read is skipped. The +snapshot is kept and counts each skipped entry as failed, and the command exits +non-zero. An error while packing or uploading still stops the backup. + +A file that is removed during the backup is counted as failed. It does not +change the exit status. + +#### `--prune` + +After a successful backup, `--prune` does two things: + +- It removes the older snapshots of each name it just backed up. It keeps only + the newest one. With `--keep-newer-than`, it keeps every snapshot newer than + that duration instead. +- It then deletes at the destination what [`prune`](#prune) deletes. + +`--prune` also needs permission to list, read and delete at the destination. + +`--keep-newer-than` without `--prune` is an error. So is a value that is not a +duration. Both are reported before the backup starts. + +The prune does not run after a backup that may have missed data. An older +snapshot may then hold the only copy of what was missed. This applies when: + +- `--skip-errors` skipped a file or directory that could not be read. +- A configured path held nothing, as the mount point of an unmounted volume + does. +- An earlier snapshot of the same name found a mount point that this backup did + not find. + +In each case every older snapshot is kept. The command says why, names the path +and exits non-zero. It does so on every backup until the cause is gone. + +A mount point here is a directory at or below a configured path that is on a +different filesystem from its parent. It counts only when the backup finds an +entry in it that is not excluded. + +A backup does not find a mount point in these cases: + +- The volume is not mounted. +- The directory holds no entry that the backup reads. +- The directory is now excluded. +- The directory is no longer under a configured path. + +If the cause is intended, such as a retired volume, remove the older snapshots +yourself with `snapshot purge`. Take a path that is empty on purpose out of the +config. + +### `snapshot list` + +Lists the snapshots in the local index and the snapshots at the destination, as +one table. Before the table, the command prints the snapshot names and paths in +the config. + +It needs permission to list and read at the destination. It needs no private +key. + +The table has these columns: + +- `SNAPSHOT ID` +- `TIMESTAMP`: in UTC. +- `COMPRESSED SIZE`: the total size of the blobs the snapshot uses. +- `UNCOMPRESSED SIZE`: the total size of its distinct chunks before compression. +- `NEW CHUNK SIZE`: the size of the chunks this snapshot stored that were not + stored before. + +A snapshot at the destination that is not in the local index is shown as +``. The dots stand for the first 12 characters of its remote +key. Its last two columns read ``, because those sizes come from +the local index. Its timestamp is the time its backup finished. + +The command reports these below the table: + +- Snapshots that are in the local index but have no manifest at the destination. + Run `vaultik prune` to drop them from the local index. +- Remote-only snapshots whose manifest cannot be read. They are left out of the + table. +- Remote-only snapshots beyond the first 1000. They are left out of the table. +- A local index that was used with a different destination. See + [Changing the destination](#changing-the-destination). + +When the destination cannot be listed, the command warns and shows the local +index alone. It still exits zero. It does the same when the destination has no +marker file although the local index holds snapshots. + +`--json` prints a JSON array instead. Each entry has these fields: + +- `id`: the snapshot ID. It is empty for a remote-only snapshot. +- `remote_key`: the full 64-character remote key. +- `timestamp` +- `compressed_size` +- `uncompressed_size` and `new_chunk_size`: left out for a remote-only snapshot. +- `locally_tracked`: whether the snapshot is in the local index. +- `remote_present`: whether the snapshot is at the destination. It is `null` + when the destination could not be listed. + +Under `--json`, every warning above goes to stderr. + +### `snapshot verify ` + +Checks a snapshot at the destination. The argument is a snapshot ID, a remote +key or the leading part of a remote key. + +Without `--deep`, it checks two things: + +- Every blob the manifest lists is present, with the size the manifest records. +- The snapshot's encrypted database is present. + +This reads no blob contents and needs no private key. It needs permission to +read at the destination, and to list it when you give a remote key. + +Flags: + +- `--deep`: download, decrypt and re-hash every blob. See below. +- `--json`: print the result as JSON. + +The command exits non-zero when a check fails or cannot be made. + +#### `--deep` + +`--deep` needs the age private key. Set it in the `VAULTIK_AGE_SECRET_KEY` +environment variable, or as `age_secret_key` in the config file. The environment +variable wins when both are set. See [The private key](#the-private-key). + +`--deep` does these checks, in order: + +- It downloads and decrypts the snapshot's database. It checks that this is the + database of the snapshot you named. +- It checks that the manifest and the database list the same blobs, with the + same sizes. +- It checks that every blob is present with the recorded size. +- It downloads and decrypts every blob. It checks the hash of each chunk, and + the hash that names the blob. + +It stops at the first blob that fails. + +`--deep` proves that the stored data is intact and decrypts with your key. It +does not prove who wrote the data. + +`--deep` writes the decrypted database to a temporary directory and removes it +afterwards. See [Temporary files](#temporary-files). + +### `snapshot purge` + +Removes old snapshots. You must give exactly one of `--keep-latest` and +`--older-than`. + +It needs the [lock](#locking), and permission to list and delete at the +destination. It needs no private key. + +Flags: + +- `--keep-latest`: keep only the newest snapshot of each name. +- `--older-than `: remove every snapshot older than this. +- `--snapshot `: act only on snapshots with this name. Repeat the flag for + more names. +- `--force`: do not ask for confirmation. + +The command acts only on snapshots in the local index. `--keep-latest` keeps the +newest snapshot of each name, not the newest overall. + +The command removes from the local index each snapshot that the flags select. It +also deletes the snapshot's directory under `metadata/` at the destination. Its +blobs stay. Run `vaultik prune` afterwards to delete the blobs that no snapshot +uses any more. + +The command first drops from the local index each snapshot whose manifest is +gone from the destination. It fails and removes nothing when the destination +cannot be listed or has no marker file. + +### `snapshot remove ` + +Removes one snapshot. The argument must be a snapshot ID. A remote key is not +accepted. + +It needs the [lock](#locking), and permission to list and delete at the +destination. It needs no private key. + +Flags: + +- `--force`, `-f`: do not ask for confirmation. +- `--dry-run`: say what would be removed, and remove nothing. +- `--local-only`: remove the snapshot from the local index only. The destination + is not changed. +- `--json`: print the result as JSON. This also skips the confirmation. + +The command removes the snapshot from the local index. It also deletes the +snapshot's directory under `metadata/` at the destination. Its blobs stay. Run +`vaultik prune` afterwards to delete the blobs that no snapshot uses any more. + +An ID that is in neither the local index nor the destination is an error. With +`--local-only`, the snapshot must be in the local index. + +The deletion at the destination can fail, for example when the destination is +unreachable or has no marker file. What happens then depends on where the +snapshot was: + +- A snapshot in the local index is still removed from it, with a warning. Run + the same command again later to delete its directory at the destination. + `vaultik prune` does not delete it. +- For a snapshot that is only at the destination, the failure is an error. + +### `snapshot restore [paths...]` + +Restores files from a snapshot into ``. The first argument is a +snapshot ID, a remote key or the leading part of a remote key. + +It needs: + +- The age private key, as for `snapshot verify --deep`. See + [The private key](#the-private-key). +- Permission to read at the destination, and to list it when you give a remote + key. +- Write access to ``. The command creates the directory when it does + not exist. +- Free temporary space for the decrypted blobs. See + [Temporary files](#temporary-files). + +It does not need `age_recipients`, and it does not take the lock. + +Each file is restored under its original absolute path inside ``. +For example, `/home/user/notes.txt` is restored to +`/home/user/notes.txt`. + +With no paths, the command restores the whole snapshot. Each path names a file +or a directory by its original absolute path: ```sh -# zsh: load for the current session +vaultik snapshot restore ~/vaultik-restore /home/user/documents +``` + +A path that matches nothing is named in a warning. The command restores the +other paths and then exits non-zero. + +The command restores these: + +- regular files, symlinks and directories, empty directories included +- permissions and modification times +- owners and groups, when it runs as root + +Without root, restored files belong to the user who runs the command. The +command then warns once if any owner or group differs from the snapshot. + +A file that already exists in `` is replaced. Running the same +restore again starts from the beginning. + +Restore into a new directory that only you can write, not a shared one such as +`/tmp`. No other user can then change the files while they are written. + +Flags: + +- `--verify`: after the restore, read every restored file and check its chunk + hashes. + +With `--skip-errors`, a file that cannot be restored is reported and skipped, +and the command exits non-zero. `--verify` does not check a skipped file. + +### `prune` + +Removes what is no longer needed, from the local index and from the destination. + +It needs the [lock](#locking), and permission to list, read and delete at the +destination. It needs no private key. + +It works in three steps: + +- It drops from the local index each snapshot whose manifest is gone from the + destination. +- It removes unfinished snapshots from the local index. It also removes the + file, chunk and blob records that no snapshot uses. +- It reads every manifest at the destination. It then deletes each blob that no + manifest lists and that is older than `prune_grace_period`. + +The third step also deletes a directory under `metadata/` that has no manifest. +A backup leaves one when it stops before it uploads its manifest. Such a +directory is deleted only when all of these hold: + +- Every object in it is older than `prune_grace_period`. +- Its snapshot is not in the local index. +- Its name has the form of a remote key. + +`prune_grace_period` defaults to `24h`. A newer blob is kept, because a backup +that is still running may need it. See +[Sharing a destination](#sharing-a-destination) for what the grace period does +not protect. + +The command deletes nothing at the destination in these cases: + +- The destination cannot be listed. +- The destination has no marker file, although the local index holds snapshots. +- A manifest cannot be read. + +It exits non-zero when a blob or a directory cannot be deleted. + +The time to read the manifests grows with the number of snapshots. + +Flags: + +- `--force`: do not ask for confirmation before deleting at the destination. +- `--json`: print the result as JSON. This also skips the confirmation. + +The JSON document has these fields: + +- `blobs_found`: the unused blobs old enough to delete. +- `blobs_deleted` +- `blobs_failed`: blobs that could not be deleted. Left out when zero. +- `bytes_freed` +- `blobs_too_new`: unused blobs that were kept. They are newer than the grace + period, or the destination gave no time for them. +- `metadata_dirs_deleted` +- `metadata_dirs_failed`: left out when zero. + +### `info` + +Prints the system details, the settings in use and figures from the local index. +The figures are the size of the index and its counts of snapshots, blobs and +files. + +### `remote info` + +Reports what the destination holds: + +- the kind of destination and its location +- the size of each snapshot's database and manifest +- the number and size of all blobs +- the number and size of the blobs that manifests list +- the number and size of the blobs that no manifest lists + +It needs permission to list and read at the destination. It needs no private +key. + +When a manifest cannot be read, the figures for blobs that no manifest lists are +reported as unknown. That snapshot's own blob count and size are unknown too. + +The command skips a name under `metadata/` that is not a remote key, with a +warning. It does not count a file under `blobs/` whose name is not a blob hash. + +`--json` prints the report as JSON. Unknown figures are `null`. +`unreadable_manifests` lists the remote key of each manifest that could not be +read. `skipped_manifest_count` counts the manifests under skipped names. + +### `remote nuke --force` + +Deletes everything `vaultik` stored at the destination. This cannot be undone. + +It needs `--force`, the [lock](#locking), and permission to list and delete at +the destination. + +It deletes these, in order: + +- every object under `metadata/` +- every object under `blobs/` +- the marker file + +It also removes each snapshot from the local index, once that snapshot's +directory at the destination is deleted. + +The command fails before it deletes anything when the destination cannot be +listed, or has no marker file although the local index holds snapshots. + +If objects remain afterwards, the command names each one and exits non-zero. +Running it again deletes what is left. + +### `database delete` + +Deletes the local index file. The destination is not changed. + +It needs a config file that loads, because the config gives the path of the +index. + +The next backup then reads every file again. It skips the upload of a blob that +the destination already holds under the same name. + +Use this command after you point the config at a different destination. See +[Changing the destination](#changing-the-destination). + +`--force` skips the confirmation. + +### `completion ` + +Prints a completion script for `bash`, `zsh`, `fish` or `powershell`. + +```sh +# zsh, for the current session source <(vaultik completion zsh) -# zsh: install permanently +# zsh, permanently vaultik completion zsh > "${fpath[1]}/_vaultik" -# bash: load for the current session +# bash, for the current session source <(vaultik completion bash) -# bash: install permanently (Linux) +# bash, permanently, on Linux vaultik completion bash > /etc/bash_completion.d/vaultik # fish vaultik completion fish > ~/.config/fish/completions/vaultik.fish ``` -### command details +### `version` -**`config init`**: Write a default config file with commented explanations for -every setting. Writes to the path from `--config`, `$VAULTIK_CONFIG`, or the -platform config directory (`~/Library/Application Support/vaultik/` on macOS, -`~/.config/vaultik/` on Linux, `/etc/vaultik/` as root). Refuses to overwrite an -existing file. Created with mode `0600` since it will contain credentials. +Prints the version, the commit, the commit date and the platform. It labels the +commit date `build date`. It also says when the binary is a development build, +not a release. -**`config edit`**: Open the config file in `$EDITOR` (falls back to `vi`). +## Configuration -**`config get`**: Print a config value addressed by dotted YAML path -(e.g. `vaultik config get storage_url`). Non-scalar values print as YAML. +The config file is YAML. `vaultik config init` writes one to start from, and +[`config.example.yml`](config.example.yml) is a longer example. -**`config set`**: Set a scalar config value by dotted YAML path -(e.g. `vaultik config set compression_level 9`, -`vaultik config set storage_url "file:///mnt/backups"`). Comments and -formatting in the file are preserved. Missing keys along the path are -created: as a list when the next path element is a number (so -`vaultik config set age_recipients.0 age1...` works on a file without -`age_recipients`), otherwise as a map. If the config loads, a change -after which it would not, such as `vaultik config set compression_level 99`, -is refused and the file is left unchanged. A config that does not load, -such as the one `config init` writes before `storage_url` is set or one -with a `${ENV:...}` reference to a variable not set in this shell, is -checked one setting at a time: only a change after which the changed -setting cannot be read, such as one giving it the wrong type (a single -value where a list belongs or a word where a number belongs), is refused. +A command refuses to start when the config does not load. The `config` commands, +`completion` and `version` are the exceptions. The config loads when all of +these hold: -**`snapshot create`**: Perform incremental backup of configured snapshots. -* Optional snapshot names argument to create specific snapshots (default: all) -* On macOS, the terminal application running vaultik needs Full Disk Access - (System Settings → Privacy & Security → Full Disk Access) to read - TCC-protected directories; without it the backup aborts with a permission - error that explains how to fix it -* `--cron`: Silent on total success; warnings and errors are still printed - (for crontab) -* `--prune`: After backup, drop older snapshots of each backed-up name, - remove orphaned blobs older than `prune_grace_period` from remote storage, - and delete each directory under `metadata/` that has no - `manifest.json.zst` once every object in it is older than - `prune_grace_period` and its snapshot is not in the local index, as `prune` - does. By default keeps only the latest - snapshot per name; use `--keep-newer-than` for a rolling window. When - `--skip-errors` skipped a file or directory it could not read or list, the - prune does not run and every older snapshot is kept, since one of them may - be the only one that holds what was skipped; the command says so and exits - non-zero. When the backup found nothing in a configured path, as in the - mount point of a volume that is not mounted, the prune does not run - either: every older snapshot is kept, and the command names the path and - exits non-zero, on every run until the path holds something again. Take a - path that is empty on purpose out of the config, or remove the older - snapshots by hand with `snapshot purge`. When an earlier snapshot of the - same name in the local index found a volume mounted at or below a - configured path (a directory on a different filesystem from its parent, - with an entry in it that the backup does not exclude) and this backup does - not, the prune does not run either: every older snapshot is kept, and the - command names the directory and exits non-zero, on every run until the - volume is mounted again. A backup also does not find the volume when it - finds no such entry in its directory, as when an automounted volume cannot - be mounted, or when that directory is no longer backed up, because it is - excluded or its configured path was removed. If any of this is on purpose, - as for a retired volume, remove the older snapshots by hand with - `snapshot purge`. A file removed during the backup does not stop the - prune. -* `--keep-newer-than `: With `--prune`, keep snapshots newer than - this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). - Without `--prune`, or with a value that is not a valid duration, the - command fails before the backup starts. -* Fails before it writes anything when a `file://` destination directory - does not exist, or when the local index holds snapshots and the - destination answers that it has no `vaultik-destination.txt`. When that - look-up fails for any reason other than not found, for example when S3 - refuses it to credentials that allow only PUT, the check is skipped (see - [remote storage layout](#remote-storage-layout)) -* Fails, before anything is pruned, when a configured path is a symlink - whose target does not exist, as one into a volume that is not mounted +- Every entry in `age_recipients` is an age X25519 public key. +- `snapshots` has at least one name, and every name has at least one path. +- A destination is set, with credentials when it is S3. +- Every other setting below is within its limits. -**`snapshot list`**: Show every snapshot known to this host — the union -of the local index and the backup destination store — with timestamps -and three sizes per snapshot (compressed remote size; total -uncompressed chunk size; size of the chunks that snapshot stored that -were not stored before). Both chunk sizes are recorded when the -snapshot's backup runs. +`vaultik` warns when group or others can read the config file. -Listing the destination store does **not** require the age secret key, -so it works in vaultik's intended configuration, where the backed-up -host holds only the public key. A host that has lost its local index -can still see what it has backed up. +A value written as `${ENV:NAME}` is replaced by the environment variable `NAME`. +This keeps a credential out of the file. -What that host cannot see is a remote-only snapshot's name. The -snapshot ID is hashed at the storage boundary and the manifest records -only the hash, so hostname and snapshot name exist solely in the local -index and in the encrypted per-snapshot database. Snapshots found only -on the destination store are therefore listed as -`>` and show `` in -the uncompressed and "new chunk" columns, which come only from the -local index. Their timestamp and compressed size are real, -read from the manifest. +A leading `~` is expanded to your home directory in snapshot paths, in +`index_path` and in a `file://~/...` destination. -Snapshots in the local index with no counterpart on the destination -store are reported below the table as drift, with the `vaultik prune` -invocation that reconciles them. If the local index is bound to a -different destination than the configured one (after a change to -`storage_url`, or `s3.endpoint`, `s3.bucket` or `s3.prefix`), the -command says so instead and gives the same two ways out that `prune` -gives: revert the configuration, or run `vaultik database delete`. +### Where the config file is -If the destination store cannot be listed (unmounted volume, -permission denied, network down), the command warns, falls back to the -local index alone, and still exits zero. It does the same when the -destination answers that it has no `vaultik-destination.txt` while the -local index holds snapshots stored there, as when a volume mounted at -the destination is unplugged and leaves its empty mount point behind. -* `--json`: Output in JSON format. Each entry carries `locally_tracked` - (whether the snapshot is in the local index), `remote_key` (the full - 64-character storage key), and `remote_present` (whether it was seen - on the destination store, or `null` if the destination could not be - listed). Warnings about an unlistable destination, a local index - bound to a different destination (with both destinations and the - same two ways out), unreadable manifests, and a truncated listing all - go to stderr through the logger, so stdout stays a single parseable - document. +`vaultik` uses the first of these that applies: -**`snapshot verify`**: Verify snapshot integrity. -* Default (shallow): checks that every blob the manifest lists is present in - storage with the size the manifest records, and that the encrypted database is - present. It does not read blob contents. -* `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the - encrypted metadata database -* Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a - remote-only snapshot's remote key (or an unambiguous leading part of it) -* `--json`: Output results as JSON +- the path given with `--config` +- the path in the `VAULTIK_CONFIG` environment variable +- `config.yml` in the platform config directory, which is `~/.config/vaultik/` + on Linux and `~/Library/Application Support/vaultik/` on macOS +- `/etc/vaultik/config.yml` -**`snapshot purge`**: Remove old snapshots based on criteria. Retention is -per-snapshot-name (`--keep-latest` keeps the latest of each name, not the -latest globally). -* `--keep-latest`: Keep only the most recent snapshot of each name -* `--older-than `: Remove snapshots older than duration (e.g. `30d`, - `4w`, `6mo`, `1y`; `m` is minutes, `mo` is months) -* `--snapshot `: Restrict to specific snapshot names (repeat for multiple) -* `--force`: Skip confirmation prompt +A path from `--config` or `VAULTIK_CONFIG` must exist. -**`snapshot remove`**: Remove one snapshot. By default this removes the -snapshot from the local index and strips the snapshot's metadata from -the backup destination store. Blobs are NOT touched — deleting blobs -requires reading every remaining remote manifest (the destination store -may hold snapshots this host doesn't know about), which is what -`vaultik prune` does. The command prints the `vaultik prune` invocation -to run as a follow-up only when it removed the snapshot's metadata from -the destination store, and never under `--json`. An ID that names no -snapshot in the local index or on the destination store is an error; -with `--local-only`, the snapshot must be in the local index. Local row -cleanup (files, chunks, blobs the snapshot was the last referrer for) -runs automatically. If removing the snapshot's metadata from the -destination store fails (for example because the store is unreachable, -or the local index holds snapshots and the destination answers that it -has no `vaultik-destination.txt`), the local-DB removal still completes and a -warning is emitted; run `vaultik snapshot remove ` again -once the store is reachable to remove the snapshot's metadata from it -(`vaultik prune` does not). For a snapshot that is only on the -destination store, that failure is an error. To wipe everything on the -destination in one go, use `vaultik remote nuke --force`. -* `--local-only`: Skip remote cleanup; only touch the local index -* `--dry-run`: Show what would be deleted without deleting -* `--force`: Skip confirmation prompt -* `--json`: Output result as JSON. Also skips the confirmation prompt, as - `--force` does. +### Settings -**`snapshot restore`**: Restore files from a backup snapshot. -* Requires `VAULTIK_AGE_SECRET_KEY` environment variable -* Accepts a snapshot ID, or — for a snapshot only on the destination - store — its remote key (or an unambiguous leading part of it) as shown - by `snapshot list`. See - [restoring on another machine](#restoring-on-another-machine). -* Optional path arguments to restore specific files/directories (default: all). - A path that matches nothing in the snapshot is named in a warning, and the - command exits non-zero after restoring the paths that match. -* Preserves file permissions, timestamps, ownership (ownership requires root), - symlinks, and empty directories -* `--verify`: After restoring, verify every restored file's chunk hashes - match. A file skipped under `--skip-errors` is reported as failed to - restore and is not verified. +- `age_recipients` + - The age public keys that backups are encrypted to. + - Default: none. `snapshot create` needs at least one. No other command + needs one. + - Any one of the matching private keys can decrypt. + - Adding a key later does not re-encrypt what is already stored. See + [Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). +- `age_secret_key` + - The age private key. + - Default: not set. + - Only `snapshot restore` and `snapshot verify --deep` use it. + - The `VAULTIK_AGE_SECRET_KEY` environment variable overrides it. + - Do not set it on a machine you back up. See + [The private key](#the-private-key). +- `snapshots` + - The named sets of paths to back up. Required. + - `snapshots..paths`: the paths to back up under this name. + - `snapshots..exclude`: exclude patterns for this name only. +- `exclude` + - Exclude patterns that apply to every snapshot. See + [Exclude patterns](#exclude-patterns). + - Default: none. +- `storage_url` + - The destination. See [Destinations](#destinations). + - Required, unless the `s3` settings name the destination. +- `s3.access_key_id` and `s3.secret_access_key` + - The S3 credentials. Required for an S3 destination. +- `s3.region` + - The S3 region. Default: `us-east-1`. +- `s3.part_size` + - The size of each part of a multipart upload. Default: `5MiB`. + - It must be between `5MiB` and `5GiB`. + - A blob too large for 10,000 parts of this size is uploaded in larger + parts. +- `s3.endpoint`, `s3.bucket`, `s3.prefix` and `s3.use_ssl` + - An older way to name an S3 destination. They are used only when + `storage_url` is not set. + - `s3.endpoint` and `s3.bucket` are then required. + - `s3.use_ssl: false` selects plain HTTP for an endpoint written without a + scheme. +- `chunk_size` + - The average size of a chunk. Default: `10MB`. Minimum: `1MB`. + - No chunk is larger than four times this size. Only a file's last chunk can + be smaller than a quarter of it. + - A smaller size finds more duplicate content and makes the index larger. +- `blob_size_limit` + - The largest size of a blob. Default: `10GB`. + - It must be at least four times `chunk_size`. +- `compression_level` + - The zstd compression level, from `1` to `19`. Default: `3`. + - Levels `1` to `2`, `3` to `5`, `6` to `9` and `10` to `19` form four + groups. Levels in one group compress the same way. +- `prune_grace_period` + - How old unused data must be before `prune` deletes it. Default: `24h`. + - It applies to a blob that no manifest lists. It also applies to every + object in a directory under `metadata/` that has no manifest. + - `0s` deletes such data at once. A negative value is refused. + - See [Sharing a destination](#sharing-a-destination). +- `hostname` + - The hostname used in snapshot IDs. Default: the system hostname. + - Only the part before the first `.` is used. +- `index_path` + - The path of the local index. + - Default: `index.sqlite` in the platform data directory. That is + `~/.local/share/vaultik/` on Linux and + `~/Library/Application Support/vaultik/` on macOS. -**`prune`**: Tidy up everything that isn't needed. Runs three passes: -(1) reconcile the local index against the destination store — any -local snapshot whose remote metadata is missing is dropped from the -local index; (2) delete orphaned local rows (files, chunks, blobs no -longer referenced by any completed snapshot); (3) list every remote -manifest on the destination store to compute the still-referenced blob -set and delete any blob not in that set that is older than -`prune_grace_period` (default `24h`). A newer one is kept and counted: it -may belong to a backup still running, on this host or another sharing the -destination store, that has not yet uploaded its manifest. The grace period -does not protect a blob such a backup reuses from an earlier run that stopped -before its manifest: that blob keeps the time of its first upload, so it can -be deleted. Step (3) also deletes each directory under `metadata/` that has -no `manifest.json.zst`, as a backup leaves when it stops after uploading -`db.zst.age` but before its manifest, once every object in it is older than -`prune_grace_period` and its snapshot is not in the local index. An object the -listing gives no time for keeps its directory, and a directory whose name is -not a remote key is left alone. Step (3) reads all remote manifests — network -cost scales with the number of snapshots. -`snapshot create --prune` runs the same cleanup automatically; this is the -manual entry point for the same work. If the destination store cannot -be listed, or the local index holds snapshots and the destination answers -that it has no `vaultik-destination.txt` (an unplugged volume's empty mount -point), prune fails and deletes nothing. `snapshot purge` does the same. If a -blob or such a directory cannot be deleted, the command (and -`snapshot create --prune`) exits non-zero. -* `--force`: Skip confirmation prompt -* `--json`: Output stats as JSON. Also skips the confirmation prompt, as - `--force` does. A blob that cannot be deleted is counted in - `blobs_failed`, and the document is printed before the non-zero exit. - A blob kept for being newer than `prune_grace_period`, or because the - listing of the destination gives no time for it, is counted in - `blobs_too_new` and not in `blobs_found`. A directory under `metadata/` - with no manifest is counted in `metadata_dirs_deleted` once deleted, or - in `metadata_dirs_failed` if it cannot be. +Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB` +and `GiB` are powers of 1024. -**`info`**: Display system configuration, storage settings, encryption -recipients, and local database statistics. +### Environment variables -**`remote info`**: Show storage backend type and location plus detailed -remote storage inventory: per-snapshot metadata sizes, blob counts, and -orphaned blob detection. A name under `metadata/` that is not a remote -key is skipped with a warning and is not printed. If a listed -`manifest.json.zst` cannot be read, or sits under a skipped name, the -orphaned blob figures are reported as unknown; `--json` gives them as -`null`, lists the remote key of each unreadable manifest in -`unreadable_manifests` and counts the manifests under skipped names in -`skipped_manifest_count`. An unreadable manifest also leaves its -snapshot's blob count and blob size unknown, `null` in `--json`. A file -under `blobs/` whose name is not a blob hash is not counted as a blob, -since `prune` never deletes it. -* `--json`: Output as JSON +- `VAULTIK_AGE_SECRET_KEY`: the age private key. See + [The private key](#the-private-key). +- `VAULTIK_CONFIG`: the path of the config file. `--config` overrides it. +- `VAULTIK_INDEX_PATH`: the path of the local index. It overrides `index_path`. +- `RCLONE_CONFIG`: the path of the rclone config file, for an `rclone://` + destination. +- `EDITOR`: the editor that `config edit` starts. +- `TMPDIR`: the directory for temporary files. +- `NO_COLOR`: when set, status lines have no color. +- `VAULTIK_CPUPROFILE`: write a CPU profile to this path. For debugging. +- `VAULTIK_MEMPROFILE`: write a heap profile to this path on exit. For + debugging. -**`remote nuke`**: Delete every object under `metadata/` and `blobs/` on -the backup destination store, whatever its name, and then -`vaultik-destination.txt`, leaving the bucket prefix empty. Destructive -and irreversible. This is the single supported way to wipe the entire -destination store. A snapshot is removed from the local index only once -its metadata is deleted from the destination store. If the destination -store cannot be listed, or the local index holds snapshots and the -destination answers that it has no `vaultik-destination.txt` (an unplugged -volume's empty mount point), the command fails before it removes anything. If a -listing afterwards still finds anything under `metadata/` or `blobs/`, -the command names each object left and exits non-zero; running it again -deletes what is left. -* `--force`: Required to confirm destruction. +### The private key -**`database delete`**: Delete the local SQLite state database file -entirely. Remote storage is unaffected; the next backup will do a full -scan and re-deduplicate against existing remote blobs, and the local -index will re-bind to the currently configured storage destination. -Use this after changing `storage_url`, or `s3.endpoint`, `s3.bucket` or -`s3.prefix`, to a different destination. -* `--force`: Skip confirmation prompt +`snapshot restore` and `snapshot verify --deep` decrypt data, so they need the +age private key. No other command needs it. ---- +Give the key in one of two ways: -## storage backends +- the `VAULTIK_AGE_SECRET_KEY` environment variable +- `age_secret_key` in the config file -vaultik supports three storage backends, selected via the `storage_url` config field: +The environment variable wins when both are set. -**S3** (`s3://bucket/prefix?endpoint=host®ion=us-east-1`): Any S3-compatible -object store. Credentials are read from `s3.access_key_id` and -`s3.secret_access_key` in the config file. +The value can be the whole file that `age-keygen` writes. Comments are ignored, +and every key in the file is used. -**Local filesystem** (`file:///path/to/backup`): Stores blobs and metadata on -a local or mounted filesystem. Useful for testing or backing up to a NAS. The -path must be absolute; a relative one is refused when the config loads. -Percent-escapes in the path are decoded, so `my%20stick` and `my stick` name -the same directory. The directory must exist before the first backup: vaultik -never creates it. A directory on a removable volume is gone while the volume is -not mounted, so a backup then fails instead of writing to the disk under the -mount point. The mount point itself stays behind when the volume is unplugged: -there only `vaultik-destination.txt` stops such a backup, and only while the -local index holds snapshots. +Set the variable from the file, so that the key is not typed into your shell +history: -**Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud -providers. Requires rclone to be configured separately (`rclone config`). - -An upload cut off part-way leaves nothing under the object's name on S3, which -shows an object only once its upload has completed, and on the local filesystem -backend, which writes a temporary file and renames it into place. Rclone -remotes with a server-side move (local and sftp among them) are written under a -temporary name ending in `.partial` and moved into place. Rclone remotes without -one are written in place: where such a remote shows a file while it is still -being written, a killed upload can leave a truncated object under its name, -which a later backup takes for complete. A leftover `.partial` file is ignored -and can be deleted. - -A destination shared between hosts or users, each with its own local index, -relies on `prune_grace_period` to keep one's prune from deleting the blobs of -another's backup still running, or its directory under `metadata/` that has -no manifest yet, so it must be longer than the longest backup run. It does not -protect a blob that backup reuses from an earlier run that -stopped before its manifest, which the prune can delete. To not depend on the -grace period, give each host its own prefix (or directory). See -[locking](#locking). - -Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is -still supported for backward compatibility. `storage_url` takes precedence if -both are set. - ---- - -## architecture - -For an implementation-level view of the internals — the data model, the -`fx` dependency-injection wiring, and the scanner — see -[`ARCHITECTURE.md`](ARCHITECTURE.md). - -### remote storage layout - -``` -// -├── vaultik-destination.txt # Marker written by every backup -├── blobs/ -│ └── // -└── metadata/ - └── / - ├── db.zst.age # Encrypted binary SQLite database - └── manifest.json.zst # Unencrypted blob list (for pruning) +```sh +export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" ``` -* `vaultik-destination.txt` holds one fixed line of text, with no host - name, path or time. While the local index holds snapshots stored - there, a destination that answers a look-up of it with not found is - treated as not present: `snapshot create` refuses to write to it, - `prune`, `snapshot purge` and `remote nuke` fail, and `snapshot list` - and `snapshot remove` behave as for a destination that cannot be - listed. This keeps an unplugged volume's empty mount point from being - taken for a destination whose snapshots are all gone. Mount - the volume if it is removable; if the destination was emptied on - purpose, run `vaultik database delete` and back up again. - When the look-up fails for any reason other than not found, the check - is skipped and the command goes on. S3 refuses it on every run to - credentials that allow only PUT, so a host with such credentials does - not notice a destination emptied since its last backup, and its next - backup would leave out the data its local index records as already - stored. After the destination is emptied, run - `vaultik database delete` on such a host before it backs up again. -* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash -* `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted) - containing all file metadata, chunk mappings, and relationships for the snapshot -* `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling - pruning without the private key +Do not put the private key on a machine you back up. Anyone who breaks into that +machine could then decrypt its backups. Supply the key only on the machine you +restore on. -Snapshot IDs follow the human-readable format -`__` (e.g. -`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the -destination store in plaintext. Each snapshot's metadata directory is named -with its ``, a one-way double SHA-256 hash of the ID, so a plain -listing of the store shows no hostname or snapshot name. The hash uses no -secret, though, so an observer who guesses a candidate hostname and snapshot -name can hash it and confirm the snapshot is present; the remote key keeps -names out of a listing but does not hide them from a guess. The backup time is -not hidden either: manifest.json.zst carries a plaintext timestamp, and object -modification times are visible at the storage layer regardless. For example, -`server1_home_2025-06-01T12:00:00Z` is stored under -`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`. -See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the -derivation, and [Security Considerations](docs/REPOSTRUCTURE.md#security-considerations) -(including [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks)) for what the -format does and does not protect. +### Exclude patterns -### data flow +A pattern excludes a file or a whole directory from a backup. Patterns are +matched against the path below the configured path. -**backup:** +- A pattern that starts with `/` matches from the configured path. In a snapshot + of `~`, `/Library/Caches` matches only `~/Library/Caches`. +- Any other pattern matches at every depth. `.cache` matches every file or + directory named `.cache`. +- `*` and `?` match within one name. `**` matches across directories. +- A pattern that matches a directory excludes everything in it. -1. Open local SQLite index, load known files and chunks into memory -2. Walk source directories, compare mtime/size/mode against index -3. For changed/new files: chunk using content-defined chunking (FastCDC) -4. For symlinks and directories: record metadata (no chunking) -5. For each chunk: hash, check dedup, add to blob packer -6. When blob reaches size threshold: compress (zstd), encrypt (age), upload -7. Build snapshot metadata database, compress, encrypt, upload -8. Create unencrypted blob manifest for pruning support +`exclude` at the top of the config applies to every snapshot. `exclude` under a +snapshot name adds patterns for that snapshot. -**restore:** +## Destinations -1. Download and decrypt `metadata//db.zst.age` -2. Open the binary SQLite database -3. Query files (optionally filtered by paths) -4. Download and decrypt required blobs -5. Extract chunks, reconstruct files -6. Restore permissions, timestamps, ownership, symlinks +`storage_url` selects one of three kinds of destination. -**prune:** +### S3 -1. List all snapshot manifests -2. Build set of all referenced blob hashes -3. List all blobs in storage -4. Delete any blob not in the referenced set that is older than - `prune_grace_period` -5. Delete each directory under `metadata/` with no manifest whose objects - are all older than `prune_grace_period`, unless its snapshot is in the - local index +``` +s3://bucket/prefix?endpoint=s3.example.com®ion=us-east-1 +``` -### chunking and deduplication +Any S3-compatible object store works. -* Content-defined chunking using the FastCDC algorithm -* Average chunk size: configurable (default 10MB) -* Deduplication at file level (unchanged files skipped, unless a chunk - the file lists is held by no uploaded blob) and chunk level (identical - chunks across files stored once) -* Multiple chunks packed into blobs to reduce object count +- `endpoint` is the host of the store. The default is `s3.amazonaws.com`. +- `region` overrides `s3.region`. +- `ssl=false` selects plain HTTP for an endpoint written without a scheme. +- No other parameter is accepted. -### encryption +The credentials come from `s3.access_key_id` and `s3.secret_access_key` in the +config. A URL that carries credentials is refused. -* Asymmetric encryption using age (X25519 + ChaCha20-Poly1305) -* Only the public key is needed on the source host -* Each blob and each metadata database is encrypted independently -* Multiple recipients supported (encrypt to multiple keys) +### Local directory -### compression +``` +file:///path/to/backup +``` -* zstd compression at configurable level (1-19, default 3). The level is - accepted as 1-19 but maps onto zstd's four internal speed presets: - 1-2 fastest, 3-5 default, 6-9 better, 10-19 best. Levels within the - same band compress identically. -* Applied before encryption at the blob level +- The path must be absolute. +- A percent-escape in the path is decoded, so `my%20stick` names `my stick`. +- The directory must exist before the first backup. `vaultik` never creates it. ---- +A directory on a removable volume is gone while the volume is not mounted. A +backup then fails. It does not write to the disk under the mount point. -## configuration reference +This does not hold when the destination is the mount point itself. The empty +mount point stays behind when the volume is unplugged. There only +[the marker file](#the-marker-file) stops the backup. -Run `vaultik config init` to generate a fully commented config file; a -complete annotated example also lives in -[`config.example.yml`](config.example.yml). Key fields: +### rclone -| Field | Default | Description | -|-------|---------|-------------| -| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty | -| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. | -| `snapshots` | (required) | Named snapshot definitions with paths and excludes | -| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) | -| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) | -| `exclude` | | Global exclude patterns (applied to all snapshots) | -| `chunk_size` | `10MB` | Average chunk size for content-defined chunking | -| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` | -| `compression_level` | `3` | zstd compression level (1-19) | -| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once | -| `hostname` | system hostname | Hostname used in snapshot IDs | -| `index_path` | platform data dir | Local SQLite index path | +``` +rclone://remote/path +``` ---- +This uses any storage provider that rclone supports. Define the remote first +with `rclone config`. `vaultik` reads rclone's own config file. -## limitations +The URL takes no parameters and no credentials. -* **No extended attributes (xattrs).** ACLs, macOS Finder metadata, - quarantine flags, SELinux labels, and other extended attributes are not - backed up or restored. -* **No hard link detection.** Two hard links to the same inode are backed - up as independent files. Content deduplication means the data is stored - once, but the hard link relationship is lost on restore. -* **No sparse file support.** Sparse files are fully materialized during - backup. A 100 GB sparse VM disk that is mostly zeros will consume the - full (compressed) size in storage. -* **No bandwidth limiting.** Uploads and downloads use whatever bandwidth - is available. There is no `--bwlimit` flag yet. -* **No parallel blob downloads during restore.** Blobs are fetched - sequentially. Restore speed is bound by single-stream throughput. -* **Device nodes, named pipes, and sockets are silently skipped.** Only - regular files, directories, and symlinks are backed up. -* **Before 1.0, an update can make the local index unusable.** Vaultik - supports schema migrations, but none are added before 1.0 because - there is no installed base yet. If an update leaves your local index - unusable, run `vaultik database delete` and then a full backup; remote - storage is unaffected. After 1.0, the local index is migrated when - vaultik is updated. See - [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). -* **Files that change during backup may be inconsistent.** There is no - filesystem snapshot or freeze. If a file is modified between the scan - and chunk phases, the backed-up copy may reflect a partial write. -* **Ownership restoration requires root.** File uid/gid are recorded - and restored, but `chown` requires elevated privileges. Without root, - files are restored with the current user's ownership. +### Interrupted uploads ---- +An upload that is cut off must not leave a partial object under the final name. +A later backup would take that object for complete. -## roadmap +- S3 shows an object only once its upload has finished. +- The local directory destination writes a temporary file and then renames it. +- An rclone remote that can move objects is written the same way. The temporary + name ends in `.partial`. +- An rclone remote that cannot move objects is written in place. On such a + remote, a killed upload can leave a partial object under the final name. -Work planned after 1.0. Loosely ordered by priority. +A leftover `.partial` file is ignored. You can delete it. -### correctness and operability +### Destination permissions -* **Outside security audit.** Before 1.0 the encryption and - blob-generation code was reviewed: every bug the review found was - fixed, and the risks it accepted are listed in - [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside - audit has been done. age + zstd + content-defined chunking - is mostly off-the-shelf pieces, but the seams (key handling, - recipient parsing, manifest trust boundary, restore-time identity - validation) need an outside read. -* **Verify restored content end-to-end in CI.** The current - integration test does this for a small synthetic snapshot but - not at scale. A nightly job against a multi-GB representative - snapshot would catch silent regressions in the chunker, packer, - or restore planner. +Each command needs different permissions at the destination. -### performance +- `snapshot create` needs only permission to write objects. A machine you back + up can therefore hold credentials that can neither read nor delete. +- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list + and read. +- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete. +- `prune` and `snapshot create --prune` list, read and delete. -* **Parallel blob downloads during restore.** Single-stream right - now. With a fast S3 endpoint and a multi-core machine restore is - bound by per-blob fetch + decrypt + decompress; running N of - those in parallel against the disk cache would close most of the - remaining gap. Needs to interact correctly with the locality - planner and sweeper. -* **Bandwidth limiting (`--bwlimit`).** Both upload and download. - Useful for backing up over a shared link. Tricky to make work - correctly with the parallel-download story. -* **Restart of interrupted restore.** Today restore is restartable - in the sense that re-running it overwrites partial output; it - doesn't resume from where it stopped or skip already-present - files. A `--resume` mode that checks targets before fetching - blobs would matter for very large restores. -* **Daemon mode.** A long-running mode that watches for file - changes so frequent backups, such as hourly, skip the full scan. - It adds little for the usual runs from cron every 12 to 36 hours. - See [issue #204](https://git.eeqj.de/sneak/vaultik/issues/204). +A machine that runs `snapshot create --prune` can delete its own backups. To +keep it to write-only credentials, run `prune` from a separate, trusted machine. -### usability +A backup tries to look up the marker file and each blob before it writes. It +goes on when the destination refuses those look-ups. -* **Man pages and richer `--help` examples.** Cobra generates - basic help; man pages would be a separate target. -* **`vaultik snapshot diff `** — show which files changed - between two snapshots without restoring either. -* **Status reporting hook for `--cron`.** When a backup fails - silently in cron, the user has no idea. A configurable - webhook / email / `notify-send` hook on completion (success and - failure) would close the loop. +### The marker file -### infrastructure +Every backup writes `vaultik-destination.txt` at the top of the destination. The +file holds one fixed line of text. -* **Schema migrations after 1.0.** Migrations are supported, but none - are added before 1.0 because there is no installed base yet. After - 1.0, each schema change is a new migration, so an existing local - index is migrated when vaultik is updated (see - [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations)). -* **Storage backend coverage tests.** S3, file://, and rclone:// - all share the Storer interface but the rclone path is the least - exercised in CI. +The marker tells an emptied or missing destination apart from a new one. An +unplugged volume leaves an empty directory at its mount point. Without the +marker, that directory would look like a destination whose snapshots are all +gone. ---- +The check applies while the local index holds snapshots. When the destination +then answers that the marker does not exist, commands treat the destination as +not present: -## output style +- `snapshot create` refuses to write to it. +- `prune`, `snapshot purge` and `remote nuke` fail. +- `snapshot list` shows the local index alone. +- `snapshot remove` deletes nothing at the destination. -Every command's user-facing output is governed by `internal/ui`, in one -of two ways. Color is enabled when the stream written to is a TTY and -the `NO_COLOR` environment variable is unset (https://no-color.org/). +To fix this, mount the volume. If you emptied the destination on purpose, run +`vaultik database delete` and back up again. -* **Status, progress, warnings, and errors** go through the `internal/ui` - message methods below: marker-prefixed, colored on a TTY, and — except - warnings and errors — silenced by `--quiet`. This is the operational - narration of the long-running commands (`snapshot create`, `prune`, - `snapshot restore`, and the like) and the confirmations of - `config init`, `config set`, and `database delete`. -* **The data a command exists to produce** is written plain, with no - marker and no color, because a marker would corrupt a table or a - parsed document. This covers the `version`, `info`, `remote info` and - `snapshot verify` reports, the `snapshot list` table, `config get` - values, and every `--json` document. `--quiet` silences the human - reports and tables (`version`, `info`, `remote info`, `snapshot - verify`, `snapshot list`) but never the machine-consumed `config get` - value or the `--json` documents, which a script depends on. The - confirmation prompts of `database delete`, `prune`, `snapshot purge` - and `snapshot remove`, each with what it asks to delete, are also - written this way and always shown: they are an interactive exchange - the operator must see. +The check is skipped when the look-up fails for any other reason. S3 refuses the +look-up to credentials that can only write. A machine with such credentials +therefore does not notice an emptied destination. Its next backup would not +upload data that its local index says is stored. After you empty a destination, +run `vaultik database delete` on such a machine before it backs up again. -`internal/ui` writes to stdout; it is the output the user asked for. The -exceptions are the startup banner and the error a failed command ends -with, which go to stderr. Structured log records are a different thing -and go through `internal/log`, which writes to stderr (see "stdout and -stderr" above). +### Changing the destination -Message classes: +The local index belongs to one destination. It records which blobs are stored +there. -| Class | Marker | Alignment | Use for | -|-------|--------|-----------|---------| -| Banner | none | column 0 | The startup line printed once per invocation, on stderr | -| Begin | `》` (white) | column 0 | An operation is about to start (present-continuous verb) | -| Complete | `》` (green) | column 0 | An operation just finished (past-tense verb) | -| Info | `》` (white) | column 0 | Neutral status update | -| Notice | `》` (cyan) | column 0 | Important note that is not a warning | -| Warning | `⚠️ Warning:` (orange/yellow) | column 0 | Recoverable problem | -| Error | `🛑 ERROR:` (red) | column 0 | Operation aborted | -| Progress | ` 》` (white) | column 2 | Heartbeat or per-item status during a long-running operation | -| Detail | ` 》` (white) | column 2 | Continuation/sub-line of a preceding Complete (visually identical to Progress) | +The first command that changes data saves the configured destination in the +local index. Later, these commands refuse to run when the config names a +different destination: -Conventions: +- `snapshot create` +- `snapshot purge` +- `snapshot remove` +- `prune` +- `remote nuke` -* Messages are complete English sentences ending with a period. -* Fully qualify terms — say "backup destination store" instead of - "storage", "snapshot source files enumeration" instead of "scan", - "local index database" instead of "database". -* Every operation that emits a Complete also emits a corresponding - Begin. Operations that print only a Begin (because completion is - obvious from a later Begin) should be rare and intentional. -* Use natural verb tense to signal state: "Uploading" for Begin, - "Uploaded" for Complete. Never write the words "begin" or "complete" - in the body — the marker color already conveys that. -* All elapsed and remaining-time fields are explicitly scoped to their - subject: write "blob upload elapsed: 30s, blob upload ETA: 03:15:00 - (est remain 14s)", never just "elapsed 30s, ETA 14s". -* "ETA" means an absolute clock time (when the operation will finish), - not a remaining-duration. Use `ui.Time()` for the former and - `ui.Duration()` for the latter, and label both. -* `ui.Time` formats same-day times as `HH:MM:SS` and other-day times as - `YYYY-MM-DD HH:MM:SS`. No timezone — local time is implied. +The destination that is compared is `storage_url`. When `storage_url` is not +set, it is `s3.endpoint`, `s3.bucket` and `s3.prefix`. -Value colorizers in `internal/ui` colorize specific value types -consistently. Compose messages from these helpers rather than embedding -ANSI escapes inline: +There are two ways out: -| Helper | Color | Use for | -|--------|-------|---------| -| `Hex` | cyan | Blob hashes, chunk hashes (truncated to 12 chars + `...`) | -| `Snapshot` | bold cyan | Snapshot IDs (untruncated) | -| `Path` | blue | Filesystem paths | -| `Size` | magenta | Byte counts (human-readable) | -| `Speed` | magenta | Bytes-per-second rates | -| `Duration` | yellow | Elapsed or remaining time | -| `Time` | yellow | Absolute clock times | -| `Count` | magenta | Integer counts with thousands separators | -| `Percent` | magenta | Percentages | +- Change the config back to the earlier destination. +- Run `vaultik database delete`. The next backup then starts a new index for the + new destination. -When `NO_COLOR` is set or output is not a TTY, all helpers return plain -text and the marker prefixes (`》`, `Warning:`, `ERROR:`) emit without -ANSI escapes. The emoji prefixes on Warning and Error are always emitted -regardless of color setting (emoji are not color). +### Sharing a destination -## requirements +Give each machine and each user its own destination, such as its own prefix or +directory. The rest of this section applies only when two of them share one. -* Go 1.26 or later -* Docker, with a reachable daemon, to test, lint, check, or commit: - `script/test` and `script/lint` build the `test` and `lint` phases of - the `Dockerfile`, and `make check` and the pre-commit hook both run - them. A `golangci-lint` installed on `PATH` is not a substitute and is - never used on a host, whatever its version. -* S3-compatible object storage (or local filesystem, or rclone remote) +A prune from one machine can meet a backup from another that is still running. A +backup uploads its blobs first, then its database, and its manifest last. Until +the manifest exists, no manifest lists the new blobs. -## development workflow +`prune_grace_period` protects those blobs. A prune keeps a blob that no manifest +lists until the blob is older than the grace period. It keeps a directory under +`metadata/` that has no manifest for the same time. Set the grace period longer +than the longest backup of any machine or user that shares the destination. -All changes follow this workflow. No exceptions. +The grace period does not protect one case. A backup does not upload a blob that +is already at the destination. Such a blob keeps the time of its first upload. +It can be left over from an earlier backup that stopped before its manifest. A +prune from another machine can then delete that blob while the new backup runs. +The new snapshot cannot be restored afterwards. -1. Create a feature branch off `main`. -2. Write tests. -3. Write the implementation. -4. Fix implementation errors until it compiles and tests pass. -5. Fix linting errors (`make lint`). -6. Update documentation and README as required by the change. -7. Format code (`make fmt`). -8. Run `make check` (lint + fmt-check + test). Fix any issues. Repeat until clean. -9. Commit on the branch. -10. Merge to `main`. -11. Push. +`remote nuke` ignores the grace period. -Do not commit directly to `main`. Do not skip steps. +## Locking -Repository policies for AI agents are in [`AGENTS.md`](AGENTS.md). +Commands that change data take a lock, so that only one of them runs at a time. +These commands take it: -## Entrypoints +- `snapshot create` +- `snapshot purge` +- `snapshot remove` +- `prune` +- `remote nuke` -This repository adheres to the +A second one does not wait. It exits at once with an "already running" error. + +The lock is the file `vaultik.pid` in the platform data directory. That is +`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/` +on macOS. + +The lock covers one user on one machine. It does not stop another user or +another machine from writing to the same destination. See +[Sharing a destination](#sharing-a-destination). + +No other command takes the lock. `snapshot list`, `snapshot verify`, +`snapshot restore`, `info` and `remote info` run while a backup is in progress. + +## Output + +### stdout and stderr + +stdout carries the output you asked for: + +- status and progress lines +- tables and reports +- JSON documents +- `config get` values +- completion scripts + +stderr carries the rest: + +- the startup banner +- log records, including those from `--verbose` and `--debug` +- the error that a failed command ends with + +So `vaultik snapshot list > out.txt` captures the list and leaves the +diagnostics on your terminal. + +Log records are colored lines when stderr is a terminal. They are JSON, one +object per line, when stderr is redirected. + +Status lines are colored only on a terminal, and never when `NO_COLOR` is set. + +### `--json` + +Under `--json`, stdout holds one JSON document and nothing else. The banner and +the status lines are left out, and warnings go to stderr. A command can +therefore be piped as it is: + +```sh +vaultik snapshot list --json | jq . +``` + +### `--quiet` and `--cron` + +`--quiet` leaves out the banner, the status lines, the tables and the reports. +`--cron` on `snapshot create` does the same. + +Neither flag hides these: + +- warnings and errors +- `config get` values +- JSON documents +- confirmation prompts + +### Exit status + +- `0`: the command succeeded. +- `1`: the command failed. +- `130`: the command was interrupted by Ctrl-C or `SIGTERM`. + +An interrupted command removes its temporary files before it exits. It gives up +after 30 seconds. + +## Temporary files + +Temporary files go to `$TMPDIR`, or to `/tmp` when `TMPDIR` is not set. + +A backup needs free temporary space for these: + +- One blob at a time. Each blob is written whole to a temporary file before it + is uploaded, so allow about `blob_size_limit`. +- About twice that for an rclone remote that cannot stream uploads. +- Copies of the local index, made when the snapshot's database is prepared. + +With `TMPDIR` not set, SQLite writes one of those index copies to `/var/tmp`. + +A restore keeps the downloaded blobs on disk, decrypted, until no remaining file +needs them. `snapshot restore` and `snapshot verify --deep` also write the +decrypted snapshot database there. + +These files are readable only by their owner, and are removed when the command +ends. A `kill -9` or a power loss leaves them behind. Delete any leftover +`vaultik-*` entry in the temporary directory yourself. + +## What the destination reveals + +File contents, names, paths and metadata are encrypted. Someone who can read the +destination still learns these: + +- when each backup finished +- how many blobs each snapshot uses, and their sizes +- which blobs two snapshots share +- whether a hostname and snapshot name they guess exist there + +The last point holds because a remote key is a plain hash of the snapshot ID, +made without a secret. + +[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the +destination, what it protects and the risks it accepts. + +## Limitations + +- **No extended attributes.** ACLs, macOS Finder metadata, quarantine flags and + SELinux labels are not backed up. +- **No hard links.** Two hard links to one file are backed up as two files. The + content is stored once, but a restore creates two separate files. +- **No sparse files.** A sparse file is backed up at its full size, zeros + included, and restored as a full file. +- **No bandwidth limit.** Uploads and downloads use all the bandwidth there is. +- **One download at a time.** A restore downloads blobs one after another. +- **No device nodes, named pipes or sockets.** They are skipped without a + message. +- **No consistent view of changing files.** A file that changes while it is read + can be stored in a mixed state. +- **Owners need root.** A restore sets owners and groups only when it runs as + root. +- **No resume.** An interrupted restore starts again from the beginning. +- **The local index can become unusable after an update.** This applies before + version 1.0. Run `vaultik database delete` and then a full backup. The + destination is not affected. See + [Schema migrations](docs/DATAMODEL.md#schema-migrations). + +## Design + +A backup passes your files through four parts of the program: + +- The `Scanner` walks the configured paths and picks the files to read. +- The `Chunker` cuts each of those files into chunks. +- The `Packer` collects new chunks into blobs. It compresses and encrypts each + blob. +- The `Storer` uploads each blob to the destination. + +The backup then uploads the snapshot's database and its manifest. The database +is a trimmed, encrypted copy of the local index. + +These documents give the detail: + +- [ARCHITECTURE.md](ARCHITECTURE.md) describes how the code is organised. +- [docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index. +- [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the + destination. + +## Development + +Two documents give the rules for work on this repository: + +- [`AGENTS.md`](AGENTS.md) holds the rules for AI agents. +- [`REPO_POLICIES.md`](REPO_POLICIES.md) holds the repository policies. + +### Requirements + +- Go 1.26.1 or later. +- Docker, with a daemon you can reach. The tests and the linter run only inside + Docker. + +### Workflow + +- Make every change on a feature branch. +- Write the tests with the change. +- Update the documentation that the change affects. +- Run `make fmt`, then `make check`, before each commit. + +`make check` runs the tests, the linter and the format check. All three must +pass. + +### Entrypoints + +This repository follows the [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) -standard: normalized scripts in `script/` are the entrypoints for the -development workflow, and the Makefile targets are thin shims that call -them. We provide: +standard. The scripts in `script/` are the entrypoints. The `Makefile` targets +only call them. -* `script/bootstrap` — install all development dependencies (Go, Go - module download). A host without Go gets the `go.mod` version through - `script/install-go`, in `.tool/go`, which `script/bootstrap` itself, - the `Makefile`, `script/fmt`, `script/fmt-check`, `script/precommit` - and `script/release` add to their `PATH`. It - deliberately does not install `golangci-lint`; see `script/lint` - below. -* `script/setup` — make a fresh clone ready for development: runs - `script/bootstrap`, then `script/install-precommit` -* `script/projectname` — print the project name (used for the Docker - image tag) -* `script/version` — print the version string to bake into the binary. - The `Makefile`'s `LDFLAGS` call this. See [releasing](#releasing) for - the rules. -* `script/install-goreleaser` — install the pinned `goreleaser` into - `.tool/bin` from a sha256-verified release archive. Idempotent, and - called by `script/bootstrap`; the release workflow calls it directly - because it needs `goreleaser` but not the Docker daemon - `script/bootstrap` insists on. -* `script/install-go` — install the Go toolchain named by `go.mod`'s - `go` directive into `.tool/go` from a sha256-verified `go.dev` - archive, for Linux or macOS on amd64 or arm64. Idempotent. On a CI - runner it also puts `.tool/go/bin` on `PATH` for the steps that - follow. Called by the release workflow, which needs a host Go for - `goreleaser` to shell out to, and by `script/bootstrap` on a host - without Go. `actions/setup-go` is not used because it verifies the - downloaded toolchain against no value in this repo. Bumping Go edits - `go.mod`, the checksums in this script, and the two `golang` digests - in the `Dockerfile` together. -* `script/release` — cross-compile and publish the release artifacts - with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose - version is not the pinned one, because a different version would build - a different release from the same tag. -* `script/release-snapshot` — the same build with no publishing and no - tagging, into `./dist` -* `script/test` — run the test suite by building the `test` phase of - the `Dockerfile` (verbose rerun on failure). This runs *everything*: - there is no separate integration target and no build-tagged subset - held back, so the full round-trip tests in - `internal/vaultik/integration_test.go` run on every invocation. The - 90s `-timeout` is a hang backstop rather than a performance budget; it - applies per test binary, to test execution only. -* `script/lint` — lint by building the `lint` phase of the - `Dockerfile`, which runs `golangci-lint config verify` and then - `golangci-lint run --config .golangci.yml ./...` as build steps in - the digest-pinned `golangci-lint` image. Nothing lints on the host. - That `FROM` line is the only pin of the linter version, and it changes - in the same commit as a re-vendored `.golangci.yml`. -* `script/lint-fix` — apply the linter's autofixes (rewrites files), - using the same pinned image, parsed out of the `lint` phase's `FROM` - line. It cannot be a build step, because fixes have to land in the - worktree, so it bind-mounts the tree into a `docker run` and therefore - needs a *local* daemon. It is a developer convenience and never a - gate: no gate reads its exit status. Run `make lint` afterwards to - find out whether the tree is clean. -* `script/fmt` — format all code (writes) -* `script/fmt-check` — check formatting (read-only). It runs `gofmt` on - the host, over every Go file outside `.tool`. -* `script/check` — run `script/test`, `script/lint`, and - `script/fmt-check`. -* `script/docker` — build the Docker image tagged via - `script/projectname`, stamped with the version - `git describe --tags --always --dirty` gives on the host, or `unknown` - outside a git checkout. The image's build stage depends on the `lint` - and `test` phases, so this lints and tests too. -* `script/cibuild` — CI entrypoint: runs `script/bootstrap`, - `script/check`, and then the same image build as `script/docker`. - `.gitea/workflows/check.yml` runs it on every push. +- `script/bootstrap`: installs what development needs. It installs Go from + `script/install-go` when the machine has none, and it fails without Docker. +- `script/setup`: prepares a fresh clone. It runs `script/bootstrap` and + `script/install-precommit`. +- `script/test`: runs the whole test suite, by building the `test` stage of the + `Dockerfile`. +- `script/lint`: runs the linter, by building the `lint` stage of the + `Dockerfile`. The linter never runs on your machine. +- `script/lint-fix`: applies the linter's automatic fixes to your files. It + needs a local Docker daemon. Run `script/lint` afterwards. +- `script/fmt`: formats the Go code. +- `script/fmt-check`: checks the formatting and changes nothing. +- `script/check`: runs `script/test`, `script/lint` and `script/fmt-check`. +- `script/precommit`: runs `go mod tidy`, `go fmt` and `script/check`. The git + pre-commit hook calls it. +- `script/install-precommit`: installs that hook. +- `script/docker`: builds the Docker image. The build runs the linter and the + tests too. +- `script/cibuild`: what CI runs on every push. It runs `script/bootstrap`, + `script/check` and the image build. +- `script/version`: prints the version to build into the binary. +- `script/projectname`: prints the project name, which tags the Docker image. +- `script/install-go`: installs the Go version that `go.mod` names into + `.tool/go`. It checks the download against a checksum in the script. +- `script/install-goreleaser`: installs the pinned `goreleaser` into + `.tool/bin`, checked the same way. +- `script/release`: builds and publishes the release files. +- `script/release-snapshot`: builds the release files into `./dist` and + publishes nothing. - Every `docker build` in these scripts passes `--no-cache`, because on - an unchanged tree a cached check layer is replayed without running and - the build still exits 0. A plain `docker build .` is therefore no - evidence that the checks ran. The cost is that `script/cibuild` runs - the `lint` and `test` phases twice: once in `script/check` and again - in the image build. -* `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must - not change files), then `script/check` -* `script/install-precommit` — install the git pre-commit hook that - runs `script/precommit` +Every `docker build` in these scripts uses `--no-cache`. A cached layer would +skip the tests or the linter and still report success. -## releasing +The test suite has no separate integration target. Every test runs every time. +Each test binary is stopped after 90 seconds. -### version numbers +To change the Go version, change these together: `go.mod`, the version and +checksums in `script/install-go`, and the two `golang` image digests in the +`Dockerfile`. -The version a binary reports comes from git, not from a constant in a -file. It is `git describe --tags --always --dirty`, which -`script/version` runs for the `Makefile`, and `script/docker` and -`script/cibuild` run themselves: +### Messages to the user -* `HEAD` is exactly on a tag → that tag, such as `v1.0.0`. -* a commit after a tag → `--g`. -* no tag reachable → the short commit sha. -* any of these, with uncommitted changes to tracked files → a `-dirty` - suffix, because a modified checkout of a tag is not that tag. +Status lines, warnings and errors go through `internal/ui`. The data a command +exists to print is written plain, with no marker and no color. That covers +tables, reports, JSON documents and `config get` values. -A `docker build .` of a clone, with no build arguments, runs the same -`git describe` (without `--dirty`) on the `.git` in its build context, -so it stamps the same value for a clean commit; the build fails if the -context carries `.git` and no version, commit or commit date comes out. -A binary built without -git metadata reports `dev`, or `unknown` when `script/docker` or -`script/cibuild` built it outside a git checkout. +`internal/ui` has these kinds of message: -`goreleaser` stamps a release binary with the tag minus its leading -`v`, so the tag `v1.0.0` produces `vaultik 1.0.0`, matching the archive -name `vaultik_1.0.0_linux_amd64.tar.gz`. `goreleaser --snapshot` stamps -`dev-<12 chars of the commit sha>` rather than inventing the next patch -number. `vaultik version` calls a build a development build when its -version is `dev`, `unknown`, `dev-`, the short commit sha or -`--g`, with or without `-dirty`; only a plain tag, -such as `v1.0.0` or `1.0.0`, is a release. If `script/version` cannot -be run at all, `make` stops with an error instead of building an -unversioned binary, and a binary that somehow carries an empty version -string still reports itself as a development build. +- **Banner**: the startup line. No marker. +- **Begin**: an operation is starting. Marker `》`. +- **Complete**: an operation has finished. Marker `》`, in green. +- **Info**: a neutral status line. Marker `》`. +- **Notice**: a note the user should see. Marker `》`, in cyan. +- **Warning**: a problem the command survives. Prefix `⚠️ Warning:`. +- **Error**: the command stops. Prefix `🛑 ERROR:`. +- **Progress** and **Detail**: an indented line under another. Marker ` 》`. -### cutting a release +Messages follow these conventions: -Releases are cut by CI from a tag, not from a workstation: +- A message is a full sentence that ends with a period. +- Name things in full. Write "backup destination store", not "storage". +- An operation that prints a Complete also prints a Begin. +- Use the tense to show the state: "Uploading" for a Begin, "Uploaded" for a + Complete. +- Say what a time belongs to, as in "blob upload elapsed: 30s". +- "ETA" is a clock time, not a remaining time. Print it with `ui.Time`, and + print a remaining time with `ui.Duration`. -``` -git tag -a v1.2.3 -m 'v1.2.3' -git push origin v1.2.3 -``` +Build a message from the value helpers in `internal/ui`, and never write color +codes by hand: -`.gitea/workflows/release.yml` triggers on `v*` tags, installs a Go -toolchain and the pinned `goreleaser`, and runs `script/release`, which -builds -`linux,darwin × amd64,arm64` archives plus `checksums.txt` and publishes -them to this repository's Gitea releases as a draft. `.goreleaser.yaml` -has a `gitea_urls:` block pointing at `https://git.eeqj.de/api/v1`; -without it `goreleaser` would talk to the GitHub API. +- `Hex`: a blob or chunk hash, cut to 12 characters. +- `Snapshot`: a snapshot ID. +- `Path`: a filesystem path. +- `Size`: a byte count. +- `Speed`: a transfer rate, in bits per second. +- `Duration`: an elapsed or remaining time. +- `Time`: a clock time, in local time. +- `Count`: a whole number, with thousands separators. +- `Percent`: a percentage. -The workflow needs one repository Actions secret: +## Releasing -| Secret | What it is | -| --------------- | ------------------------------------------------------------------------------------------------------- | -| `RELEASE_TOKEN` | A Gitea access token with `write:repository` scope, owned by an account that can publish releases here. | +### Version numbers -It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic -token is deliberately not used: it is not guaranteed to carry release -write access. +The version a binary reports comes from git. It is the output of +`git describe --tags --always --dirty`: -The Go toolchain that compiles the released binaries is installed by -`script/install-go`, which downloads the version named by `go.mod` -(currently `1.26.1`, the same version the `Dockerfile` builder stage -pins by digest) and refuses the archive unless its sha256 matches the -value committed in the script. `goreleaser` shells out to `go` for every -cross-compile, so without that step the release would either fail -outright or ship binaries built by whatever unpinned toolchain the -runner happened to carry. +- On a tagged commit, it is the tag, such as `v1.0.0`. +- On a commit after a tag, it is `--g`. +- With no tag, it is the short commit. +- With uncommitted changes to tracked files, `-dirty` is added. -To rehearse the whole build without publishing or tagging anything: +`script/version` gives this value to the `Makefile`. `script/docker` and +`script/cibuild` compute it themselves. -``` -make release-snapshot -``` +A binary built without git reports `dev`. An image built by `script/docker` or +`script/cibuild` outside a git checkout reports `unknown`. -Artifacts land in `./dist`, which is gitignored. +`goreleaser` builds the release binaries. It uses the tag without its leading +`v`, so the tag `v1.0.0` gives `vaultik 1.0.0`. `script/release-snapshot` stamps +`dev-` followed by 12 characters of the commit. -Release artifacts are not signed, carry no SBOM, and are not built -reproducibly; the archives contain the binary, `LICENSE`, and -`README.md` only (no shell completions or man page). +`vaultik version` calls a build a release only when its version is a plain tag, +such as `v1.0.0` or `1.0.0`. Every other version is a development build. + +`make` stops with an error when `script/version` gives no version. + +### The release workflow + +`.gitea/workflows/release.yml` runs when a tag that starts with `v` is pushed. +It does this: + +- It installs Go with `script/install-go` and `goreleaser` with + `script/install-goreleaser`. +- It runs `script/release`. +- `script/release` builds archives for Linux and macOS on `amd64` and `arm64`, + and a `checksums.txt`. +- It publishes them to this repository's releases, as a draft. + +The workflow needs one repository secret, `RELEASE_TOKEN`. It is a Gitea access +token with the `write:repository` scope, from an account that can publish +releases here. + +`script/release` refuses a `goreleaser` on `PATH` that is not the pinned +version. + +`make release-snapshot` builds the same files into `./dist` and publishes +nothing. + +Each archive holds the binary, `LICENSE` and `README.md`. The release files are +not signed, have no SBOM and are not built reproducibly. ## license diff --git a/TODO.md b/TODO.md index c32824f..849ebf8 100644 --- a/TODO.md +++ b/TODO.md @@ -1316,5 +1316,64 @@ owner's; a `v*` tag runs the release build in # Future Steps -Work planned after 1.0 is listed in the README -[roadmap](README.md#roadmap). +Work planned after 1.0, loosely ordered by priority. + +## Correctness and operability + +- **Outside security audit.** Before 1.0 the encryption and + blob-generation code was reviewed. Every bug the review found was + fixed. The risks it accepted are listed in + [Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside + audit has been done. age, zstd and content-defined chunking are mostly + off-the-shelf pieces. The code that joins them needs an outside read: + - key handling + - recipient parsing + - manifest trust boundary + - restore-time identity validation +- **Verify restored content end-to-end in CI.** The current integration + test does this for a small synthetic snapshot but not at scale. A + nightly job against a multi-GB representative snapshot would catch + silent regressions in the chunker, packer, or restore planner. + +## Performance + +- **Parallel blob downloads during restore.** A restore downloads one + blob at a time now. With a fast S3 endpoint and a multi-core machine, + restore speed is bound by the fetch, decryption and decompression of + each blob. Running several of those in parallel against the disk + cache would close most of the remaining gap. This needs to interact + correctly with the locality planner and the sweeper. +- **Bandwidth limiting (`--bwlimit`).** For both upload and download. + It is useful for backing up over a shared link. It is tricky to make + work correctly with parallel downloads. +- **Restart of interrupted restore.** Today a restore can be started + again, and it then overwrites the partial output. It does not resume + from where it stopped, and it does not skip files that are already + present. A `--resume` mode that checks targets before fetching blobs + would matter for very large restores. +- **Daemon mode.** A long-running mode that watches for file changes, + so that frequent backups, such as hourly ones, skip the full scan. It + adds little for the usual backups from cron every 12 to 36 hours. See + [issue #204](https://git.eeqj.de/sneak/vaultik/issues/204). + +## Usability + +- **Man pages and richer `--help` examples.** Cobra generates basic + help. Man pages would be a separate target. +- **`vaultik snapshot diff `.** Show which files changed between + two snapshots, without restoring either. +- **Status reporting hook for `--cron`.** When a backup fails silently + in cron, the user has no idea. A configurable hook on completion, for + both success and failure, would close the loop. It could be a + webhook, an email or `notify-send`. + +## Infrastructure + +- **Schema migrations after 1.0.** Migrations are supported, but none + are added before 1.0 because there is no installed base yet. After + 1.0, each schema change is a new migration, so an existing local + index is migrated when `vaultik` is updated. See + [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). +- **Storage backend coverage tests.** S3, `file://` and `rclone://` all + share the `Storer` interface, but the rclone path is the least + exercised in CI. diff --git a/config.example.yml b/config.example.yml index b23252f..d78982b 100644 --- a/config.example.yml +++ b/config.example.yml @@ -1,16 +1,31 @@ # vaultik configuration file example -# This file shows all available configuration options with their default values -# Copy this file and uncomment/modify the values you need +# This file shows every configuration option. The values are examples, not +# defaults; where an option has a default, its comment gives it. +# Copy this file and change the values you need # Age recipient public keys for encryption # Backups are encrypted to these public keys. snapshot create needs at least # one; listing, verifying and restoring do not -# Generate with: age-keygen | grep "public key" +# Generate a key pair, and show its public key, with: +# age-keygen -o vaultik_backup_private_key.txt +# grep 'public key' vaultik_backup_private_key.txt +# Keep the private key file: restoring needs it. Do not pipe age-keygen +# straight into grep, which would discard the private key age_recipients: - age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz +# Age private key for decryption +# Only snapshot restore and snapshot verify --deep need it. The +# VAULTIK_AGE_SECRET_KEY environment variable overrides this setting +# Do not set it on a machine you back up: prefer the environment variable, +# supplied only on the machine you restore on +# Default: not set +#age_secret_key: YOUR_AGE_PRIVATE_KEY + # Named snapshots - each snapshot can contain multiple paths # Each snapshot gets its own ID and can have snapshot-specific excludes +# Required: at least one snapshot with at least one path. Only the config +# commands, version and completion run without one snapshots: testing: paths: @@ -258,6 +273,11 @@ exclude: # Storage URL - use either this OR the s3 section below # Supports: s3://bucket/prefix, file:///path, rclone://remote/path +# An s3:// URL also needs s3.access_key_id and s3.secret_access_key below +# A file:// path must be absolute, and the directory must already exist: +# vaultik never creates it +# An rclone:// URL needs the remote to be defined in rclone's own config +# (rclone config) storage_url: "rclone://myremote/path/to/backups" # S3-compatible storage configuration @@ -283,6 +303,8 @@ storage_url: "rclone://myremote/path/to/backups" # #region: us-east-1 # # # Use SSL/TLS for S3 connections +# # Only read when storage_url is not set and the endpoint has no scheme; +# # with an s3:// storage_url, use ?ssl=false in the URL instead # # Default: true # #use_ssl: true # @@ -295,6 +317,7 @@ storage_url: "rclone://myremote/path/to/backups" # Path to local SQLite index database # This database tracks file state for incremental backups +# The VAULTIK_INDEX_PATH environment variable overrides this setting # Default: the platform data directory, e.g. # macOS: ~/Library/Application Support/vaultik/index.sqlite # Linux: ~/.local/share/vaultik/index.sqlite diff --git a/docs/DATAMODEL.md b/docs/DATAMODEL.md index 3b197e3..efbbd74 100644 --- a/docs/DATAMODEL.md +++ b/docs/DATAMODEL.md @@ -1,308 +1,418 @@ -# Vaultik Data Model +# vaultik data model -## Overview +`vaultik` keeps a SQLite database on the backed-up machine, called the local +index. This document describes its tables and how each command uses them. -Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication. +The local index has two jobs: -**Important Notes:** +- It records which files, chunks and blobs are already stored. The next backup + then does not store them again. +- It is the source of each snapshot's own database. A backup uploads a trimmed, + encrypted copy of the index with every snapshot. -- **Version Compatibility**: In rare cases, you may need to use the same version - of Vaultik to restore a backup as was used to create it. This ensures - compatibility with the metadata format stored in S3. +A restore does not need the local index. It reads the copy that was uploaded +with the snapshot. -## Schema Migrations +## Terms -Vaultik supports schema migrations. They are the numbered SQL files in -`internal/database/schema/`, embedded in the binary: `000.sql` creates the -`schema_migrations` table, which records each migration that has run, and -`001.sql` creates the application tables. `database.New` opens a database and -applies, in order, every migration that database has not yet recorded. +- **Snapshot**: one backup of the paths listed under one name in the config. +- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that + content. +- **Blob**: one object at the destination. It holds many chunks, compressed and + then encrypted. +- **Destination**: the place the backups are stored. +- **Manifest**: the unencrypted list of the blobs one snapshot uses. +- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory + at the destination. -**Before 1.0** no migrations are added, because nothing is installed anywhere -yet. A schema change edits `001.sql` (and the code that uses the affected -tables) directly. A local database created before the change has already -recorded `001.sql` as run, so it keeps the old schema and can become unusable; -`vaultik database delete` followed by a full backup rebuilds it. - -**After 1.0** each schema change is a new numbered file, so an existing local -database is migrated the first time an updated vaultik opens it. A file that -has shipped in a release is never edited. - -## Database Tables - -### 1. `files` -Stores metadata about files in the filesystem being backed up. - -**Columns:** -- `id` (TEXT PRIMARY KEY) - UUID for the file record -- `path` (TEXT NOT NULL UNIQUE) - Absolute file path -- `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix epoch -- `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to 999999999 -- `size` (INTEGER NOT NULL) - File size in bytes -- `mode` (INTEGER NOT NULL) - Unix file permissions and type -- `uid` (INTEGER NOT NULL) - User ID of file owner -- `gid` (INTEGER NOT NULL) - Group ID of file owner -- `link_target` (TEXT) - Symlink target path (NULL for regular files) - -**Indexes:** -- `idx_files_path` on `path` for efficient lookups - -**Purpose:** Tracks file metadata to detect changes between backup runs. Used for incremental backup decisions. The UUID primary key provides stable references that don't change if files are moved. - -### 2. `chunks` -Stores information about content-defined chunks created from files. - -**Columns:** -- `chunk_hash` (TEXT PRIMARY KEY) - SHA256 hash of chunk content -- `size` (INTEGER NOT NULL) - Chunk size in bytes +## Where the index is -**Purpose:** Enables deduplication by tracking unique chunks across all files. - -### 3. `file_chunks` -Maps files to their constituent chunks in order. +The index is the file `index.sqlite` in the platform data directory. That is +`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/` +on macOS. The `index_path` setting and the `VAULTIK_INDEX_PATH` environment +variable change the path. -**Columns:** -- `file_id` (TEXT) - File ID (FK to files.id) -- `idx` (INTEGER) - Chunk index within file (0-based) -- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash) -- PRIMARY KEY (`file_id`, `idx`) +The file is readable only by its owner, because it lists every backed-up path. +`vaultik` sets that mode each time it opens the index. -**Purpose:** Allows reconstruction of files from chunks during restore. +## Schema migrations -### 4. `chunk_files` -Reverse mapping showing which files contain each chunk. - -**Columns:** -- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash) -- `file_id` (TEXT) - File ID (FK to files.id) -- `file_offset` (INTEGER) - Byte offset of chunk within file -- `length` (INTEGER) - Length of chunk in bytes -- PRIMARY KEY (`chunk_hash`, `file_id`) +`vaultik` supports schema migrations. They are the numbered SQL files in +`internal/database/schema/`, which are built into the binary. -**Purpose:** Supports efficient queries for chunk usage and deduplication statistics. +- `000.sql` creates the `schema_migrations` table. That table records each + migration that has run. +- `001.sql` creates every other table. -### 5. `blobs` -Stores information about packed, compressed, and encrypted blob files. +`database.New` opens the index. It then applies, in order, every migration that +the index has not recorded. -**Columns:** -- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts -- `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (NULL until finalized); see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs) -- `created_ts` (INTEGER NOT NULL) - Creation timestamp -- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress) -- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression -- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage: when the blob was already stored, under another compression level or other recipients, this can differ from the size of the local copy -- `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded) +Before version 1.0, no migration is added, because nothing is installed anywhere +yet. A schema change edits `001.sql` directly, with the code that uses the +changed tables. -**Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs. +This has one consequence. An index created before such a change has already +recorded `001.sql` as run, so it keeps the old schema and can become unusable. +To rebuild it, run `vaultik database delete` and then a full backup. -### 6. `blob_chunks` -Maps chunks to the blobs that contain them. - -**Columns:** -- `blob_id` (TEXT) - Blob ID (FK to blobs.id) -- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash) -- `offset` (INTEGER) - Byte offset of chunk within blob (before compression) -- `length` (INTEGER) - Length of chunk in bytes -- PRIMARY KEY (`blob_id`, `chunk_hash`) - -**Purpose:** Enables chunk retrieval from blobs during restore operations. - -### 7. `snapshots` -Tracks backup snapshots. - -**Columns:** -- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g. `server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the snapshot name, and an RFC 3339 UTC timestamp) -- `hostname` (TEXT) - Hostname where backup was created -- `vaultik_version` (TEXT) - Version of Vaultik used -- `vaultik_git_revision` (TEXT) - Git revision of Vaultik used -- `started_at` (INTEGER) - Start timestamp -- `completed_at` (INTEGER) - Completion timestamp (NULL if in progress) -- `file_count` (INTEGER) - Number of files in snapshot -- `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not stored before -- `new_chunk_size` (INTEGER) - Total size of those chunks before compression; `snapshot list` shows it as NEW CHUNK SIZE -- `blob_count` (INTEGER) - Number of blobs this snapshot created -- `total_size` (INTEGER) - Total size of all files -- `uncompressed_chunk_size` (INTEGER) - Total size before compression of the distinct chunks the snapshot's files reference, recorded when its backup ran; `snapshot list` shows it as UNCOMPRESSED SIZE -- `blob_size` (INTEGER) - Total compressed size of all referenced blobs -- `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced blobs -- `compression_ratio` (REAL) - Compression ratio achieved -- `compression_level` (INTEGER) - Compression level used for this snapshot -- `upload_bytes` (INTEGER) - Total bytes uploaded during this snapshot -- `upload_duration_ms` (INTEGER) - Total milliseconds spent uploading to S3 - -**Purpose:** Provides snapshot metadata and statistics including version tracking for compatibility. - -### 8. `snapshot_files` -Maps snapshots to the files they contain. - -**Columns:** -- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) -- `file_id` (TEXT) - File ID (FK to files.id) -- PRIMARY KEY (`snapshot_id`, `file_id`) - -**Purpose:** Records which files are included in each snapshot. - -### 9. `snapshot_blobs` -Maps snapshots to the blobs they reference. - -**Columns:** -- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) -- `blob_id` (TEXT) - Blob ID (FK to blobs.id) -- `blob_hash` (TEXT) - Denormalized blob hash for manifest generation -- PRIMARY KEY (`snapshot_id`, `blob_id`) - -**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation. - -### 10. `snapshot_mount_points` -The directories at or below a configured path, the configured path itself included, that a snapshot's backup found an entry in and, once it had listed them, found on a different filesystem from their parent directory (a different device number), as a mount point is. An automount point whose volume cannot be mounted is not recorded, because the backup finds no entry in it. - -**Columns:** -- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) -- `path` (TEXT) - Absolute path of the directory -- PRIMARY KEY (`snapshot_id`, `path`) - -**Purpose:** Lets `snapshot create --prune` skip the prune when a backup does not find a mount point that an earlier snapshot of the same name found, as when the volume is not mounted. - -### 11. `uploads` -Tracks blob upload metrics. - -**Columns:** -- `blob_hash` (TEXT PRIMARY KEY) - Hash of uploaded blob -- `snapshot_id` (TEXT NOT NULL) - The snapshot that triggered this upload (FK to snapshots.id) -- `uploaded_at` (INTEGER) - Upload timestamp -- `size` (INTEGER) - Size of uploaded blob -- `duration_ms` (INTEGER) - Upload duration in milliseconds - -**Purpose:** Performance monitoring and tracking which blobs were newly created (uploaded) during each snapshot. - -## Data Flow and Operations - -### 1. Backup Process - -1. **File Scanning** - - `INSERT OR REPLACE INTO files` - Update file metadata - - `SELECT * FROM files WHERE path = ?` - Check if file has changed - - `INSERT INTO snapshot_files` - Add file to current snapshot - - `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a different filesystem from its parent that the walk found an entry in - -2. **Chunking** (for changed files) - - `INSERT OR IGNORE INTO chunks` - Store new chunks - - `INSERT INTO file_chunks` - Map chunks to file - - `INSERT INTO chunk_files` - Create reverse mapping - -3. **Blob Packing** - - `INSERT INTO blobs` - Create blob record with UUID (blob_hash NULL) - - `INSERT INTO blob_chunks` - Associate chunks with blob immediately - - `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing - -4. **Upload** - - `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (the upload is skipped when the blob is already stored) - - `INSERT INTO uploads` - Record upload metrics with snapshot_id - - `INSERT INTO snapshot_blobs` - Associate blob with snapshot - -5. **Snapshot Completion** - - `UPDATE snapshots SET completed_at = ?, stats...` - Finalize snapshot - - Generate and upload blob manifest from `snapshot_blobs` - -### 2. Incremental Backup - -1. **Change Detection** - - `SELECT * FROM files WHERE path = ?` - Get previous file metadata - - Compare mtime, size, mode to detect changes - - Re-chunk a file that lists a chunk no uploaded blob holds, even when its metadata is unchanged - - Skip unchanged files but still add to `snapshot_files` - -2. **Chunk Reuse** - - `SELECT * FROM blob_chunks WHERE chunk_hash = ?` - Find existing chunks - - `INSERT INTO snapshot_blobs` - Reference existing blobs for unchanged files - -### 3. Snapshot Metadata Export - -After a snapshot is completed: -1. Copy database to temporary file -2. Clean temporary database to contain only current snapshot data -3. VACUUM the trimmed database so deleted rows leave no pages behind -4. Compress with zstd and encrypt with age -5. Upload to S3 as `metadata/{remote-key}/db.zst.age` -6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst` +After version 1.0, each schema change is a new numbered file. An existing index +is then migrated the first time an updated `vaultik` opens it. A file that was +part of a release is never edited. -The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext. The hash uses no secret, so a guessed hostname and snapshot name can still be confirmed against a listing; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its [Accepted Risks](REPOSTRUCTURE.md#accepted-risks). +Migrations apply to the local index only. A snapshot's database at the +destination is read as it was written. Each snapshot records the `vaultik` +version that wrote it. -### 4. Restore Process +## Tables -The restore process doesn't use the local database. Instead: -1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from S3 -2. Downloads the blobs holding the chunks of the files being restored, found through that database's `blob_chunks` table; the manifest is not read -3. Reconstructs files from decrypted and decompressed chunks +Times are whole seconds since the Unix epoch, in UTC. The two exceptions are +noted at their columns. -### 5. Pruning +### `files` -1. **Identify Unreferenced Blobs** - - Query blobs not referenced by any remaining snapshot - - Delete from S3, once older than `prune_grace_period`, and from the local - database -2. **Delete Metadata Directories With No Manifest** - - Delete each directory under `metadata/` that has no `manifest.json.zst` - and no local snapshot record, once every object in it is older than - `prune_grace_period` +One row for each backed-up path: a regular file, a symlink or a directory. -### 6. Incomplete Snapshot Cleanup +- `id`: a UUID. Primary key. +- `path`: the absolute path. Unique. +- `source_path`: the configured path that the scan found this entry under. A + restore does not read it. +- `mtime`: the modification time. +- `mtime_nsec`: the nanoseconds within that second. +- `size`: the size in bytes. +- `mode`: the permission bits and the file type. +- `uid`: the owner's user ID. +- `gid`: the owner's group ID. +- `link_target`: the target of a symlink. It is `NULL` for other entries. -Before each backup: -1. Query incomplete snapshots (where `completed_at IS NULL`) -2. Delete each one and all its associations, without checking S3 for its metadata -3. Clean up orphaned files, chunks, and blobs +`idx_files_path` indexes `path`. -## Repository Pattern +There is one row for each path, and it is updated when the file changes. The +row is shared by every snapshot in the local index that lists the path. An +older snapshot's own view of the file is in that snapshot's database at the +destination. -Vaultik uses a repository pattern for database access: - -- `FileRepository` - CRUD operations for files and file metadata -- `ChunkRepository` - CRUD operations for content chunks -- `FileChunkRepository` - Manage file-to-chunk mappings -- `ChunkFileRepository` - Manage chunk-to-file reverse mappings -- `BlobRepository` - Manage blob lifecycle (creation, finalization, upload) -- `BlobChunkRepository` - Manage blob-to-chunk associations -- `SnapshotRepository` - Manage snapshots and their relationships -- `UploadRepository` - Track blob upload metrics +### `chunks` -Each repository provides methods like: -- `Create()` - Insert new record -- `GetByID()` / `GetByPath()` / `GetByHash()` - Retrieve records -- `Update()` - Update existing records -- `Delete()` - Remove records -- Specialized queries for each entity type (e.g., `DeleteOrphaned()`, `GetIncompleteByHostname()`) +One row for each distinct chunk. -## Transaction Management +- `chunk_hash`: the SHA-256 hash of the chunk's content, in hex. Primary key. +- `size`: the size in bytes. -All database operations that modify multiple tables are wrapped in transactions: +### `file_chunks` + +The chunks that make up each file, in order. + +- `file_id`: the file. References `files.id`. +- `idx`: the position of the chunk within the file, counted from 0. +- `chunk_hash`: the chunk. References `chunks.chunk_hash`. + +The primary key is (`file_id`, `idx`). `idx_file_chunks_chunk_hash` indexes +`chunk_hash`. + +### `chunk_files` + +The reverse of `file_chunks`: the files that contain each chunk. + +- `chunk_hash`: the chunk. References `chunks.chunk_hash`. +- `file_id`: the file. References `files.id`. +- `file_offset`: the byte offset of the chunk within the file. +- `length`: the length of the chunk in bytes. + +The primary key is (`chunk_hash`, `file_id`). `idx_chunk_files_file_id` indexes +`file_id`. + +### `blobs` + +One row for each blob. + +- `id`: a UUID, given when packing of the blob starts. Primary key. +- `blob_hash`: the name of the blob at the destination. Unique. Until the blob + is closed, it holds a placeholder that starts with `temp-placeholder-`. +- `created_ts`: when packing started. +- `finished_ts`: when the blob was closed. It is `NULL` before that. +- `uncompressed_size`: the total size of the blob's chunks. +- `compressed_size`: the size after compression and encryption. +- `uploaded_ts`: when the upload finished. It is `NULL` before that. + +`blob_hash` is `hex(SHA256(SHA256(uncompressed contents)))`. See +[REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs). + +Once a blob is uploaded, `compressed_size` is the size of the object at the +destination. A blob that was already there can have a different size from the +local copy. That happens when it was written with another compression level or +other recipients. + +### `blob_chunks` + +The chunks that each blob holds. + +- `blob_id`: the blob. References `blobs.id`. +- `chunk_hash`: the chunk. References `chunks.chunk_hash`. +- `offset`: the byte offset of the chunk within the uncompressed blob. +- `length`: the length of the chunk in bytes. + +The primary key is (`blob_id`, `chunk_hash`). `idx_blob_chunks_chunk_hash` +indexes `chunk_hash`. + +A restore finds the blob for each chunk through this table. + +### `snapshots` + +One row for each snapshot. + +- `id`: the snapshot ID. Primary key. +- `hostname`: the hostname of the machine that made the backup. +- `vaultik_version`: the `vaultik` version that made it. +- `vaultik_git_revision`: the git commit of that version. +- `started_at`: when the backup started. +- `completed_at`: when the backup finished. It is `NULL` until then. +- `file_count`: the number of files and symlinks in the snapshot. Directories + are not counted. +- `chunk_count`: the number of chunks this snapshot stored that were not stored + before. +- `new_chunk_size`: the total size of those chunks before compression. +- `blob_count`: the number of blobs this snapshot created. +- `total_size`: the total size of all files in the snapshot. +- `uncompressed_chunk_size`: the total size, before compression, of the + distinct chunks the snapshot's files use. +- `blob_size`: the total compressed size of the blobs the snapshot uses. +- `blob_uncompressed_size`: the total uncompressed size of those blobs. +- `compression_ratio`: the compression ratio of those blobs. +- `compression_level`: the compression level the backup used. +- `upload_bytes`: the bytes uploaded during the backup. +- `upload_duration_ms`: the time spent uploading, in milliseconds. + +The snapshot ID has the form `__`. An +example is `server1_home_2025-06-01T12:00:00Z`. It has three parts: + +- the hostname, up to its first `.` +- the snapshot name +- the start time, in UTC + +`snapshot list` shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE` and +`new_chunk_size` as `NEW CHUNK SIZE`. + +A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list` +does not show it. The next backup deletes it, and so does `prune`. + +### `snapshot_files` + +The files in each snapshot. + +- `snapshot_id`: the snapshot. References `snapshots.id`. +- `file_id`: the file. References `files.id`. + +The primary key is (`snapshot_id`, `file_id`). `idx_snapshot_files_file_id` +indexes `file_id`. + +### `snapshot_blobs` + +The blobs that each snapshot uses. + +- `snapshot_id`: the snapshot. References `snapshots.id`. +- `blob_id`: the blob. References `blobs.id`. +- `blob_hash`: a copy of the blob's hash. + +The primary key is (`snapshot_id`, `blob_id`). `idx_snapshot_blobs_blob_id` +indexes `blob_id`. + +The manifest is built from this table. + +### `snapshot_mount_points` + +The mount points that a snapshot's backup found. + +- `snapshot_id`: the snapshot. References `snapshots.id`. +- `path`: the absolute path of the directory. + +The primary key is (`snapshot_id`, `path`). + +A directory is recorded here when all of these hold: + +- It is at or below a configured path. +- The backup found an entry in it that is not excluded. +- It is on a different filesystem from its parent directory. + +The device number is read after the directory has been listed. Listing an +automount point mounts its volume, so the number is that of the volume. An +automount point whose volume cannot be mounted is not recorded, because the +backup finds no entry in it. + +`snapshot create --prune` reads this table. It skips the prune when an earlier +snapshot of the same name recorded a mount point that the new backup did not +find. That happens when the volume is not mounted. + +### `uploads` + +One row for each blob that a backup uploaded. + +- `blob_hash`: the blob. References `blobs.blob_hash`. Primary key. +- `snapshot_id`: the snapshot whose backup uploaded it. References + `snapshots.id`. +- `uploaded_at`: when the upload started. It is written as a date and time, not + as seconds. +- `size`: the size of the uploaded blob. +- `duration_ms`: the duration of the upload, in milliseconds. + +`idx_uploads_snapshot_id` indexes `snapshot_id`. + +A blob that was already at the destination gets no row here. + +### `local_meta` + +Settings that belong to this copy of the index. + +- `key`: the name of the setting. Primary key. +- `value`: its value. + +The only key is `storage_url`. It holds the destination that the index was +first used with. A command that changes data refuses to run when the config +names a different destination. + +### `schema_migrations` + +The migrations that have run. + +- `version`: the number of the migration. Primary key. +- `applied_at`: when it ran. It is written as a date and time, not as seconds. + +## How commands use the tables + +### Backup + +A backup works on the tables in this order: + +- It deletes unfinished snapshots, and the rows that no snapshot uses. +- It creates a row in `snapshots`, with `completed_at` set to `NULL`. +- It deletes the `blobs` rows that were never uploaded. +- It walks each path and compares every entry with its row in `files`. +- It links each unchanged file to the snapshot in `snapshot_files`. +- It records each mount point in `snapshot_mount_points`. +- It reads each changed file and stores it. See below. +- It fills `snapshot_blobs` with every blob the snapshot uses. +- It stores the snapshot's totals in `snapshots`. +- It uploads the snapshot's metadata. +- It sets `completed_at`. + +A file counts as changed when its row in `files` differs from the file on disk +in any of these: + +- size +- modification time +- mode +- owner or group +- link target + +A file is also read again when it lists a chunk that no uploaded blob holds. + +Storing a changed file writes these rows: + +- A row in `blobs`, when packing of a blob starts. +- When the blob is closed, in one transaction: + - a row in `chunks` for each new chunk in the blob + - the blob's rows in `blob_chunks` + - the blob's `blob_hash`, sizes and `finished_ts` +- When the blob is stored at the destination, in one transaction: + - `uploaded_ts` and `compressed_size` in `blobs` + - a row in `snapshot_blobs` + - a row in `uploads`, unless the blob was already at the destination +- The row in `files`, with its `file_chunks`, `chunk_files` and + `snapshot_files` rows. + +The row in `files` is written last. It is written only after every blob that +holds the file's new chunks is stored. + +### Snapshot metadata + +The metadata of a snapshot is made from a copy of the index: + +- The index is copied into a temporary directory with `VACUUM INTO`. +- The rows of every other snapshot are deleted from the copy. So are the files + and blobs that this snapshot does not use. A chunk is then deleted when no + remaining file and no remaining blob uses it. +- The `local_meta` and `schema_migrations` tables are kept as they are. The + uploaded database therefore names the destination, and lists the migrations + that have run. +- `VACUUM` rewrites the copy, so that no deleted row is left in the file. +- The copy is compressed with zstd and encrypted with age. +- The manifest is built from the copy's `snapshot_blobs` table. +- The copy is uploaded as `metadata//db.zst.age`. +- The manifest is uploaded as `metadata//manifest.json.zst`. + +The copy is made through SQLite, not by copying the file. The index runs in WAL +mode, so rows that are already committed can still be in the `-wal` file beside +it. + +### Restore + +A restore does not use the local index. + +- It downloads and decrypts `db.zst.age` into a temporary directory. +- It opens that file read-only. It applies no migration to it. +- It reads the files to restore from `files`. +- It finds each file's chunks in `file_chunks`. +- It finds the blob and offset of each chunk in `blob_chunks`. +- It finds the name of each blob in `blobs`. + +It does not read the manifest. + +### Prune + +`prune` changes the local index in two ways: + +- It deletes each snapshot whose manifest is gone from the destination. +- It deletes unfinished snapshots, and the rows that no snapshot uses. + +The second step removes unused rows in this order: + +- files that are in no snapshot +- blobs that are in no snapshot +- `blob_chunks` rows whose chunk is gone +- chunks that no file and no blob uses + +`prune` then deletes unused data at the destination. That step reads the +manifests, not the local index. It reads the local index only to keep a +directory under `metadata/` whose snapshot is still recorded there. + +## Code + +Each repository type in `internal/database` serves one or more tables: + +- `FileRepository`: `files` +- `ChunkRepository`: `chunks` +- `FileChunkRepository`: `file_chunks` +- `ChunkFileRepository`: `chunk_files` +- `BlobRepository`: `blobs` +- `BlobChunkRepository`: `blob_chunks` +- `SnapshotRepository`: + - `snapshots` + - `snapshot_files` + - `snapshot_blobs` + - `snapshot_mount_points` +- `UploadRepository`: `uploads` +- `LocalMetaRepository`: `local_meta` + +No repository type serves `schema_migrations`. The code that applies the +migrations reads and writes it. + +`Repositories` holds one of each type. + +`Repositories.WithTx` runs a function in one transaction: ```go err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error { - // Multiple repository operations using tx + // several repository calls that use tx }) ``` -This ensures consistency, especially important for operations like: -- Creating file-chunk mappings -- Associating chunks with blobs -- Updating snapshot statistics +The transaction commits when the function returns no error. Otherwise it is +rolled back. -## Performance Considerations +## SQLite settings -1. **Indexes**: - - Primary keys are automatically indexed - - `idx_files_path` on `files(path)` for efficient file lookups - -2. **Prepared Statements**: All queries use prepared statements for performance and security - -3. **Batch Operations**: Where possible, operations are batched within transactions - -4. **Write-Ahead Logging**: The local index runs in SQLite WAL mode with a 10-second busy timeout, so a read-only command such as `snapshot list` can read it while a backup writes to it. Committed rows can sit in the `-wal` file beside the index until a checkpoint, so the metadata export copies the index through SQLite (`VACUUM INTO`), not as a file - -## Data Integrity - -1. **Foreign Keys**: Enforced through CASCADE DELETE and application-level repository methods -2. **Unique Constraints**: Chunk hashes, file paths, and blob hashes are unique -3. **Null Handling**: Nullable fields clearly indicate in-progress operations -4. **Timestamp Tracking**: All major operations record timestamps for auditing \ No newline at end of file +- The index runs in WAL mode. A command that only reads, such as + `snapshot list`, can read the index while a backup writes to it. +- A connection waits up to 10 seconds for a lock before it fails. +- Foreign keys are enforced. +- Each open database uses one connection, so its queries run one at a time. +- Queries pass their values as parameters. diff --git a/docs/REPOSTRUCTURE.md b/docs/REPOSTRUCTURE.md index d2f003a..74db889 100644 --- a/docs/REPOSTRUCTURE.md +++ b/docs/REPOSTRUCTURE.md @@ -1,184 +1,416 @@ -# Vaultik S3 Repository Structure +# vaultik destination layout -This document describes the structure and organization of data stored in the S3 bucket by Vaultik. +This document describes what `vaultik` stores at the destination. It covers the +layout, what the layout protects, and the risks it accepts. -## Overview +The destination is the place the backups are stored. It is an S3 bucket and +prefix, a local directory or an rclone remote. The layout is the same on all +three. -Vaultik stores all backup data in an S3-compatible object store. The repository consists of two main components, plus a marker file: -1. **Blobs** - The actual backup data (content-addressed, encrypted) -2. **Metadata** - Snapshot information and manifests (partially encrypted) +## Terms -## Directory Structure +- **Snapshot**: one backup of the paths listed under one name in the config. +- **Snapshot ID**: the name of a snapshot, in the form + `__`. +- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that + content. +- **Blob**: one object under `blobs/`. It holds many chunks. +- **Local index**: the SQLite database on the backed-up machine. +- **Recipient**: an age public key that backups are encrypted to. + +## Layout ``` -// +/ ├── vaultik-destination.txt ├── blobs/ │ └── / │ └── / -│ └── +│ └── └── metadata/ └── / ├── db.zst.age └── manifest.json.zst ``` -The metadata subdirectory is named with the **remote key**, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See [Remote Key Derivation](#remote-key-derivation). +The destination holds three things: -## Marker File (`vaultik-destination.txt`) +- `vaultik-destination.txt`, the marker file +- `blobs/`, which holds the file contents +- `metadata/`, which holds one directory for each snapshot -Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. While the local index holds snapshots stored there, a destination that answers a look-up of the marker with not found is treated as not present: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last. +## The marker file -When the look-up of the marker fails for any reason other than not found, the check is skipped and the command goes on. S3 refuses it on every run to credentials that allow only PUT, so a host with such credentials does not notice a destination emptied since its last backup, and its next backup would leave out the data its local index records as already stored. After the destination is emptied, run `vaultik database delete` on such a host before it backs up again. +Every backup writes `vaultik-destination.txt` before it uploads anything else. +The file holds one fixed line of text. It names no host, path or time. -## Blobs Directory (`blobs/`) +The marker tells an emptied or missing destination apart from a new one. An +unplugged volume leaves an empty directory at its mount point. Without the +marker, that directory would look like a destination whose snapshots are all +gone. -### Structure -- **Path format**: `blobs///` -- **Example**: `blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678` -- **Sharding**: The two-level directory structure (using the first 4 characters of the hash) prevents any single directory from containing too many objects +The check applies while the local index holds snapshots. When the destination +then answers that the marker does not exist, commands treat the destination as +not present: -### Content -- **What it contains**: Packed collections of content-defined chunks from files -- **Format**: Zstandard compressed, then Age encrypted -- **Encryption**: Always encrypted with Age using the configured recipients -- **Naming**: Content-addressed. The blob's name is `hex(SHA256(SHA256(uncompressed blob contents)))` — the double SHA-256 of the concatenated chunk data, computed before compression and encryption, not a hash of the stored (compressed, encrypted) bytes. One consequence: only a holder of the age private key can check a stored blob's integrity, because matching a blob to its name means decrypting and decompressing it first — which is what `restore` and `verify --deep` do. Implemented in `internal/blobgen` (`DoubleSHA256`). This is the canonical description of blob naming; other documents and comments point here. +- `snapshot create` refuses to write to it. +- `prune`, `snapshot purge` and `remote nuke` fail. +- `snapshot list` shows the local index alone. +- `snapshot remove` deletes nothing at the destination. -### Why Encrypted -Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy. +The check is skipped when the look-up fails for any other reason. S3 refuses +the look-up to credentials that can only write. A machine with such credentials +therefore does not notice an emptied destination. Its next backup would not +upload data that its local index says is stored. After a destination is emptied, +run `vaultik database delete` on such a machine before it backs up again. -## Metadata Directory (`metadata/`) +`remote nuke` deletes the marker last. -Each snapshot has its own subdirectory. The directory is **not** named with the human-readable snapshot ID; it is named with the remote key — a one-way hash of that ID. The human ID is never written to the destination store as a directory name (see [Remote Key Derivation](#remote-key-derivation)). +## Blobs -### Snapshot ID Format +A blob holds the content of many chunks, one after another. -The human-readable snapshot ID is used in CLI arguments, log lines, and the local database. It is not written to the destination store. -- **Format**: `__` (or `_` if no - name was specified) -- **Example**: `laptop_home_2024-01-15T14:30:52Z` -- **Components**: - - Short hostname (everything before the first dot is stripped from the FQDN) - - Snapshot name from the configured `snapshots:` map (optional) - - RFC3339 UTC timestamp +- **Path**: `blobs///`. +- **Example**: + `blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`. +- **Format**: the chunk data, compressed with zstd and then encrypted with age. +- **Recipients**: the public keys in `age_recipients` when the blob was written. -This ID reveals the hostname, the configured snapshot name, and the backup time, so it is never used as the on-disk directory name — the remote key is used instead. +The two directory levels keep any one directory from holding too many objects. -### Remote Key Derivation +### Blob names -The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string. The hash is not reversible, but it uses no secret: an observer who guesses a candidate hostname and snapshot name can hash it the same way and confirm whether that snapshot is present. The remote key keeps names out of a plain listing; it does not hide them from a guess. Implemented in `internal/snapshot/remotekey.go`. +A blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. The contents are +the chunk data, joined in order, before compression and encryption. The name is +not a hash of the stored bytes. + +This has two consequences: + +- Checking a blob against its name needs the private key. The blob must be + decrypted and decompressed first. `snapshot restore` and + `snapshot verify --deep` do this. +- Without the key, only a blob's presence and size can be checked. + `snapshot verify` does this. + +`blobgen.DoubleSHA256` in `internal/blobgen` computes the name. + +### Uploads + +A backup looks a blob up by name before it uploads it. It does not upload a +blob that is already there. + +When the destination refuses the look-up, the backup uploads the blob. An +object with that name is then replaced by one that holds the same chunk data. + +## Metadata + +Each snapshot has one directory under `metadata/`. The directory is named with +the snapshot's remote key, not with its snapshot ID. + +### Snapshot IDs + +A snapshot ID has the form `__`. An example +is `laptop_home_2024-01-15T14:30:52Z`. It has three parts: + +- the hostname, up to its first `.` +- the snapshot name from the config +- the time the backup started, in UTC + +The snapshot ID is used on the command line, in log records and in the local +index. It reveals the hostname, the snapshot name and the backup time. It is +therefore never written to the destination without encryption. + +### Remote keys + +The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot ID)))`. It is 64 hex +characters long. `internal/snapshot/remotekey.go` computes it. + +For example: -Worked example: - Snapshot ID: `server1_home_2025-06-01T12:00:00Z` - Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa` -- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/` +- Directory: + `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/` -A plain listing of the destination store therefore shows only these hashes, not the hostname or snapshot name of any backup — but because the hash uses no secret, a guessed hostname and snapshot name can be hashed and confirmed against the listing. The same remote key is stored in the manifest's `snapshot_id` field. +A listing of the destination therefore shows no hostname and no snapshot name. -### Files in Each Snapshot Directory +The hash cannot be reversed, but it uses no secret. Someone who guesses a +snapshot ID can hash it the same way and see whether that snapshot exists. The +remote key keeps names out of a listing. It does not hide them from a guess. -#### `db.zst.age` - Encrypted Database -- **What it contains**: Pruned binary SQLite database for this snapshot -- **Format**: Binary SQLite → Zstandard compressed → Age encrypted -- **Encryption**: Encrypted with Age -- **Purpose**: Contains full file metadata, chunk mappings, and all relationships -- **Why encrypted**: Contains sensitive metadata like file paths, permissions, and ownership +### `db.zst.age` + +The snapshot's database. A restore needs it. + +- **Contents**: a SQLite database. It is a copy of the local index without the + rows of other snapshots. It lists every file in the snapshot with its + metadata, its chunks, and the blob that holds each chunk. +- **Format**: the database file, compressed with zstd and then encrypted with + age. + +It is encrypted because it holds file paths, permissions and owners. +[DATAMODEL.md](DATAMODEL.md) describes its tables. + +### `manifest.json.zst` + +The list of the blobs the snapshot uses. + +- **Format**: JSON, compressed with zstd. It is not encrypted. -#### `manifest.json.zst` - Unencrypted Blob Manifest -- **What it contains**: JSON list of all blob hashes referenced by this snapshot -- **Format**: JSON → Zstandard compressed (NOT encrypted) -- **Encryption**: NOT encrypted -- **Purpose**: Enables pruning operations without requiring decryption keys -- **Structure**: ```json { - "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa", - "timestamp": "2025-06-01T12:00:00Z", - "blob_count": 42, - "total_compressed_size": 1048576, - "blobs": [ - { "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 24576 }, - { "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 32768 } - ] + "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa", + "timestamp": "2025-06-01T12:04:31Z", + "blob_count": 2, + "total_compressed_size": 57344, + "blobs": [ + { + "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", + "compressed_size": 24576 + }, + { + "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", + "compressed_size": 32768 + } + ] } ``` -`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is written in the clear. -### Why Manifest is Unencrypted -The manifest must be readable without the private key to enable: -1. **Pruning operations** - Identifying unreferenced blobs for deletion -2. **Storage analysis** - Understanding space usage without decryption -3. **Verification** - Checking blob existence without decryption -4. **Cross-snapshot deduplication analysis** - Finding shared blobs between snapshots +- `snapshot_id`: the remote key, not the snapshot ID. +- `timestamp`: when the manifest was written, at the end of the backup. +- `blob_count`: the number of blobs. +- `total_compressed_size`: the total size of those blobs, in bytes. +- `blobs`: the hash and the size of each blob. -The manifest contains the remote key, the backup timestamp, the blob count and total compressed size, and each blob's hash and compressed size. It contains no file names, paths, or other decrypted metadata. +The manifest holds no file name and no path. -## Security Considerations +The manifest is not encrypted so that these commands work without the private +key: -### What's Encrypted -- **All file content** (in blobs) -- **All file metadata** (paths, permissions, timestamps, ownership in db.zst.age) -- **File-to-chunk mappings** (in db.zst.age) +- `prune`, which must know which blobs are still in use +- `snapshot verify`, which checks that each blob is present +- `snapshot list` and `remote info`, which report what the destination holds -### What's Not Encrypted -- **The remote key** — directory names and the manifest `snapshot_id`, a one-way hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation)) -- **The backup timestamp** (in manifest.json.zst) -- **Blob hashes and their compressed sizes** (in manifest.json.zst) -- **Blob count and total compressed size per snapshot** (in manifest.json.zst) +## What is protected -### Privacy Implications -From the unencrypted data, an observer of the destination store can determine: -- **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear -- How many blobs each snapshot references, and the total compressed size -- The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns) -- **Whether a guessed hostname and snapshot name are present** — the remote key is an unkeyed hash, so an observer holding candidate names can hash each one and match it against the directory listing. The human ID is never published, so it cannot be read off directly, but it can be confirmed by guessing. +These are encrypted: -Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless. +- file contents, in the blobs +- in `db.zst.age`: + - file names and paths + - permissions and owners + - modification times + - which chunks make up which file -An observer cannot determine: -- The hostname or snapshot name of any backup by reading it off the store — the directory name and the manifest `snapshot_id` are unkeyed hashes of the human ID, so the text is never published (though a guessed name can be confirmed, as above) -- File names or paths -- File contents -- File permissions or ownership -- Directory structure -- Which chunks belong to which files +These are not encrypted: -### Accepted Risks +- the remote key of each snapshot +- the time each backup finished, in the manifest +- the hash and the compressed size of each blob, in the manifest +- the number of blobs and their total size for each snapshot, in the manifest -These are known, deliberate properties of the format and the tooling, recorded so an operator can weigh them rather than discover them. +Someone who can read the destination therefore learns these: -1. **No proof of authorship.** Restore and `verify --deep` prove that data decrypts with the age private key and matches its unkeyed content hashes. They do not prove who wrote it: anyone who knows a recipient public key and can replace objects on the destination can substitute a snapshot they built. The recipient string is not stored at the destination, but a compromised backed-up host has it. Defences live on the destination side — bucket versioning or object lock, credentials for the source host that cannot delete or overwrite existing versions, and pruning from a trusted host. Note that S3 `PutObject` overwrites an existing key, so PUT permission alone is not append-only. -2. **Compression reveals sizes.** Blobs and `db.zst.age` are zstd-compressed then age-encrypted; the manifest is compressed only. age does not pad, so an object's size is the exact compressed length of its contents. All new chunks packed into one blob share a single zstd stream (8 MiB window, 4 MiB at compression levels 1-2), and a blob is closed at `blob_size_limit` and at the end of each configured path. Because a stored chunk is never packed again, someone who can write into a backed-up file and watch blob sizes learns something only when their controlled data and a secret land in the same chunk of a file that keeps changing. Advice: back up any outsider-writable directory as its own snapshot. -3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of 2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them. -4. **Decrypted data on local disk.** Several commands stage plaintext under `$TMPDIR`: `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` and removed on success, but a `kill -9` or a power loss leaves them behind — delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` should be trusted to the same degree as the restore target. -5. **Store permissions differ per command.** The backed-up host needs only PUT to run `snapshot create`: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — `snapshot verify`, `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`, `snapshot remove`, and `remote nuke` also delete. The recommended cron line uses `--prune`, which runs `prune` on the backed-up host, so granting that host `--prune` gives it credentials that can delete its own backups. To keep the source host to PUT only, prune from a separate trusted host instead. -6. **Changing recipients does not re-encrypt existing data.** Deduplicated chunks and same-named blobs already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run `vaultik database delete` and take a full backup to a fresh destination or prefix. -7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients. Long-lived ciphertext held by a third party (the destination operator) has no fallback if X25519 is ever broken: there is no second recipient type and no post-quantum option. +- when each backup finished +- how many blobs each snapshot uses, and how large they are +- which blobs two snapshots share +- whether a snapshot ID they guess exists -## Consistency Guarantees +They do not learn these: -1. **Blobs are immutable** - Once written, a blob is never modified -2. **Blobs are written before metadata** - A snapshot's metadata is only written after all its blobs are successfully uploaded -3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst are written as complete files -4. **A snapshot is marked complete in the local DB only after its metadata is uploaded** - `finalizeSnapshotMetadata` runs `ExportSnapshotMetadata` first and records completion (`MarkSnapshotComplete`) only once the export succeeds (see the backup data flow in [ARCHITECTURE.md](../ARCHITECTURE.md)). A crash during the export therefore leaves the snapshot incomplete, so the next backup's `PruneDatabase` drops it and re-backs-up its data, rather than leaving a completed-looking row in the local index with no matching metadata on the destination store. (A crash in the brief moment after the export succeeds but before completion is recorded leaves a fully-restorable snapshot on the destination that the local index drops as incomplete on the next run; `snapshot list` then reports it honestly as remote-only, which is the safe direction: the destination copy stays restorable.) +- any hostname or snapshot name, unless they guess it +- file names or paths +- file contents +- permissions or owners +- the directory structure +- which chunks belong to which file -## Pruning Safety +The times and sizes could not be hidden by encrypting the manifest. The +destination itself shows when each object was written and how large it is. -The prune operation is safe because: -1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references. Under `metadata/` it deletes only a directory that has no `manifest.json.zst`, whose name is a remote key, and that no local snapshot record claims -2. Manifests are unencrypted and can be read without keys -3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced -4. It deletes an unreferenced blob only once the blob is older than `prune_grace_period` (default `24h`), and a directory under `metadata/` with no manifest only once every object in it is. A backup uploads its blobs and its `db.zst.age` before its manifest, so a prune running meanwhile, on this host or another sharing the destination, leaves them as long as no backup run takes longer than the grace period. The grace period does not protect a blob a backup reuses from an earlier run that stopped before its manifest: the backup skips the upload of a blob already on the destination, so that blob keeps the time of its first upload, and a prune before the backup's manifest is written can delete it; the new snapshot then cannot be restored. `remote nuke` has no such check +## Accepted risks -## Restoration Requirements +These are known properties of the format. They are written down so that you can +weigh them. -To restore from a backup, you need: -1. **The Age private key** - To decrypt blobs and database -2. **The snapshot metadata** - Both files from the snapshot's metadata directory -3. **All referenced blobs** - As listed in the manifest +### No proof of who wrote the data -The restoration process: -1. Download and decrypt the database dump to understand file structure -2. Download and decrypt the required blobs -3. Reconstruct files from their chunks -4. Restore file metadata (permissions, timestamps, etc.) \ No newline at end of file +A restore and `snapshot verify --deep` prove two things. The data decrypts with +your private key, and it matches its hashes. They do not prove who wrote it. + +The hashes use no secret. Someone who knows a recipient's public key and can +replace objects at the destination can put a snapshot of their own there. The +public key is not stored at the destination, but a backed-up machine that is +broken into has it. + +The defences are on the destination's side: + +- object versioning or object lock on the bucket +- credentials for the backed-up machine that cannot delete or overwrite +- running `prune` from a trusted machine + +An S3 `PutObject` replaces an existing object. Permission to write alone +therefore does not stop overwrites. + +### Sizes are visible + +An object's size follows exactly from the size of its compressed contents. + +All the new chunks in one blob share one zstd stream. The stream looks back 8 +MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at +`blob_size_limit` and at the end of each configured path. A chunk that is +already stored is never packed again. + +Data that an outsider writes compresses smaller when it matches a secret in the +same chunk, so the blob gets smaller. The blob's size therefore shows the +outsider whether their data matched the secret. + +This works only when all of these hold: + +- The outsider can write into a backed-up file. +- The outsider can see the blob sizes at the destination. +- The outsider's data and the secret land in the same chunk. +- The file keeps changing. + +Back up a directory that outsiders can write to as a snapshot of its own. + +### Chunk boundaries use no secret + +The FastCDC parameters are fixed and public. `chunk_size` sets the average chunk +size. No chunk is larger than four times this size. Only a file's last chunk can +be smaller than a quarter of it. With the default of 10 MB, those limits are 40 +MB and 2.5 MB. A file of 2.5 MB or less is one chunk. + +At the default `blob_size_limit` of 10 GB, a blob holds hundreds of chunks. The +size of the blob then does not show the size of any one chunk. A limit close to +`chunk_size` begins to show chunk sizes. + +### Decrypted data on the local disk + +These commands write unencrypted data to the temporary directory: + +- `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/`. There + is no limit on their total size. +- `snapshot restore` writes the decrypted database to + `vaultik-restore-*/snapshot.db`. +- `snapshot verify --deep` writes the decrypted database to + `vaultik-verify-*/snapshot.db`. +- `snapshot create` writes a copy of the local index to + `vaultik-snapshot-*/snapshot.db`. + +These files are readable only by their owner, and are removed when the command +ends. A `kill -9` or a power loss leaves them behind. Delete any leftover +`vaultik-*` entry in the temporary directory yourself. + +Trust the temporary directory as much as the directory you restore into. + +### Commands need different permissions + +`snapshot create` needs only permission to write objects. Other commands need +more: + +- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list + and read. +- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete. +- `prune` lists, reads and deletes. + +`snapshot create --prune` runs a prune on the backed-up machine. That machine +then holds credentials that can delete its own backups. To keep it to +write-only credentials, run `prune` from a separate, trusted machine. + +### New recipients cannot read old data + +Changing `age_recipients` does not re-encrypt what is already stored. A chunk +that is already stored is not written again, so it stays encrypted to the +earlier recipients. + +A new snapshot that uses such chunks cannot be restored with a new recipient's +key alone. To make everything readable by a new key, run +`vaultik database delete` and take a full backup to a new destination. + +### X25519 recipients only + +`vaultik` accepts age X25519 public keys only. It rejects age SSH and plugin +recipients. + +The encrypted data is held by a third party for a long time. There is no second +kind of recipient to fall back on if X25519 is broken, and no post-quantum +option. + +## Order of writes + +A backup writes in a fixed order: + +- the marker file +- the blobs +- `db.zst.age` +- `manifest.json.zst` + +A snapshot's metadata is written only after all its blobs are uploaded. The +manifest is written last. A directory under `metadata/` that has no manifest +therefore belongs to a backup that did not finish. + +The local index marks a snapshot complete only after both metadata files are +uploaded. A backup that stops during that upload leaves the snapshot +unfinished. The next backup removes it from the local index and stores its data +again. + +One short gap remains. A backup can stop after the upload and before the +snapshot is marked complete. The destination then holds a snapshot that can be +restored, and the local index does not list it. `snapshot list` shows it as +remote only. + +An upload that is cut off leaves no partial object under the final name on S3 +or on a local directory. The same holds for an rclone remote that can move +objects. An rclone remote that cannot move objects can keep a partial object. + +## Pruning + +`prune` deletes two kinds of object at the destination: + +- a blob that no manifest lists +- a directory under `metadata/` that has no manifest + +It reads the manifests, which are not encrypted, so it needs no private key. + +These rules keep it from deleting data that is in use: + +- It keeps every blob that any manifest lists. +- It deletes nothing when any manifest cannot be downloaded or decoded. +- It deletes a blob only once the blob is older than `prune_grace_period`. +- It deletes a directory only once every object in it is that old. +- It keeps a directory whose snapshot is in the local index. +- It leaves alone a directory whose name is not a remote key. + +`prune_grace_period` defaults to `24h`. It protects a backup that is still +running. Such a backup has uploaded blobs and perhaps `db.zst.age`, but not yet +its manifest. A prune from this machine or another keeps those objects, as long +as no backup takes longer than the grace period. + +The grace period does not protect one case. A backup does not upload a blob +that is already at the destination, so that blob keeps the time of its first +upload. The blob can be left over from an earlier backup that stopped before +its manifest. A prune that runs before the new manifest is written can then +delete that blob. The new snapshot cannot be restored afterwards. + +To avoid this case, give each machine and each user its own destination. + +`remote nuke` applies none of these rules. + +## What a restore needs + +- the age private key +- the snapshot's `db.zst.age` +- the blobs that hold the files to restore + +A restore does not read the manifest and does not need the local index. + +It works in these steps: + +- It downloads and decrypts `db.zst.age`. +- It checks that the database is the one for the snapshot that was asked for. +- It downloads and decrypts the blobs it needs. It checks each blob against its + name. +- It writes each file from its chunks. +- It sets each file's permissions and times, and its owner when it runs as + root. diff --git a/internal/cli/config.go b/internal/cli/config.go index 2a18fc1..b99f731 100644 --- a/internal/cli/config.go +++ b/internal/cli/config.go @@ -289,12 +289,14 @@ func newConfigInitCommand() *cobra.Command { Use: "init", Short: "Write a default config file", Long: `Creates a default configuration file with commented explanations -for every setting. If a config file already exists at the target path, -the command refuses to overwrite it. +for the settings it lists. If a config file already exists at the target +path, the command refuses to overwrite it. The file is created with mode +0600, because it can hold credentials. The config is written to the path from --config, $VAULTIK_CONFIG, or -the platform default config directory (e.g. ~/Library/Application Support/ -on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`, +config.yml in the platform default config directory +(~/Library/Application Support/vaultik/ on macOS, ~/.config/vaultik/ on +Linux, /etc/vaultik/ as root).`, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { path := configPathForInit() @@ -332,7 +334,7 @@ on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`, func newConfigEditCommand() *cobra.Command { return &cobra.Command{ Use: "edit", - Short: "Open the config file in $EDITOR", + Short: "Open the config file in $EDITOR (vi when it is not set)", Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { path, err := ResolveConfigPath() diff --git a/internal/cli/config_test.go b/internal/cli/config_test.go index 7dccfa0..c928501 100644 --- a/internal/cli/config_test.go +++ b/internal/cli/config_test.go @@ -61,8 +61,8 @@ func TestDefaultConfigTemplateParses(t *testing.T) { // testRecipient is a valid age public key. const testRecipient = "age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj" -// TestConfigSetRecipientOnFreshConfig follows the README quickstart: on the -// file `config init` writes, `config set age_recipients.0` and +// TestConfigSetRecipientOnFreshConfig follows the README's getting started +// steps: on the file `config init` writes, `config set age_recipients.0` and // `config set storage_url` give a config that loads with that recipient. // Neither is refused, although the config does not load until both are set; // once it loads, a recipient that is not a valid key is. diff --git a/internal/cli/prune.go b/internal/cli/prune.go index 6b6843a..b74e45b 100644 --- a/internal/cli/prune.go +++ b/internal/cli/prune.go @@ -17,7 +17,11 @@ func NewPruneCommand() *cobra.Command { unreferenced blobs and directories under metadata/ with no manifest from the backup destination store. -Local cleanup drops incomplete snapshots and any files, chunks, or +Needs permission to list, read and delete objects on the destination +store. It does not need the age private key. + +Local cleanup drops snapshots whose manifest is gone from the +destination store, incomplete snapshots, and any files, chunks, or blobs no longer referenced by a completed snapshot. Remote cleanup scans every snapshot manifest in the destination store, builds the set of still-referenced blob hashes, and deletes any blob not in that diff --git a/internal/cli/remote.go b/internal/cli/remote.go index 4bf403f..3dbc486 100644 --- a/internal/cli/remote.go +++ b/internal/cli/remote.go @@ -40,11 +40,13 @@ func newRemoteNukeCommand() *cobra.Command { cmd := &cobra.Command{ Use: "nuke", Short: "Delete ALL snapshot metadata and blobs from the backup destination store", - Long: `Removes every snapshot's metadata and every blob from remote -storage. After this command completes successfully the bucket prefix is -empty and the next backup starts from scratch. + Long: `Removes every snapshot's metadata, every blob and the destination +marker from the backup destination store, and each removed snapshot from +the local index. After this command completes successfully the bucket +prefix is empty and the next backup starts from scratch. -This is destructive and irreversible. Requires --force.`, +This is destructive and irreversible. Requires --force, and permission +to list and delete objects on the destination store.`, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { if !force { @@ -75,7 +77,10 @@ func newRemoteInfoCommand() *cobra.Command { - Size of all snapshot metadata (per snapshot and total) - Count and total size of all blobs - Count and size of referenced blobs (from all manifests) -- Count and size of orphaned blobs (not referenced by any manifest)`, +- Count and size of orphaned blobs (not referenced by any manifest) + +Needs permission to list and read the destination store. It does not +need the age private key.`, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { // Use unified config resolution diff --git a/internal/cli/root.go b/internal/cli/root.go index ba1fcbc..843d3ca 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -35,9 +35,10 @@ func NewRootCommand() *cobra.Command { cmd := &cobra.Command{ Use: "vaultik", Short: "Secure incremental backup tool with asymmetric encryption", - Long: `vaultik is a secure incremental backup tool that encrypts data using age -public keys and uploads to S3-compatible storage. No private keys are needed -on the source system.`, + Long: `vaultik is an incremental backup tool that encrypts data to age public +keys and uploads it to an S3-compatible object store, a local directory or +an rclone remote. The backed-up machine needs no private key: only +'snapshot restore' and 'snapshot verify --deep' need one.`, SilenceUsage: true, // Bare 'vaultik' (no subcommand): print help. The banner is // printed once at process startup by Entry, before cobra @@ -50,13 +51,15 @@ on the source system.`, // Add global flags cmd.PersistentFlags().StringVar(&rootFlags.ConfigPath, "config", "", - "Path to config file (default: $VAULTIK_CONFIG or platform config dir)") + "Path to config file (default: $VAULTIK_CONFIG, then the platform "+ + "config directory, then /etc/vaultik/config.yml)") cmd.PersistentFlags().BoolVarP(&rootFlags.Verbose, "verbose", "v", false, "Enable verbose output") cmd.PersistentFlags().BoolVar(&rootFlags.Debug, "debug", false, "Enable debug output") cmd.PersistentFlags().BoolVarP(&rootFlags.Quiet, "quiet", "q", false, - "Suppress non-error output") + "Suppress status output (warnings, errors, JSON documents, "+ + "'config get' values and confirmation prompts are still printed)") cmd.PersistentFlags().BoolVar(&rootFlags.SkipErrors, "skip-errors", false, "Skip files that cannot be read when creating a snapshot, or "+ "that cannot be restored when restoring, instead of aborting "+ diff --git a/internal/cli/snapshot.go b/internal/cli/snapshot.go index 8996d35..ed4737b 100644 --- a/internal/cli/snapshot.go +++ b/internal/cli/snapshot.go @@ -60,6 +60,21 @@ func NewSnapshotCommand() *cobra.Command { return cmd } +// snapshotCreateLong is the long help text of 'snapshot create'. +const snapshotCreateLong = `Creates new snapshots of the configured directories. + +If snapshot names are provided, only those snapshots are created. +If no names are provided, all configured snapshots are created. + +Needs at least one public key in age_recipients, and permission to write +objects at the backup destination store. It does not need the age private +key. With --prune it also needs permission to list, read and delete objects +there. + +The config is read from the path given by --config or VAULTIK_CONFIG; +otherwise from the platform config directory (~/.config/vaultik/config.yml +on Linux), then /etc/vaultik/config.yml.` + // newSnapshotCreateCommand creates the 'snapshot create' subcommand func newSnapshotCreateCommand() *cobra.Command { opts := &vaultik.SnapshotCreateOptions{} @@ -67,15 +82,8 @@ func newSnapshotCreateCommand() *cobra.Command { cmd := &cobra.Command{ Use: "create [snapshot-names...]", Short: "Create new snapshots", - Long: `Creates new snapshots of the configured directories. - -If snapshot names are provided, only those snapshots are created. -If no names are provided, all configured snapshots are created. - -The config is read from the path given by --config or VAULTIK_CONFIG; -otherwise from the platform config directory (~/.config/vaultik/config.yml -on Linux), then /etc/vaultik/config.yml.`, - Args: cobra.ArbitraryArgs, + Long: snapshotCreateLong, + Args: cobra.ArbitraryArgs, RunE: func(cmd *cobra.Command, args []string) error { err := opts.Validate() if err != nil { @@ -156,8 +164,15 @@ func newSnapshotListCommand() *cobra.Command { Use: "list", Aliases: []string{"ls"}, Short: "List all snapshots", - Long: "Lists all snapshots with their ID, timestamp, and compressed size", - Args: cobra.NoArgs, + Long: `Lists the snapshots in the local index and on the backup destination +store, with their ID, timestamp, and sizes. + +Needs permission to list and read the destination store. It does not need +the age private key. A snapshot found only on the destination store is +listed by its remote key, because its ID is stored only in the local index +and in the snapshot's encrypted database. If the destination store cannot +be listed, the command warns and lists the local index alone.`, + Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { return runVaultikApp(cmd, readOnly, false, "Failed to list snapshots", @@ -179,11 +194,19 @@ func newSnapshotPurgeCommand() *cobra.Command { cmd := &cobra.Command{ Use: "purge", Short: "Purge old snapshots", - Long: `Removes snapshots based on age or count criteria. + Long: `Removes snapshots based on age or count criteria. Exactly one of +--keep-latest and --older-than is required. -Retention is per-snapshot-name: --keep-latest keeps the latest of each -configured snapshot name, not the latest globally. Use --snapshot to -restrict the operation to specific snapshot names.`, +Retention is per-snapshot-name: --keep-latest keeps the latest snapshot of +each name, not the latest globally. Use --snapshot to restrict the +operation to specific snapshot names. + +Only snapshots in the local index are removed. Each one is removed from +the local index and its metadata is deleted from the backup destination +store. Blobs are not touched: run 'vaultik prune' afterwards. + +Needs permission to list and delete objects on the destination store. If +the destination store cannot be listed, nothing is removed.`, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { // Validate flags @@ -222,14 +245,29 @@ func newSnapshotVerifyCommand() *cobra.Command { cmd := &cobra.Command{ Use: "verify ", Short: "Check a snapshot's blobs are present with the listed size", - Long: "Checks that every blob the snapshot's manifest lists is present\n" + - "in storage with the size the manifest records, and that the\n" + - "snapshot's encrypted database is present. It does not read blob\n" + - "contents; use --deep to download, decrypt, and re-hash every blob\n" + - "to detect corruption -- integrity, not who wrote it.\n\n" + - "The snapshot may be named by its ID or, on a host with no local\n" + - "index, by the remote key that 'snapshot list' prints for a\n" + - "remote-only snapshot (an unambiguous leading part is enough).", + Long: `Checks that every blob the snapshot's manifest lists is present +in storage with the size the manifest records, and that the +snapshot's encrypted database is present. It does not read blob +contents and does not need the age private key. + +Needs permission to read the backup destination store, and to list it +when a remote key is given. + +Use --deep to download, decrypt, and re-hash every blob to detect +corruption -- integrity, not who wrote it. + +--deep requires the age private key, in the VAULTIK_AGE_SECRET_KEY +environment variable or as age_secret_key in the config file; the +environment variable wins. The variable may hold the whole age-keygen +file, with its comments and all of its keys. Read the key from the +file rather than typing it, so that it does not land in your shell +history: + + export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" + +The snapshot may be named by its ID or, on a host with no local +index, by the remote key that 'snapshot list' prints for a +remote-only snapshot (an unambiguous leading part is enough).`, Args: requireSnapshotIDArg, RunE: func(cmd *cobra.Command, args []string) error { snapshotID := args[0] @@ -264,7 +302,10 @@ func newSnapshotVerifyCommand() *cobra.Command { }, } - cmd.Flags().BoolVar(&opts.Deep, "deep", false, "Download and verify blob hashes") + cmd.Flags().BoolVar(&opts.Deep, "deep", false, + "Download, decrypt and re-hash every blob; requires the age private "+ + "key in VAULTIK_AGE_SECRET_KEY, or as age_secret_key in the config "+ + "file") cmd.Flags().BoolVar(&opts.JSON, "json", false, "Output verification results as JSON") return cmd @@ -288,8 +329,10 @@ know about), which is what 'vaultik prune' does. The command prints the 'vaultik prune' invocation to run as a follow-up only when it removed the snapshot's metadata from the destination store, and never under --json. -An ID that names no snapshot in the local index or on the destination -store is an error. +The argument must be a snapshot ID: a remote key is not accepted. An ID +that names no snapshot in the local index or on the destination store is +an error. Needs permission to list and delete objects on the destination +store, unless --local-only is given. Use --local-only to skip the remote half (e.g. when you want to forget a snapshot locally without touching the destination store); the snapshot diff --git a/internal/cli/snapshot_restore.go b/internal/cli/snapshot_restore.go index 8609b90..80d6d08 100644 --- a/internal/cli/snapshot_restore.go +++ b/internal/cli/snapshot_restore.go @@ -37,13 +37,21 @@ The snapshot may be named by its ID or, when restoring on a host with no local index, by the remote key that 'snapshot list' prints for a remote-only snapshot (an unambiguous leading part is enough). -Requires the age private key in the VAULTIK_AGE_SECRET_KEY environment -variable. The variable may hold the whole age-keygen file (comments and -all of its identities are accepted); read it from the file rather than -typing the key, so it does not land in your shell history: +Each file is restored under its original absolute path inside the target +directory. File owners are restored only when the command runs as root. + +Requires the age private key, in the VAULTIK_AGE_SECRET_KEY environment +variable or as age_secret_key in the config file; the environment +variable wins. The variable may hold the whole age-keygen file, with its +comments and all of its keys. Read the key from the file rather than +typing it, so that it does not land in your shell history: export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" +Also needs permission to read the backup destination store (and to list +it when a remote key is given), and free space in the temporary directory +for the decrypted blobs. + Examples: # Restore entire snapshot vaultik snapshot restore myhost_docs_2025-01-01T12:00:00Z /restore diff --git a/internal/storage/s3_test.go b/internal/storage/s3_test.go index 8902bc7..7758b38 100644 --- a/internal/storage/s3_test.go +++ b/internal/storage/s3_test.go @@ -161,7 +161,7 @@ func TestS3ListingCutShortIsAnError(t *testing.T) { } // TestS3URLPrefixKeyLayout pins the bucket keys an s3:// URL reads and -// writes: the README's remote storage layout, with the prefix joined to +// writes: the layout in docs/REPOSTRUCTURE.md, with the prefix joined to // each key by one "/". s3://b/p and s3://b/p/ must be the same // destination, or a host that writes the URL the other way finds no // snapshots. The listed object is put straight into the bucket, as