From 38b588f658181f010894275d1788bc823427eab5 Mon Sep 17 00:00:00 2001 From: sneak Date: Tue, 10 Nov 2026 21:38:20 +0000 Subject: [PATCH] Run make fmt over the Markdown Only prettier's output; no text change. Model: opus-5-5 --- AGENTS.md | 147 ++- ARCHITECTURE.md | 153 ++- README.md | 1347 +++++++++++++------------ TODO.md | 2160 +++++++++++++++++++---------------------- docs/DATAMODEL.md | 209 ++-- docs/REPOSTRUCTURE.md | 284 ++++-- 6 files changed, 2244 insertions(+), 2056 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 2f253ae..2d82840 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,127 +4,122 @@ Version: 2025-06-08 # Instructions and Contextual Information -* Be direct, robotic, expert, accurate, and professional. +- Be direct, robotic, expert, accurate, and professional. -* Do not butter me up or kiss my ass. +- Do not butter me up or kiss my ass. -* Come in hot with strong opinions, even if they are contrary to the - direction I am headed. +- Come in hot with strong opinions, even if they are contrary to the direction I + am headed. -* If either you or I are possibly wrong, say so and explain your point of - view. +- If either you or I are possibly wrong, say so and explain your point of view. -* Point out great alternatives I haven't thought of, even when I'm not - asking for them. +- Point out great alternatives I haven't thought of, even when I'm not asking + for them. -* Treat me like the world's leading expert in every situation and every +- Treat me like the world's leading expert in every situation and every conversation, and deliver the absolute best recommendations. -* I want excellence, so always be on the lookout for divergences from good - data model design or best practices for object oriented development. +- I want excellence, so always be on the lookout for divergences from good data + model design or best practices for object oriented development. -* IMPORTANT: This is production code, not a research or teaching exercise. +- IMPORTANT: This is production code, not a research or teaching exercise. Deliver professional-level results, not prototypes. -* Please read and understand the `README.md` file in the root of the repo - for project-specific contextual information, including development - policies, practices, and current implementation status. +- Please read and understand the `README.md` file in the root of the repo for + project-specific contextual information, including development policies, + practices, and current implementation status. -* Be proactive in suggesting improvements or refactorings in places where we +- Be proactive in suggesting improvements or refactorings in places where we diverge from best practices for clean, modular, maintainable code. # Policies 1. Before committing, tests must pass (`make test`), linting must pass - (`make lint`), and code must be formatted (`make fmt`). For go, those + (`make lint`), and code must be formatted (`make fmt`). For go, those makefile targets should use `go fmt` and `go test -v ./...` and - `golangci-lint run`. Each Makefile target does exactly one thing — to - run lint + fmt-check + test together (the standard pre-commit gate), - use `make check`. + `golangci-lint run`. Each Makefile target does exactly one thing — to run + lint + fmt-check + test together (the standard pre-commit gate), use + `make check`. -2. Always write a `Makefile` with the default target being `test`, and with - a `fmt` target that formats the code. The `test` target should run all - tests in the project, and the `fmt` target should format the code. - `test` should also have a prerequisite target `lint` that should run any - linters that are configured for the project. +2. Always write a `Makefile` with the default target being `test`, and with a + `fmt` target that formats the code. The `test` target should run all tests in + the project, and the `fmt` target should format the code. `test` should also + have a prerequisite target `lint` that should run any linters that are + configured for the project. -3. After each completed bugfix or feature, the code must be committed. Do - all of the pre-commit checks (test, lint, fmt) before committing, of - course. +3. After each completed bugfix or feature, the code must be committed. Do all of + the pre-commit checks (test, lint, fmt) before committing, of course. 4. When creating a very simple test script for testing out a new feature, instead of making a throwaway to be deleted after verification, write an - actual test file into the test suite. It doesn't need to be very big or + actual test file into the test suite. It doesn't need to be very big or complex, but it should be a real test that can be run. 5. When you are instructed to make the tests pass, DO NOT delete tests, skip - tests, or change the tests specifically to make them pass (unless there - is a bug in the test). This is cheating, and it is bad. You should only - be modifying the test if it is incorrect or if the test is no longer - relevant. In almost all cases, you should be fixing the code that is - being tested, or updating the tests to match a refactored implementation. + tests, or change the tests specifically to make them pass (unless there is a + bug in the test). This is cheating, and it is bad. You should only be + modifying the test if it is incorrect or if the test is no longer relevant. + In almost all cases, you should be fixing the code that is being tested, or + updating the tests to match a refactored implementation. 6. When dealing with dates and times or timestamps, always use, display, and - store UTC. Set the local timezone to UTC on startup. If the user needs - to see the time in a different timezone, store the user's timezone in a - separate field and convert the UTC time to the user's timezone when - displaying it. For internal use and internal applications and - administrative purposes, always display UTC. + store UTC. Set the local timezone to UTC on startup. If the user needs to see + the time in a different timezone, store the user's timezone in a separate + field and convert the UTC time to the user's timezone when displaying it. For + internal use and internal applications and administrative purposes, always + display UTC. 7. Always write tests, even if they are extremely simple and just check for - correct syntax (ability to compile/import). If you are writing a new - feature, write a test for it. You don't need to target complete - coverage, but you should at least test any new functionality you add. If - you are fixing a bug, write a test first that reproduces the bug, and - then fix the bug in the code. + correct syntax (ability to compile/import). If you are writing a new feature, + write a test for it. You don't need to target complete coverage, but you + should at least test any new functionality you add. If you are fixing a bug, + write a test first that reproduces the bug, and then fix the bug in the code. -8. When implementing new features, be aware of potential side-effects (such - as state files on disk, data in the database, etc.) and ensure that it is +8. When implementing new features, be aware of potential side-effects (such as + state files on disk, data in the database, etc.) and ensure that it is possible to mock or stub these side-effects in tests. -9. Always use structured logging. Log any relevant state/context with the - messages (but do not log secrets). If the log stream is not a terminal, +9. Always use structured logging. Log any relevant state/context with the + messages (but do not log secrets). If the log stream is not a terminal, output the structured logs in jsonl format. 10. Avoid using bare strings or numbers in code, especially if they appear - anywhere more than once. Always define a constant (usually at the top - of the file) and give it a descriptive name, then use that constant in - the code instead of the bare string or number. + anywhere more than once. Always define a constant (usually at the top of the + file) and give it a descriptive name, then use that constant in the code + instead of the bare string or number. 11. You do not need to summarize your changes in the chat after making them. - Making the changes and committing them is sufficient. If anything out - of the ordinary happened, please explain it, but in the normal case - where you found and fixed the bug, or implemented the feature, there is - no need for the end-of-change summary. + Making the changes and committing them is sufficient. If anything out of the + ordinary happened, please explain it, but in the normal case where you found + and fixed the bug, or implemented the feature, there is no need for the + end-of-change summary. -12. Do not create additional files in the root directory of the project - without asking permission first. Configuration files, documentation, and - build files are acceptable in the root, but source code and other files - should be organized in appropriate subdirectories. +12. Do not create additional files in the root directory of the project without + asking permission first. Configuration files, documentation, and build files + are acceptable in the root, but source code and other files should be + organized in appropriate subdirectories. 13. Pre-1.0: NEVER add a database migration. Migrations are supported, but - nothing is installed anywhere yet, so there is nothing to migrate. To - change the schema, edit `internal/database/schema/001.sql` (and any - code that touches the affected tables) directly. After 1.0, each schema - change is a new numbered file in that directory and a released file is - never edited; an existing local database is then migrated when vaultik - is updated. Before 1.0, a local database left on an older schema is - deleted and re-created by a full backup. See - [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). + nothing is installed anywhere yet, so there is nothing to migrate. To change + the schema, edit `internal/database/schema/001.sql` (and any code that + touches the affected tables) directly. After 1.0, each schema change is a + new numbered file in that directory and a released file is never edited; an + existing local database is then migrated when vaultik is updated. Before + 1.0, a local database left on an older schema is deleted and re-created by a + full backup. See [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). -14. Never use `git add -A`. Stage only the files you intentionally - changed. +14. Never use `git add -A`. Stage only the files you intentionally changed. -15. Commit messages carry no attribution or advertising trailers for the - tool that helped write the code, or for its vendor: the owner is the - sole author of code written with a tool. +15. Commit messages carry no attribution or advertising trailers for the tool + that helped write the code, or for its vendor: the owner is the sole author + of code written with a tool. 16. Run the whole test suite with `make test` every time, and read its full - output. Never run `go test`, a single test or a single package, and - never grep the output. + output. Never run `go test`, a single test or a single package, and never + grep the output. 17. Do not stop working on a task until the definition of done given in the initial instruction is met: all of the work, not part or most of it. -18. For estimates: backing up over 2.5Gbit/s ethernet to an S3 server - backed by 2000MB/sec SSD takes about 4 seconds per gigabyte. +18. For estimates: backing up over 2.5Gbit/s ethernet to an S3 server backed by + 2000MB/sec SSD takes about 4 seconds per gigabyte. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b41dc22..83bfe36 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,10 +1,14 @@ # Vaultik Architecture -This document describes the internal architecture of Vaultik, focusing on the data model, type instantiation, and the relationships between core modules. +This document describes the internal architecture of Vaultik, focusing on the +data model, type instantiation, and the relationships between core modules. ## Overview -Vaultik is a backup system that uses content-defined chunking for deduplication and packs chunks into large, compressed, encrypted blobs for efficient cloud storage. The system is built around dependency injection using [uber-go/fx](https://github.com/uber-go/fx). +Vaultik is a backup system that uses content-defined chunking for deduplication +and packs chunks into large, compressed, encrypted blobs for efficient cloud +storage. The system is built around dependency injection using +[uber-go/fx](https://github.com/uber-go/fx). ## Data Flow @@ -53,57 +57,88 @@ The database tracks five primary entities and their relationships: ### Entity Descriptions #### File (`database.File`) -Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration: + +Represents a file, directory, or symlink in the backup system. Stores metadata +needed for restoration: + - Path, mtime - Size, mode, ownership (uid, gid) - Symlink target (if applicable) -It also stores `source_path`, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it. +It also stores `source_path`, the source directory the scan found it under, made +absolute and with symlinks resolved. Restore does not read it. #### Chunk (`database.Chunk`) -A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm: + +A content-addressed unit of data. Files are split into variable-size chunks +using the FastCDC algorithm: + - `ChunkHash`: SHA256 hash of chunk content (primary key) - `Size`: Chunk size in bytes -Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for the 10MB default average). +Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for +the 10MB default average). #### FileChunk (`database.FileChunk`) + Maps files to their constituent chunks: + - `FileID`: Reference to the file - `Idx`: Position of this chunk within the file (0-indexed) - `ChunkHash`: Reference to the chunk #### Blob (`database.Blob`) -The final storage unit uploaded to S3. Contains many compressed and encrypted chunks: + +The final storage unit uploaded to S3. Contains many compressed and encrypted +chunks: + - `ID`: UUID assigned at creation -- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs)) +- `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before + compression and encryption (see + [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs-directory-blobs)) - `UncompressedSize`: Total raw chunk data before compression - `CompressedSize`: Size after zstd compression and age encryption - `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps Blob creation process: + 1. Chunks are accumulated (up to MaxBlobSize, typically 10GB) 2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256 -3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file -4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes -5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted +3. Concurrently, the same bytes are compressed with zstd, then encrypted with + age (recipients configured in config), and written to a temporary file +4. On finalize, the blob's name is the double SHA-256 of the uncompressed + contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, + encrypted bytes +5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and + then deleted -A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`. +A backup needs free temporary space, because each blob is written whole to a +temporary file before it is uploaded (up to about `blob_size_limit`; an rclone +destination that cannot stream uploads needs about twice that) and the metadata +export writes copies of the local index. Temporary files go to `$TMPDIR` +(default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to +`/var/tmp`. #### BlobChunk (`database.BlobChunk`) + Maps chunks to their position within blobs: + - `BlobID`: Reference to the blob - `ChunkHash`: Reference to the chunk - `Offset`: Byte offset within the uncompressed blob - `Length`: Chunk size #### Snapshot (`database.Snapshot`) + Represents a point-in-time backup: -- `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g. `server1_home_2025-06-01T12:00:00Z`) + +- `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g. + `server1_home_2025-06-01T12:00:00Z`) - Tracks file count, chunk count, blob count, sizes, compression ratio - `CompletedAt`: Null until snapshot finishes successfully #### SnapshotFile / SnapshotBlob + Join tables linking snapshots to their files and blobs. ### Relationship Summary @@ -140,21 +175,27 @@ fx.New( ### Key Type Instantiation Points #### 1. Config (`config.Config`) + - **Created by**: `config.Module` via `config.LoadConfig()` - **When**: Application startup (fx DI) -- **Contains**: All configuration from YAML file (S3 credentials, encryption keys, paths, etc.) +- **Contains**: All configuration from YAML file (S3 credentials, encryption + keys, paths, etc.) #### 2. Database (`database.DB`) + - **Created by**: `database.Module` via `database.New()` - **When**: Application startup (fx DI) - **Contains**: SQLite connection, path reference #### 3. Repositories (`database.Repositories`) + - **Created by**: `database.Module` via `database.NewRepositories()` - **When**: Application startup (fx DI) -- **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots, etc.) +- **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots, + etc.) #### 4. Vaultik (`vaultik.Vaultik`) + - **Created by**: `vaultik.New(VaultikParams)` - **When**: Application startup (fx DI) - **Contains**: All dependencies for backup operations @@ -176,11 +217,13 @@ type Vaultik struct { ``` #### 5. SnapshotManager (`snapshot.SnapshotManager`) + - **Created by**: `snapshot.Module` via `snapshot.NewSnapshotManager()` - **When**: Application startup (fx DI) - **Responsibility**: Creates/completes snapshots, exports metadata to S3 #### 6. Scanner (`snapshot.Scanner`) + - **Created by**: `ScannerFactory(ScannerParams)` - **When**: Each `CreateSnapshot()` call - **Contains**: Chunker, Packer, progress reporter @@ -194,20 +237,22 @@ scanner := v.ScannerFactory(snapshot.ScannerParams{ ``` #### 7. Chunker (`chunker.Chunker`) + - **Created by**: `chunker.NewChunker(avgChunkSize)` - **When**: Inside `snapshot.NewScanner()` - **Configuration**: - - `avgChunkSize`: From config (default 10MB) - - `minChunkSize`: avgChunkSize / 4 - - `maxChunkSize`: avgChunkSize * 4 + - `avgChunkSize`: From config (default 10MB) + - `minChunkSize`: avgChunkSize / 4 + - `maxChunkSize`: avgChunkSize \* 4 #### 8. Packer (`blob.Packer`) + - **Created by**: `blob.NewPacker(PackerConfig)` - **When**: Inside `snapshot.NewScanner()` - **Configuration**: - - `MaxBlobSize`: Maximum blob size before finalization (typically 10GB) - - `CompressionLevel`: zstd level (1-19) - - `Recipients`: age public keys for encryption + - `MaxBlobSize`: Maximum blob size before finalization (typically 10GB) + - `CompressionLevel`: zstd level (1-19) + - `Recipients`: age public keys for encryption ```go // snapshot/scanner.go: NewScanner() @@ -224,17 +269,22 @@ packer, err := blob.NewPacker(packerCfg) ## Module Responsibilities ### `internal/cli` -Entry point for fx application. Combines all modules and handles signal interrupts. + +Entry point for fx application. Combines all modules and handles signal +interrupts. Key functions: + - `NewApp(AppOptions)` → Creates fx.App with all modules - `RunApp(ctx, app)` → Starts app, handles graceful shutdown - `RunWithApp(ctx, opts)` → Convenience wrapper ### `internal/vaultik` + Main orchestrator containing all dependencies and command implementations. Key methods: + - `New(VaultikParams)` → Constructor (fx DI) - `CreateSnapshot(opts)` → Main backup operation - `ListSnapshots(jsonOutput)` → List available snapshots @@ -242,27 +292,34 @@ Key methods: - `PurgeSnapshots(...)` → Remove old snapshots ### `internal/chunker` + Content-defined chunking using FastCDC algorithm. Key types: + - `Chunk` → Hash, Data, Offset, Size - `Chunker` → avgChunkSize, minChunkSize, maxChunkSize Key methods: + - `NewChunker(avgChunkSize)` → Constructor -- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback (preferred) +- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback + (preferred) - `ChunkReader(reader)` → Return all chunks at once (memory-intensive) ### `internal/blob` + Blob packing: accumulates chunks, compresses, encrypts, tracks metadata. Key types: + - `Packer` → Thread-safe blob accumulator - `ChunkRef` → Hash + Data for adding to packer - `FinishedBlob` → Completed blob ready for upload - `BlobWithReader` → FinishedBlob + io.Reader for streaming upload Key methods: + - `NewPacker(PackerConfig)` → Constructor - `AddChunk(ChunkRef)` → Add chunk to current blob - `FinalizeBlob()` → Compress, encrypt, hash current blob @@ -272,35 +329,48 @@ Key methods: ### `internal/snapshot` #### Scanner + Orchestrates the backup process for a directory. Key methods: + - `NewScanner(ScannerConfig)` → Constructor (creates Chunker + Packer) - `Scan(ctx, path, snapshotID)` → Main scan operation Scan phases: + 1. **Phase 1**: Walk directory, identify files needing processing -2. **Phase 1b**: Count as deleted the files that the previous snapshot of the same name lists, the walk did not find, and are gone from disk; a file that is excluded, or inside an excluded directory, is not counted +2. **Phase 1b**: Count as deleted the files that the previous snapshot of the + same name lists, the walk did not find, and are gone from disk; a file that + is excluded, or inside an excluded directory, is not counted 3. **Phase 2**: Process files (chunk → pack → upload) #### SnapshotManager + Manages snapshot lifecycle and metadata export. Key methods: + - `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record -- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot references +- `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot + references - `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3 -- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a successful export -- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark complete (no export between) +- `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a + successful export +- `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark + complete (no export between) ### `internal/database` + SQLite database for local index. Single-writer mode for thread safety. Key types: + - `DB` → Database connection wrapper - `Repositories` → Collection of all repository interfaces Repository interfaces: + - `FilesRepository` → CRUD for File records - `ChunksRepository` → CRUD for Chunk records - `BlobsRepository` → CRUD for Blob records @@ -359,11 +429,16 @@ CreateSnapshot(opts) ## Deduplication Strategy -1. **File-level**: Files and symlinks unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked +1. **File-level**: Files and symlinks unchanged since last backup are skipped + (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target), + unless the file lists a chunk that no uploaded blob holds; such a file is + re-chunked -2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded. +2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash + already exists in the database, the chunk data is not re-uploaded. -3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a blob are skipped. +3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a + blob are skipped. ## Storage Layout in S3 @@ -382,17 +457,16 @@ bucket/ └── manifest.json.zst # Blob list (for pruning/verification) ``` -A destination that answers a look-up of `vaultik-destination.txt` with not -found is treated as not present while the local index holds snapshots stored -there (`checkDestinationMarker` in `internal/vaultik/destination_marker.go`): +A destination that answers a look-up of `vaultik-destination.txt` with not found +is treated as not present while the local index holds snapshots stored there +(`checkDestinationMarker` in `internal/vaultik/destination_marker.go`): `snapshot create` refuses to write to it, `prune`, `snapshot purge` and `remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a destination that cannot be listed. This keeps those commands from taking an -unplugged volume's empty mount point for a destination whose snapshots are -all gone. When the look-up fails for any reason other than not found, for -example when S3 refuses it to credentials that allow only PUT, the check is -skipped, so a host with such credentials does not notice an emptied -destination; see +unplugged volume's empty mount point for a destination whose snapshots are all +gone. When the look-up fails for any reason other than not found, for example +when S3 refuses it to credentials that allow only PUT, the check is skipped, so +a host with such credentials does not notice an emptied destination; see [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt). The `file://` backend never creates its destination directory, so a write to a missing one fails. @@ -407,5 +481,6 @@ derivation and a worked example. - `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`. - `Scanner`: Uses `packerMu` mutex to coordinate blob finalization. -- `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread safety. +- `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread + safety. - `Repositories.WithTx()`: Handles transaction lifecycle automatically. diff --git a/README.md b/README.md index 80576de..07a4b9b 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # vaultik (ваултик) -`vaultik` is an incremental backup tool written in Go. It encrypts data -using an `age` public key and uploads each encrypted blob directly to a -remote S3-compatible object store. It requires no private keys, secrets, or -credentials (other than those required to PUT to encrypted object storage, -such as S3 API keys) stored on the backed-up system. +`vaultik` is an incremental backup tool written in Go. It encrypts data using an +`age` public key and uploads each encrypted blob directly to a remote +S3-compatible object store. It requires no private keys, secrets, or credentials +(other than those required to PUT to encrypted object storage, such as S3 API +keys) stored on the backed-up system. ## quickstart @@ -42,36 +42,36 @@ vaultik snapshot list Features: -* modern encryption ([age](https://age-encryption.org/), X25519 + ChaCha20-Poly1305) -* content-defined chunking with deduplication (FastCDC) -* incremental backups (a file is re-chunked only when it changed or a - chunk it lists is held by no uploaded blob) -* multithreaded zstd compression at configurable levels -* content-addressed immutable storage -* local state tracking in SQLite (enables write-only incremental backups) -* no mutable remote metadata -* no plaintext file paths or metadata in remote storage -* packs small files into large blobs (keeps S3 operation counts down) -* backs up regular files, symlinks, empty directories, and file permissions -* pluggable storage backends: S3, local filesystem, rclone (70+ providers) -* pure Go (no CGO), cross-compiles to linux/darwin × amd64/arm64 +- modern encryption ([age](https://age-encryption.org/), X25519 + + ChaCha20-Poly1305) +- content-defined chunking with deduplication (FastCDC) +- incremental backups (a file is re-chunked only when it changed or a chunk it + lists is held by no uploaded blob) +- multithreaded zstd compression at configurable levels +- content-addressed immutable storage +- local state tracking in SQLite (enables write-only incremental backups) +- no mutable remote metadata +- no plaintext file paths or metadata in remote storage +- packs small files into large blobs (keeps S3 operation counts down) +- backs up regular files, symlinks, empty directories, and file permissions +- pluggable storage backends: S3, local filesystem, rclone (70+ providers) +- pure Go (no CGO), cross-compiles to linux/darwin × amd64/arm64 ## why Other backup tools like `restic`, `borg`, and `duplicity` are designed for environments where the source host can store secrets and has access to -decryption keys. `vaultik` is for environments where you don't want to -store backup decryption keys on your hosts — only public keys for -encryption. +decryption keys. `vaultik` is for environments where you don't want to store +backup decryption keys on your hosts — only public keys for encryption. Requirements that no existing tool meets: -* open source -* no passphrases or private keys on the source host -* incremental -* compressed -* encrypted -* s3 compatible without an intermediate step or tool +- open source +- no passphrases or private keys on the source host +- incremental +- compressed +- encrypted +- s3 compatible without an intermediate step or tool ## daily use @@ -101,17 +101,17 @@ relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup another one is still running, and its directory under `metadata/` that has no manifest yet, so it must be longer than the longest backup run. It does not protect a blob that backup reuses from an earlier run that stopped before its -manifest: the prune can delete such a blob, and the new snapshot then cannot -be restored. To not depend on the grace period, give each host its own prefix -or directory. See [locking](#locking). +manifest: the prune can delete such a blob, and the new snapshot then cannot be +restored. To not depend on the grace period, give each host its own prefix or +directory. See [locking](#locking). ## restoring on another machine -Restoring on a host that never ran the backup — a replacement machine -after the original is gone — is the case vaultik is built for. That host -needs only three things: the `vaultik` binary, the age **private** key, -and the storage credentials for the destination. It does **not** need the -local index, the original config file, or the original hostname. +Restoring on a host that never ran the backup — a replacement machine after the +original is gone — is the case vaultik is built for. That host needs only three +things: the `vaultik` binary, the age **private** key, and the storage +credentials for the destination. It does **not** need the local index, the +original config file, or the original hostname. ```sh # install @@ -127,17 +127,16 @@ vaultik config set s3.secret_access_key "..." vaultik snapshot list ``` -`snapshot list` reads the destination store without the private key. A -snapshot that is not in this host's (empty) local index is shown as -remote-only: its row is identified by `` rather than by -a `hostname_name_timestamp` name, because the name lives only in the -local index and the encrypted database and cannot be recovered from the -store. Its timestamp and compressed size are real. (See the `snapshot -list` description under [command details](#command-details) for the full -explanation.) +`snapshot list` reads the destination store without the private key. A snapshot +that is not in this host's (empty) local index is shown as remote-only: its row +is identified by `` rather than by a `hostname_name_timestamp` +name, because the name lives only in the local index and the encrypted database +and cannot be recovered from the store. Its timestamp and compressed size are +real. (See the `snapshot list` description under +[command details](#command-details) for the full explanation.) -Use that remote key — the hex printed inside ``, or the -full `remote_key` from `snapshot list --json` — to restore and verify: +Use that remote key — the hex printed inside ``, or the full +`remote_key` from `snapshot list --json` — to restore and verify: ```sh # put the private key file in the environment (reading it from the file @@ -154,10 +153,10 @@ vaultik snapshot restore --verify ~/vaultik-restore vaultik snapshot verify --deep ``` -`age_recipients` (the public key) is not needed to restore — only the -private key in `VAULTIK_AGE_SECRET_KEY`. Both the abbreviated key printed -in the table and the full 64-character key from `--json` are accepted; a -leading part of the key is enough as long as it is unambiguous. +`age_recipients` (the public key) is not needed to restore — only the private +key in `VAULTIK_AGE_SECRET_KEY`. Both the abbreviated key printed in the table +and the full 64-character key from `--json` are accepted; a leading part of the +key is enough as long as it is unambiguous. --- @@ -187,83 +186,92 @@ vaultik version ### global flags -* `--config `: Path to config file (default: `$VAULTIK_CONFIG`, then platform config dir, then `/etc/vaultik/config.yml`) -* `--verbose`, `-v`: Enable verbose output (on stderr — see below) -* `--debug`: Enable debug output (on stderr — see below) -* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner) -* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run. A run that skipped anything still finishes but exits non-zero: `snapshot create` keeps the snapshot and counts each file or directory it skipped as failed. A file removed during the backup is also counted as failed but does not change the exit status. +- `--config `: Path to config file (default: `$VAULTIK_CONFIG`, then + platform config dir, then `/etc/vaultik/config.yml`) +- `--verbose`, `-v`: Enable verbose output (on stderr — see below) +- `--debug`: Enable debug output (on stderr — see below) +- `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner) +- `--skip-errors`: Skip files that cannot be read when creating a snapshot, or + that cannot be restored when restoring, instead of aborting. Packing and + storage errors while creating a snapshot (which would leave a chunk recorded + but not stored) still abort the run. A run that skipped anything still + finishes but exits non-zero: `snapshot create` keeps the snapshot and counts + each file or directory it skipped as failed. A file removed during the backup + is also counted as failed but does not change the exit status. ### locking -Commands that write persistent state — `snapshot create`, `snapshot -purge`, `snapshot remove`, `prune`, and `remote nuke` — take a -process-wide lock at `$XDG_DATA_HOME/vaultik/vaultik.pid` -(`~/.local/share/vaultik/vaultik.pid` on Linux) for the whole run. Only -one of them runs at a time: a second one exits immediately with an -"already running" error rather than waiting, so two writers can never -corrupt the local index. +Commands that write persistent state — `snapshot create`, `snapshot purge`, +`snapshot remove`, `prune`, and `remote nuke` — take a process-wide lock at +`$XDG_DATA_HOME/vaultik/vaultik.pid` (`~/.local/share/vaultik/vaultik.pid` on +Linux) for the whole run. Only one of them runs at a time: a second one exits +immediately with an "already running" error rather than waiting, so two writers +can never corrupt the local index. -The lock covers one user on one machine. It does not stop another user, -or another host, from writing to the same destination store at the same -time. A backup uploads its blobs, and then its `db.zst.age`, before its -manifest, so a prune from elsewhere at that moment finds blobs no snapshot -lists and a directory under `metadata/` with no manifest. It -deletes such a blob only once it is older than `prune_grace_period` (default -`24h`), and such a directory only once every object in it is, so the grace -period must be longer than the longest backup run of any host or user sharing -the destination. The grace period does not protect a blob a backup reuses -from an earlier run that stopped before its manifest: a backup skips the -upload of a blob already on the destination, so that blob keeps the time of -its first upload, and a prune from elsewhere during the backup can delete it; -the new snapshot then cannot be restored. `remote nuke` deletes everything -regardless. To not depend on the grace period, give each host or user its own -prefix (or directory) on the destination. +The lock covers one user on one machine. It does not stop another user, or +another host, from writing to the same destination store at the same time. A +backup uploads its blobs, and then its `db.zst.age`, before its manifest, so a +prune from elsewhere at that moment finds blobs no snapshot lists and a +directory under `metadata/` with no manifest. It deletes such a blob only once +it is older than `prune_grace_period` (default `24h`), and such a directory only +once every object in it is, so the grace period must be longer than the longest +backup run of any host or user sharing the destination. The grace period does +not protect a blob a backup reuses from an earlier run that stopped before its +manifest: a backup skips the upload of a blob already on the destination, so +that blob keeps the time of its first upload, and a prune from elsewhere during +the backup can delete it; the new snapshot then cannot be restored. +`remote nuke` deletes everything regardless. To not depend on the grace period, +give each host or user its own prefix (or directory) on the destination. Read-only commands — `info`, `snapshot list`, `snapshot verify`, and -`remote info` — do not take the lock and are never blocked, so they run -even while a backup is in progress. `snapshot restore` does not take the -lock either: it writes only to the target directory you name, not the -local index or the destination store. `config`, `database delete`, -`completion`, and `version` do not take the lock. +`remote info` — do not take the lock and are never blocked, so they run even +while a backup is in progress. `snapshot restore` does not take the lock either: +it writes only to the target directory you name, not the local index or the +destination store. `config`, `database delete`, `completion`, and `version` do +not take the lock. ### stdout and stderr -Log output — everything from `--verbose` and `--debug`, and every -warning and error the logger emits — goes to **stderr**, and so does the -startup banner. stdout carries the output you asked for: tables, the -documents produced by `--json`, `config get` values, and completion -scripts. +Log output — everything from `--verbose` and `--debug`, and every warning and +error the logger emits — goes to **stderr**, and so does the startup banner. +stdout carries the output you asked for: tables, the documents produced by +`--json`, `config get` values, and completion scripts. -This means `vaultik snapshot list --verbose > out.txt` captures the -listing and leaves the diagnostics on your terminal. To capture both, -redirect stderr as well (`> out.txt 2> log.txt`, or `> out.txt 2>&1` to -interleave them). +This means `vaultik snapshot list --verbose > out.txt` captures the listing and +leaves the diagnostics on your terminal. To capture both, redirect stderr as +well (`> out.txt 2> log.txt`, or `> out.txt 2>&1` to interleave them). -The split is what makes `--json` usable from a script. Warnings and -errors are never suppressed — not by `--quiet`, not by `--cron` — so a -logger on stdout would eventually land a log line inside a JSON -document and break the parse. A config file with group- or -world-readable permissions is enough to trigger it. +The split is what makes `--json` usable from a script. Warnings and errors are +never suppressed — not by `--quiet`, not by `--cron` — so a logger on stdout +would eventually land a log line inside a JSON document and break the parse. A +config file with group- or world-readable permissions is enough to trigger it. -Format follows the stream: when stderr is a terminal the records are -colorized one-liners, and when it is redirected or piped they are -JSON, one object per line. +Format follows the stream: when stderr is a terminal the records are colorized +one-liners, and when it is redirected or piped they are JSON, one object per +line. -Under `--json`, stdout holds the document and nothing else. The startup -banner is suppressed, as `--quiet` and `--cron` suppress it, and the -progress narration a command would otherwise print — such as the stale -local records `prune` reconciles away — is suppressed too, so it cannot -land ahead of the document. Every `--json` command therefore pipes on -its own, with no additional flag: `vaultik snapshot list --json | jq .` -and `vaultik prune --json | jq .` both work as written. +Under `--json`, stdout holds the document and nothing else. The startup banner +is suppressed, as `--quiet` and `--cron` suppress it, and the progress narration +a command would otherwise print — such as the stale local records `prune` +reconciles away — is suppressed too, so it cannot land ahead of the document. +Every `--json` command therefore pipes on its own, with no additional flag: +`vaultik snapshot list --json | jq .` and `vaultik prune --json | jq .` both +work as written. ### environment variables -* `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `snapshot restore` and `snapshot verify --deep`). May hold the whole `age-keygen` file — comments and every identity in it are accepted. Set it from the file, e.g. `export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"`, so the key is not typed into your shell history. -* `VAULTIK_CONFIG`: Path to config file (overridden by `--config`) -* `VAULTIK_INDEX_PATH`: Override local SQLite index path -* `VAULTIK_CPUPROFILE`: Write a CPU profile to this path for the duration of the run (development/debugging) -* `VAULTIK_MEMPROFILE`: Write a heap profile to this path when the run exits (development/debugging) +- `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for + `snapshot restore` and `snapshot verify --deep`). May hold the whole + `age-keygen` file — comments and every identity in it are accepted. Set it + from the file, e.g. + `export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"`, so + the key is not typed into your shell history. +- `VAULTIK_CONFIG`: Path to config file (overridden by `--config`) +- `VAULTIK_INDEX_PATH`: Override local SQLite index path +- `VAULTIK_CPUPROFILE`: Write a CPU profile to this path for the duration of the + run (development/debugging) +- `VAULTIK_MEMPROFILE`: Write a heap profile to this path when the run exits + (development/debugging) ### shell completion @@ -294,282 +302,279 @@ existing file. Created with mode `0600` since it will contain credentials. **`config edit`**: Open the config file in `$EDITOR` (falls back to `vi`). -**`config get`**: Print a config value addressed by dotted YAML path -(e.g. `vaultik config get storage_url`). Non-scalar values print as YAML. +**`config get`**: Print a config value addressed by dotted YAML path (e.g. +`vaultik config get storage_url`). Non-scalar values print as YAML. -**`config set`**: Set a scalar config value by dotted YAML path -(e.g. `vaultik config set compression_level 9`, -`vaultik config set storage_url "file:///mnt/backups"`). Comments and -formatting in the file are preserved. Missing keys along the path are -created: as a list when the next path element is a number (so +**`config set`**: Set a scalar config value by dotted YAML path (e.g. +`vaultik config set compression_level 9`, +`vaultik config set storage_url "file:///mnt/backups"`). Comments and formatting +in the file are preserved. Missing keys along the path are created: as a list +when the next path element is a number (so `vaultik config set age_recipients.0 age1...` works on a file without -`age_recipients`), otherwise as a map. If the config loads, a change -after which it would not, such as `vaultik config set compression_level 99`, -is refused and the file is left unchanged. A config that does not load, -such as the one `config init` writes before `storage_url` is set or one -with a `${ENV:...}` reference to a variable not set in this shell, is -checked one setting at a time: only a change after which the changed -setting cannot be read, such as one giving it the wrong type (a single -value where a list belongs or a word where a number belongs), is refused. +`age_recipients`), otherwise as a map. If the config loads, a change after which +it would not, such as `vaultik config set compression_level 99`, is refused and +the file is left unchanged. A config that does not load, such as the one +`config init` writes before `storage_url` is set or one with a `${ENV:...}` +reference to a variable not set in this shell, is checked one setting at a time: +only a change after which the changed setting cannot be read, such as one giving +it the wrong type (a single value where a list belongs or a word where a number +belongs), is refused. **`snapshot create`**: Perform incremental backup of configured snapshots. -* Optional snapshot names argument to create specific snapshots (default: all) -* On macOS, the terminal application running vaultik needs Full Disk Access + +- Optional snapshot names argument to create specific snapshots (default: all) +- On macOS, the terminal application running vaultik needs Full Disk Access (System Settings → Privacy & Security → Full Disk Access) to read TCC-protected directories; without it the backup aborts with a permission error that explains how to fix it -* `--cron`: Silent on total success; warnings and errors are still printed - (for crontab) -* `--prune`: After backup, drop older snapshots of each backed-up name, - remove orphaned blobs older than `prune_grace_period` from remote storage, - and delete each directory under `metadata/` that has no - `manifest.json.zst` once every object in it is older than - `prune_grace_period` and its snapshot is not in the local index, as `prune` - does. By default keeps only the latest - snapshot per name; use `--keep-newer-than` for a rolling window. When - `--skip-errors` skipped a file or directory it could not read or list, the - prune does not run and every older snapshot is kept, since one of them may - be the only one that holds what was skipped; the command says so and exits - non-zero. When the backup found nothing in a configured path, as in the - mount point of a volume that is not mounted, the prune does not run - either: every older snapshot is kept, and the command names the path and - exits non-zero, on every run until the path holds something again. Take a - path that is empty on purpose out of the config, or remove the older - snapshots by hand with `snapshot purge`. When an earlier snapshot of the - same name in the local index found a volume mounted at or below a - configured path (a directory on a different filesystem from its parent, - with an entry in it that the backup does not exclude) and this backup does - not, the prune does not run either: every older snapshot is kept, and the - command names the directory and exits non-zero, on every run until the - volume is mounted again. A backup also does not find the volume when it - finds no such entry in its directory, as when an automounted volume cannot - be mounted, or when that directory is no longer backed up, because it is - excluded or its configured path was removed. If any of this is on purpose, - as for a retired volume, remove the older snapshots by hand with - `snapshot purge`. A file removed during the backup does not stop the - prune. -* `--keep-newer-than `: With `--prune`, keep snapshots newer than - this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). - Without `--prune`, or with a value that is not a valid duration, the - command fails before the backup starts. -* Fails before it writes anything when a `file://` destination directory - does not exist, or when the local index holds snapshots and the - destination answers that it has no `vaultik-destination.txt`. When that - look-up fails for any reason other than not found, for example when S3 - refuses it to credentials that allow only PUT, the check is skipped (see +- `--cron`: Silent on total success; warnings and errors are still printed (for + crontab) +- `--prune`: After backup, drop older snapshots of each backed-up name, remove + orphaned blobs older than `prune_grace_period` from remote storage, and delete + each directory under `metadata/` that has no `manifest.json.zst` once every + object in it is older than `prune_grace_period` and its snapshot is not in the + local index, as `prune` does. By default keeps only the latest snapshot per + name; use `--keep-newer-than` for a rolling window. When `--skip-errors` + skipped a file or directory it could not read or list, the prune does not run + and every older snapshot is kept, since one of them may be the only one that + holds what was skipped; the command says so and exits non-zero. When the + backup found nothing in a configured path, as in the mount point of a volume + that is not mounted, the prune does not run either: every older snapshot is + kept, and the command names the path and exits non-zero, on every run until + the path holds something again. Take a path that is empty on purpose out of + the config, or remove the older snapshots by hand with `snapshot purge`. When + an earlier snapshot of the same name in the local index found a volume mounted + at or below a configured path (a directory on a different filesystem from its + parent, with an entry in it that the backup does not exclude) and this backup + does not, the prune does not run either: every older snapshot is kept, and the + command names the directory and exits non-zero, on every run until the volume + is mounted again. A backup also does not find the volume when it finds no such + entry in its directory, as when an automounted volume cannot be mounted, or + when that directory is no longer backed up, because it is excluded or its + configured path was removed. If any of this is on purpose, as for a retired + volume, remove the older snapshots by hand with `snapshot purge`. A file + removed during the backup does not stop the prune. +- `--keep-newer-than `: With `--prune`, keep snapshots newer than this + duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without + `--prune`, or with a value that is not a valid duration, the command fails + before the backup starts. +- Fails before it writes anything when a `file://` destination directory does + not exist, or when the local index holds snapshots and the destination answers + that it has no `vaultik-destination.txt`. When that look-up fails for any + reason other than not found, for example when S3 refuses it to credentials + that allow only PUT, the check is skipped (see [remote storage layout](#remote-storage-layout)) -* Fails, before anything is pruned, when a configured path is a symlink - whose target does not exist, as one into a volume that is not mounted +- Fails, before anything is pruned, when a configured path is a symlink whose + target does not exist, as one into a volume that is not mounted -**`snapshot list`**: Show every snapshot known to this host — the union -of the local index and the backup destination store — with timestamps -and three sizes per snapshot (compressed remote size; total -uncompressed chunk size; size of the chunks that snapshot stored that -were not stored before). Both chunk sizes are recorded when the -snapshot's backup runs. +**`snapshot list`**: Show every snapshot known to this host — the union of the +local index and the backup destination store — with timestamps and three sizes +per snapshot (compressed remote size; total uncompressed chunk size; size of the +chunks that snapshot stored that were not stored before). Both chunk sizes are +recorded when the snapshot's backup runs. -Listing the destination store does **not** require the age secret key, -so it works in vaultik's intended configuration, where the backed-up -host holds only the public key. A host that has lost its local index -can still see what it has backed up. +Listing the destination store does **not** require the age secret key, so it +works in vaultik's intended configuration, where the backed-up host holds only +the public key. A host that has lost its local index can still see what it has +backed up. -What that host cannot see is a remote-only snapshot's name. The -snapshot ID is hashed at the storage boundary and the manifest records -only the hash, so hostname and snapshot name exist solely in the local -index and in the encrypted per-snapshot database. Snapshots found only -on the destination store are therefore listed as -`>` and show `` in -the uncompressed and "new chunk" columns, which come only from the -local index. Their timestamp and compressed size are real, -read from the manifest. +What that host cannot see is a remote-only snapshot's name. The snapshot ID is +hashed at the storage boundary and the manifest records only the hash, so +hostname and snapshot name exist solely in the local index and in the encrypted +per-snapshot database. Snapshots found only on the destination store are +therefore listed as `>` and show +`` in the uncompressed and "new chunk" columns, which come only +from the local index. Their timestamp and compressed size are real, read from +the manifest. -Snapshots in the local index with no counterpart on the destination -store are reported below the table as drift, with the `vaultik prune` -invocation that reconciles them. If the local index is bound to a -different destination than the configured one (after a change to -`storage_url`, or `s3.endpoint`, `s3.bucket` or `s3.prefix`), the -command says so instead and gives the same two ways out that `prune` -gives: revert the configuration, or run `vaultik database delete`. +Snapshots in the local index with no counterpart on the destination store are +reported below the table as drift, with the `vaultik prune` invocation that +reconciles them. If the local index is bound to a different destination than the +configured one (after a change to `storage_url`, or `s3.endpoint`, `s3.bucket` +or `s3.prefix`), the command says so instead and gives the same two ways out +that `prune` gives: revert the configuration, or run `vaultik database delete`. -If the destination store cannot be listed (unmounted volume, -permission denied, network down), the command warns, falls back to the -local index alone, and still exits zero. It does the same when the -destination answers that it has no `vaultik-destination.txt` while the -local index holds snapshots stored there, as when a volume mounted at -the destination is unplugged and leaves its empty mount point behind. -* `--json`: Output in JSON format. Each entry carries `locally_tracked` - (whether the snapshot is in the local index), `remote_key` (the full - 64-character storage key), and `remote_present` (whether it was seen - on the destination store, or `null` if the destination could not be - listed). Warnings about an unlistable destination, a local index - bound to a different destination (with both destinations and the - same two ways out), unreadable manifests, and a truncated listing all - go to stderr through the logger, so stdout stays a single parseable - document. +If the destination store cannot be listed (unmounted volume, permission denied, +network down), the command warns, falls back to the local index alone, and still +exits zero. It does the same when the destination answers that it has no +`vaultik-destination.txt` while the local index holds snapshots stored there, as +when a volume mounted at the destination is unplugged and leaves its empty mount +point behind. + +- `--json`: Output in JSON format. Each entry carries `locally_tracked` (whether + the snapshot is in the local index), `remote_key` (the full 64-character + storage key), and `remote_present` (whether it was seen on the destination + store, or `null` if the destination could not be listed). Warnings about an + unlistable destination, a local index bound to a different destination (with + both destinations and the same two ways out), unreadable manifests, and a + truncated listing all go to stderr through the logger, so stdout stays a + single parseable document. **`snapshot verify`**: Verify snapshot integrity. -* Default (shallow): checks that every blob the manifest lists is present in + +- Default (shallow): checks that every blob the manifest lists is present in storage with the size the manifest records, and that the encrypted database is present. It does not read blob contents. -* `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the +- `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the encrypted metadata database -* Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a +- Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a remote-only snapshot's remote key (or an unambiguous leading part of it) -* `--json`: Output results as JSON +- `--json`: Output results as JSON **`snapshot purge`**: Remove old snapshots based on criteria. Retention is -per-snapshot-name (`--keep-latest` keeps the latest of each name, not the -latest globally). -* `--keep-latest`: Keep only the most recent snapshot of each name -* `--older-than `: Remove snapshots older than duration (e.g. `30d`, - `4w`, `6mo`, `1y`; `m` is minutes, `mo` is months) -* `--snapshot `: Restrict to specific snapshot names (repeat for multiple) -* `--force`: Skip confirmation prompt +per-snapshot-name (`--keep-latest` keeps the latest of each name, not the latest +globally). -**`snapshot remove`**: Remove one snapshot. By default this removes the -snapshot from the local index and strips the snapshot's metadata from -the backup destination store. Blobs are NOT touched — deleting blobs -requires reading every remaining remote manifest (the destination store -may hold snapshots this host doesn't know about), which is what -`vaultik prune` does. The command prints the `vaultik prune` invocation -to run as a follow-up only when it removed the snapshot's metadata from -the destination store, and never under `--json`. An ID that names no -snapshot in the local index or on the destination store is an error; -with `--local-only`, the snapshot must be in the local index. Local row -cleanup (files, chunks, blobs the snapshot was the last referrer for) -runs automatically. If removing the snapshot's metadata from the -destination store fails (for example because the store is unreachable, -or the local index holds snapshots and the destination answers that it -has no `vaultik-destination.txt`), the local-DB removal still completes and a -warning is emitted; run `vaultik snapshot remove ` again -once the store is reachable to remove the snapshot's metadata from it -(`vaultik prune` does not). For a snapshot that is only on the -destination store, that failure is an error. To wipe everything on the -destination in one go, use `vaultik remote nuke --force`. -* `--local-only`: Skip remote cleanup; only touch the local index -* `--dry-run`: Show what would be deleted without deleting -* `--force`: Skip confirmation prompt -* `--json`: Output result as JSON. Also skips the confirmation prompt, as +- `--keep-latest`: Keep only the most recent snapshot of each name +- `--older-than `: Remove snapshots older than duration (e.g. `30d`, + `4w`, `6mo`, `1y`; `m` is minutes, `mo` is months) +- `--snapshot `: Restrict to specific snapshot names (repeat for multiple) +- `--force`: Skip confirmation prompt + +**`snapshot remove`**: Remove one snapshot. By default this removes the snapshot +from the local index and strips the snapshot's metadata from the backup +destination store. Blobs are NOT touched — deleting blobs requires reading every +remaining remote manifest (the destination store may hold snapshots this host +doesn't know about), which is what `vaultik prune` does. The command prints the +`vaultik prune` invocation to run as a follow-up only when it removed the +snapshot's metadata from the destination store, and never under `--json`. An ID +that names no snapshot in the local index or on the destination store is an +error; with `--local-only`, the snapshot must be in the local index. Local row +cleanup (files, chunks, blobs the snapshot was the last referrer for) runs +automatically. If removing the snapshot's metadata from the destination store +fails (for example because the store is unreachable, or the local index holds +snapshots and the destination answers that it has no `vaultik-destination.txt`), +the local-DB removal still completes and a warning is emitted; run +`vaultik snapshot remove ` again once the store is reachable to +remove the snapshot's metadata from it (`vaultik prune` does not). For a +snapshot that is only on the destination store, that failure is an error. To +wipe everything on the destination in one go, use `vaultik remote nuke --force`. + +- `--local-only`: Skip remote cleanup; only touch the local index +- `--dry-run`: Show what would be deleted without deleting +- `--force`: Skip confirmation prompt +- `--json`: Output result as JSON. Also skips the confirmation prompt, as `--force` does. **`snapshot restore`**: Restore files from a backup snapshot. -* Requires `VAULTIK_AGE_SECRET_KEY` environment variable -* Accepts a snapshot ID, or — for a snapshot only on the destination - store — its remote key (or an unambiguous leading part of it) as shown - by `snapshot list`. See - [restoring on another machine](#restoring-on-another-machine). -* Optional path arguments to restore specific files/directories (default: all). + +- Requires `VAULTIK_AGE_SECRET_KEY` environment variable +- Accepts a snapshot ID, or — for a snapshot only on the destination store — its + remote key (or an unambiguous leading part of it) as shown by `snapshot list`. + See [restoring on another machine](#restoring-on-another-machine). +- Optional path arguments to restore specific files/directories (default: all). A path that matches nothing in the snapshot is named in a warning, and the command exits non-zero after restoring the paths that match. -* Preserves file permissions, timestamps, ownership (ownership requires root), +- Preserves file permissions, timestamps, ownership (ownership requires root), symlinks, and empty directories -* `--verify`: After restoring, verify every restored file's chunk hashes - match. A file skipped under `--skip-errors` is reported as failed to - restore and is not verified. +- `--verify`: After restoring, verify every restored file's chunk hashes match. + A file skipped under `--skip-errors` is reported as failed to restore and is + not verified. -**`prune`**: Tidy up everything that isn't needed. Runs three passes: -(1) reconcile the local index against the destination store — any -local snapshot whose remote metadata is missing is dropped from the -local index; (2) delete orphaned local rows (files, chunks, blobs no -longer referenced by any completed snapshot); (3) list every remote -manifest on the destination store to compute the still-referenced blob -set and delete any blob not in that set that is older than -`prune_grace_period` (default `24h`). A newer one is kept and counted: it +**`prune`**: Tidy up everything that isn't needed. Runs three passes: (1) +reconcile the local index against the destination store — any local snapshot +whose remote metadata is missing is dropped from the local index; (2) delete +orphaned local rows (files, chunks, blobs no longer referenced by any completed +snapshot); (3) list every remote manifest on the destination store to compute +the still-referenced blob set and delete any blob not in that set that is older +than `prune_grace_period` (default `24h`). A newer one is kept and counted: it may belong to a backup still running, on this host or another sharing the -destination store, that has not yet uploaded its manifest. The grace period -does not protect a blob such a backup reuses from an earlier run that stopped -before its manifest: that blob keeps the time of its first upload, so it can -be deleted. Step (3) also deletes each directory under `metadata/` that has -no `manifest.json.zst`, as a backup leaves when it stops after uploading +destination store, that has not yet uploaded its manifest. The grace period does +not protect a blob such a backup reuses from an earlier run that stopped before +its manifest: that blob keeps the time of its first upload, so it can be +deleted. Step (3) also deletes each directory under `metadata/` that has no +`manifest.json.zst`, as a backup leaves when it stops after uploading `db.zst.age` but before its manifest, once every object in it is older than `prune_grace_period` and its snapshot is not in the local index. An object the -listing gives no time for keeps its directory, and a directory whose name is -not a remote key is left alone. Step (3) reads all remote manifests — network -cost scales with the number of snapshots. -`snapshot create --prune` runs the same cleanup automatically; this is the -manual entry point for the same work. If the destination store cannot -be listed, or the local index holds snapshots and the destination answers -that it has no `vaultik-destination.txt` (an unplugged volume's empty mount -point), prune fails and deletes nothing. `snapshot purge` does the same. If a -blob or such a directory cannot be deleted, the command (and +listing gives no time for keeps its directory, and a directory whose name is not +a remote key is left alone. Step (3) reads all remote manifests — network cost +scales with the number of snapshots. `snapshot create --prune` runs the same +cleanup automatically; this is the manual entry point for the same work. If the +destination store cannot be listed, or the local index holds snapshots and the +destination answers that it has no `vaultik-destination.txt` (an unplugged +volume's empty mount point), prune fails and deletes nothing. `snapshot purge` +does the same. If a blob or such a directory cannot be deleted, the command (and `snapshot create --prune`) exits non-zero. -* `--force`: Skip confirmation prompt -* `--json`: Output stats as JSON. Also skips the confirmation prompt, as - `--force` does. A blob that cannot be deleted is counted in - `blobs_failed`, and the document is printed before the non-zero exit. - A blob kept for being newer than `prune_grace_period`, or because the - listing of the destination gives no time for it, is counted in - `blobs_too_new` and not in `blobs_found`. A directory under `metadata/` - with no manifest is counted in `metadata_dirs_deleted` once deleted, or - in `metadata_dirs_failed` if it cannot be. + +- `--force`: Skip confirmation prompt +- `--json`: Output stats as JSON. Also skips the confirmation prompt, as + `--force` does. A blob that cannot be deleted is counted in `blobs_failed`, + and the document is printed before the non-zero exit. A blob kept for being + newer than `prune_grace_period`, or because the listing of the destination + gives no time for it, is counted in `blobs_too_new` and not in `blobs_found`. + A directory under `metadata/` with no manifest is counted in + `metadata_dirs_deleted` once deleted, or in `metadata_dirs_failed` if it + cannot be. **`info`**: Display system configuration, storage settings, encryption recipients, and local database statistics. -**`remote info`**: Show storage backend type and location plus detailed -remote storage inventory: per-snapshot metadata sizes, blob counts, and -orphaned blob detection. A name under `metadata/` that is not a remote -key is skipped with a warning and is not printed. If a listed -`manifest.json.zst` cannot be read, or sits under a skipped name, the -orphaned blob figures are reported as unknown; `--json` gives them as -`null`, lists the remote key of each unreadable manifest in -`unreadable_manifests` and counts the manifests under skipped names in -`skipped_manifest_count`. An unreadable manifest also leaves its -snapshot's blob count and blob size unknown, `null` in `--json`. A file -under `blobs/` whose name is not a blob hash is not counted as a blob, -since `prune` never deletes it. -* `--json`: Output as JSON +**`remote info`**: Show storage backend type and location plus detailed remote +storage inventory: per-snapshot metadata sizes, blob counts, and orphaned blob +detection. A name under `metadata/` that is not a remote key is skipped with a +warning and is not printed. If a listed `manifest.json.zst` cannot be read, or +sits under a skipped name, the orphaned blob figures are reported as unknown; +`--json` gives them as `null`, lists the remote key of each unreadable manifest +in `unreadable_manifests` and counts the manifests under skipped names in +`skipped_manifest_count`. An unreadable manifest also leaves its snapshot's blob +count and blob size unknown, `null` in `--json`. A file under `blobs/` whose +name is not a blob hash is not counted as a blob, since `prune` never deletes +it. -**`remote nuke`**: Delete every object under `metadata/` and `blobs/` on -the backup destination store, whatever its name, and then -`vaultik-destination.txt`, leaving the bucket prefix empty. Destructive -and irreversible. This is the single supported way to wipe the entire -destination store. A snapshot is removed from the local index only once -its metadata is deleted from the destination store. If the destination -store cannot be listed, or the local index holds snapshots and the -destination answers that it has no `vaultik-destination.txt` (an unplugged -volume's empty mount point), the command fails before it removes anything. If a -listing afterwards still finds anything under `metadata/` or `blobs/`, -the command names each object left and exits non-zero; running it again -deletes what is left. -* `--force`: Required to confirm destruction. +- `--json`: Output as JSON -**`database delete`**: Delete the local SQLite state database file -entirely. Remote storage is unaffected; the next backup will do a full -scan and re-deduplicate against existing remote blobs, and the local -index will re-bind to the currently configured storage destination. -Use this after changing `storage_url`, or `s3.endpoint`, `s3.bucket` or -`s3.prefix`, to a different destination. -* `--force`: Skip confirmation prompt +**`remote nuke`**: Delete every object under `metadata/` and `blobs/` on the +backup destination store, whatever its name, and then `vaultik-destination.txt`, +leaving the bucket prefix empty. Destructive and irreversible. This is the +single supported way to wipe the entire destination store. A snapshot is removed +from the local index only once its metadata is deleted from the destination +store. If the destination store cannot be listed, or the local index holds +snapshots and the destination answers that it has no `vaultik-destination.txt` +(an unplugged volume's empty mount point), the command fails before it removes +anything. If a listing afterwards still finds anything under `metadata/` or +`blobs/`, the command names each object left and exits non-zero; running it +again deletes what is left. + +- `--force`: Required to confirm destruction. + +**`database delete`**: Delete the local SQLite state database file entirely. +Remote storage is unaffected; the next backup will do a full scan and +re-deduplicate against existing remote blobs, and the local index will re-bind +to the currently configured storage destination. Use this after changing +`storage_url`, or `s3.endpoint`, `s3.bucket` or `s3.prefix`, to a different +destination. + +- `--force`: Skip confirmation prompt --- ## storage backends -vaultik supports three storage backends, selected via the `storage_url` config field: +vaultik supports three storage backends, selected via the `storage_url` config +field: **S3** (`s3://bucket/prefix?endpoint=host®ion=us-east-1`): Any S3-compatible object store. Credentials are read from `s3.access_key_id` and `s3.secret_access_key` in the config file. -**Local filesystem** (`file:///path/to/backup`): Stores blobs and metadata on -a local or mounted filesystem. Useful for testing or backing up to a NAS. The -path must be absolute; a relative one is refused when the config loads. -Percent-escapes in the path are decoded, so `my%20stick` and `my stick` name -the same directory. The directory must exist before the first backup: vaultik -never creates it. A directory on a removable volume is gone while the volume is -not mounted, so a backup then fails instead of writing to the disk under the -mount point. The mount point itself stays behind when the volume is unplugged: -there only `vaultik-destination.txt` stops such a backup, and only while the -local index holds snapshots. +**Local filesystem** (`file:///path/to/backup`): Stores blobs and metadata on a +local or mounted filesystem. Useful for testing or backing up to a NAS. The path +must be absolute; a relative one is refused when the config loads. +Percent-escapes in the path are decoded, so `my%20stick` and `my stick` name the +same directory. The directory must exist before the first backup: vaultik never +creates it. A directory on a removable volume is gone while the volume is not +mounted, so a backup then fails instead of writing to the disk under the mount +point. The mount point itself stays behind when the volume is unplugged: there +only `vaultik-destination.txt` stops such a backup, and only while the local +index holds snapshots. **Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud providers. Requires rclone to be configured separately (`rclone config`). An upload cut off part-way leaves nothing under the object's name on S3, which shows an object only once its upload has completed, and on the local filesystem -backend, which writes a temporary file and renames it into place. Rclone -remotes with a server-side move (local and sftp among them) are written under a +backend, which writes a temporary file and renames it into place. Rclone remotes +with a server-side move (local and sftp among them) are written under a temporary name ending in `.partial` and moved into place. Rclone remotes without one are written in place: where such a remote shows a file while it is still being written, a killed upload can leave a truncated object under its name, @@ -578,12 +583,11 @@ and can be deleted. A destination shared between hosts or users, each with its own local index, relies on `prune_grace_period` to keep one's prune from deleting the blobs of -another's backup still running, or its directory under `metadata/` that has -no manifest yet, so it must be longer than the longest backup run. It does not -protect a blob that backup reuses from an earlier run that -stopped before its manifest, which the prune can delete. To not depend on the -grace period, give each host its own prefix (or directory). See -[locking](#locking). +another's backup still running, or its directory under `metadata/` that has no +manifest yet, so it must be longer than the longest backup run. It does not +protect a blob that backup reuses from an earlier run that stopped before its +manifest, which the prune can delete. To not depend on the grace period, give +each host its own prefix (or directory). See [locking](#locking). Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is still supported for backward compatibility. `storage_url` takes precedence if @@ -593,8 +597,8 @@ both are set. ## architecture -For an implementation-level view of the internals — the data model, the -`fx` dependency-injection wiring, and the scanner — see +For an implementation-level view of the internals — the data model, the `fx` +dependency-injection wiring, and the scanner — see [`ARCHITECTURE.md`](ARCHITECTURE.md). ### remote storage layout @@ -610,44 +614,45 @@ For an implementation-level view of the internals — the data model, the └── manifest.json.zst # Unencrypted blob list (for pruning) ``` -* `vaultik-destination.txt` holds one fixed line of text, with no host - name, path or time. While the local index holds snapshots stored - there, a destination that answers a look-up of it with not found is - treated as not present: `snapshot create` refuses to write to it, - `prune`, `snapshot purge` and `remote nuke` fail, and `snapshot list` - and `snapshot remove` behave as for a destination that cannot be - listed. This keeps an unplugged volume's empty mount point from being - taken for a destination whose snapshots are all gone. Mount - the volume if it is removable; if the destination was emptied on - purpose, run `vaultik database delete` and back up again. - When the look-up fails for any reason other than not found, the check - is skipped and the command goes on. S3 refuses it on every run to - credentials that allow only PUT, so a host with such credentials does - not notice a destination emptied since its last backup, and its next - backup would leave out the data its local index records as already - stored. After the destination is emptied, run +- `vaultik-destination.txt` holds one fixed line of text, with no host name, + path or time. While the local index holds snapshots stored there, a + destination that answers a look-up of it with not found is treated as not + present: `snapshot create` refuses to write to it, `prune`, `snapshot purge` + and `remote nuke` fail, and `snapshot list` and `snapshot remove` behave as + for a destination that cannot be listed. This keeps an unplugged volume's + empty mount point from being taken for a destination whose snapshots are all + gone. Mount the volume if it is removable; if the destination was emptied on + purpose, run `vaultik database delete` and back up again. When the look-up + fails for any reason other than not found, the check is skipped and the + command goes on. S3 refuses it on every run to credentials that allow only + PUT, so a host with such credentials does not notice a destination emptied + since its last backup, and its next backup would leave out the data its local + index records as already stored. After the destination is emptied, run `vaultik database delete` on such a host before it backs up again. -* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash -* `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted) - containing all file metadata, chunk mappings, and relationships for the snapshot -* `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling +- Blobs are two-level directory sharded using the first 4 hex chars of the blob + hash +- `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted) + containing all file metadata, chunk mappings, and relationships for the + snapshot +- `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling pruning without the private key Snapshot IDs follow the human-readable format `__` (e.g. `server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the -destination store in plaintext. Each snapshot's metadata directory is named -with its ``, a one-way double SHA-256 hash of the ID, so a plain -listing of the store shows no hostname or snapshot name. The hash uses no -secret, though, so an observer who guesses a candidate hostname and snapshot -name can hash it and confirm the snapshot is present; the remote key keeps -names out of a listing but does not hide them from a guess. The backup time is -not hidden either: manifest.json.zst carries a plaintext timestamp, and object -modification times are visible at the storage layer regardless. For example, +destination store in plaintext. Each snapshot's metadata directory is named with +its ``, a one-way double SHA-256 hash of the ID, so a plain listing +of the store shows no hostname or snapshot name. The hash uses no secret, +though, so an observer who guesses a candidate hostname and snapshot name can +hash it and confirm the snapshot is present; the remote key keeps names out of a +listing but does not hide them from a guess. The backup time is not hidden +either: manifest.json.zst carries a plaintext timestamp, and object modification +times are visible at the storage layer regardless. For example, `server1_home_2025-06-01T12:00:00Z` is stored under `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`. See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the -derivation, and [Security Considerations](docs/REPOSTRUCTURE.md#security-considerations) +derivation, and +[Security Considerations](docs/REPOSTRUCTURE.md#security-considerations) (including [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks)) for what the format does and does not protect. @@ -680,89 +685,88 @@ format does and does not protect. 3. List all blobs in storage 4. Delete any blob not in the referenced set that is older than `prune_grace_period` -5. Delete each directory under `metadata/` with no manifest whose objects - are all older than `prune_grace_period`, unless its snapshot is in the - local index +5. Delete each directory under `metadata/` with no manifest whose objects are + all older than `prune_grace_period`, unless its snapshot is in the local + index ### chunking and deduplication -* Content-defined chunking using the FastCDC algorithm -* Average chunk size: configurable (default 10MB) -* Deduplication at file level (unchanged files skipped, unless a chunk - the file lists is held by no uploaded blob) and chunk level (identical - chunks across files stored once) -* Multiple chunks packed into blobs to reduce object count +- Content-defined chunking using the FastCDC algorithm +- Average chunk size: configurable (default 10MB) +- Deduplication at file level (unchanged files skipped, unless a chunk the file + lists is held by no uploaded blob) and chunk level (identical chunks across + files stored once) +- Multiple chunks packed into blobs to reduce object count ### encryption -* Asymmetric encryption using age (X25519 + ChaCha20-Poly1305) -* Only the public key is needed on the source host -* Each blob and each metadata database is encrypted independently -* Multiple recipients supported (encrypt to multiple keys) +- Asymmetric encryption using age (X25519 + ChaCha20-Poly1305) +- Only the public key is needed on the source host +- Each blob and each metadata database is encrypted independently +- Multiple recipients supported (encrypt to multiple keys) ### compression -* zstd compression at configurable level (1-19, default 3). The level is - accepted as 1-19 but maps onto zstd's four internal speed presets: - 1-2 fastest, 3-5 default, 6-9 better, 10-19 best. Levels within the - same band compress identically. -* Applied before encryption at the blob level +- zstd compression at configurable level (1-19, default 3). The level is + accepted as 1-19 but maps onto zstd's four internal speed presets: 1-2 + fastest, 3-5 default, 6-9 better, 10-19 best. Levels within the same band + compress identically. +- Applied before encryption at the blob level --- ## configuration reference -Run `vaultik config init` to generate a fully commented config file; a -complete annotated example also lives in -[`config.example.yml`](config.example.yml). Key fields: +Run `vaultik config init` to generate a fully commented config file; a complete +annotated example also lives in [`config.example.yml`](config.example.yml). Key +fields: -| Field | Default | Description | -|-------|---------|-------------| -| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty | -| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. | -| `snapshots` | (required) | Named snapshot definitions with paths and excludes | -| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) | -| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) | -| `exclude` | | Global exclude patterns (applied to all snapshots) | -| `chunk_size` | `10MB` | Average chunk size for content-defined chunking | -| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` | -| `compression_level` | `3` | zstd compression level (1-19) | -| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once | -| `hostname` | system hostname | Hostname used in snapshot IDs | -| `index_path` | platform data dir | Local SQLite index path | +| Field | Default | Description | +| -------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty | +| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. | +| `snapshots` | (required) | Named snapshot definitions with paths and excludes | +| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) | +| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) | +| `exclude` | | Global exclude patterns (applied to all snapshots) | +| `chunk_size` | `10MB` | Average chunk size for content-defined chunking | +| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` | +| `compression_level` | `3` | zstd compression level (1-19) | +| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once | +| `hostname` | system hostname | Hostname used in snapshot IDs | +| `index_path` | platform data dir | Local SQLite index path | --- ## limitations -* **No extended attributes (xattrs).** ACLs, macOS Finder metadata, - quarantine flags, SELinux labels, and other extended attributes are not - backed up or restored. -* **No hard link detection.** Two hard links to the same inode are backed - up as independent files. Content deduplication means the data is stored - once, but the hard link relationship is lost on restore. -* **No sparse file support.** Sparse files are fully materialized during - backup. A 100 GB sparse VM disk that is mostly zeros will consume the - full (compressed) size in storage. -* **No bandwidth limiting.** Uploads and downloads use whatever bandwidth - is available. There is no `--bwlimit` flag yet. -* **No parallel blob downloads during restore.** Blobs are fetched - sequentially. Restore speed is bound by single-stream throughput. -* **Device nodes, named pipes, and sockets are silently skipped.** Only - regular files, directories, and symlinks are backed up. -* **Before 1.0, an update can make the local index unusable.** Vaultik - supports schema migrations, but none are added before 1.0 because - there is no installed base yet. If an update leaves your local index - unusable, run `vaultik database delete` and then a full backup; remote - storage is unaffected. After 1.0, the local index is migrated when - vaultik is updated. See - [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). -* **Files that change during backup may be inconsistent.** There is no - filesystem snapshot or freeze. If a file is modified between the scan - and chunk phases, the backed-up copy may reflect a partial write. -* **Ownership restoration requires root.** File uid/gid are recorded - and restored, but `chown` requires elevated privileges. Without root, - files are restored with the current user's ownership. +- **No extended attributes (xattrs).** ACLs, macOS Finder metadata, quarantine + flags, SELinux labels, and other extended attributes are not backed up or + restored. +- **No hard link detection.** Two hard links to the same inode are backed up as + independent files. Content deduplication means the data is stored once, but + the hard link relationship is lost on restore. +- **No sparse file support.** Sparse files are fully materialized during backup. + A 100 GB sparse VM disk that is mostly zeros will consume the full + (compressed) size in storage. +- **No bandwidth limiting.** Uploads and downloads use whatever bandwidth is + available. There is no `--bwlimit` flag yet. +- **No parallel blob downloads during restore.** Blobs are fetched sequentially. + Restore speed is bound by single-stream throughput. +- **Device nodes, named pipes, and sockets are silently skipped.** Only regular + files, directories, and symlinks are backed up. +- **Before 1.0, an update can make the local index unusable.** Vaultik supports + schema migrations, but none are added before 1.0 because there is no installed + base yet. If an update leaves your local index unusable, run + `vaultik database delete` and then a full backup; remote storage is + unaffected. After 1.0, the local index is migrated when vaultik is updated. + See [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). +- **Files that change during backup may be inconsistent.** There is no + filesystem snapshot or freeze. If a file is modified between the scan and + chunk phases, the backed-up copy may reflect a partial write. +- **Ownership restoration requires root.** File uid/gid are recorded and + restored, but `chown` requires elevated privileges. Without root, files are + restored with the current user's ownership. --- @@ -772,161 +776,150 @@ Work planned after 1.0. Loosely ordered by priority. ### correctness and operability -* **Outside security audit.** Before 1.0 the encryption and - blob-generation code was reviewed: every bug the review found was - fixed, and the risks it accepted are listed in - [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside - audit has been done. age + zstd + content-defined chunking - is mostly off-the-shelf pieces, but the seams (key handling, - recipient parsing, manifest trust boundary, restore-time identity - validation) need an outside read. -* **Verify restored content end-to-end in CI.** The current - integration test does this for a small synthetic snapshot but - not at scale. A nightly job against a multi-GB representative - snapshot would catch silent regressions in the chunker, packer, - or restore planner. +- **Outside security audit.** Before 1.0 the encryption and blob-generation code + was reviewed: every bug the review found was fixed, and the risks it accepted + are listed in [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks). No + outside audit has been done. age + zstd + content-defined chunking is mostly + off-the-shelf pieces, but the seams (key handling, recipient parsing, manifest + trust boundary, restore-time identity validation) need an outside read. +- **Verify restored content end-to-end in CI.** The current integration test + does this for a small synthetic snapshot but not at scale. A nightly job + against a multi-GB representative snapshot would catch silent regressions in + the chunker, packer, or restore planner. ### performance -* **Parallel blob downloads during restore.** Single-stream right - now. With a fast S3 endpoint and a multi-core machine restore is - bound by per-blob fetch + decrypt + decompress; running N of - those in parallel against the disk cache would close most of the - remaining gap. Needs to interact correctly with the locality - planner and sweeper. -* **Bandwidth limiting (`--bwlimit`).** Both upload and download. - Useful for backing up over a shared link. Tricky to make work - correctly with the parallel-download story. -* **Restart of interrupted restore.** Today restore is restartable - in the sense that re-running it overwrites partial output; it - doesn't resume from where it stopped or skip already-present - files. A `--resume` mode that checks targets before fetching - blobs would matter for very large restores. -* **Daemon mode.** A long-running mode that watches for file - changes so frequent backups, such as hourly, skip the full scan. - It adds little for the usual runs from cron every 12 to 36 hours. - See [issue #204](https://git.eeqj.de/sneak/vaultik/issues/204). +- **Parallel blob downloads during restore.** Single-stream right now. With a + fast S3 endpoint and a multi-core machine restore is bound by per-blob fetch + + decrypt + decompress; running N of those in parallel against the disk cache + would close most of the remaining gap. Needs to interact correctly with the + locality planner and sweeper. +- **Bandwidth limiting (`--bwlimit`).** Both upload and download. Useful for + backing up over a shared link. Tricky to make work correctly with the + parallel-download story. +- **Restart of interrupted restore.** Today restore is restartable in the sense + that re-running it overwrites partial output; it doesn't resume from where it + stopped or skip already-present files. A `--resume` mode that checks targets + before fetching blobs would matter for very large restores. +- **Daemon mode.** A long-running mode that watches for file changes so frequent + backups, such as hourly, skip the full scan. It adds little for the usual runs + from cron every 12 to 36 hours. See + [issue #204](https://git.eeqj.de/sneak/vaultik/issues/204). ### usability -* **Man pages and richer `--help` examples.** Cobra generates - basic help; man pages would be a separate target. -* **`vaultik snapshot diff `** — show which files changed - between two snapshots without restoring either. -* **Status reporting hook for `--cron`.** When a backup fails - silently in cron, the user has no idea. A configurable - webhook / email / `notify-send` hook on completion (success and - failure) would close the loop. +- **Man pages and richer `--help` examples.** Cobra generates basic help; man + pages would be a separate target. +- **`vaultik snapshot diff `** — show which files changed between two + snapshots without restoring either. +- **Status reporting hook for `--cron`.** When a backup fails silently in cron, + the user has no idea. A configurable webhook / email / `notify-send` hook on + completion (success and failure) would close the loop. ### infrastructure -* **Schema migrations after 1.0.** Migrations are supported, but none - are added before 1.0 because there is no installed base yet. After - 1.0, each schema change is a new migration, so an existing local - index is migrated when vaultik is updated (see - [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations)). -* **Storage backend coverage tests.** S3, file://, and rclone:// - all share the Storer interface but the rclone path is the least - exercised in CI. +- **Schema migrations after 1.0.** Migrations are supported, but none are added + before 1.0 because there is no installed base yet. After 1.0, each schema + change is a new migration, so an existing local index is migrated when vaultik + is updated (see [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations)). +- **Storage backend coverage tests.** S3, file://, and rclone:// all share the + Storer interface but the rclone path is the least exercised in CI. --- ## output style -Every command's user-facing output is governed by `internal/ui`, in one -of two ways. Color is enabled when the stream written to is a TTY and -the `NO_COLOR` environment variable is unset (https://no-color.org/). +Every command's user-facing output is governed by `internal/ui`, in one of two +ways. Color is enabled when the stream written to is a TTY and the `NO_COLOR` +environment variable is unset (https://no-color.org/). -* **Status, progress, warnings, and errors** go through the `internal/ui` +- **Status, progress, warnings, and errors** go through the `internal/ui` message methods below: marker-prefixed, colored on a TTY, and — except - warnings and errors — silenced by `--quiet`. This is the operational - narration of the long-running commands (`snapshot create`, `prune`, - `snapshot restore`, and the like) and the confirmations of - `config init`, `config set`, and `database delete`. -* **The data a command exists to produce** is written plain, with no - marker and no color, because a marker would corrupt a table or a - parsed document. This covers the `version`, `info`, `remote info` and - `snapshot verify` reports, the `snapshot list` table, `config get` - values, and every `--json` document. `--quiet` silences the human - reports and tables (`version`, `info`, `remote info`, `snapshot - verify`, `snapshot list`) but never the machine-consumed `config get` - value or the `--json` documents, which a script depends on. The - confirmation prompts of `database delete`, `prune`, `snapshot purge` - and `snapshot remove`, each with what it asks to delete, are also - written this way and always shown: they are an interactive exchange - the operator must see. + warnings and errors — silenced by `--quiet`. This is the operational narration + of the long-running commands (`snapshot create`, `prune`, `snapshot restore`, + and the like) and the confirmations of `config init`, `config set`, and + `database delete`. +- **The data a command exists to produce** is written plain, with no marker and + no color, because a marker would corrupt a table or a parsed document. This + covers the `version`, `info`, `remote info` and `snapshot verify` reports, the + `snapshot list` table, `config get` values, and every `--json` document. + `--quiet` silences the human reports and tables (`version`, `info`, + `remote info`, `snapshot verify`, `snapshot list`) but never the + machine-consumed `config get` value or the `--json` documents, which a script + depends on. The confirmation prompts of `database delete`, `prune`, + `snapshot purge` and `snapshot remove`, each with what it asks to delete, are + also written this way and always shown: they are an interactive exchange the + operator must see. `internal/ui` writes to stdout; it is the output the user asked for. The -exceptions are the startup banner and the error a failed command ends -with, which go to stderr. Structured log records are a different thing -and go through `internal/log`, which writes to stderr (see "stdout and -stderr" above). +exceptions are the startup banner and the error a failed command ends with, +which go to stderr. Structured log records are a different thing and go through +`internal/log`, which writes to stderr (see "stdout and stderr" above). Message classes: -| Class | Marker | Alignment | Use for | -|-------|--------|-----------|---------| -| Banner | none | column 0 | The startup line printed once per invocation, on stderr | -| Begin | `》` (white) | column 0 | An operation is about to start (present-continuous verb) | -| Complete | `》` (green) | column 0 | An operation just finished (past-tense verb) | -| Info | `》` (white) | column 0 | Neutral status update | -| Notice | `》` (cyan) | column 0 | Important note that is not a warning | -| Warning | `⚠️ Warning:` (orange/yellow) | column 0 | Recoverable problem | -| Error | `🛑 ERROR:` (red) | column 0 | Operation aborted | -| Progress | ` 》` (white) | column 2 | Heartbeat or per-item status during a long-running operation | -| Detail | ` 》` (white) | column 2 | Continuation/sub-line of a preceding Complete (visually identical to Progress) | +| Class | Marker | Alignment | Use for | +| -------- | ------------------------------ | --------- | ------------------------------------------------------------------------------ | +| Banner | none | column 0 | The startup line printed once per invocation, on stderr | +| Begin | `》` (white) | column 0 | An operation is about to start (present-continuous verb) | +| Complete | `》` (green) | column 0 | An operation just finished (past-tense verb) | +| Info | `》` (white) | column 0 | Neutral status update | +| Notice | `》` (cyan) | column 0 | Important note that is not a warning | +| Warning | `⚠️ Warning:` (orange/yellow) | column 0 | Recoverable problem | +| Error | `🛑 ERROR:` (red) | column 0 | Operation aborted | +| Progress | ` 》` (white) | column 2 | Heartbeat or per-item status during a long-running operation | +| Detail | ` 》` (white) | column 2 | Continuation/sub-line of a preceding Complete (visually identical to Progress) | Conventions: -* Messages are complete English sentences ending with a period. -* Fully qualify terms — say "backup destination store" instead of - "storage", "snapshot source files enumeration" instead of "scan", - "local index database" instead of "database". -* Every operation that emits a Complete also emits a corresponding - Begin. Operations that print only a Begin (because completion is - obvious from a later Begin) should be rare and intentional. -* Use natural verb tense to signal state: "Uploading" for Begin, - "Uploaded" for Complete. Never write the words "begin" or "complete" - in the body — the marker color already conveys that. -* All elapsed and remaining-time fields are explicitly scoped to their - subject: write "blob upload elapsed: 30s, blob upload ETA: 03:15:00 - (est remain 14s)", never just "elapsed 30s, ETA 14s". -* "ETA" means an absolute clock time (when the operation will finish), - not a remaining-duration. Use `ui.Time()` for the former and - `ui.Duration()` for the latter, and label both. -* `ui.Time` formats same-day times as `HH:MM:SS` and other-day times as +- Messages are complete English sentences ending with a period. +- Fully qualify terms — say "backup destination store" instead of "storage", + "snapshot source files enumeration" instead of "scan", "local index database" + instead of "database". +- Every operation that emits a Complete also emits a corresponding Begin. + Operations that print only a Begin (because completion is obvious from a later + Begin) should be rare and intentional. +- Use natural verb tense to signal state: "Uploading" for Begin, "Uploaded" for + Complete. Never write the words "begin" or "complete" in the body — the marker + color already conveys that. +- All elapsed and remaining-time fields are explicitly scoped to their subject: + write "blob upload elapsed: 30s, blob upload ETA: 03:15:00 (est remain 14s)", + never just "elapsed 30s, ETA 14s". +- "ETA" means an absolute clock time (when the operation will finish), not a + remaining-duration. Use `ui.Time()` for the former and `ui.Duration()` for the + latter, and label both. +- `ui.Time` formats same-day times as `HH:MM:SS` and other-day times as `YYYY-MM-DD HH:MM:SS`. No timezone — local time is implied. -Value colorizers in `internal/ui` colorize specific value types -consistently. Compose messages from these helpers rather than embedding -ANSI escapes inline: +Value colorizers in `internal/ui` colorize specific value types consistently. +Compose messages from these helpers rather than embedding ANSI escapes inline: -| Helper | Color | Use for | -|--------|-------|---------| -| `Hex` | cyan | Blob hashes, chunk hashes (truncated to 12 chars + `...`) | -| `Snapshot` | bold cyan | Snapshot IDs (untruncated) | -| `Path` | blue | Filesystem paths | -| `Size` | magenta | Byte counts (human-readable) | -| `Speed` | magenta | Bytes-per-second rates | -| `Duration` | yellow | Elapsed or remaining time | -| `Time` | yellow | Absolute clock times | -| `Count` | magenta | Integer counts with thousands separators | -| `Percent` | magenta | Percentages | +| Helper | Color | Use for | +| ---------- | --------- | --------------------------------------------------------- | +| `Hex` | cyan | Blob hashes, chunk hashes (truncated to 12 chars + `...`) | +| `Snapshot` | bold cyan | Snapshot IDs (untruncated) | +| `Path` | blue | Filesystem paths | +| `Size` | magenta | Byte counts (human-readable) | +| `Speed` | magenta | Bytes-per-second rates | +| `Duration` | yellow | Elapsed or remaining time | +| `Time` | yellow | Absolute clock times | +| `Count` | magenta | Integer counts with thousands separators | +| `Percent` | magenta | Percentages | -When `NO_COLOR` is set or output is not a TTY, all helpers return plain -text and the marker prefixes (`》`, `Warning:`, `ERROR:`) emit without -ANSI escapes. The emoji prefixes on Warning and Error are always emitted -regardless of color setting (emoji are not color). +When `NO_COLOR` is set or output is not a TTY, all helpers return plain text and +the marker prefixes (`》`, `Warning:`, `ERROR:`) emit without ANSI escapes. The +emoji prefixes on Warning and Error are always emitted regardless of color +setting (emoji are not color). ## requirements -* Go 1.26 or later -* Docker, with a reachable daemon, to test, lint, check, or commit: - `script/test` and `script/lint` build the `test` and `lint` phases of - the `Dockerfile`, and `make check` and the pre-commit hook both run - them. A `golangci-lint` installed on `PATH` is not a substitute and is - never used on a host, whatever its version. -* S3-compatible object storage (or local filesystem, or rclone remote) +- Go 1.26 or later +- Docker, with a reachable daemon, to test, lint, check, or commit: + `script/test` and `script/lint` build the `test` and `lint` phases of the + `Dockerfile`, and `make check` and the pre-commit hook both run them. A + `golangci-lint` installed on `PATH` is not a substitute and is never used on a + host, whatever its version. +- S3-compatible object storage (or local filesystem, or rclone remote) ## development workflow @@ -939,7 +932,8 @@ All changes follow this workflow. No exceptions. 5. Fix linting errors (`make lint`). 6. Update documentation and README as required by the change. 7. Format code (`make fmt`). -8. Run `make check` (lint + fmt-check + test). Fix any issues. Repeat until clean. +8. Run `make check` (lint + fmt-check + test). Fix any issues. Repeat until + clean. 9. Commit on the branch. 10. Merge to `main`. 11. Push. @@ -953,127 +947,114 @@ Repository policies for AI agents are in [`AGENTS.md`](AGENTS.md). This repository adheres to the [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) standard: normalized scripts in `script/` are the entrypoints for the -development workflow, and the Makefile targets are thin shims that call -them. We provide: +development workflow, and the Makefile targets are thin shims that call them. We +provide: -* `script/bootstrap` — install all development dependencies (Go, Go - module download, and node, yarn and `prettier` for Markdown). Node - comes from the host, or else at a pinned version through `nvm`. A - host without Go gets the `go.mod` version through - `script/install-go`, in `.tool/go`, which `script/bootstrap` itself, - the `Makefile`, `script/fmt`, `script/fmt-check`, `script/precommit` - and `script/release` add to their `PATH`. It - deliberately does not install `golangci-lint`; see `script/lint` - below. -* `script/setup` — make a fresh clone ready for development: runs +- `script/bootstrap` — install all development dependencies (Go, Go module + download, and node, yarn and `prettier` for Markdown). Node comes from the + host, or else at a pinned version through `nvm`. A host without Go gets the + `go.mod` version through `script/install-go`, in `.tool/go`, which + `script/bootstrap` itself, the `Makefile`, `script/fmt`, `script/fmt-check`, + `script/precommit` and `script/release` add to their `PATH`. It deliberately + does not install `golangci-lint`; see `script/lint` below. +- `script/setup` — make a fresh clone ready for development: runs `script/bootstrap`, then `script/install-precommit` -* `script/projectname` — print the project name (used for the Docker - image tag) -* `script/version` — print the version string to bake into the binary. - The `Makefile`'s `LDFLAGS` call this. See [releasing](#releasing) for - the rules. -* `script/install-goreleaser` — install the pinned `goreleaser` into - `.tool/bin` from a sha256-verified release archive. Idempotent, and - called by `script/bootstrap`; the release workflow calls it directly - because it needs `goreleaser` but not the Docker daemon - `script/bootstrap` insists on. -* `script/install-go` — install the Go toolchain named by `go.mod`'s - `go` directive into `.tool/go` from a sha256-verified `go.dev` - archive, for Linux or macOS on amd64 or arm64. Idempotent. On a CI - runner it also puts `.tool/go/bin` on `PATH` for the steps that - follow. Called by the release workflow, which needs a host Go for - `goreleaser` to shell out to, and by `script/bootstrap` on a host - without Go. `actions/setup-go` is not used because it verifies the - downloaded toolchain against no value in this repo. Bumping Go edits - `go.mod`, the checksums in this script, and the two `golang` digests - in the `Dockerfile` together. -* `script/release` — cross-compile and publish the release artifacts - with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose - version is not the pinned one, because a different version would build - a different release from the same tag. -* `script/release-snapshot` — the same build with no publishing and no - tagging, into `./dist` -* `script/test` — run the test suite by building the `test` phase of - the `Dockerfile` (verbose rerun on failure). This runs *everything*: - there is no separate integration target and no build-tagged subset - held back, so the full round-trip tests in - `internal/vaultik/integration_test.go` run on every invocation. The - 90s `-timeout` is a hang backstop rather than a performance budget; it - applies per test binary, to test execution only. -* `script/lint` — lint by building the `lint` phase of the - `Dockerfile`, which runs `golangci-lint config verify` and then - `golangci-lint run --config .golangci.yml ./...` as build steps in - the digest-pinned `golangci-lint` image. Nothing lints on the host. - That `FROM` line is the only pin of the linter version, and it changes - in the same commit as a re-vendored `.golangci.yml`. -* `script/lint-fix` — apply the linter's autofixes (rewrites files), - using the same pinned image, parsed out of the `lint` phase's `FROM` - line. It cannot be a build step, because fixes have to land in the - worktree, so it bind-mounts the tree into a `docker run` and therefore - needs a *local* daemon. It is a developer convenience and never a - gate: no gate reads its exit status. Run `make lint` afterwards to - find out whether the tree is clean. -* `script/fmt` — format all code (writes): Go with `go fmt`, Markdown - with `prettier` -* `script/fmt-check` — check formatting (read-only). It runs `gofmt` on - the host, over every Go file outside `.tool`, and `prettier --check` - over every Markdown file. -* `script/check` — run `script/test`, `script/lint`, and - `script/fmt-check`. -* `script/docker` — build the Docker image tagged via - `script/projectname`, stamped with the version - `git describe --tags --always --dirty` gives on the host, or `unknown` - outside a git checkout. The image's build stage depends on the `lint` - and `test` phases, so this lints and tests too. -* `script/cibuild` — CI entrypoint: runs `script/bootstrap`, - `script/check`, and then the same image build as `script/docker`. - `.gitea/workflows/check.yml` runs it on every push. +- `script/projectname` — print the project name (used for the Docker image tag) +- `script/version` — print the version string to bake into the binary. The + `Makefile`'s `LDFLAGS` call this. See [releasing](#releasing) for the rules. +- `script/install-goreleaser` — install the pinned `goreleaser` into `.tool/bin` + from a sha256-verified release archive. Idempotent, and called by + `script/bootstrap`; the release workflow calls it directly because it needs + `goreleaser` but not the Docker daemon `script/bootstrap` insists on. +- `script/install-go` — install the Go toolchain named by `go.mod`'s `go` + directive into `.tool/go` from a sha256-verified `go.dev` archive, for Linux + or macOS on amd64 or arm64. Idempotent. On a CI runner it also puts + `.tool/go/bin` on `PATH` for the steps that follow. Called by the release + workflow, which needs a host Go for `goreleaser` to shell out to, and by + `script/bootstrap` on a host without Go. `actions/setup-go` is not used + because it verifies the downloaded toolchain against no value in this repo. + Bumping Go edits `go.mod`, the checksums in this script, and the two `golang` + digests in the `Dockerfile` together. +- `script/release` — cross-compile and publish the release artifacts with the + pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose version is not the + pinned one, because a different version would build a different release from + the same tag. +- `script/release-snapshot` — the same build with no publishing and no tagging, + into `./dist` +- `script/test` — run the test suite by building the `test` phase of the + `Dockerfile` (verbose rerun on failure). This runs _everything_: there is no + separate integration target and no build-tagged subset held back, so the full + round-trip tests in `internal/vaultik/integration_test.go` run on every + invocation. The 90s `-timeout` is a hang backstop rather than a performance + budget; it applies per test binary, to test execution only. +- `script/lint` — lint by building the `lint` phase of the `Dockerfile`, which + runs `golangci-lint config verify` and then + `golangci-lint run --config .golangci.yml ./...` as build steps in the + digest-pinned `golangci-lint` image. Nothing lints on the host. That `FROM` + line is the only pin of the linter version, and it changes in the same commit + as a re-vendored `.golangci.yml`. +- `script/lint-fix` — apply the linter's autofixes (rewrites files), using the + same pinned image, parsed out of the `lint` phase's `FROM` line. It cannot be + a build step, because fixes have to land in the worktree, so it bind-mounts + the tree into a `docker run` and therefore needs a _local_ daemon. It is a + developer convenience and never a gate: no gate reads its exit status. Run + `make lint` afterwards to find out whether the tree is clean. +- `script/fmt` — format all code (writes): Go with `go fmt`, Markdown with + `prettier` +- `script/fmt-check` — check formatting (read-only). It runs `gofmt` on the + host, over every Go file outside `.tool`, and `prettier --check` over every + Markdown file. +- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`. +- `script/docker` — build the Docker image tagged via `script/projectname`, + stamped with the version `git describe --tags --always --dirty` gives on the + host, or `unknown` outside a git checkout. The image's build stage depends on + the `lint` and `test` phases, so this lints and tests too. +- `script/cibuild` — CI entrypoint: runs `script/bootstrap`, `script/check`, and + then the same image build as `script/docker`. `.gitea/workflows/check.yml` + runs it on every push. - Every `docker build` in these scripts passes `--no-cache`, because on - an unchanged tree a cached check layer is replayed without running and - the build still exits 0. A plain `docker build .` is therefore no - evidence that the checks ran. The cost is that `script/cibuild` runs - the `lint` and `test` phases twice: once in `script/check` and again - in the image build. -* `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must - not change files), then `script/check` -* `script/install-precommit` — install the git pre-commit hook that - runs `script/precommit` + Every `docker build` in these scripts passes `--no-cache`, because on an + unchanged tree a cached check layer is replayed without running and the + build still exits 0. A plain `docker build .` is therefore no evidence that + the checks ran. The cost is that `script/cibuild` runs the `lint` and `test` + phases twice: once in `script/check` and again in the image build. + +- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must not + change files), then `script/check` +- `script/install-precommit` — install the git pre-commit hook that runs + `script/precommit` ## releasing ### version numbers -The version a binary reports comes from git, not from a constant in a -file. It is `git describe --tags --always --dirty`, which -`script/version` runs for the `Makefile`, and `script/docker` and -`script/cibuild` run themselves: +The version a binary reports comes from git, not from a constant in a file. It +is `git describe --tags --always --dirty`, which `script/version` runs for the +`Makefile`, and `script/docker` and `script/cibuild` run themselves: -* `HEAD` is exactly on a tag → that tag, such as `v1.0.0`. -* a commit after a tag → `--g`. -* no tag reachable → the short commit sha. -* any of these, with uncommitted changes to tracked files → a `-dirty` - suffix, because a modified checkout of a tag is not that tag. +- `HEAD` is exactly on a tag → that tag, such as `v1.0.0`. +- a commit after a tag → `--g`. +- no tag reachable → the short commit sha. +- any of these, with uncommitted changes to tracked files → a `-dirty` suffix, + because a modified checkout of a tag is not that tag. A `docker build .` of a clone, with no build arguments, runs the same -`git describe` (without `--dirty`) on the `.git` in its build context, -so it stamps the same value for a clean commit; the build fails if the -context carries `.git` and no version, commit or commit date comes out. -A binary built without +`git describe` (without `--dirty`) on the `.git` in its build context, so it +stamps the same value for a clean commit; the build fails if the context carries +`.git` and no version, commit or commit date comes out. A binary built without git metadata reports `dev`, or `unknown` when `script/docker` or `script/cibuild` built it outside a git checkout. -`goreleaser` stamps a release binary with the tag minus its leading -`v`, so the tag `v1.0.0` produces `vaultik 1.0.0`, matching the archive -name `vaultik_1.0.0_linux_amd64.tar.gz`. `goreleaser --snapshot` stamps -`dev-<12 chars of the commit sha>` rather than inventing the next patch -number. `vaultik version` calls a build a development build when its -version is `dev`, `unknown`, `dev-`, the short commit sha or -`--g`, with or without `-dirty`; only a plain tag, -such as `v1.0.0` or `1.0.0`, is a release. If `script/version` cannot -be run at all, `make` stops with an error instead of building an -unversioned binary, and a binary that somehow carries an empty version -string still reports itself as a development build. +`goreleaser` stamps a release binary with the tag minus its leading `v`, so the +tag `v1.0.0` produces `vaultik 1.0.0`, matching the archive name +`vaultik_1.0.0_linux_amd64.tar.gz`. `goreleaser --snapshot` stamps +`dev-<12 chars of the commit sha>` rather than inventing the next patch number. +`vaultik version` calls a build a development build when its version is `dev`, +`unknown`, `dev-`, the short commit sha or `--g`, with +or without `-dirty`; only a plain tag, such as `v1.0.0` or `1.0.0`, is a +release. If `script/version` cannot be run at all, `make` stops with an error +instead of building an unversioned binary, and a binary that somehow carries an +empty version string still reports itself as a development build. ### cutting a release @@ -1084,13 +1065,12 @@ git tag -a v1.2.3 -m 'v1.2.3' git push origin v1.2.3 ``` -`.gitea/workflows/release.yml` triggers on `v*` tags, installs a Go -toolchain and the pinned `goreleaser`, and runs `script/release`, which -builds -`linux,darwin × amd64,arm64` archives plus `checksums.txt` and publishes -them to this repository's Gitea releases as a draft. `.goreleaser.yaml` -has a `gitea_urls:` block pointing at `https://git.eeqj.de/api/v1`; -without it `goreleaser` would talk to the GitHub API. +`.gitea/workflows/release.yml` triggers on `v*` tags, installs a Go toolchain +and the pinned `goreleaser`, and runs `script/release`, which builds +`linux,darwin × amd64,arm64` archives plus `checksums.txt` and publishes them to +this repository's Gitea releases as a draft. `.goreleaser.yaml` has a +`gitea_urls:` block pointing at `https://git.eeqj.de/api/v1`; without it +`goreleaser` would talk to the GitHub API. The workflow needs one repository Actions secret: @@ -1098,18 +1078,16 @@ The workflow needs one repository Actions secret: | --------------- | ------------------------------------------------------------------------------------------------------- | | `RELEASE_TOKEN` | A Gitea access token with `write:repository` scope, owned by an account that can publish releases here. | -It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic -token is deliberately not used: it is not guaranteed to carry release -write access. +It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic token is +deliberately not used: it is not guaranteed to carry release write access. The Go toolchain that compiles the released binaries is installed by -`script/install-go`, which downloads the version named by `go.mod` -(currently `1.26.1`, the same version the `Dockerfile` builder stage -pins by digest) and refuses the archive unless its sha256 matches the -value committed in the script. `goreleaser` shells out to `go` for every -cross-compile, so without that step the release would either fail -outright or ship binaries built by whatever unpinned toolchain the -runner happened to carry. +`script/install-go`, which downloads the version named by `go.mod` (currently +`1.26.1`, the same version the `Dockerfile` builder stage pins by digest) and +refuses the archive unless its sha256 matches the value committed in the script. +`goreleaser` shells out to `go` for every cross-compile, so without that step +the release would either fail outright or ship binaries built by whatever +unpinned toolchain the runner happened to carry. To rehearse the whole build without publishing or tagging anything: @@ -1119,9 +1097,9 @@ make release-snapshot Artifacts land in `./dist`, which is gitignored. -Release artifacts are not signed, carry no SBOM, and are not built -reproducibly; the archives contain the binary, `LICENSE`, and -`README.md` only (no shell completions or man page). +Release artifacts are not signed, carry no SBOM, and are not built reproducibly; +the archives contain the binary, `LICENSE`, and `README.md` only (no shell +completions or man page). ## license @@ -1129,7 +1107,8 @@ reproducibly; the archives contain the binary, `LICENSE`, and ## author -Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in Berlin in the summer of 2025. +Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in +Berlin in the summer of 2025. Released as a free software gift to the world, no strings attached. diff --git a/TODO.md b/TODO.md index 69d31aa..894cf52 100644 --- a/TODO.md +++ b/TODO.md @@ -1,12 +1,12 @@ # Workflow -* branch (from `main`) -* do the work in Next Step -* move Next Step to the top of Completed Steps -* move the top item of Future Steps into Next Step -* commit (`TODO.md` changes in the same commit as the work) -* merge to `main` if the branch is not protected, otherwise open a PR -* push +- branch (from `main`) +- do the work in Next Step +- move Next Step to the top of Completed Steps +- move the top item of Future Steps into Next Step +- commit (`TODO.md` changes in the same commit as the work) +- merge to `main` if the branch is not protected, otherwise open a PR +- push # Status @@ -15,1317 +15,1205 @@ pre-1.0 # Next Step The 1.0 work is complete on `next`: the scope settled on -[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) has landed, -and so has every fix from the defect hunt before the release candidate -([issue #288](https://git.eeqj.de/sneak/vaultik/issues/288)). What is -left is merging `next` to `main` and tagging the release, both the -owner's; a `v*` tag runs the release build in -`.gitea/workflows/release.yml`. +[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) has landed, and so +has every fix from the defect hunt before the release candidate +([issue #288](https://git.eeqj.de/sneak/vaultik/issues/288)). What is left is +merging `next` to `main` and tagging the release, both the owner's; a `v*` tag +runs the release build in `.gitea/workflows/release.yml`. # Completed Steps -- 2026-11-10: Printed the status and progress lines of `remote info` - and `snapshot verify` through `internal/ui`, with its marker and - color, as the README describes - ([issue #410](https://git.eeqj.de/sneak/vaultik/issues/410)). Their - reports stay plain. +- 2026-11-10: Printed the status and progress lines of `remote info` and + `snapshot verify` through `internal/ui`, with its marker and color, as the + README describes ([issue #410](https://git.eeqj.de/sneak/vaultik/issues/410)). + Their reports stay plain. -- 2026-11-10: Made `config init` write the `apps` snapshot, which backs - up `/Applications`, only on macOS - ([issue #408](https://git.eeqj.de/sneak/vaultik/issues/408)). On - Linux the first `snapshot create` stopped on the missing directory. +- 2026-11-10: Made `config init` write the `apps` snapshot, which backs up + `/Applications`, only on macOS + ([issue #408](https://git.eeqj.de/sneak/vaultik/issues/408)). On Linux the + first `snapshot create` stopped on the missing directory. -- 2026-10-15: Stopped `snapshot create --prune` pruning after a backup - in which a volume mounted below a configured path in an earlier - snapshot of the same name was not mounted - ([issue #395](https://git.eeqj.de/sneak/vaultik/issues/395)). Such a - prune could delete every snapshot still holding the volume's files. - Each snapshot now records in the local index the directories it found - on a different filesystem from their parent and found an entry in that - the backup does not exclude. +- 2026-10-15: Stopped `snapshot create --prune` pruning after a backup in which + a volume mounted below a configured path in an earlier snapshot of the same + name was not mounted + ([issue #395](https://git.eeqj.de/sneak/vaultik/issues/395)). Such a prune + could delete every snapshot still holding the volume's files. Each snapshot + now records in the local index the directories it found on a different + filesystem from their parent and found an entry in that the backup does not + exclude. -- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup - that found nothing in a configured path - ([issue #388](https://git.eeqj.de/sneak/vaultik/issues/388)), as in - the mount point of a volume that is not mounted. Such a prune could - delete every snapshot still holding that path's files. A configured - path that is a symlink whose target is gone, as one into such a - volume, now fails the backup, with or without `--prune`. +- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup that + found nothing in a configured path + ([issue #388](https://git.eeqj.de/sneak/vaultik/issues/388)), as in the mount + point of a volume that is not mounted. Such a prune could delete every + snapshot still holding that path's files. A configured path that is a symlink + whose target is gone, as one into such a volume, now fails the backup, with or + without `--prune`. -- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup - that skipped files it could not read - ([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a - prune could delete the only snapshot still holding those files. +- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup that + skipped files it could not read + ([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a prune + could delete the only snapshot still holding those files. -- 2026-10-14: Made `snapshot create --skip-errors` exit non-zero, - instead of 0, when it skips anything it cannot read +- 2026-10-14: Made `snapshot create --skip-errors` exit non-zero, instead of 0, + when it skips anything it cannot read ([issue #310](https://git.eeqj.de/sneak/vaultik/issues/310)). -- 2026-10-14: Kept the local index when `remote nuke` cannot list or - find the destination store - ([issue #292](https://git.eeqj.de/sneak/vaultik/issues/292)). Nuke - had cleared the local snapshot records anyway; it now fails before - removing anything. +- 2026-10-14: Kept the local index when `remote nuke` cannot list or find the + destination store + ([issue #292](https://git.eeqj.de/sneak/vaultik/issues/292)). Nuke had cleared + the local snapshot records anyway; it now fails before removing anything. -- 2026-10-14: Stopped counting directories and unchanged symlinks as - files to back up - ([issue #305](https://git.eeqj.de/sneak/vaultik/issues/305)). +- 2026-10-14: Stopped counting directories and unchanged symlinks as files to + back up ([issue #305](https://git.eeqj.de/sneak/vaultik/issues/305)). -- 2026-10-14: Refused a backup destination that an unplugged volume - left empty, as the owner ruled - ([issue #239](https://git.eeqj.de/sneak/vaultik/issues/239)). Every - backup now writes `vaultik-destination.txt` at the destination root, - and while the local index holds snapshots stored there, a destination - without it is treated as not present. +- 2026-10-14: Refused a backup destination that an unplugged volume left empty, + as the owner ruled + ([issue #239](https://git.eeqj.de/sneak/vaultik/issues/239)). Every backup now + writes `vaultik-destination.txt` at the destination root, and while the local + index holds snapshots stored there, a destination without it is treated as not + present. -- 2026-10-14: Warned about restored ownership only when an entry's - owner or group differs from the recorded one - ([issue #316](https://git.eeqj.de/sneak/vaultik/issues/316)); a - restore by a non-root user had warned even when none did. +- 2026-10-14: Warned about restored ownership only when an entry's owner or + group differs from the recorded one + ([issue #316](https://git.eeqj.de/sneak/vaultik/issues/316)); a restore by a + non-root user had warned even when none did. -- 2026-10-14: Made `snapshot list` report a changed destination instead - of advising `prune`, which refuses in that case +- 2026-10-14: Made `snapshot list` report a changed destination instead of + advising `prune`, which refuses in that case ([issue #317](https://git.eeqj.de/sneak/vaultik/issues/317)). -- 2026-10-14: Counted a file as deleted by a backup only if the - previous snapshot of the same name listed it - ([issue #304](https://git.eeqj.de/sneak/vaultik/issues/304)); files - that snapshot did not list had been counted too. +- 2026-10-14: Counted a file as deleted by a backup only if the previous + snapshot of the same name listed it + ([issue #304](https://git.eeqj.de/sneak/vaultik/issues/304)); files that + snapshot did not list had been counted too. -- 2026-10-14: Named each prefix once in restore, verify and purge - errors, some of which repeated it - ([issue #327](https://git.eeqj.de/sneak/vaultik/issues/327)). A deep - verify log line that named S3 whatever the destination now names - storage. +- 2026-10-14: Named each prefix once in restore, verify and purge errors, some + of which repeated it + ([issue #327](https://git.eeqj.de/sneak/vaultik/issues/327)). A deep verify + log line that named S3 whatever the destination now names storage. -- 2026-10-14: Refused credentials in a storage URL's endpoint or rclone - remote name, which were accepted and could then be printed +- 2026-10-14: Refused credentials in a storage URL's endpoint or rclone remote + name, which were accepted and could then be printed ([issue #322](https://git.eeqj.de/sneak/vaultik/issues/322)). -- 2026-10-14: Ran `$EDITOR` through the shell in `config edit`, so a - value with arguments, such as `code --wait`, no longer fails +- 2026-10-14: Ran `$EDITOR` through the shell in `config edit`, so a value with + arguments, such as `code --wait`, no longer fails ([issue #315](https://git.eeqj.de/sneak/vaultik/issues/315)). -- 2026-10-14: Made `snapshot remove` fail, instead of exiting 0, when - it cannot delete the metadata of a snapshot that is only on the - destination store - ([issue #340](https://git.eeqj.de/sneak/vaultik/issues/340)). The - README and the help now say when it prints the `vaultik prune` - follow-up. +- 2026-10-14: Made `snapshot remove` fail, instead of exiting 0, when it cannot + delete the metadata of a snapshot that is only on the destination store + ([issue #340](https://git.eeqj.de/sneak/vaultik/issues/340)). The README and + the help now say when it prints the `vaultik prune` follow-up. -- 2026-10-14: Labelled the totals in the local index cleanup before - each backup as totals - ([issue #306](https://git.eeqj.de/sneak/vaultik/issues/306)). Each +- 2026-10-14: Labelled the totals in the local index cleanup before each backup + as totals ([issue #306](https://git.eeqj.de/sneak/vaultik/issues/306)). Each was shown as `(N remain)`, which read as rows left to clean up. -- 2026-10-13: Freed a cached blob once no file still to restore needs - it ([issue #324](https://git.eeqj.de/sneak/vaultik/issues/324)); one - also used by a file outside the selected paths had stayed in the - temporary cache until the restore ended. +- 2026-10-13: Freed a cached blob once no file still to restore needs it + ([issue #324](https://git.eeqj.de/sneak/vaultik/issues/324)); one also used by + a file outside the selected paths had stayed in the temporary cache until the + restore ended. - 2026-10-13: Kept the whole startup error when it contains `): ` - ([issue #320](https://git.eeqj.de/sneak/vaultik/issues/320)). Such - an error was cut at its last `): `, so the `blob_size_limit` one lost - the setting's name. + ([issue #320](https://git.eeqj.de/sneak/vaultik/issues/320)). Such an error + was cut at its last `): `, so the `blob_size_limit` one lost the setting's + name. -- 2026-10-13: Silenced the status lines of `prune` and snapshot removal - under `--cron` and `--quiet` +- 2026-10-13: Silenced the status lines of `prune` and snapshot removal under + `--cron` and `--quiet` ([issue #294](https://git.eeqj.de/sneak/vaultik/issues/294)). - 2026-10-13: Read the rclone config file that `RCLONE_CONFIG` names - ([issue #321](https://git.eeqj.de/sneak/vaultik/issues/321)); a - remote defined only there had been reported as not found. + ([issue #321](https://git.eeqj.de/sneak/vaultik/issues/321)); a remote defined + only there had been reported as not found. -- 2026-10-13: Made `/**/` in an exclude pattern match zero directories, - as in gitignore - ([issue #338](https://git.eeqj.de/sneak/vaultik/issues/338)); +- 2026-10-13: Made `/**/` in an exclude pattern match zero directories, as in + gitignore ([issue #338](https://git.eeqj.de/sneak/vaultik/issues/338)); `a/**/*.log` had missed `a/x.log`. -- 2026-10-13: Made the `chunk_size` and `blob_size_limit` defaults and - the `chunk_size` minimum 1000-based, like the size parser and the - docs ([issue #313](https://git.eeqj.de/sneak/vaultik/issues/313)). +- 2026-10-13: Made the `chunk_size` and `blob_size_limit` defaults and the + `chunk_size` minimum 1000-based, like the size parser and the docs + ([issue #313](https://git.eeqj.de/sneak/vaultik/issues/313)). -- 2026-10-13: Kept a snapshot's remote metadata when `snapshot purge` - cannot remove it from the local index - ([issue #337](https://git.eeqj.de/sneak/vaultik/issues/337)); the - purge had deleted the metadata anyway and exited 0. +- 2026-10-13: Kept a snapshot's remote metadata when `snapshot purge` cannot + remove it from the local index + ([issue #337](https://git.eeqj.de/sneak/vaultik/issues/337)); the purge had + deleted the metadata anyway and exited 0. -- 2026-10-13: Made `snapshot restore` fail for a requested path that - matches nothing, which it had ignored +- 2026-10-13: Made `snapshot restore` fail for a requested path that matches + nothing, which it had ignored ([issue #309](https://git.eeqj.de/sneak/vaultik/issues/309)). -- 2026-10-13: Made `remote nuke` delete every object under `metadata/` - and `blobs/` - ([issue #291](https://git.eeqj.de/sneak/vaultik/issues/291)); it had - left some, such as a stray `.DS_Store`, while reporting the store - empty. +- 2026-10-13: Made `remote nuke` delete every object under `metadata/` and + `blobs/` ([issue #291](https://git.eeqj.de/sneak/vaultik/issues/291)); it had + left some, such as a stray `.DS_Store`, while reporting the store empty. -- 2026-10-13: Made `config set` create a list for an index under a - missing key - ([issue #308](https://git.eeqj.de/sneak/vaultik/issues/308)); it had - written a map, after which no command could load the config. +- 2026-10-13: Made `config set` create a list for an index under a missing key + ([issue #308](https://git.eeqj.de/sneak/vaultik/issues/308)); it had written a + map, after which no command could load the config. -- 2026-10-13: Listed a `file://` destination through a symlinked - `metadata/` or `blobs/`, which had listed as empty - ([issue #296](https://git.eeqj.de/sneak/vaultik/issues/296)). A link - to a missing target now fails the listing +- 2026-10-13: Listed a `file://` destination through a symlinked `metadata/` or + `blobs/`, which had listed as empty + ([issue #296](https://git.eeqj.de/sneak/vaultik/issues/296)). A link to a + missing target now fails the listing ([issue #332](https://git.eeqj.de/sneak/vaultik/issues/332)). -- 2026-10-13: Made `snapshot verify --deep --json` report its blob - counts ([issue #314](https://git.eeqj.de/sneak/vaultik/issues/314)); - a failed run had reported 0 missing and 0 verified. +- 2026-10-13: Made `snapshot verify --deep --json` report its blob counts + ([issue #314](https://git.eeqj.de/sneak/vaultik/issues/314)); a failed run had + reported 0 missing and 0 verified. -- 2026-10-13: Limited and retried the module download in the Docker - build ([issue #351](https://git.eeqj.de/sneak/vaultik/issues/351)), - which had hung for as long as a stalled connection to the module - proxy lasted. +- 2026-10-13: Limited and retried the module download in the Docker build + ([issue #351](https://git.eeqj.de/sneak/vaultik/issues/351)), which had hung + for as long as a stalled connection to the module proxy lasted. -- 2026-10-13: Recorded each snapshot's uncompressed size when its - backup runs - ([issue #360](https://git.eeqj.de/sneak/vaultik/issues/360)). A later - backup could change an older snapshot's UNCOMPRESSED SIZE in - `snapshot list`. +- 2026-10-13: Recorded each snapshot's uncompressed size when its backup runs + ([issue #360](https://git.eeqj.de/sneak/vaultik/issues/360)). A later backup + could change an older snapshot's UNCOMPRESSED SIZE in `snapshot list`. -- 2026-10-13: Stopped `snapshot restore --skip-errors --verify` - reporting a file that failed to restore again as a checksum failure +- 2026-10-13: Stopped `snapshot restore --skip-errors --verify` reporting a file + that failed to restore again as a checksum failure ([issue #323](https://git.eeqj.de/sneak/vaultik/issues/323)). -- 2026-10-13: Recorded the stored size of a blob a backup finds already - stored, not the size of its local copy, which can differ, so - `snapshot verify` could fail on a good snapshot +- 2026-10-13: Recorded the stored size of a blob a backup finds already stored, + not the size of its local copy, which can differ, so `snapshot verify` could + fail on a good snapshot ([issue #300](https://git.eeqj.de/sneak/vaultik/issues/300)). -- 2026-10-13: Stopped the local index and the blob upload step ending - the process on a failed close of query results or a failed temp file - cleanup ([issue #345](https://git.eeqj.de/sneak/vaultik/issues/345)); - both are now returned as errors. +- 2026-10-13: Stopped the local index and the blob upload step ending the + process on a failed close of query results or a failed temp file cleanup + ([issue #345](https://git.eeqj.de/sneak/vaultik/issues/345)); both are now + returned as errors. -- 2026-10-13: Closed the snapshot database's storage reader too when - restore or deep verify is interrupted +- 2026-10-13: Closed the snapshot database's storage reader too when restore or + deep verify is interrupted ([issue #356](https://git.eeqj.de/sneak/vaultik/issues/356)). -- 2026-10-13: Made `prune` and `remote nuke` fail when a delete on the - backup destination store fails +- 2026-10-13: Made `prune` and `remote nuke` fail when a delete on the backup + destination store fails ([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)). - 2026-10-13: Made an S3 listing that the server cuts short an error - ([issue #297](https://git.eeqj.de/sneak/vaultik/issues/297)); `prune` - had taken it as complete and could delete blobs still in use. + ([issue #297](https://git.eeqj.de/sneak/vaultik/issues/297)); `prune` had + taken it as complete and could delete blobs still in use. -- 2026-10-13: Made shallow `snapshot verify` report a blob it could not - check apart from a missing one - ([issue #318](https://git.eeqj.de/sneak/vaultik/issues/318)); any - error checking a blob had counted as missing, its cause dropped. +- 2026-10-13: Made shallow `snapshot verify` report a blob it could not check + apart from a missing one + ([issue #318](https://git.eeqj.de/sneak/vaultik/issues/318)); any error + checking a blob had counted as missing, its cause dropped. -- 2026-10-13: Restored the other files when a file's chunk is in no - blob under `--skip-errors` - ([issue #326](https://git.eeqj.de/sneak/vaultik/issues/326)); that - file had stopped the restore before anything was restored. +- 2026-10-13: Restored the other files when a file's chunk is in no blob under + `--skip-errors` ([issue #326](https://git.eeqj.de/sneak/vaultik/issues/326)); + that file had stopped the restore before anything was restored. -- 2026-10-13: Recorded each snapshot's new chunk size when its backup - runs ([issue #311](https://git.eeqj.de/sneak/vaultik/issues/311)). - `snapshot list` could show a NEW CHUNK SIZE of 0 B for a snapshot - whose change was a modified file. +- 2026-10-13: Recorded each snapshot's new chunk size when its backup runs + ([issue #311](https://git.eeqj.de/sneak/vaultik/issues/311)). `snapshot list` + could show a NEW CHUNK SIZE of 0 B for a snapshot whose change was a modified + file. -- 2026-10-13: Made a backup interrupted during a local index write exit - as interrupted - ([issue #301](https://git.eeqj.de/sneak/vaultik/issues/301)); it could - end with `FATAL: failed to rollback transaction` and exit 1 instead of - 130. +- 2026-10-13: Made a backup interrupted during a local index write exit as + interrupted ([issue #301](https://git.eeqj.de/sneak/vaultik/issues/301)); it + could end with `FATAL: failed to rollback transaction` and exit 1 instead + of 130. -- 2026-10-13: Printed `snapshot list` and `snapshot purge` sizes like - every other command - ([issue #312](https://git.eeqj.de/sneak/vaultik/issues/312)). They, - and the `bytes` value in terminal log lines, were 1024-based but +- 2026-10-13: Printed `snapshot list` and `snapshot purge` sizes like every + other command ([issue #312](https://git.eeqj.de/sneak/vaultik/issues/312)). + They, and the `bytes` value in terminal log lines, were 1024-based but labelled KB or MB, and now use 1000-based units. -- 2026-10-13: Closed each blob's storage reader when restore or deep - verify is interrupted, which ends a stalled read unless it is stuck in - the kernel ([issue #325](https://git.eeqj.de/sneak/vaultik/issues/325)). +- 2026-10-13: Closed each blob's storage reader when restore or deep verify is + interrupted, which ends a stalled read unless it is stuck in the kernel + ([issue #325](https://git.eeqj.de/sneak/vaultik/issues/325)). -- 2026-10-13: Made a mistyped subcommand under `snapshot`, `config`, - `remote`, `database` or `completion` an error - ([issue #319](https://git.eeqj.de/sneak/vaultik/issues/319)); each had - printed its help and exited 0. +- 2026-10-13: Made a mistyped subcommand under `snapshot`, `config`, `remote`, + `database` or `completion` an error + ([issue #319](https://git.eeqj.de/sneak/vaultik/issues/319)); each had printed + its help and exited 0. -- 2026-10-13: Made an exclude pattern starting with `**/` match - directly under the snapshot path too, as in gitignore +- 2026-10-13: Made an exclude pattern starting with `**/` match directly under + the snapshot path too, as in gitignore ([issue #303](https://git.eeqj.de/sneak/vaultik/issues/303)). - 2026-10-13: Refused a relative `file://` storage URL and decoded its - percent-escapes - ([issue #299](https://git.eeqj.de/sneak/vaultik/issues/299)); a - relative path had moved with the working directory. + percent-escapes ([issue #299](https://git.eeqj.de/sneak/vaultik/issues/299)); + a relative path had moved with the working directory. - 2026-10-13: Removed the blob temp file when a backup stops part-way - ([issue #302](https://git.eeqj.de/sneak/vaultik/issues/302)); the - half-written blob had stayed in `$TMPDIR`. + ([issue #302](https://git.eeqj.de/sneak/vaultik/issues/302)); the half-written + blob had stayed in `$TMPDIR`. - 2026-10-13: Checked `--keep-newer-than` before a backup starts - ([issue #307](https://git.eeqj.de/sneak/vaultik/issues/307)); an - invalid value had been read only by the prune after the backup, so - retention never ran. + ([issue #307](https://git.eeqj.de/sneak/vaultik/issues/307)); an invalid value + had been read only by the prune after the backup, so retention never ran. -- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that - is not a blob as an orphaned blob - ([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray - file such as `.DS_Store` was reported as an orphaned blob and the - report advised running `prune`, which skips such a file and deleted - nothing. `remote info` now takes its blob list from the same listing - `prune` uses, in the table and in `--json`. +- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that is not a + blob as an orphaned blob + ([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray file + such as `.DS_Store` was reported as an orphaned blob and the report advised + running `prune`, which skips such a file and deleted nothing. `remote info` + now takes its blob list from the same listing `prune` uses, in the table and + in `--json`. -- 2026-10-13: Made `snapshot remove` fail for a snapshot that does not - exist ([issue #293](https://git.eeqj.de/sneak/vaultik/issues/293)). - The command never checked the ID: it reported the snapshot removed - from the local database and from the destination store and exited 0, - under `--json`, `--dry-run` and `--local-only` too. An ID that names - no snapshot in the local index or on the destination store is now an - error; with `--local-only`, the snapshot must be in the local index. - The command now reports a removal from either side only when one - happened, and `remote_removed` is true only when metadata was deleted. +- 2026-10-13: Made `snapshot remove` fail for a snapshot that does not exist + ([issue #293](https://git.eeqj.de/sneak/vaultik/issues/293)). The command + never checked the ID: it reported the snapshot removed from the local database + and from the destination store and exited 0, under `--json`, `--dry-run` and + `--local-only` too. An ID that names no snapshot in the local index or on the + destination store is now an error; with `--local-only`, the snapshot must be + in the local index. The command now reports a removal from either side only + when one happened, and `remote_removed` is true only when metadata was + deleted. -- 2026-10-12: Bound the local index to the destination an `s3.*` config - names ([issue #298](https://git.eeqj.de/sneak/vaultik/issues/298)). - Only `storage_url` was checked, so after `s3.bucket` or `s3.prefix` - changed, the next backup treated every chunk as already stored, - uploaded nothing, and completed a snapshot that could not be restored. - Without `storage_url`, the local index is now bound to `s3.endpoint`, - `s3.bucket` and `s3.prefix`, and a change is refused as a changed - `storage_url` is. +- 2026-10-12: Bound the local index to the destination an `s3.*` config names + ([issue #298](https://git.eeqj.de/sneak/vaultik/issues/298)). Only + `storage_url` was checked, so after `s3.bucket` or `s3.prefix` changed, the + next backup treated every chunk as already stored, uploaded nothing, and + completed a snapshot that could not be restored. Without `storage_url`, the + local index is now bound to `s3.endpoint`, `s3.bucket` and `s3.prefix`, and a + change is refused as a changed `storage_url` is. - 2026-10-12: Rejected a retention duration longer than about 292 years - ([issue #289](https://git.eeqj.de/sneak/vaultik/issues/289)). The - number given to `--older-than` or `--keep-newer-than` was multiplied - by the length of its unit with no limit, so from `293y`, `15251w`, - `3559mo` or `106752d`, or a sum of parts such as `200y200y`, the - result wrapped to a negative duration. The cutoff then lay in the - future, every snapshot of the name was deleted, the one just made - included, and the command exited 0. Such a duration is now rejected + ([issue #289](https://git.eeqj.de/sneak/vaultik/issues/289)). The number given + to `--older-than` or `--keep-newer-than` was multiplied by the length of its + unit with no limit, so from `293y`, `15251w`, `3559mo` or `106752d`, or a sum + of parts such as `200y200y`, the result wrapped to a negative duration. The + cutoff then lay in the future, every snapshot of the name was deleted, the one + just made included, and the command exited 0. Such a duration is now rejected as invalid and nothing is deleted. - 2026-10-08: Made a backup cancelled under `--skip-errors` stop at once - ([issue #286](https://git.eeqj.de/sneak/vaultik/issues/286)). After - Ctrl-C or SIGTERM, phase 2 treated the cancellation error from each - remaining file like an unreadable file: it opened the file, printed an - error line for it, counted it as failed and went on to the next one. - The run now stops at the first file after the cancellation, and no - file is counted as failed because of it. + ([issue #286](https://git.eeqj.de/sneak/vaultik/issues/286)). After Ctrl-C or + SIGTERM, phase 2 treated the cancellation error from each remaining file like + an unreadable file: it opened the file, printed an error line for it, counted + it as failed and went on to the next one. The run now stops at the first file + after the cancellation, and no file is counted as failed because of it. -- 2026-10-08: Made `remote nuke` delete the `.partial` files that - uploads cut off part-way leave on the destination store - ([issue #281](https://git.eeqj.de/sneak/vaultik/issues/281)). The - file and rclone listings skip such a file, so the command left it in - place and still reported the store empty. It now removes them under - `metadata/` and `blobs/` as its last step. +- 2026-10-08: Made `remote nuke` delete the `.partial` files that uploads cut + off part-way leave on the destination store + ([issue #281](https://git.eeqj.de/sneak/vaultik/issues/281)). The file and + rclone listings skip such a file, so the command left it in place and still + reported the store empty. It now removes them under `metadata/` and `blobs/` + as its last step. -- 2026-10-08: Made a local index error while recording a directory or - symlink stop a backup under `--skip-errors` - ([issue #284](https://git.eeqj.de/sneak/vaultik/issues/284)). Phase 2 - only records such an entry in the local index, since it has no data - to open or read, but an error doing so was skipped like an unreadable - file. The snapshot then completed without the entry, and a restore - did not recreate it. The error now stops the backup, as it does - without the flag. +- 2026-10-08: Made a local index error while recording a directory or symlink + stop a backup under `--skip-errors` + ([issue #284](https://git.eeqj.de/sneak/vaultik/issues/284)). Phase 2 only + records such an entry in the local index, since it has no data to open or + read, but an error doing so was skipped like an unreadable file. The snapshot + then completed without the entry, and a restore did not recreate it. The error + now stops the backup, as it does without the flag. - 2026-10-08: Counted a file that a backup could not store as failed - ([issue #280](https://git.eeqj.de/sneak/vaultik/issues/280)). A file - that phase 1 counted and phase 2 could not open, because it was - unreadable under `--skip-errors` or removed in between, was reported - in the summary as unchanged with its bytes as backed up, and the - `snapshots` row's `file_count` and `total_size` included it. The - summary now counts it as failed, and its data total and the row leave - it out. + ([issue #280](https://git.eeqj.de/sneak/vaultik/issues/280)). A file that + phase 1 counted and phase 2 could not open, because it was unreadable under + `--skip-errors` or removed in between, was reported in the summary as + unchanged with its bytes as backed up, and the `snapshots` row's `file_count` + and `total_size` included it. The summary now counts it as failed, and its + data total and the row leave it out. -- 2026-10-08: Made `remote info` report a snapshot's blob count and - blob size as unknown when its manifest cannot be read - ([issue #272](https://git.eeqj.de/sneak/vaultik/issues/272)). The - orphan figures were already unknown in that case, but the snapshot's - row still gave 0 blobs and 0 B, in the table and in `--json`. The row - now reads `unknown` and `--json` gives `null`. A directory with no - manifest still shows 0, since the orphan figures count its blobs as - orphaned. +- 2026-10-08: Made `remote info` report a snapshot's blob count and blob size as + unknown when its manifest cannot be read + ([issue #272](https://git.eeqj.de/sneak/vaultik/issues/272)). The orphan + figures were already unknown in that case, but the snapshot's row still gave 0 + blobs and 0 B, in the table and in `--json`. The row now reads `unknown` and + `--json` gives `null`. A directory with no manifest still shows 0, since the + orphan figures count its blobs as orphaned. -- 2026-10-08: Kept the progress line of a snapshot with more than one - path within 100% - ([issue #271](https://git.eeqj.de/sneak/vaultik/issues/271)). The - bytes and files processed counted every path, but the totals they - were divided by held only the current path's, so a second path - smaller than the first showed more than 100% and an ETA of `unknown`. - The totals now add up over the paths scanned so far. The rate behind - the ETA restarts when each path's scan phase ends, so that phase does - not lower the rate while the path is processed; until then the rate - is the previous path's. +- 2026-10-08: Kept the progress line of a snapshot with more than one path + within 100% ([issue #271](https://git.eeqj.de/sneak/vaultik/issues/271)). The + bytes and files processed counted every path, but the totals they were divided + by held only the current path's, so a second path smaller than the first + showed more than 100% and an ETA of `unknown`. The totals now add up over the + paths scanned so far. The rate behind the ETA restarts when each path's scan + phase ends, so that phase does not lower the rate while the path is processed; + until then the rate is the previous path's. -- 2026-10-08: Made a second `snapshot create` of one name succeed when - it starts in the same second as the first - ([issue #270](https://git.eeqj.de/sneak/vaultik/issues/270)). The - timestamp in a snapshot ID is in whole seconds, so the second run got - the first run's ID and failed with `UNIQUE constraint failed: - snapshots.id`. When the local index already has a snapshot with the - ID, the create now waits a second and takes a new timestamp. +- 2026-10-08: Made a second `snapshot create` of one name succeed when it starts + in the same second as the first + ([issue #270](https://git.eeqj.de/sneak/vaultik/issues/270)). The timestamp in + a snapshot ID is in whole seconds, so the second run got the first run's ID + and failed with `UNIQUE constraint failed: snapshots.id`. When the local index + already has a snapshot with the ID, the create now waits a second and takes a + new timestamp. - 2026-10-07: Documented that `--json` skips the confirmation prompt of `snapshot remove` and `prune` - ([issue #268](https://git.eeqj.de/sneak/vaultik/issues/268)). Both - commands delete without asking under `--json`, since a prompt on stdout - would break the JSON document, but the help and the README described - only `--force` as skipping it. The `--json` help of both commands and - their README entries now say so. + ([issue #268](https://git.eeqj.de/sneak/vaultik/issues/268)). Both commands + delete without asking under `--json`, since a prompt on stdout would break the + JSON document, but the help and the README described only `--force` as + skipping it. The `--json` help of both commands and their README entries now + say so. - 2026-10-07: Made a symlink whose target cannot be read stop the backup - ([issue #269](https://git.eeqj.de/sneak/vaultik/issues/269)). It was - left out of the snapshot with only a debug log line, even without - `--skip-errors`. It now aborts the run, or with `--skip-errors` is - skipped with the `Failed to access` error line that any other entry - the scan cannot read gets. + ([issue #269](https://git.eeqj.de/sneak/vaultik/issues/269)). It was left out + of the snapshot with only a debug log line, even without `--skip-errors`. It + now aborts the run, or with `--skip-errors` is skipped with the + `Failed to access` error line that any other entry the scan cannot read gets. -- 2026-10-07: Stopped a killed rclone upload from leaving a truncated - object under its key - ([issue #266](https://git.eeqj.de/sneak/vaultik/issues/266)). The - rclone backend wrote each object straight to its key, so killing an - upload to a local or sftp remote left a truncated object there; the - next backup found the key with `Stat`, skipped the upload and recorded - a snapshot that could not be restored. On a remote with a server-side - move, an object is now written under a temporary name ending in - `.partial` and moved into place, and listings skip such names. Remotes - without one are still written in place. +- 2026-10-07: Stopped a killed rclone upload from leaving a truncated object + under its key ([issue #266](https://git.eeqj.de/sneak/vaultik/issues/266)). + The rclone backend wrote each object straight to its key, so killing an upload + to a local or sftp remote left a truncated object there; the next backup found + the key with `Stat`, skipped the upload and recorded a snapshot that could not + be restored. On a remote with a server-side move, an object is now written + under a temporary name ending in `.partial` and moved into place, and listings + skip such names. Remotes without one are still written in place. - 2026-10-07: Made an interrupted command exit 130 and say so - ([issue #267](https://git.eeqj.de/sneak/vaultik/issues/267)). Ctrl-C - or SIGTERM during `snapshot create`, `snapshot restore` or `snapshot - verify` exited 0 with no error line (`snapshot verify --json` exited - 1, also without one), so a `--cron` run that never finished looked - like a success. A command stopped by either signal now exits 130 and - prints `interrupted before the command finished` on stderr, under - `--cron` and `--json` too. + ([issue #267](https://git.eeqj.de/sneak/vaultik/issues/267)). Ctrl-C or + SIGTERM during `snapshot create`, `snapshot restore` or `snapshot verify` + exited 0 with no error line (`snapshot verify --json` exited 1, also without + one), so a `--cron` run that never finished looked like a success. A command + stopped by either signal now exits 130 and prints + `interrupted before the command finished` on stderr, under `--cron` and + `--json` too. -- 2026-10-07: Corrected documentation, help text and comments that were - false about the code - ([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A blob - is not streamed to storage. The README, `ARCHITECTURE.md` and - `config.example.yml` now say a backup needs free temporary space, - because each blob is written whole to a temporary file before it is - uploaded (up to about `blob_size_limit`; an rclone destination that - cannot stream uploads needs about twice that) and the metadata export - writes copies of the local index. Temporary files go to `$TMPDIR` - (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those - copies to `/var/tmp`. Also corrected: the snapshot ID format, what restore - reads and how incomplete snapshots are removed in `docs/DATAMODEL.md`, - what `source_path` holds, the `index_path` and config file defaults, - what `snapshot remove` cleans up, how the release gets its Go +- 2026-10-07: Corrected documentation, help text and comments that were false + about the code ([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A + blob is not streamed to storage. The README, `ARCHITECTURE.md` and + `config.example.yml` now say a backup needs free temporary space, because each + blob is written whole to a temporary file before it is uploaded (up to about + `blob_size_limit`; an rclone destination that cannot stream uploads needs + about twice that) and the metadata export writes copies of the local index. + Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite + writes one of those copies to `/var/tmp`. Also corrected: the snapshot ID + format, what restore reads and how incomplete snapshots are removed in + `docs/DATAMODEL.md`, what `source_path` holds, the `index_path` and config + file defaults, what `snapshot remove` cleans up, how the release gets its Go toolchain, and the `script/release` and `script/fmt-check` comments. -- 2026-10-07: Cut the time the `internal/vaultik` and `internal/database` - tests take ([issue #235](https://git.eeqj.de/sneak/vaultik/issues/235)). - Most of the `internal/vaultik` time went to 24 tests that ran one at a - time only because they call `log.Initialize`; they now call it before - `t.Parallel()`, as the package's other tests do. `TestLargeDatasets` - committed each of its 1,500 inserts on its own and now makes them in - one transaction. `TestDedupOnlySnapshotRestores` gives its second - backup its own snapshot name instead of sleeping past the one-second - timestamp in the snapshot ID. +- 2026-10-07: Cut the time the `internal/vaultik` and `internal/database` tests + take ([issue #235](https://git.eeqj.de/sneak/vaultik/issues/235)). Most of the + `internal/vaultik` time went to 24 tests that ran one at a time only because + they call `log.Initialize`; they now call it before `t.Parallel()`, as the + package's other tests do. `TestLargeDatasets` committed each of its 1,500 + inserts on its own and now makes them in one transaction. + `TestDedupOnlySnapshotRestores` gives its second backup its own snapshot name + instead of sleeping past the one-second timestamp in the snapshot ID. - 2026-10-07: Made two messages say only what is true - ([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config - file that others can read was warned about as containing S3 - credentials even when it set none, as a `file://` config does. The - warning now says the file may contain S3 credentials only when - `s3.access_key_id` or `s3.secret_access_key` is set, since either may - come from a `${...}` reference rather than the file, and otherwise - says the file is readable by others. `snapshot purge` against a + ([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config file + that others can read was warned about as containing S3 credentials even when + it set none, as a `file://` config does. The warning now says the file may + contain S3 credentials only when `s3.access_key_id` or `s3.secret_access_key` + is set, since either may come from a `${...}` reference rather than the file, + and otherwise says the file is readable by others. `snapshot purge` against a destination store it could not list gave an error with `listing remote snapshots:` in it twice; the prefix now appears once. - 2026-10-07: Made `s3.part_size` set the multipart upload part size - ([issue #232](https://git.eeqj.de/sneak/vaultik/issues/232)). It was - loaded and defaulted but never passed to the S3 client, whose uploader - used a fixed 10MiB part. It now reaches the uploader for `storage_url` - and for the `s3.*` fields, and a part size S3 refuses, below 5MiB or - above 5GiB, `0` included, fails at config load. A blob too large for - S3's limit of 10,000 parts at the configured size is uploaded in larger - parts. The docs gave the default as `5MB`, which the config file reads - as 5,000,000 bytes, below the minimum; they now say `5MiB`. + ([issue #232](https://git.eeqj.de/sneak/vaultik/issues/232)). It was loaded + and defaulted but never passed to the S3 client, whose uploader used a fixed + 10MiB part. It now reaches the uploader for `storage_url` and for the `s3.*` + fields, and a part size S3 refuses, below 5MiB or above 5GiB, `0` included, + fails at config load. A blob too large for S3's limit of 10,000 parts at the + configured size is uploaded in larger parts. The docs gave the default as + `5MB`, which the config file reads as 5,000,000 bytes, below the minimum; they + now say `5MiB`. - 2026-10-07: Made per-name retention work when the hostname contains `_` - ([issue #230](https://git.eeqj.de/sneak/vaultik/issues/230)). A - snapshot ID is `hostname_name_timestamp`, and the name was read as - everything between the first and the last `_`, so with - `hostname: my_host` the name `home` came out as `host_home`. - `snapshot purge --keep-latest --snapshot home` then printed "No - snapshots to delete", and `snapshot create --prune` purged nothing - without a message. The name is now read using the hostname the - `snapshots` table stores with each snapshot, cut at its first `.` as it - is in the ID. + ([issue #230](https://git.eeqj.de/sneak/vaultik/issues/230)). A snapshot ID is + `hostname_name_timestamp`, and the name was read as everything between the + first and the last `_`, so with `hostname: my_host` the name `home` came out + as `host_home`. `snapshot purge --keep-latest --snapshot home` then printed + "No snapshots to delete", and `snapshot create --prune` purged nothing without + a message. The name is now read using the hostname the `snapshots` table + stores with each snapshot, cut at its first `.` as it is in the ID. -- 2026-10-07: Made `remote info` stop reporting a snapshot's blobs as - orphaned when its manifest cannot be read, and stop printing raw - names from under `metadata/` - ([issue #228](https://git.eeqj.de/sneak/vaultik/issues/228)). A - manifest it failed to read was skipped, so that snapshot's blobs were - counted as orphaned and the report advised running `vaultik prune`. - The orphan figures are now unknown in that case, with no prune - advice, and `--json` gives them as `null` with the unreadable remote - keys in `unreadable_manifests`. A name under `metadata/` that is not - 64 lowercase hex characters is now skipped with a warning instead of - being printed, control characters included. A manifest under a - skipped name is then not read either, so it also leaves the orphan +- 2026-10-07: Made `remote info` stop reporting a snapshot's blobs as orphaned + when its manifest cannot be read, and stop printing raw names from under + `metadata/` ([issue #228](https://git.eeqj.de/sneak/vaultik/issues/228)). A + manifest it failed to read was skipped, so that snapshot's blobs were counted + as orphaned and the report advised running `vaultik prune`. The orphan figures + are now unknown in that case, with no prune advice, and `--json` gives them as + `null` with the unreadable remote keys in `unreadable_manifests`. A name under + `metadata/` that is not 64 lowercase hex characters is now skipped with a + warning instead of being printed, control characters included. A manifest + under a skipped name is then not read either, so it also leaves the orphan figures unknown, and `--json` counts such manifests in - `skipped_manifest_count`. A directory with no manifest in it, as left - by an interrupted backup, leaves the figures known. + `skipped_manifest_count`. A directory with no manifest in it, as left by an + interrupted backup, leaves the figures known. - 2026-10-07: Made `config set` keep a string that looks like a number - ([issue #229](https://git.eeqj.de/sneak/vaultik/issues/229)). It wrote - every value unquoted, and `config.Load` reads the file through untyped - YAML, so an access key `00112233` loaded as `38043` and a hostname `007` - as `7`. A value for a string setting in `config.Config` is now tagged as - a YAML string, which the file quotes wherever YAML would read a number or - a boolean; other settings are still written unquoted. + ([issue #229](https://git.eeqj.de/sneak/vaultik/issues/229)). It wrote every + value unquoted, and `config.Load` reads the file through untyped YAML, so an + access key `00112233` loaded as `38043` and a hostname `007` as `7`. A value + for a string setting in `config.Config` is now tagged as a YAML string, which + the file quotes wherever YAML would read a number or a boolean; other settings + are still written unquoted. -- 2026-10-07: Made a backup notice a file rewritten with its size - unchanged and a new mtime in the same second as the one in the index - ([issue #226](https://git.eeqj.de/sneak/vaultik/issues/226)). The - `files` table held mtime in whole seconds and the scanner compared - whole seconds, so every later snapshot kept the old content. A new - `mtime_nsec` column holds the nanoseconds within the second that - `mtime` holds, and the scanner compares the full mtime. A local index - created before the change lacks the column and is rebuilt with - `vaultik database delete` and a full backup. +- 2026-10-07: Made a backup notice a file rewritten with its size unchanged and + a new mtime in the same second as the one in the index + ([issue #226](https://git.eeqj.de/sneak/vaultik/issues/226)). The `files` + table held mtime in whole seconds and the scanner compared whole seconds, so + every later snapshot kept the old content. A new `mtime_nsec` column holds the + nanoseconds within the second that `mtime` holds, and the scanner compares the + full mtime. A local index created before the change lacks the column and is + rebuilt with `vaultik database delete` and a full backup. - 2026-10-07: Made taking the process-wide lock atomic - ([issue #227](https://git.eeqj.de/sneak/vaultik/issues/227)). The lock - read `vaultik.pid`, checked whether that PID was alive and then wrote - its own, so two writers started together could both pass the check and - both run. It is now an `flock` on `vaultik.pid`, held until the run - ends; the kernel drops it when the process exits, so a crash leaves no - lock behind. A clean exit now empties the file instead of deleting it, - because deleting it would let two later runs each lock a different - file. + ([issue #227](https://git.eeqj.de/sneak/vaultik/issues/227)). The lock read + `vaultik.pid`, checked whether that PID was alive and then wrote its own, so + two writers started together could both pass the check and both run. It is now + an `flock` on `vaultik.pid`, held until the run ends; the kernel drops it when + the process exits, so a crash leaves no lock behind. A clean exit now empties + the file instead of deleting it, because deleting it would let two later runs + each lock a different file. -- 2026-10-06: Made `snapshot remove --json` write only its document to - stdout when the destination store cannot be reached - ([issue #251](https://git.eeqj.de/sneak/vaultik/issues/251)). Its - warning that the snapshot's metadata was left on the destination store - went to stdout ahead of the document, breaking `| jq` on a command that - exited 0. Under `--json` the warning now reaches stderr only, through - the logger. The warning, the README and the command's help said - `vaultik prune` would finish the cleanup, but `prune` never removes - snapshot metadata; they now say to run `vaultik snapshot remove` for the - snapshot again once the destination store is reachable. +- 2026-10-06: Made `snapshot remove --json` write only its document to stdout + when the destination store cannot be reached + ([issue #251](https://git.eeqj.de/sneak/vaultik/issues/251)). Its warning that + the snapshot's metadata was left on the destination store went to stdout ahead + of the document, breaking `| jq` on a command that exited 0. Under `--json` + the warning now reaches stderr only, through the logger. The warning, the + README and the command's help said `vaultik prune` would finish the cleanup, + but `prune` never removes snapshot metadata; they now say to run + `vaultik snapshot remove` for the snapshot again once the destination store is + reachable. -- 2026-10-06: Made the backup summary and the `snapshots` row count each - file, byte and upload once - ([issue #225](https://git.eeqj.de/sneak/vaultik/issues/225)). The - scanner added a file's bytes again for each new chunk and counted a - file as unchanged for each chunk already stored, so a first backup - reported twice its size and "backed up" could go negative. Upload - figures came from the progress reporter, which `--cron` turns off, and - `blob_count` counted earlier paths' blobs again for each later path. - The scanner now counts uploads itself; `blob_size`, - `blob_uncompressed_size` and `compression_ratio` describe the blobs - the snapshot references, and `docs/DATAMODEL.md` now says - `chunk_count` and `blob_count` count what the run added. +- 2026-10-06: Made the backup summary and the `snapshots` row count each file, + byte and upload once + ([issue #225](https://git.eeqj.de/sneak/vaultik/issues/225)). The scanner + added a file's bytes again for each new chunk and counted a file as unchanged + for each chunk already stored, so a first backup reported twice its size and + "backed up" could go negative. Upload figures came from the progress reporter, + which `--cron` turns off, and `blob_count` counted earlier paths' blobs again + for each later path. The scanner now counts uploads itself; `blob_size`, + `blob_uncompressed_size` and `compression_ratio` describe the blobs the + snapshot references, and `docs/DATAMODEL.md` now says `chunk_count` and + `blob_count` count what the run added. -- 2026-10-06: Made command output follow the README's stdout and stderr - rules ([issue #224](https://git.eeqj.de/sneak/vaultik/issues/224)). The - startup banner went to stdout, so a `completion` script or a - `config get` value started with it; the banner now goes to stderr. A - failing `remote info`, `prune` or `snapshot remove` under `--json` - printed nothing on either stream, and now reports its error on stderr. - `snapshot verify --quiet` printed its whole report; it now prints - none, and a failure still reaches stderr with the same exit status. +- 2026-10-06: Made command output follow the README's stdout and stderr rules + ([issue #224](https://git.eeqj.de/sneak/vaultik/issues/224)). The startup + banner went to stdout, so a `completion` script or a `config get` value + started with it; the banner now goes to stderr. A failing `remote info`, + `prune` or `snapshot remove` under `--json` printed nothing on either stream, + and now reports its error on stderr. `snapshot verify --quiet` printed its + whole report; it now prints none, and a failure still reaches stderr with the + same exit status. -- 2026-10-06: Made a backup without `--cron` of a snapshot with two or - more `paths` complete instead of panicking with `close of closed - channel` ([issue #253](https://git.eeqj.de/sneak/vaultik/issues/253)). - `Scan` runs once per path and started and stopped the progress - reporter each time, and a second stop panics. The reporter is now - started and stopped once per snapshot, around the scans of all its - paths. +- 2026-10-06: Made a backup without `--cron` of a snapshot with two or more + `paths` complete instead of panicking with `close of closed channel` + ([issue #253](https://git.eeqj.de/sneak/vaultik/issues/253)). `Scan` runs once + per path and started and stopped the progress reporter each time, and a second + stop panics. The reporter is now started and stopped once per snapshot, around + the scans of all its paths. -- 2026-10-06: Made a restore path argument select only that path and - what is beneath it - ([issue #223](https://git.eeqj.de/sneak/vaultik/issues/223)). The - lookup matched with SQL `LIKE`, so `/home/u/doc` also restored - `doc2`, `DOC` and `doc.txt.bak`, and a `_` or `%` in the path acted - as a wildcard. A backup used the same lookup to load the known files - of each configured path, so files of a longer sibling path were - counted as deleted. `FileRepository.ListUnderPath`, which replaces - `ListByPrefix`, returns the file at the path and every file whose - path starts with the path plus `/`, compared case-sensitively. +- 2026-10-06: Made a restore path argument select only that path and what is + beneath it ([issue #223](https://git.eeqj.de/sneak/vaultik/issues/223)). The + lookup matched with SQL `LIKE`, so `/home/u/doc` also restored `doc2`, `DOC` + and `doc.txt.bak`, and a `_` or `%` in the path acted as a wildcard. A backup + used the same lookup to load the known files of each configured path, so files + of a longer sibling path were counted as deleted. + `FileRepository.ListUnderPath`, which replaces `ListByPrefix`, returns the + file at the path and every file whose path starts with the path plus `/`, + compared case-sensitively. -- 2026-10-06: Made restore return an error instead of panicking on a - malformed snapshot database - ([issue #231](https://git.eeqj.de/sneak/vaultik/issues/231)). A chunk - hash shorter than 16 characters crashed the error message naming it, - and `--verify` dereferenced a missing `chunks` row and allocated - whatever chunk size the database gave. Those messages now go through - `shortHash`, a missing row is an error, and `--verify` rejects a - negative size and hashes each chunk as a stream. +- 2026-10-06: Made restore return an error instead of panicking on a malformed + snapshot database + ([issue #231](https://git.eeqj.de/sneak/vaultik/issues/231)). A chunk hash + shorter than 16 characters crashed the error message naming it, and `--verify` + dereferenced a missing `chunks` row and allocated whatever chunk size the + database gave. Those messages now go through `shortHash`, a missing row is an + error, and `--verify` rejects a negative size and hashes each chunk as a + stream. - 2026-10-06: Made `s3://bucket/prefix` and `s3://bucket/prefix/` the same - destination ([issue #222](https://git.eeqj.de/sneak/vaultik/issues/222)). - The S3 client put the prefix directly in front of each key, so a prefix - without a trailing slash stored `prefixblobs/...`. A non-empty prefix is - now joined to every key with one `/`, giving the README's - `//blobs/...` layout. The `s3.prefix` config setting goes - through the same client and gets the same join. + destination ([issue #222](https://git.eeqj.de/sneak/vaultik/issues/222)). The + S3 client put the prefix directly in front of each key, so a prefix without a + trailing slash stored `prefixblobs/...`. A non-empty prefix is now joined to + every key with one `/`, giving the README's `//blobs/...` + layout. The `s3.prefix` config setting goes through the same client and gets + the same join. -- 2026-10-06: Made restore apply owners, modes and times in an order - that keeps them - ([issue #219](https://git.eeqj.de/sneak/vaultik/issues/219)). A - directory got its stored mode and mtime before its contents were - written, so a read-only directory came back without its files and a - non-empty one carried the time of the restore. Directories are now - created owner-only and get their stored owner, mode and mtime once the - restore loop is done, each before its parent. A file's mode is applied - after its chown, which on Linux clears setuid and setgid, and a - symlink gets its stored owner (as root) and mtime on the link itself. +- 2026-10-06: Made restore apply owners, modes and times in an order that keeps + them ([issue #219](https://git.eeqj.de/sneak/vaultik/issues/219)). A directory + got its stored mode and mtime before its contents were written, so a read-only + directory came back without its files and a non-empty one carried the time of + the restore. Directories are now created owner-only and get their stored + owner, mode and mtime once the restore loop is done, each before its parent. A + file's mode is applied after its chown, which on Linux clears setuid and + setgid, and a symlink gets its stored owner (as root) and mtime on the link + itself. -- 2026-10-06: Made `go.mod` what `go mod tidy` writes, so the - pre-commit hook no longer stops every commit - ([issue #246](https://git.eeqj.de/sneak/vaultik/issues/246)). A test - in `internal/cli` imports `github.com/spf13/pflag` directly, but - `go.mod` still marked it `// indirect`, and `script/precommit` fails - whenever the tidy changes `go.mod`. It is now in the direct `require` - block. +- 2026-10-06: Made `go.mod` what `go mod tidy` writes, so the pre-commit hook no + longer stops every commit + ([issue #246](https://git.eeqj.de/sneak/vaultik/issues/246)). A test in + `internal/cli` imports `github.com/spf13/pflag` directly, but `go.mod` still + marked it `// indirect`, and `script/precommit` fails whenever the tidy + changes `go.mod`. It is now in the direct `require` block. -- 2026-10-06: Made the README's steps for restoring on another machine - work ([issue #221](https://git.eeqj.de/sneak/vaultik/issues/221)). - `config init` wrote a placeholder recipient that `config.Load` - rejects, so every command on the new machine failed before reaching - the store. The file now has an empty `age_recipients` list, - `config.Load` accepts an empty list, and `snapshot create` refuses to - run without a recipient. +- 2026-10-06: Made the README's steps for restoring on another machine work + ([issue #221](https://git.eeqj.de/sneak/vaultik/issues/221)). `config init` + wrote a placeholder recipient that `config.Load` rejects, so every command on + the new machine failed before reaching the store. The file now has an empty + `age_recipients` list, `config.Load` accepts an empty list, and + `snapshot create` refuses to run without a recipient. -- 2026-10-06: Made `snapshot restore --skip-errors` skip the files that - need a blob it cannot download - ([issue #218](https://git.eeqj.de/sneak/vaultik/issues/218)). A missing - or damaged blob ended the restore even with the flag, after restoring - whichever files happened to come first. Every file that needs such a - blob is now reported as failed, the rest are restored, and the command - still exits non-zero. Without the flag the blob error still aborts. +- 2026-10-06: Made `snapshot restore --skip-errors` skip the files that need a + blob it cannot download + ([issue #218](https://git.eeqj.de/sneak/vaultik/issues/218)). A missing or + damaged blob ended the restore even with the flag, after restoring whichever + files happened to come first. Every file that needs such a blob is now + reported as failed, the rest are restored, and the command still exits + non-zero. Without the flag the blob error still aborts. -- 2026-10-06: Re-vendored the canonical files from `sneak/prompts` at - `dd4027b` ([issue #213](https://git.eeqj.de/sneak/vaultik/issues/213)). - Linting and testing are now the `lint` and `test` phases of the - `Dockerfile`, and the build stage depends on both. `Dockerfile.lint`, - `CHECK_EPOCH` and the tests that checked them are gone; every - `docker build` in `script/` passes `--no-cache` instead. golangci-lint - is v2.14.0, with its new findings fixed in the code, and the rules in - `CLAUDE.md` now live in `AGENTS.md`. +- 2026-10-06: Re-vendored the canonical files from `sneak/prompts` at `dd4027b` + ([issue #213](https://git.eeqj.de/sneak/vaultik/issues/213)). Linting and + testing are now the `lint` and `test` phases of the `Dockerfile`, and the + build stage depends on both. `Dockerfile.lint`, `CHECK_EPOCH` and the tests + that checked them are gone; every `docker build` in `script/` passes + `--no-cache` instead. golangci-lint is v2.14.0, with its new findings fixed in + the code, and the rules in `CLAUDE.md` now live in `AGENTS.md`. -- 2026-10-06: Made the local index actually run in WAL mode with a busy - timeout ([issue #217](https://git.eeqj.de/sneak/vaultik/issues/217)). - The connection settings were written in a form the SQLite driver - ignores, so the index ran without either and `snapshot list` or `info` - during a backup could make the backup's next write fail with - `database is locked`. They are now `_pragma=` parameters, and the - metadata export copies the open index with `VACUUM INTO`, because a - copy of the file alone misses rows still in the `-wal` file. +- 2026-10-06: Made the local index actually run in WAL mode with a busy timeout + ([issue #217](https://git.eeqj.de/sneak/vaultik/issues/217)). The connection + settings were written in a form the SQLite driver ignores, so the index ran + without either and `snapshot list` or `info` during a backup could make the + backup's next write fail with `database is locked`. They are now `_pragma=` + parameters, and the metadata export copies the open index with `VACUUM INTO`, + because a copy of the file alone misses rows still in the `-wal` file. -- 2026-10-06: Made a backup record the real uid and gid of files, - directories and symlinks - ([issue #216](https://git.eeqj.de/sneak/vaultik/issues/216)). The +- 2026-10-06: Made a backup record the real uid and gid of files, directories + and symlinks ([issue #216](https://git.eeqj.de/sneak/vaultik/issues/216)). The scanner asked the stat result for `Uid()` and `Gid()` methods, which - `*syscall.Stat_t` does not have, so every entry was stored as `0:0` - and a restore as root gave every file to root. It now reads the - `Uid` and `Gid` fields of `*syscall.Stat_t`. + `*syscall.Stat_t` does not have, so every entry was stored as `0:0` and a + restore as root gave every file to root. It now reads the `Uid` and `Gid` + fields of `*syscall.Stat_t`. -- 2026-10-06: Made a `file://` destination whose directory is missing - count as one that cannot be listed - ([issue #220](https://git.eeqj.de/sneak/vaultik/issues/220)). The file - backend listed a missing directory as an empty store, so with the - quickstart's USB stick unplugged `snapshot list` reported every local - snapshot as missing from the store, `snapshot remove` claimed to have - removed metadata it never reached, and `prune` dropped every local - snapshot record. Listing a missing destination directory is now an - error; a first backup still creates the directory. +- 2026-10-06: Made a `file://` destination whose directory is missing count as + one that cannot be listed + ([issue #220](https://git.eeqj.de/sneak/vaultik/issues/220)). The file backend + listed a missing directory as an empty store, so with the quickstart's USB + stick unplugged `snapshot list` reported every local snapshot as missing from + the store, `snapshot remove` claimed to have removed metadata it never + reached, and `prune` dropped every local snapshot record. Listing a missing + destination directory is now an error; a first backup still creates the + directory. -- 2026-10-06: Made `--older-than` and `--keep-newer-than` reject a - duration with characters outside its number-and-unit parts - ([issue #215](https://git.eeqj.de/sneak/vaultik/issues/215)). The - parser picked out the parts it recognised and skipped the rest, so - `1.0y` became zero and `--prune --keep-newer-than 1.0y` deleted every - snapshot of the backed-up names, the new one included. `1.0y`, - `1,5y`, `30 days`, `x7d` and a bare `0` are now errors; decimals still - work in Go units such as `1.5h`. +- 2026-10-06: Made `--older-than` and `--keep-newer-than` reject a duration with + characters outside its number-and-unit parts + ([issue #215](https://git.eeqj.de/sneak/vaultik/issues/215)). The parser + picked out the parts it recognised and skipped the rest, so `1.0y` became zero + and `--prune --keep-newer-than 1.0y` deleted every snapshot of the backed-up + names, the new one included. `1.0y`, `1,5y`, `30 days`, `x7d` and a bare `0` + are now errors; decimals still work in Go units such as `1.5h`. -- 2026-10-06: Made a backup re-chunk a known file that lists a chunk no - uploaded blob holds - ([issue #214](https://git.eeqj.de/sneak/vaultik/issues/214)). File - rows are shared by every snapshot and updated in place, so removing or - pruning the only snapshot that referenced a changed file's current - blob left an older snapshot keeping a row that matched the disk while - no blob held its chunks. The next backup skipped the file and - completed a snapshot that could not restore it. +- 2026-10-06: Made a backup re-chunk a known file that lists a chunk no uploaded + blob holds ([issue #214](https://git.eeqj.de/sneak/vaultik/issues/214)). File + rows are shared by every snapshot and updated in place, so removing or pruning + the only snapshot that referenced a changed file's current blob left an older + snapshot keeping a row that matched the disk while no blob held its chunks. + The next backup skipped the file and completed a snapshot that could not + restore it. - 2026-09-22: Routed the last direct-to-stdout command output through - `internal/ui` - ([issue #149](https://git.eeqj.de/sneak/vaultik/issues/149)). The - `version`, `info`, `remote info`, `config`, and `database delete` - commands wrote plain text straight to stdout, so they were unstyled and - ignored `--quiet`. Output now falls in two buckets, both governed by - `internal/ui`: status lines and confirmations go through its message - methods (styled, and `--quiet` silences them), while the data a command - exists to produce — the reports, the `snapshot list` table, `config get` - values, and the `--json` documents — is written plain. `--quiet` - silences the human reports and tables but never the `config get` value - or the `--json` documents, which a script depends on, and the - `database delete` confirmation prompt is always shown. The README - output-style section now states this rule. + `internal/ui` ([issue #149](https://git.eeqj.de/sneak/vaultik/issues/149)). + The `version`, `info`, `remote info`, `config`, and `database delete` commands + wrote plain text straight to stdout, so they were unstyled and ignored + `--quiet`. Output now falls in two buckets, both governed by `internal/ui`: + status lines and confirmations go through its message methods (styled, and + `--quiet` silences them), while the data a command exists to produce — the + reports, the `snapshot list` table, `config get` values, and the `--json` + documents — is written plain. `--quiet` silences the human reports and tables + but never the `config get` value or the `--json` documents, which a script + depends on, and the `database delete` confirmation prompt is always shown. The + README output-style section now states this rule. -- 2026-09-22: Validated blob hashes, offsets and lengths read back from - the destination before using them - ([issue #155](https://git.eeqj.de/sneak/vaultik/issues/155)). A blob - hash taken from the downloaded database or the store listing was - trusted unchecked, so a hostile remote could drive a decrypted blob to - be written outside the cache directory (a hash like `aa/../../etc`) or - crash a command with a short or negative value. `blobDiskCache.path` - now refuses any key containing a path separator, and `ReadAt` rejects a - negative offset or length and bounds with `length > size-offset` so a - sum cannot overflow past the check. A new `isBlobHash` helper (a plain - function, not a method — the packer stores `temp-placeholder-{uuid}` as - a hash) gates `FetchBlob`, shallow and deep verify; the `blobs/` and - `metadata/` listings skip a non-conforming name with a warning; and - short-hash prefixes in log and error text go through a `shortHash` - helper that cannot panic. `verify`'s chunk reader also rejects a - negative `blob_chunks` length and streams the chunk instead of - allocating a database-supplied size. `restore.go` and - `internal/database` were left untouched to avoid colliding with the - in-flight [issue #156](https://git.eeqj.de/sneak/vaultik/issues/156) - work; the cache-path and `FetchBlob` guards already stop the unsafe - write and fetch, so restore's own `buildBlobIndexes` early check is - deferred as fail-fast defense in depth. +- 2026-09-22: Validated blob hashes, offsets and lengths read back from the + destination before using them + ([issue #155](https://git.eeqj.de/sneak/vaultik/issues/155)). A blob hash + taken from the downloaded database or the store listing was trusted unchecked, + so a hostile remote could drive a decrypted blob to be written outside the + cache directory (a hash like `aa/../../etc`) or crash a command with a short + or negative value. `blobDiskCache.path` now refuses any key containing a path + separator, and `ReadAt` rejects a negative offset or length and bounds with + `length > size-offset` so a sum cannot overflow past the check. A new + `isBlobHash` helper (a plain function, not a method — the packer stores + `temp-placeholder-{uuid}` as a hash) gates `FetchBlob`, shallow and deep + verify; the `blobs/` and `metadata/` listings skip a non-conforming name with + a warning; and short-hash prefixes in log and error text go through a + `shortHash` helper that cannot panic. `verify`'s chunk reader also rejects a + negative `blob_chunks` length and streams the chunk instead of allocating a + database-supplied size. `restore.go` and `internal/database` were left + untouched to avoid colliding with the in-flight + [issue #156](https://git.eeqj.de/sneak/vaultik/issues/156) work; the + cache-path and `FetchBlob` guards already stop the unsafe write and fetch, so + restore's own `buildBlobIndexes` early check is deferred as fail-fast defense + in depth. -- 2026-09-21: Stopped an interrupted blob upload from making a later - backup deduplicate against data that was never stored - ([issue #148](https://git.eeqj.de/sneak/vaultik/issues/148)). The - packer commits a blob's `chunks`, `blob_chunks`, and `blobs` rows - before the upload is attempted, so a failed upload left chunk rows - behind and the next run skipped re-uploading them, producing a - snapshot that reported success but could not be restored. A run now - deduplicates only against chunks held by a blob whose `uploaded_ts` is - set, and at startup drops any un-uploaded blob rows (and the chunks - they orphan) so the affected data is re-chunked and re-uploaded. Blobs - recorded with no remote backend are marked uploaded so this invariant - holds uniformly. +- 2026-09-21: Stopped an interrupted blob upload from making a later backup + deduplicate against data that was never stored + ([issue #148](https://git.eeqj.de/sneak/vaultik/issues/148)). The packer + commits a blob's `chunks`, `blob_chunks`, and `blobs` rows before the upload + is attempted, so a failed upload left chunk rows behind and the next run + skipped re-uploading them, producing a snapshot that reported success but + could not be restored. A run now deduplicates only against chunks held by a + blob whose `uploaded_ts` is set, and at startup drops any un-uploaded blob + rows (and the chunks they orphan) so the affected data is re-chunked and + re-uploaded. Blobs recorded with no remote backend are marked uploaded so this + invariant holds uniformly. -- 2026-09-22: Made restore refuse any snapshot path that would write - outside the target directory - ([issue #154](https://git.eeqj.de/sneak/vaultik/issues/154)). - `restoreFile` and `verifyRestoredFiles` joined the stored path onto the - target with no containment check, so a `..` segment or an absolute path - escaped the target and a restored symlink could redirect a later child - write anywhere on disk. Every stored path is now rejected unless - `filepath.IsLocal` accepts it with the leading separator removed, and - each existing ancestor directory below the target is `Lstat`ed to refuse - descending through a symlink; honest symlinks pointing outside the tree - are still written verbatim. age decryption proves a snapshot is - readable, not honest, and restore usually runs as root. +- 2026-09-22: Made restore refuse any snapshot path that would write outside the + target directory ([issue #154](https://git.eeqj.de/sneak/vaultik/issues/154)). + `restoreFile` and `verifyRestoredFiles` joined the stored path onto the target + with no containment check, so a `..` segment or an absolute path escaped the + target and a restored symlink could redirect a later child write anywhere on + disk. Every stored path is now rejected unless `filepath.IsLocal` accepts it + with the leading separator removed, and each existing ancestor directory below + the target is `Lstat`ed to refuse descending through a symlink; honest + symlinks pointing outside the tree are still written verbatim. age decryption + proves a snapshot is readable, not honest, and restore usually runs as root. - 2026-09-21: Stopped `--json` from silencing stderr diagnostics - ([issue #112](https://git.eeqj.de/sneak/vaultik/issues/112)). `--json` - used to be folded into `Quiet`, which pinned the log level to `WARN`, - so `prune --json` gave a machine consumer no record of the local index - rows it deleted even under `--verbose`. `--json` now quiets only the - stdout UI (the JSON document must stay clean, per - [issue #108](https://git.eeqj.de/sneak/vaultik/issues/108)); the stderr - log level follows `--verbose`/`--debug` again. The coupling was - removed the same way for `snapshot verify`, `snapshot remove`, and - `remote info`, which carried it for the same outdated reason. + ([issue #112](https://git.eeqj.de/sneak/vaultik/issues/112)). `--json` used to + be folded into `Quiet`, which pinned the log level to `WARN`, so + `prune --json` gave a machine consumer no record of the local index rows it + deleted even under `--verbose`. `--json` now quiets only the stdout UI (the + JSON document must stay clean, per + [issue #108](https://git.eeqj.de/sneak/vaultik/issues/108)); the stderr log + level follows `--verbose`/`--debug` again. The coupling was removed the same + way for `snapshot verify`, `snapshot remove`, and `remote info`, which carried + it for the same outdated reason. - 2026-09-21: Stopped `prune` from reporting a failed row count as 0 ([issue #96](https://git.eeqj.de/sneak/vaultik/issues/96)). The seven - `getTableCount` reads in `PruneDatabase` discarded their error, so a - query that could not run became a plausible `0` and the before/after - delta computed from it looked like real work. Each read now logs at - warn on failure and renders as `unknown`, never `0`, so an empty table - is distinguishable from one that could not be queried. The counts have - no `--json` representation — under `--json` the summary is suppressed - entirely — so nothing there can show a false `0`. + `getTableCount` reads in `PruneDatabase` discarded their error, so a query + that could not run became a plausible `0` and the before/after delta computed + from it looked like real work. Each read now logs at warn on failure and + renders as `unknown`, never `0`, so an empty table is distinguishable from one + that could not be queried. The counts have no `--json` representation — under + `--json` the summary is suppressed entirely — so nothing there can show a + false `0`. - 2026-09-21: Made the s3 storage backend report a missing object as `storage.ErrNotFound`, like the `file` and `rclone` backends and as the - `Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw - AWS SDK error, so `errors.Is(err, storage.ErrNotFound)` was false on s3 - and callers branched differently per backend. Added a small `s3.IsNotFound` - helper (reused by `HeadObject`) and a test that a missing key maps to - `ErrNotFound` + `Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw AWS + SDK error, so `errors.Is(err, storage.ErrNotFound)` was false on s3 and + callers branched differently per backend. Added a small `s3.IsNotFound` helper + (reused by `HeadObject`) and a test that a missing key maps to `ErrNotFound` ([issue #129](https://git.eeqj.de/sneak/vaultik/issues/129)). -- 2026-09-21: Fixed `verify --deep` reporting healthy snapshots as - corrupt. Its final blob-integrity check hashed the encrypted - downloaded bytes with a single SHA256 and compared that to the blob - ID, which is the double SHA256 of the plaintext, so the two could - never match. It now hashes the decompressed plaintext and compares the - double SHA256. Added a test that backs up a real snapshot, deep-verifies - it, then flips a byte in one stored blob and confirms deep verification - then fails +- 2026-09-21: Fixed `verify --deep` reporting healthy snapshots as corrupt. Its + final blob-integrity check hashed the encrypted downloaded bytes with a single + SHA256 and compared that to the blob ID, which is the double SHA256 of the + plaintext, so the two could never match. It now hashes the decompressed + plaintext and compares the double SHA256. Added a test that backs up a real + snapshot, deep-verifies it, then flips a byte in one stored blob and confirms + deep verification then fails ([issue #131](https://git.eeqj.de/sneak/vaultik/issues/131)). -- 2026-09-21: Made `snapshot create` VACUUM the per-snapshot metadata - database through the `modernc.org/sqlite` driver instead of shelling - out to the external `sqlite` command-line binary (issue #120). A - backup no longer needs that binary on `PATH`, so `make check` passes - on a stock `go install` host; `script/bootstrap` and the `Dockerfile` - (both the test-build and the shipped runtime stage) no longer install - it, and a new test asserts the uploaded database keeps no pages from - deleted rows. Dropped the now-false note on the 2026-08-07 entry below - that said bootstrap installs it. -- 2026-09-21: Made `.gitea/workflows/check.yml` run on pushes to `main` - and `next` and on pull requests against either, so unit PRs (whose - base is `next`) and `next` itself get a CI run instead of relying on a - local `make check` +- 2026-09-21: Made `snapshot create` VACUUM the per-snapshot metadata database + through the `modernc.org/sqlite` driver instead of shelling out to the + external `sqlite` command-line binary (issue #120). A backup no longer needs + that binary on `PATH`, so `make check` passes on a stock `go install` host; + `script/bootstrap` and the `Dockerfile` (both the test-build and the shipped + runtime stage) no longer install it, and a new test asserts the uploaded + database keeps no pages from deleted rows. Dropped the now-false note on the + 2026-08-07 entry below that said bootstrap installs it. +- 2026-09-21: Made `.gitea/workflows/check.yml` run on pushes to `main` and + `next` and on pull requests against either, so unit PRs (whose base is `next`) + and `next` itself get a CI run instead of relying on a local `make check` ([issue #122](https://git.eeqj.de/sneak/vaultik/issues/122)). - 2026-09-21: Hash-verified the Go toolchain in the release workflow ([issue #105](https://git.eeqj.de/sneak/vaultik/issues/105)). New - `script/install-go` downloads the exact `go.dev` archive for `go.mod`'s - `go` directive and refuses it unless its sha256 matches a value - committed in the script; `.gitea/workflows/release.yml` calls it - instead of `actions/setup-go`, which verified the downloaded toolchain - against nothing in the repo. `GOTOOLCHAIN: local` on the release step - keeps that exact compiler from auto-switching. Bumping Go now touches - `go.mod`, the checksum, and the `Dockerfile` `golang` digest together. + `script/install-go` downloads the exact `go.dev` archive for `go.mod`'s `go` + directive and refuses it unless its sha256 matches a value committed in the + script; `.gitea/workflows/release.yml` calls it instead of `actions/setup-go`, + which verified the downloaded toolchain against nothing in the repo. + `GOTOOLCHAIN: local` on the release step keeps that exact compiler from + auto-switching. Bumping Go now touches `go.mod`, the checksum, and the + `Dockerfile` `golang` digest together. - 2026-09-21: Collapsed the two duration parsers into one and fixed the `--older-than` months example - ([issue #123](https://git.eeqj.de/sneak/vaultik/issues/123)). Two - functions named `parseDuration` existed with different grammars; + ([issue #123](https://git.eeqj.de/sneak/vaultik/issues/123)). Two functions + named `parseDuration` existed with different grammars; `snapshot purge --older-than` and `--keep-newer-than` both already went through the one in `internal/vaultik`, while the richer copy in - `internal/cli/duration.go` was reachable only from its own test. Kept - the live-path parser and deleted the unused one, so no flag's accepted - grammar changes. The trap the issue was filed over: `README.md` - documented `6m` as the months example for `--older-than`, but `m` is - minutes, so the documented command deleted every snapshot older than - six minutes on a destructive flag. Corrected the doc to `6mo` and put - both flags' help text on one example list that states `m` is minutes - and `mo` is months. The surviving parser now rejects negatives, which - it previously accepted (`-5h`) or silently made positive (`-5d`). - Table-driven tests cover every unit, `6m` as six minutes, `6mo` as 180 - days, and rejection of a bare number, an unknown unit, and a negative. + `internal/cli/duration.go` was reachable only from its own test. Kept the + live-path parser and deleted the unused one, so no flag's accepted grammar + changes. The trap the issue was filed over: `README.md` documented `6m` as the + months example for `--older-than`, but `m` is minutes, so the documented + command deleted every snapshot older than six minutes on a destructive flag. + Corrected the doc to `6mo` and put both flags' help text on one example list + that states `m` is minutes and `mo` is months. The surviving parser now + rejects negatives, which it previously accepted (`-5h`) or silently made + positive (`-5d`). Table-driven tests cover every unit, `6m` as six minutes, + `6mo` as 180 days, and rejection of a bare number, an unknown unit, and a + negative. -- 2026-08-10: Moved every lint run into its own container, as a build - step ([issue #113](https://git.eeqj.de/sneak/vaultik/issues/113)). - New root `Dockerfile.lint`, built by `script/lint`, runs - `golangci-lint run --config .golangci.yml ./...` as a `RUN` - instruction in the digest-pinned `golangci/golangci-lint` image: a - successful build of that file *is* a clean lint, and it works even - where the daemon is remote and bind mounts are impossible. That - `FROM` line is now the only pin of the linter version in the repo. +- 2026-08-10: Moved every lint run into its own container, as a build step + ([issue #113](https://git.eeqj.de/sneak/vaultik/issues/113)). New root + `Dockerfile.lint`, built by `script/lint`, runs + `golangci-lint run --config .golangci.yml ./...` as a `RUN` instruction in the + digest-pinned `golangci/golangci-lint` image: a successful build of that file + _is_ a clean lint, and it works even where the daemon is remote and bind + mounts are impossible. That `FROM` line is now the only pin of the linter + version in the repo. - This supersedes the per-worktree cache isolation landed for - [issue #99](https://git.eeqj.de/sneak/vaultik/issues/99). Isolation - fixed cross-worktree contamination but not lock contention — two - concurrent runs with entirely separate cache directories still - collided. A container per run has its own cache and its own lock, so - the whole class is gone, and with it the per-worktree cache - machinery, the lock-retry loop, and `script/lint-audit`, which - existed to catch replayed findings from a cache that no longer - exists. The host lint path went too: no escape hatch, no - `VAULTIK_LINT_IN_CONTAINER`, no version detection. Nothing lints on - the host at any version. + This supersedes the per-worktree cache isolation landed for + [issue #99](https://git.eeqj.de/sneak/vaultik/issues/99). Isolation fixed + cross-worktree contamination but not lock contention — two concurrent runs + with entirely separate cache directories still collided. A container per run + has its own cache and its own lock, so the whole class is gone, and with it + the per-worktree cache machinery, the lock-retry loop, and + `script/lint-audit`, which existed to catch replayed findings from a cache + that no longer exists. The host lint path went too: no escape hatch, no + `VAULTIK_LINT_IN_CONTAINER`, no version detection. Nothing lints on the host + at any version. - A cached build lints nothing, so the same `CHECK_EPOCH` mechanism the - product `Dockerfile` already used is what makes the green mean - something: `ARG CHECK_EPOCH` with no default below the module layers, - a `RUN [ -n "$CHECK_EPOCH" ] || exit 1` guard, the value expanded - into each check command, and a fresh `$(date +%s%N)$$` per invocation - computed as a bare assignment. `cmd/vaultik/lintdocker_test.go` - parses both Dockerfiles and both scripts and fails if any part of - that is dropped, because every way of losing it is silent. No test - asserts that no script runs the host linter: `script/lint` is the one - lint entry point and runs `golangci-lint` only inside the container, - and keeping it that way is a review matter, not something a test - proves. + A cached build lints nothing, so the same `CHECK_EPOCH` mechanism the + product `Dockerfile` already used is what makes the green mean something: + `ARG CHECK_EPOCH` with no default below the module layers, a + `RUN [ -n "$CHECK_EPOCH" ] || exit 1` guard, the value expanded into each + check command, and a fresh `$(date +%s%N)$$` per invocation computed as a + bare assignment. `cmd/vaultik/lintdocker_test.go` parses both Dockerfiles + and both scripts and fails if any part of that is dropped, because every way + of losing it is silent. No test asserts that no script runs the host linter: + `script/lint` is the one lint entry point and runs `golangci-lint` only + inside the container, and keeping it that way is a review matter, not + something a test proves. - The product `Dockerfile` lost its lint stage rather than gaining a - second linter pin: `make lint` is now `docker build`, so the stage - would have been docker-in-docker with no daemon, and calling - `golangci-lint` directly there would have restored the two-pins drift - of [issue #78](https://git.eeqj.de/sneak/vaultik/issues/78). - `make fmt-check` moved beside `make test` in the builder stage, and - `script/cibuild` now builds `Dockerfile.lint` and then `Dockerfile`, - each with its own fresh epoch. Consequence, stated rather than left - to be found: `script/docker` builds the product image only and no - longer lints; the gates are `script/check` and `script/cibuild`. + The product `Dockerfile` lost its lint stage rather than gaining a second + linter pin: `make lint` is now `docker build`, so the stage would have been + docker-in-docker with no daemon, and calling `golangci-lint` directly there + would have restored the two-pins drift of + [issue #78](https://git.eeqj.de/sneak/vaultik/issues/78). `make fmt-check` + moved beside `make test` in the builder stage, and `script/cibuild` now + builds `Dockerfile.lint` and then `Dockerfile`, each with its own fresh + epoch. Consequence, stated rather than left to be found: `script/docker` + builds the product image only and no longer lints; the gates are + `script/check` and `script/cibuild`. - `golangci-lint config verify` runs as its own epoch-keyed layer, - above the lint. `golangci-lint run` rejects a config it cannot parse - but silently ignores an unknown top-level *key*: renaming `linters:` - to `linterz:` discarded `default: all` and every threshold and still - exited 0 on a tree the real config fails. `config verify` catches - that, and it does so with the network off at this pin — checked under - `docker run --network none`, not assumed. An earlier revision omitted - it on the claim that it fetches its schema over live HTTPS; that - claim was false at v2.12.2. + `golangci-lint config verify` runs as its own epoch-keyed layer, above the + lint. `golangci-lint run` rejects a config it cannot parse but silently + ignores an unknown top-level _key_: renaming `linters:` to `linterz:` + discarded `default: all` and every threshold and still exited 0 on a tree + the real config fails. `config verify` catches that, and it does so with the + network off at this pin — checked under `docker run --network none`, not + assumed. An earlier revision omitted it on the claim that it fetches its + schema over live HTTPS; that claim was false at v2.12.2. - `script/lint-fix` is kept, reimplemented as a - bind-mounted `docker run` against the image parsed out of - `Dockerfile.lint` — it cannot be a build step, because fixes have to - land in the worktree — and marked in its header as a developer - convenience that no gate reads. + `script/lint-fix` is kept, reimplemented as a bind-mounted `docker run` + against the image parsed out of `Dockerfile.lint` — it cannot be a build + step, because fixes have to land in the worktree — and marked in its header + as a developer convenience that no gate reads. -- 2026-08-09: Finished the `--json` stdout contract and gave `make build` - a rule ([issue #108](https://git.eeqj.de/sneak/vaultik/issues/108), - [issue #110](https://git.eeqj.de/sneak/vaultik/issues/110)). Two - unrelated defects of the same shape — a command reporting something it - did not do — landed together because both are small. +- 2026-08-09: Finished the `--json` stdout contract and gave `make build` a rule + ([issue #108](https://git.eeqj.de/sneak/vaultik/issues/108), + [issue #110](https://git.eeqj.de/sneak/vaultik/issues/110)). Two unrelated + defects of the same shape — a command reporting something it did not do — + landed together because both are small. - `CleanupLocalSnapshots` wrote three prose lines to stdout with no - `--json` awareness, covering every branch of the function, so no input - avoided them and `vaultik prune --json | jq` failed even after - [issue #106](https://git.eeqj.de/sneak/vaultik/issues/106) removed the - banner. `-q` never helped either: `printlnStdout` and `stdoutf` write - straight to `Vaultik.Stdout` and never consult `Vaultik.UI`, which is - what `SetQuiet` affects. The issue offered three fixes and asked for a - decision. Taken: thread `*PruneOptions` into the function and gate each - write on `!opts.JSON`, matching `PruneBlobs` (its sibling phase, which - already takes the same struct), `RemoveSnapshot` and `remote info`, so - the package has one pattern rather than two. Rejected: moving the lines - to `log.Info`, because the logger's default level is `slog.LevelWarn`, - so that would not relocate them to stderr — it would delete them from a - plain `vaultik prune`, and the removal of rows from the local index is - not something to narrate only under `--verbose`. Also rejected: putting - the stale-record count into `PruneBlobsResult`, whose every field is - blob-scoped and which is produced by the later phase; a prune document - covering both phases is a reasonable thing to want, but it is a schema - design question and not a stream-hygiene fix. The narration is - duplicated as `log.Info` records, which `PruneBlobs` already does - alongside its own prints, so the events survive on stderr for anyone - running `--verbose`. + `CleanupLocalSnapshots` wrote three prose lines to stdout with no `--json` + awareness, covering every branch of the function, so no input avoided them + and `vaultik prune --json | jq` failed even after + [issue #106](https://git.eeqj.de/sneak/vaultik/issues/106) removed the + banner. `-q` never helped either: `printlnStdout` and `stdoutf` write + straight to `Vaultik.Stdout` and never consult `Vaultik.UI`, which is what + `SetQuiet` affects. The issue offered three fixes and asked for a decision. + Taken: thread `*PruneOptions` into the function and gate each write on + `!opts.JSON`, matching `PruneBlobs` (its sibling phase, which already takes + the same struct), `RemoveSnapshot` and `remote info`, so the package has one + pattern rather than two. Rejected: moving the lines to `log.Info`, because + the logger's default level is `slog.LevelWarn`, so that would not relocate + them to stderr — it would delete them from a plain `vaultik prune`, and the + removal of rows from the local index is not something to narrate only under + `--verbose`. Also rejected: putting the stale-record count into + `PruneBlobsResult`, whose every field is blob-scoped and which is produced + by the later phase; a prune document covering both phases is a reasonable + thing to want, but it is a schema design question and not a stream-hygiene + fix. The narration is duplicated as `log.Info` records, which `PruneBlobs` + already does alongside its own prints, so the events survive on stderr for + anyone running `--verbose`. - `make build` printed "Nothing to be done for 'build'" and exited 0 - without producing a binary: `build` was listed in `.PHONY` with no - `build:` rule anywhere, and declaring a name phony is exactly what - converts make's "No rule to make target" error into a silent success. - Fixed with `build: vaultik`, keeping `vaultik:` as the file rule. The - audit the issue asked for covers all 19 `.PHONY` names; `build` was the - only one without a rule, and `vaultik` is correctly absent from - `.PHONY`, being a real file target. + `make build` printed "Nothing to be done for 'build'" and exited 0 without + producing a binary: `build` was listed in `.PHONY` with no `build:` rule + anywhere, and declaring a name phony is exactly what converts make's "No + rule to make target" error into a silent success. Fixed with + `build: vaultik`, keeping `vaultik:` as the file rule. The audit the issue + asked for covers all 19 `.PHONY` names; `build` was the only one without a + rule, and `vaultik` is correctly absent from `.PHONY`, being a real file + target. - Tests, each verified to fail with the fix reverted rather than assumed - to: `CleanupLocalSnapshots` leaves stdout untouched under `--json` in - all three branches (stale records, none, empty index) and still emits - every line without it, so the guard cannot be satisfied by deleting the - output; `prune --json` run end to end through `Entry`, cobra and fx - over the process's real stdout descriptor against a `file://` store, - asserting exactly one JSON document, in both the stale and non-stale - branches; and a parse of the `Makefile` asserting every `.PHONY` name - has a rule and that `build` reaches the rule that produces the binary, - which keeps the audit true for names added later. That last one is a - parse rather than an invocation of `make`, since `make test` is what - runs it and shelling back into `make build` would nest a build inside - the test run. The property a parse cannot establish — that the recipe - still fails when the build fails — was verified by hand against a - deliberately broken tree: `make build` exits 2 and produces nothing. - `cmd/vaultik` gains its first test file, so `make test` now reports 16 - packages `ok` where it reported 15. + Tests, each verified to fail with the fix reverted rather than assumed to: + `CleanupLocalSnapshots` leaves stdout untouched under `--json` in all three + branches (stale records, none, empty index) and still emits every line + without it, so the guard cannot be satisfied by deleting the output; + `prune --json` run end to end through `Entry`, cobra and fx over the + process's real stdout descriptor against a `file://` store, asserting + exactly one JSON document, in both the stale and non-stale branches; and a + parse of the `Makefile` asserting every `.PHONY` name has a rule and that + `build` reaches the rule that produces the binary, which keeps the audit + true for names added later. That last one is a parse rather than an + invocation of `make`, since `make test` is what runs it and shelling back + into `make build` would nest a build inside the test run. The property a + parse cannot establish — that the recipe still fails when the build fails — + was verified by hand against a deliberately broken tree: `make build` exits + 2 and produces nothing. `cmd/vaultik` gains its first test file, so + `make test` now reports 16 packages `ok` where it reported 15. -- 2026-08-09: Stopped the startup banner from contaminating `--json` - documents ([issue #106](https://git.eeqj.de/sneak/vaultik/issues/106)). - `Entry` writes the banner to stdout before cobra parses anything, and - the flag scan that suppresses it knew `--quiet`, `-q` and `--cron` but - not `--json`, so every `--json` document arrived behind two lines of - prose and a blank line, and `vaultik snapshot list --json | jq` failed. - With the logger already on stderr from - [issue #82](https://git.eeqj.de/sneak/vaultik/issues/82), this was the - last writer that could put something on stdout that the caller did not - ask for. The design question the issue raised — extend the raw-argv - scan, or move the banner after parsing — is answered in favour of the - scan: the banner is printed first deliberately, so that it still - appears when cobra rejects the arguments and on `--help`, and after - parsing there is no single place that covers those paths. The stated - cost of the scan, that `--json` is a subcommand flag matched anywhere - in the vector, is a cost `--cron` already carries — it exists only on - `snapshot create` — so this adds an instance of an accepted - imprecision rather than a new kind, and the two error directions are - not symmetric: a false positive loses a decorative banner, a false - negative corrupts a document. Regression tests at the CLI layer, where - `internal/vaultik`'s existing guard cannot reach: one runs `Entry` - itself over the process's real stdout descriptor, through cobra and fx - to the document, made hermetic by `file://` storage; a second covers - the argument vectors of all five `--json` commands; a third asserts the - banner is still printed without a suppressing flag, so the first - cannot be satisfied by deleting the banner. Also corrected `AGENTS.md` - policy 9, which still keyed the structured-log format on stdout's - TTY-ness after #82 moved that decision to stderr — a rules file that - misdescribes the code misleads exactly the readers who trust it most. - Two smaller findings from the same review: `bytesAttrKey`'s - human-readable byte formatting silently stopped applying under an open - group, because the key reaching the comparison is group-qualified - (`transfer.bytes`), now matched on its final segment and tested both - ways; and `listEnv.stderr` in `snapshot_list_test.go`, assigned but - never read since those tests began capturing the process's stderr, is - removed. `Vaultik.Stderr` is kept — nothing writes to it today, which - its comment now says outright. +- 2026-08-09: Stopped the startup banner from contaminating `--json` documents + ([issue #106](https://git.eeqj.de/sneak/vaultik/issues/106)). `Entry` writes + the banner to stdout before cobra parses anything, and the flag scan that + suppresses it knew `--quiet`, `-q` and `--cron` but not `--json`, so every + `--json` document arrived behind two lines of prose and a blank line, and + `vaultik snapshot list --json | jq` failed. With the logger already on stderr + from [issue #82](https://git.eeqj.de/sneak/vaultik/issues/82), this was the + last writer that could put something on stdout that the caller did not ask + for. The design question the issue raised — extend the raw-argv scan, or move + the banner after parsing — is answered in favour of the scan: the banner is + printed first deliberately, so that it still appears when cobra rejects the + arguments and on `--help`, and after parsing there is no single place that + covers those paths. The stated cost of the scan, that `--json` is a subcommand + flag matched anywhere in the vector, is a cost `--cron` already carries — it + exists only on `snapshot create` — so this adds an instance of an accepted + imprecision rather than a new kind, and the two error directions are not + symmetric: a false positive loses a decorative banner, a false negative + corrupts a document. Regression tests at the CLI layer, where + `internal/vaultik`'s existing guard cannot reach: one runs `Entry` itself over + the process's real stdout descriptor, through cobra and fx to the document, + made hermetic by `file://` storage; a second covers the argument vectors of + all five `--json` commands; a third asserts the banner is still printed + without a suppressing flag, so the first cannot be satisfied by deleting the + banner. Also corrected `AGENTS.md` policy 9, which still keyed the + structured-log format on stdout's TTY-ness after #82 moved that decision to + stderr — a rules file that misdescribes the code misleads exactly the readers + who trust it most. Two smaller findings from the same review: `bytesAttrKey`'s + human-readable byte formatting silently stopped applying under an open group, + because the key reaching the comparison is group-qualified (`transfer.bytes`), + now matched on its final segment and tested both ways; and `listEnv.stderr` in + `snapshot_list_test.go`, assigned but never read since those tests began + capturing the process's stderr, is removed. `Vaultik.Stderr` is kept — nothing + writes to it today, which its comment now says outright. -- 2026-08-09: Moved the logger to stderr and fixed `TTYHandler`'s - discarded attributes - ([issue #82](https://git.eeqj.de/sneak/vaultik/issues/82), - [issue #97](https://git.eeqj.de/sneak/vaultik/issues/97)). Two defects - in `internal/log`, fixed together because both live in the handler - construction path. The first: both handlers were built over - `os.Stdout`, and `WARN`/`ERROR` are never suppressed, so a config file - with group- or world-readable permissions was enough to put a log - record inside a `--json` document and break `jq`. Diagnostics now go - to stderr, and the TTY/JSON format choice follows stderr rather than - stdout — testing the wrong stream would colorize records on a - redirected stderr whenever stdout happened to be a terminal. This is - user-visible: `--verbose` and `--debug` output moves to stderr too, - which is documented in `README.md` under "stdout and stderr". It also - let the local workaround in `internal/vaultik/snapshot_list.go` go: - `warnWhileListing` had been hand-rolling structured-log formatting to - reach a non-stdout writer, and the `jsonOutput` parameter threaded - through the remote-listing helpers existed only to choose between the - two writers. The collect-then-emit machinery around `listingWarning` - stays, but on its remaining merit — warnings emitted in key order - after `group.Wait()` are deterministic run to run, where emitting from - the fetch workers would order them by network timing. The second - defect: `TTYHandler.WithAttrs` and `WithGroup` discarded their - arguments and returned the receiver while their doc comments claimed - otherwise, so `log.With` attributes vanished on a terminal and - appeared correctly in CI — failing precisely when someone is debugging - interactively. Both now return a new handler (the receiver is never - written to, since `slog` permits concurrent derivation), attributes - persist across records, and grouping is implemented as dotted key - prefixes, which is the only honest rendering for a format with nowhere - to nest. New tests cover both, including one that feeds the same - derivation chain to the TTY and JSON handlers and compares the - attribute sets, so the two paths cannot drift apart again. Found and - filed while verifying: the startup banner is written to stdout and - `--json` does not suppress it - ([issue #106](https://git.eeqj.de/sneak/vaultik/issues/106)), which is - a separate writer on a separate path and the remaining source of - stdout contamination. +- 2026-08-09: Moved the logger to stderr and fixed `TTYHandler`'s discarded + attributes ([issue #82](https://git.eeqj.de/sneak/vaultik/issues/82), + [issue #97](https://git.eeqj.de/sneak/vaultik/issues/97)). Two defects in + `internal/log`, fixed together because both live in the handler construction + path. The first: both handlers were built over `os.Stdout`, and `WARN`/`ERROR` + are never suppressed, so a config file with group- or world-readable + permissions was enough to put a log record inside a `--json` document and + break `jq`. Diagnostics now go to stderr, and the TTY/JSON format choice + follows stderr rather than stdout — testing the wrong stream would colorize + records on a redirected stderr whenever stdout happened to be a terminal. This + is user-visible: `--verbose` and `--debug` output moves to stderr too, which + is documented in `README.md` under "stdout and stderr". It also let the local + workaround in `internal/vaultik/snapshot_list.go` go: `warnWhileListing` had + been hand-rolling structured-log formatting to reach a non-stdout writer, and + the `jsonOutput` parameter threaded through the remote-listing helpers existed + only to choose between the two writers. The collect-then-emit machinery around + `listingWarning` stays, but on its remaining merit — warnings emitted in key + order after `group.Wait()` are deterministic run to run, where emitting from + the fetch workers would order them by network timing. The second defect: + `TTYHandler.WithAttrs` and `WithGroup` discarded their arguments and returned + the receiver while their doc comments claimed otherwise, so `log.With` + attributes vanished on a terminal and appeared correctly in CI — failing + precisely when someone is debugging interactively. Both now return a new + handler (the receiver is never written to, since `slog` permits concurrent + derivation), attributes persist across records, and grouping is implemented as + dotted key prefixes, which is the only honest rendering for a format with + nowhere to nest. New tests cover both, including one that feeds the same + derivation chain to the TTY and JSON handlers and compares the attribute sets, + so the two paths cannot drift apart again. Found and filed while verifying: + the startup banner is written to stdout and `--json` does not suppress it + ([issue #106](https://git.eeqj.de/sneak/vaultik/issues/106)), which is a + separate writer on a separate path and the remaining source of stdout + contamination. - 2026-08-09: Made the tagged-release path actually work on Gitea - ([issue #65](https://git.eeqj.de/sneak/vaultik/issues/65)). Three - independent blockers, one of which was the whole - release: `.goreleaser.yaml` had no `gitea_urls:` block, so goreleaser - defaulted to the GitHub API and a `goreleaser release` from this repo - would have failed or published where nobody is looking. It now points - at `https://git.eeqj.de/api/v1`. The version is the second: it was a - hardcoded `VERSION := 1.0.0-rc.1` in the `Makefile`, so every local - build claimed to be a release candidate that had never been tagged and - did not exist, while `git tag -l` was empty and `internal/globals` - defaulted to `dev`. Version now comes from git via the new - `script/version` — the exact tag with a leading `v` stripped (so a - `make` build and a goreleaser build of one commit report the same - string, and it matches the archive names), otherwise `dev-<12-char - sha>`, with `-dirty` appended in either case when tracked files are - modified. Untracked files are deliberately not counted, matching - `git describe --dirty`. The same honesty was owed by the snapshot - path: `snapshot.version_template` was `{{ incpatch .Version }}-next`, - which manufactures a release number from the last tag and, with no - tags at all, from goreleaser's fabricated `v0.0.0`; it now emits the - same `dev-`. The one non-obvious consequence is that - `internal/cli/version.go` gated its "this is a development build" - notice on the version being exactly `dev`, so the moment untagged - builds began carrying a commit sha that notice would have gone silent - and an unreleased binary would have read as a release — the gate is - now `globals.IsDevVersion`, which is a predicate over a string rather - than a comparison against a global precisely so it can be tested, and - it is tested at the boundary (`1.0.0-dev` is a release, `dev-` - is not). Release automation is the third blocker: a tag-triggered - `.gitea/workflows/release.yml` runs the build in CI rather than from - a laptop, with `fetch-depth: 0` because a shallow checkout has no - tags and would silently mislabel the release, and with the - `RELEASE_TOKEN` repository secret passed as `GITEA_TOKEN` (documented - in `README.md`; the runner's automatic token is not used because it - is not guaranteed to carry release write scope). `script/release` - unsets any `GITHUB_TOKEN`/`GITLAB_TOKEN` it finds, since goreleaser - chooses its forge from whichever token variable is set and refuses to - run when it sees more than one — a runner-provided token must not get - to decide where these artifacts are published. `make release` and - `make release-snapshot`, the last two Makefile targets that were not - shims, now call `script/release` and `script/release-snapshot`, which - resolve goreleaser exactly the way `script/lint` resolves the linter: - a `PATH` binary is used only at the pinned version, never as a silent - fallback. `script/bootstrap` installs it, from a sha256-verified - GitHub release archive per `REPO_POLICIES.md`, via a separate - `script/install-goreleaser` — separate because `script/bootstrap` - hard-fails without a usable Docker daemon by design, and the release - runner needs goreleaser without needing Docker. Verified by running - the thing rather than reading it: `make release-snapshot` produced - four archives and `checksums.txt`, and the linux/amd64 binary from - `dist/` reports `dev-` with the development-build notice. Tag - handling was exercised in a throwaway repository rather than by - tagging this one; no tag was created here, since that is the owner's - call. Signing, SBOM, reproducible builds, completions and a man page - are out of scope by the issue. + ([issue #65](https://git.eeqj.de/sneak/vaultik/issues/65)). Three independent + blockers, one of which was the whole release: `.goreleaser.yaml` had no + `gitea_urls:` block, so goreleaser defaulted to the GitHub API and a + `goreleaser release` from this repo would have failed or published where + nobody is looking. It now points at `https://git.eeqj.de/api/v1`. The version + is the second: it was a hardcoded `VERSION := 1.0.0-rc.1` in the `Makefile`, + so every local build claimed to be a release candidate that had never been + tagged and did not exist, while `git tag -l` was empty and `internal/globals` + defaulted to `dev`. Version now comes from git via the new `script/version` — + the exact tag with a leading `v` stripped (so a `make` build and a goreleaser + build of one commit report the same string, and it matches the archive names), + otherwise `dev-<12-char sha>`, with `-dirty` appended in either case when + tracked files are modified. Untracked files are deliberately not counted, + matching `git describe --dirty`. The same honesty was owed by the snapshot + path: `snapshot.version_template` was `{{ incpatch .Version }}-next`, which + manufactures a release number from the last tag and, with no tags at all, from + goreleaser's fabricated `v0.0.0`; it now emits the same `dev-`. The one + non-obvious consequence is that `internal/cli/version.go` gated its "this is a + development build" notice on the version being exactly `dev`, so the moment + untagged builds began carrying a commit sha that notice would have gone silent + and an unreleased binary would have read as a release — the gate is now + `globals.IsDevVersion`, which is a predicate over a string rather than a + comparison against a global precisely so it can be tested, and it is tested at + the boundary (`1.0.0-dev` is a release, `dev-` is not). Release + automation is the third blocker: a tag-triggered + `.gitea/workflows/release.yml` runs the build in CI rather than from a laptop, + with `fetch-depth: 0` because a shallow checkout has no tags and would + silently mislabel the release, and with the `RELEASE_TOKEN` repository secret + passed as `GITEA_TOKEN` (documented in `README.md`; the runner's automatic + token is not used because it is not guaranteed to carry release write scope). + `script/release` unsets any `GITHUB_TOKEN`/`GITLAB_TOKEN` it finds, since + goreleaser chooses its forge from whichever token variable is set and refuses + to run when it sees more than one — a runner-provided token must not get to + decide where these artifacts are published. `make release` and + `make release-snapshot`, the last two Makefile targets that were not shims, + now call `script/release` and `script/release-snapshot`, which resolve + goreleaser exactly the way `script/lint` resolves the linter: a `PATH` binary + is used only at the pinned version, never as a silent fallback. + `script/bootstrap` installs it, from a sha256-verified GitHub release archive + per `REPO_POLICIES.md`, via a separate `script/install-goreleaser` — separate + because `script/bootstrap` hard-fails without a usable Docker daemon by + design, and the release runner needs goreleaser without needing Docker. + Verified by running the thing rather than reading it: `make release-snapshot` + produced four archives and `checksums.txt`, and the linux/amd64 binary from + `dist/` reports `dev-` with the development-build notice. Tag handling + was exercised in a throwaway repository rather than by tagging this one; no + tag was created here, since that is the owner's call. Signing, SBOM, + reproducible builds, completions and a man page are out of scope by the issue. -- 2026-08-09: Isolated the lint cache per worktree and context-gated the - native lint path (issues #99, #80). One defect seen twice: - `script/lint` decided whether it could skip the pinned image by asking - what version was on `PATH` rather than where it was running, and cache - isolation is part of that same question. The cache was one directory - per repo, shared by every worktree on the host, so two checkouts with - identical Go file contents collided and golangci-lint replayed the - stored analysis — paths and all. The loud direction of that failure - (a clean tree failed by a dirty sibling) is the harmless one; the - silent direction, a dirty tree **passed** by a clean sibling, is a - sixth way for a gate here to report a green it did not earn. The cache - is now keyed on a digest of the worktree path, and every run is - audited by the new `script/lint-audit`, which rejects output citing any - file that is not in the tree being linted — a backstop that runs on - clean output too, because that is the case nobody investigates. Caches - record the worktree they belong to and are collected when it - disappears, so throwaway worktrees do not accumulate them; the whole - tree lives under `XDG_CACHE_HOME` and is disposable. The - `parallel golangci-lint is running` refusal is now a bounded retry - rather than a verdict: it is not a lint result, and exiting non-zero - on it is indistinguishable to a caller from real findings (#88 showed - a private cache does not remove that contention). The native path now - requires `VAULTIK_LINT_IN_CONTAINER=1`, set only by the `Dockerfile` - lint stage, in addition to matching the pin, so a developer's locally - installed 2.12.2 no longer bypasses the digest pin; `/.dockerenv` was - rejected as the signal because `dockerd` creates it for `docker run` - and it is not reliably present during a BuildKit `docker build`, which - is the case the exception exists for. Version detection uses - `golangci-lint version --short` with the old banner scrape kept only - as a fallback. `script/bootstrap` no longer prints `bootstrap - complete` on a machine that cannot run the gate: a missing docker, or - one whose daemon is unreachable, is a hard failure naming exactly what +- 2026-08-09: Isolated the lint cache per worktree and context-gated the native + lint path (issues #99, #80). One defect seen twice: `script/lint` decided + whether it could skip the pinned image by asking what version was on `PATH` + rather than where it was running, and cache isolation is part of that same + question. The cache was one directory per repo, shared by every worktree on + the host, so two checkouts with identical Go file contents collided and + golangci-lint replayed the stored analysis — paths and all. The loud direction + of that failure (a clean tree failed by a dirty sibling) is the harmless one; + the silent direction, a dirty tree **passed** by a clean sibling, is a sixth + way for a gate here to report a green it did not earn. The cache is now keyed + on a digest of the worktree path, and every run is audited by the new + `script/lint-audit`, which rejects output citing any file that is not in the + tree being linted — a backstop that runs on clean output too, because that is + the case nobody investigates. Caches record the worktree they belong to and + are collected when it disappears, so throwaway worktrees do not accumulate + them; the whole tree lives under `XDG_CACHE_HOME` and is disposable. The + `parallel golangci-lint is running` refusal is now a bounded retry rather than + a verdict: it is not a lint result, and exiting non-zero on it is + indistinguishable to a caller from real findings (#88 showed a private cache + does not remove that contention). The native path now requires + `VAULTIK_LINT_IN_CONTAINER=1`, set only by the `Dockerfile` lint stage, in + addition to matching the pin, so a developer's locally installed 2.12.2 no + longer bypasses the digest pin; `/.dockerenv` was rejected as the signal + because `dockerd` creates it for `docker run` and it is not reliably present + during a BuildKit `docker build`, which is the case the exception exists for. + Version detection uses `golangci-lint version --short` with the old banner + scrape kept only as a fallback. `script/bootstrap` no longer prints + `bootstrap complete` on a machine that cannot run the gate: a missing docker, + or one whose daemon is unreachable, is a hard failure naming exactly what breaks. Verification was by reproduction rather than inspection — two - concurrent lints from two worktrees of differing cleanliness, a real - run made to report an outside path, a matching linter shimmed onto - `PATH`, and a `PATH` with docker removed — and is recorded on the pull - request. + concurrent lints from two worktrees of differing cleanliness, a real run made + to report an outside path, a matching linter shimmed onto `PATH`, and a `PATH` + with docker removed — and is recorded on the pull request. - 2026-08-09: Closed the fifth false-green mechanism (issues #93, #69). - `script/test` omitted `-count=1`, so Go's test result cache could - satisfy the gate outright: a second back-to-back `make test` printed - the full set of 14 `ok` lines, every one marked `(cached)`, having - executed no test at all. Since `ok (cached)` is an `ok` line, - the "14 `ok` lines means the suite ran" signal this repo leans on was - forgeable, one level below the Docker layer cache that #85 addressed. - Fixed with `-count=1` unconditionally rather than only in the - container, because the pre-commit hook runs the same script and a + `script/test` omitted `-count=1`, so Go's test result cache could satisfy the + gate outright: a second back-to-back `make test` printed the full set of 14 + `ok` lines, every one marked `(cached)`, having executed no test at all. Since + `ok (cached)` is an `ok` line, the "14 `ok` lines means the suite ran" + signal this repo leans on was forgeable, one level below the Docker layer + cache that #85 addressed. Fixed with `-count=1` unconditionally rather than + only in the container, because the pre-commit hook runs the same script and a gate honest only in CI is dishonest where people rely on it most; - `test-coverage` got the same flag, and `script/check` inherits it by - calling `script/test`. In the same area, `make test-integration` was - deleted rather than made real: no file in the repo carried a build - tag, so `-tags=integration` selected nothing and the target was an - exact duplicate of `make test`. Tagging a subset was rejected because - the entire suite runs in well under a minute, and a scheme whose - failure mode is "some tests silently stopped running" is a poor trade - for those seconds in a repo with this particular history. The - `-timeout` was raised from 30s after measuring rather than after - assuming: the standing claim that cold-cache compilation is charged - against `-timeout` is **false**, disproved by a containerised run - that spent 46s compiling and still reported per-package durations - within noise of a warm host run. `-timeout` reaches the test binary - as `-test.timeout` and its clock starts inside `testing.M.Run`, after - the build. The real exposure was margin, not compilation. The 120s - landed on is a **deliberate, documented divergence** from - `REPO_POLICIES.md:192`, which mandates 30s, and from that file's - canonical recipe at `:212-214`; the divergence is recorded in - `script/test`'s comment because `REPO_POLICIES.md` is org-canonical - and not editable here, and issue #101 proposes amending the policy - text upstream. Numbers and the full verification are recorded once, - on the pull request, and are deliberately not restated here. + `test-coverage` got the same flag, and `script/check` inherits it by calling + `script/test`. In the same area, `make test-integration` was deleted rather + than made real: no file in the repo carried a build tag, so + `-tags=integration` selected nothing and the target was an exact duplicate of + `make test`. Tagging a subset was rejected because the entire suite runs in + well under a minute, and a scheme whose failure mode is "some tests silently + stopped running" is a poor trade for those seconds in a repo with this + particular history. The `-timeout` was raised from 30s after measuring rather + than after assuming: the standing claim that cold-cache compilation is charged + against `-timeout` is **false**, disproved by a containerised run that spent + 46s compiling and still reported per-package durations within noise of a warm + host run. `-timeout` reaches the test binary as `-test.timeout` and its clock + starts inside `testing.M.Run`, after the build. The real exposure was margin, + not compilation. The 120s landed on is a **deliberate, documented divergence** + from `REPO_POLICIES.md:192`, which mandates 30s, and from that file's + canonical recipe at `:212-214`; the divergence is recorded in `script/test`'s + comment because `REPO_POLICIES.md` is org-canonical and not editable here, and + issue #101 proposes amending the policy text upstream. Numbers and the full + verification are recorded once, on the pull request, and are deliberately not + restated here. -- 2026-08-09: Triaged all fifteen stale remote branches (issue #71) and - deleted fourteen of them; the full per-branch disposition with - evidence is recorded on that issue. Method mattered more than the - outcome here: a three-dot `git diff main...branch` diffs from the - merge base, so it replays everything that landed on `main` after the - branch diverged and makes any old branch look like it holds unlanded - work. That artifact is what made `golangci-v2.12.2` appear to carry - 126 files of unpushed changes when its tree was byte-identical to - `main`'s. Every containment claim here therefore rests on two-dot tip - diffs, tree-hash equality, `git cherry`, and `git branch -r --merged`. - Nine branches were plain ancestors of `main` with zero `git cherry` - `+` commits. `golangci-v2.12.2` had landed squashed as `cc58583`, - whose tree hash equals the branch tip's exactly; note the hash - recorded in the issue had gone stale because `main` advanced, so the - check had to be redone rather than repeated. - `fix/sync-snapshot-cleanup` was redundant, its one line already on - `main` in `syncWithRemote`. `feature/restore-progress-bar` was - superseded by `printRestoreProgress` and the disk-backed blob cache, - and had become actively regressive — it would have deleted - `internal/blobgen/compress_test.go`, the #28 regression test that - landed separately. The two branches this issue was filed for both - turned out to be closed questions that `main` had already moved past - by a recorded decision, so neither was landed and no regression test - was owed: `ctime` no longer exists anywhere in the codebase after - `1c72a37` removed the column, the `File.CTime` field and every use - (#54/#55), and change detection compares size, mtime, mode, uid and - gid only, exactly as `ARCHITECTURE.md` documents — so the - silently-skipped-file data-loss risk that made this a 1.0 item does - not exist. The SQL allow-list branch would have reverted `bfd7334`, - which replaced that very allow-list with regex sanitisation on review - feedback, and would have broken `getTableCount("snapshots")` because - its allow-list omits that table. `feature/daemon-mode` is untouched - and deferred to #94 pending an owner decision, so it is the one - branch besides `main` still on the remote. The stale `TODO.md` entry - named in the issue needed no fix: `e496aa3` had already removed it. - No product code changed. -- 2026-08-09: Adopted the remaining upstream `CHECK_EPOCH` hardening - (issue #91), closing the gap #85 knowingly left open. Four changes, - all four decided as adopt upstream in `sneak/prompts` #26. (1) Each - check stage now asserts `[ -n "$CHECK_EPOCH" ] || exit 1` before - running anything, so a build that supplies no `--build-arg` fails - instead of lying. This is the item that mattered: an unset `ARG` is - an empty string and an empty string is a stable cache key, so the - second and every later bare `docker build .` on an unchanged tree - replayed all three check layers and still exited 0 — and `docker - build .` is the command `REPO_POLICIES.md` names verbatim as a thing - that must be green, so the documented command was precisely the one - that lied. Failed steps are never cached, which is what makes the - guard fire on every invocation rather than once. (2) The epoch is now - expanded into each check command rather than left as a bare - declaration, so the cache miss no longer depends on BuildKit's - unreferenced-`ARG` handling staying as it is, and the value appears - in the build log. (3) `script/cibuild` uses - `epoch="$(date +%s%N)$$"`, unique per invocation rather than per - second; `%N` alone is insufficient because busybox drops it silently - and exits 0, and `$$` is what makes the guarantee hold regardless. - The bare-assignment form is kept deliberately — inlined in an - argument, a failing substitution does not abort under `set -eu` and - would yield an empty constant epoch, restoring the exact false green - being fixed. (4) `script/docker` passes the same fresh arg, so the - two entrypoints cannot disagree about whether the tree is green; - local builds are almost always warm, which made it the likelier - fooling in practice. The `ARG` placement from #85 is unchanged, below - `apk add`, `COPY go.mod go.sum` and `go mod download`, so dependency - layers still cache and the build is not cold. Verified by negative - control rather than inspection — a bare `docker build .` run twice - back to back, plus back-to-back pairs of both scripts and a host-side - `make check`; the measurements are recorded once, in the PR - verification comment, rather than restated here. `.golangci.yml`, the - lint-stage `FROM` line and its digest, `script/lint`, - `REPO_POLICIES.md` and `.gitea/workflows/check.yml` are all - untouched. -- 2026-08-09: Stopped `script/cibuild` from reporting a green it did - not earn (issue #85). A bare `docker build .` let Docker serve the - check layers from the layer cache whenever the tree had not changed: - the checks never executed and the build still exited 0. The fix is an - `ARG CHECK_EPOCH` declared immediately above the check `RUN`s in both - the lint stage and the builder stage (`ARG` scope is per-stage, so - each declares its own), with `script/cibuild` assigning - `epoch="$(date +%s)"` and passing `--build-arg CHECK_EPOCH="$epoch"`. - The assignment is separate on purpose: under `set -eu` a command - substitution that fails inside an argument does not abort the script, - which would leave an empty constant `CHECK_EPOCH` and restore the - very false green being fixed. Placement is the rest of the point — - the `ARG` sits below the `apk add`, `COPY go.mod go.sum`, and `go mod - download` layers, so only the checks are invalidated and the - dependency layers still cache. The guarantee is conditional on a - fresh value rather than absolute: a bare `docker build .` gets an - empty `CHECK_EPOCH` and can still serve the check layers from cache, - which `README.md` and the `Dockerfile` now say plainly, with issue - #91 tracking the upstream hardening (expanded `ARG` form, unset - guard, per-invocation epoch, `script/docker`) that would close it. - Verified by re-running the reproduction plus the withheld-`--build-arg` - counterfactual; the measurements are recorded once, in the PR #89 - verification comment, rather than restated here. `.golangci.yml`, the - lint-stage `FROM` line and its digest, `script/lint`, and +- 2026-08-09: Triaged all fifteen stale remote branches (issue #71) and deleted + fourteen of them; the full per-branch disposition with evidence is recorded on + that issue. Method mattered more than the outcome here: a three-dot + `git diff main...branch` diffs from the merge base, so it replays everything + that landed on `main` after the branch diverged and makes any old branch look + like it holds unlanded work. That artifact is what made `golangci-v2.12.2` + appear to carry 126 files of unpushed changes when its tree was byte-identical + to `main`'s. Every containment claim here therefore rests on two-dot tip + diffs, tree-hash equality, `git cherry`, and `git branch -r --merged`. Nine + branches were plain ancestors of `main` with zero `git cherry` `+` commits. + `golangci-v2.12.2` had landed squashed as `cc58583`, whose tree hash equals + the branch tip's exactly; note the hash recorded in the issue had gone stale + because `main` advanced, so the check had to be redone rather than repeated. + `fix/sync-snapshot-cleanup` was redundant, its one line already on `main` in + `syncWithRemote`. `feature/restore-progress-bar` was superseded by + `printRestoreProgress` and the disk-backed blob cache, and had become actively + regressive — it would have deleted `internal/blobgen/compress_test.go`, the + #28 regression test that landed separately. The two branches this issue was + filed for both turned out to be closed questions that `main` had already moved + past by a recorded decision, so neither was landed and no regression test was + owed: `ctime` no longer exists anywhere in the codebase after `1c72a37` + removed the column, the `File.CTime` field and every use (#54/#55), and change + detection compares size, mtime, mode, uid and gid only, exactly as + `ARCHITECTURE.md` documents — so the silently-skipped-file data-loss risk that + made this a 1.0 item does not exist. The SQL allow-list branch would have + reverted `bfd7334`, which replaced that very allow-list with regex + sanitisation on review feedback, and would have broken + `getTableCount("snapshots")` because its allow-list omits that table. + `feature/daemon-mode` is untouched and deferred to #94 pending an owner + decision, so it is the one branch besides `main` still on the remote. The + stale `TODO.md` entry named in the issue needed no fix: `e496aa3` had already + removed it. No product code changed. +- 2026-08-09: Adopted the remaining upstream `CHECK_EPOCH` hardening (issue + #91), closing the gap #85 knowingly left open. Four changes, all four decided + as adopt upstream in `sneak/prompts` #26. (1) Each check stage now asserts + `[ -n "$CHECK_EPOCH" ] || exit 1` before running anything, so a build that + supplies no `--build-arg` fails instead of lying. This is the item that + mattered: an unset `ARG` is an empty string and an empty string is a stable + cache key, so the second and every later bare `docker build .` on an unchanged + tree replayed all three check layers and still exited 0 — and `docker build .` + is the command `REPO_POLICIES.md` names verbatim as a thing that must be + green, so the documented command was precisely the one that lied. Failed steps + are never cached, which is what makes the guard fire on every invocation + rather than once. (2) The epoch is now expanded into each check command rather + than left as a bare declaration, so the cache miss no longer depends on + BuildKit's unreferenced-`ARG` handling staying as it is, and the value appears + in the build log. (3) `script/cibuild` uses `epoch="$(date +%s%N)$$"`, unique + per invocation rather than per second; `%N` alone is insufficient because + busybox drops it silently and exits 0, and `$$` is what makes the guarantee + hold regardless. The bare-assignment form is kept deliberately — inlined in an + argument, a failing substitution does not abort under `set -eu` and would + yield an empty constant epoch, restoring the exact false green being fixed. + (4) `script/docker` passes the same fresh arg, so the two entrypoints cannot + disagree about whether the tree is green; local builds are almost always warm, + which made it the likelier fooling in practice. The `ARG` placement from #85 + is unchanged, below `apk add`, `COPY go.mod go.sum` and `go mod download`, so + dependency layers still cache and the build is not cold. Verified by negative + control rather than inspection — a bare `docker build .` run twice back to + back, plus back-to-back pairs of both scripts and a host-side `make check`; + the measurements are recorded once, in the PR verification comment, rather + than restated here. `.golangci.yml`, the lint-stage `FROM` line and its + digest, `script/lint`, `REPO_POLICIES.md` and `.gitea/workflows/check.yml` are + all untouched. +- 2026-08-09: Stopped `script/cibuild` from reporting a green it did not earn + (issue #85). A bare `docker build .` let Docker serve the check layers from + the layer cache whenever the tree had not changed: the checks never executed + and the build still exited 0. The fix is an `ARG CHECK_EPOCH` declared + immediately above the check `RUN`s in both the lint stage and the builder + stage (`ARG` scope is per-stage, so each declares its own), with + `script/cibuild` assigning `epoch="$(date +%s)"` and passing + `--build-arg CHECK_EPOCH="$epoch"`. The assignment is separate on purpose: + under `set -eu` a command substitution that fails inside an argument does not + abort the script, which would leave an empty constant `CHECK_EPOCH` and + restore the very false green being fixed. Placement is the rest of the point — + the `ARG` sits below the `apk add`, `COPY go.mod go.sum`, and + `go mod download` layers, so only the checks are invalidated and the + dependency layers still cache. The guarantee is conditional on a fresh value + rather than absolute: a bare `docker build .` gets an empty `CHECK_EPOCH` and + can still serve the check layers from cache, which `README.md` and the + `Dockerfile` now say plainly, with issue #91 tracking the upstream hardening + (expanded `ARG` form, unset guard, per-invocation epoch, `script/docker`) that + would close it. Verified by re-running the reproduction plus the + withheld-`--build-arg` counterfactual; the measurements are recorded once, in + the PR #89 verification comment, rather than restated here. `.golangci.yml`, + the lint-stage `FROM` line and its digest, `script/lint`, and `.gitea/workflows/check.yml` are all untouched. -- 2026-08-09: Corrected the `Vaultik.UI` doc comment (issue #84). It - claimed the cli layer replaces the writer with a discarding one in - `--cron` mode; the actual mechanism is `UI.SetQuiet(true)` in - `setupGlobals`, which drops Begin/Complete/Info/Notice/Detail/ - Progress/Banner but still emits Warning and Error. The `--cron` line - in `README.md` said "Silent unless error", which understated what - survives, and now names warnings too. The other `--cron` comments - (`internal/log/log.go`, `internal/cli/snapshot.go`, - `internal/vaultik/snapshot.go`) were audited and already accurate. - Comments and docs only, no behavior change. -- 2026-08-09: Made `snapshot list` list the destination store without - the private key (issue #64). The listing is now the union of the - local index and a single streamed listing of the `metadata/` prefix, - with no `age_secret_key` gate — the manifest is unencrypted, so a - host holding only the public key can enumerate its own backups and a - host that lost its local index can still see them. A remote-only - snapshot's hostname and name are deliberately not recovered (they are - not recoverable without the private key, and making them so would - undo the privacy property tracked in issue #81); such rows are - labelled by an abbreviation of their remote key and carry the real - timestamp and compressed size from the manifest, with `` - in the two columns that require the local index. Local-only snapshots - are reported as drift, and the hint now names `vaultik prune`, which - exists, instead of `vaultik snapshot cleanup`, which does not. - `reportRemoteDrift` collapsed into the merged view. Every remote - manifest read in the codebase now goes through - `downloadManifestByKey`, so issue #81 has one call site to change. - Review rework: snapshot timestamps now normalize to UTC in - `scanSnapshotRows`, the one place they enter the domain, so the merged - TIMESTAMP column cannot show local time for a locally tracked row and - UTC for a remote-only row on a non-UTC host; `GetIncompleteByHostname` - was folded onto that same scanner. `--json` now reports the - unreadable-manifest count and the 1000-row truncation on stderr - instead of returning a silently short document (the document's shape - is unchanged). The two per-snapshot `log.Warn` calls on the listing - path now route through the same JSON-aware writer as the existing - workaround, so one corrupt manifest can no longer put a log line on - stdout ahead of the document and break `| jq` — still a local - workaround pending issue #82. Verified with `script/cibuild` and with - an uncached `make check` (`0 issues.`, no cached test packages), plus - end to end against a `file://` destination with no secret key present. +- 2026-08-09: Corrected the `Vaultik.UI` doc comment (issue #84). It claimed the + cli layer replaces the writer with a discarding one in `--cron` mode; the + actual mechanism is `UI.SetQuiet(true)` in `setupGlobals`, which drops + Begin/Complete/Info/Notice/Detail/ Progress/Banner but still emits Warning and + Error. The `--cron` line in `README.md` said "Silent unless error", which + understated what survives, and now names warnings too. The other `--cron` + comments (`internal/log/log.go`, `internal/cli/snapshot.go`, + `internal/vaultik/snapshot.go`) were audited and already accurate. Comments + and docs only, no behavior change. +- 2026-08-09: Made `snapshot list` list the destination store without the + private key (issue #64). The listing is now the union of the local index and a + single streamed listing of the `metadata/` prefix, with no `age_secret_key` + gate — the manifest is unencrypted, so a host holding only the public key can + enumerate its own backups and a host that lost its local index can still see + them. A remote-only snapshot's hostname and name are deliberately not + recovered (they are not recoverable without the private key, and making them + so would undo the privacy property tracked in issue #81); such rows are + labelled by an abbreviation of their remote key and carry the real timestamp + and compressed size from the manifest, with `` in the two columns + that require the local index. Local-only snapshots are reported as drift, and + the hint now names `vaultik prune`, which exists, instead of + `vaultik snapshot cleanup`, which does not. `reportRemoteDrift` collapsed into + the merged view. Every remote manifest read in the codebase now goes through + `downloadManifestByKey`, so issue #81 has one call site to change. Review + rework: snapshot timestamps now normalize to UTC in `scanSnapshotRows`, the + one place they enter the domain, so the merged TIMESTAMP column cannot show + local time for a locally tracked row and UTC for a remote-only row on a + non-UTC host; `GetIncompleteByHostname` was folded onto that same scanner. + `--json` now reports the unreadable-manifest count and the 1000-row truncation + on stderr instead of returning a silently short document (the document's shape + is unchanged). The two per-snapshot `log.Warn` calls on the listing path now + route through the same JSON-aware writer as the existing workaround, so one + corrupt manifest can no longer put a log line on stdout ahead of the document + and break `| jq` — still a local workaround pending issue #82. Verified with + `script/cibuild` and with an uncached `make check` (`0 issues.`, no cached + test packages), plus end to end against a `file://` destination with no secret + key present. - 2026-08-09: Closed the gap between `make lint` and CI (issue #78). - `script/lint` now runs the digest-pinned `golangci-lint` image taken - from the `Dockerfile` lint stage, which is the single source of truth - for the linter version; the duplicate pin in the `Makefile` `deps` - target and the unpinned `golangci-lint` install in `script/bootstrap` - are gone. A `golangci-lint` on `PATH` is used only when its version is - exactly the pinned one (which is how the lint stage runs it inside the - container); anything else goes through Docker, and a missing or - unreachable Docker daemon is a hard error rather than a silent - fallback. Only the **lint** leg of `make check` became equivalent to - `script/cibuild`; its tests and `gofmt` still run on the host against - the host toolchain, as `README.md` states. An earlier version of this - entry claimed `make check` was "as trustworthy as `script/cibuild`" - outright, which overstated it; corrected under issue #80. -- 2026-08-09: Finished the lint remediation under the canonical - `.golangci.yml` (issue #61, which also unblocks issue #59). The - remaining findings were fixed behavior-preservingly: `wsl_v5` - whitespace, `sqlclosecheck`, and `prealloc`. The `sqlclosecheck` sites - now close `sql.Rows` in a deferred closure instead of via the - `CloseRows` helper, which the linter could not see through. Only the - `revive` package-name findings remain suppressed, with per-site - `//nolint` directives; the package-rename question behind them is - tracked in issue #76. Verified with `script/cibuild`, which exits 0 — - that is the only trustworthy gate, because `script/lint` runs whatever - `golangci-lint` happens to be on `PATH` rather than the pinned - v2.12.2 that CI and the `Dockerfile` use, so `make check` can report - green on findings CI still fails. That tooling gap is tracked in issue - #78. -- 2026-08-09: The earlier next step "reconcile the uncommitted - `ARCHITECTURE.md` edits on `main`" needed no work: the working tree is - clean and `ARCHITECTURE.md` is committed on `main`. + `script/lint` now runs the digest-pinned `golangci-lint` image taken from the + `Dockerfile` lint stage, which is the single source of truth for the linter + version; the duplicate pin in the `Makefile` `deps` target and the unpinned + `golangci-lint` install in `script/bootstrap` are gone. A `golangci-lint` on + `PATH` is used only when its version is exactly the pinned one (which is how + the lint stage runs it inside the container); anything else goes through + Docker, and a missing or unreachable Docker daemon is a hard error rather than + a silent fallback. Only the **lint** leg of `make check` became equivalent to + `script/cibuild`; its tests and `gofmt` still run on the host against the host + toolchain, as `README.md` states. An earlier version of this entry claimed + `make check` was "as trustworthy as `script/cibuild`" outright, which + overstated it; corrected under issue #80. +- 2026-08-09: Finished the lint remediation under the canonical `.golangci.yml` + (issue #61, which also unblocks issue #59). The remaining findings were fixed + behavior-preservingly: `wsl_v5` whitespace, `sqlclosecheck`, and `prealloc`. + The `sqlclosecheck` sites now close `sql.Rows` in a deferred closure instead + of via the `CloseRows` helper, which the linter could not see through. Only + the `revive` package-name findings remain suppressed, with per-site `//nolint` + directives; the package-rename question behind them is tracked in issue #76. + Verified with `script/cibuild`, which exits 0 — that is the only trustworthy + gate, because `script/lint` runs whatever `golangci-lint` happens to be on + `PATH` rather than the pinned v2.12.2 that CI and the `Dockerfile` use, so + `make check` can report green on findings CI still fails. That tooling gap is + tracked in issue #78. +- 2026-08-09: The earlier next step "reconcile the uncommitted `ARCHITECTURE.md` + edits on `main`" needed no work: the working tree is clean and + `ARCHITECTURE.md` is committed on `main`. - 2026-08-07: Updated golangci-lint to v2.12.2 everywhere it is pinned - (`Dockerfile` lint stage, `Makefile` deps target), replaced - `.golangci.yml` with the canonical config (v2 schema, `default: all`), - and remediated the bulk of the lint findings it surfaced (issue #61): - behavior-preserving fixes across every package, 2,990 findings down to - 80. `make test` and `make fmt-check` were green at that point but - `make lint` was still red; the commit message claiming `make check` - was green was wrong. -- 2026-08-07: Added the standard `.golangci.yml` and `.editorconfig` - (issue #59); lint findings under the new config are tracked in issue - #61. -- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, - Makefile shims, README Entrypoints section -- 2026-07-02: Consolidated CLI verbs, retired overlapping commands; bound - the local index to its backup destination URL. + (`Dockerfile` lint stage, `Makefile` deps target), replaced `.golangci.yml` + with the canonical config (v2 schema, `default: all`), and remediated the bulk + of the lint findings it surfaced (issue #61): behavior-preserving fixes across + every package, 2,990 findings down to 80. `make test` and `make fmt-check` + were green at that point but `make lint` was still red; the commit message + claiming `make check` was green was wrong. +- 2026-08-07: Added the standard `.golangci.yml` and `.editorconfig` (issue + #59); lint findings under the new config are tracked in issue #61. +- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile + shims, README Entrypoints section +- 2026-07-02: Consolidated CLI verbs, retired overlapping commands; bound the + local index to its backup destination URL. - 2026-06-28: snapshot rm now removes metadata only and prints the prune command; restore skips chown when running as non-root. -- 2026-06-26: Snapshot IDs hashed at the storage boundary; snapshot list - made resilient to bad remote entries. -- 2026-06-24: Collapsed snapshot prune into vaultik prune; restore streams - blobs to disk and restores files in blob-locality order; cron output - fixes. +- 2026-06-26: Snapshot IDs hashed at the storage boundary; snapshot list made + resilient to bad remote entries. +- 2026-06-24: Collapsed snapshot prune into vaultik prune; restore streams blobs + to disk and restores files in blob-locality order; cron output fixes. - 2026-06-17: Restore overhaul: ReadAt chunk reads from cached blobs, - reference-counted blob sweeper, integration tests; new internal/ui - output layer, banner, and progress lines. + reference-counted blob sweeper, integration tests; new internal/ui output + layer, banner, and progress lines. - 2025-12-18: Added ARCHITECTURE.md and godoc coverage for exported API. -- 2025-07-26: End-to-end integration tests; manifest format refactor; - renamed backup to snapshot; afero filesystem abstraction. -- 2025-07-20: Initial design and implementation: cobra + fx CLI skeleton, - SQLite index database, UUID blob storage with streaming chunking. +- 2025-07-26: End-to-end integration tests; manifest format refactor; renamed + backup to snapshot; afero filesystem abstraction. +- 2025-07-20: Initial design and implementation: cobra + fx CLI skeleton, SQLite + index database, UUID blob storage with streaming chunking. # Future Steps -Work planned after 1.0 is listed in the README -[roadmap](README.md#roadmap). +Work planned after 1.0 is listed in the README [roadmap](README.md#roadmap). diff --git a/docs/DATAMODEL.md b/docs/DATAMODEL.md index 3b197e3..5dd2af9 100644 --- a/docs/DATAMODEL.md +++ b/docs/DATAMODEL.md @@ -2,7 +2,9 @@ ## Overview -Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication. +Vaultik uses a local SQLite database to track file metadata, chunk mappings, and +blob associations during the backup process. This database serves as an index +for incremental backups and enables efficient deduplication. **Important Notes:** @@ -25,19 +27,23 @@ recorded `001.sql` as run, so it keeps the old schema and can become unusable; `vaultik database delete` followed by a full backup rebuilds it. **After 1.0** each schema change is a new numbered file, so an existing local -database is migrated the first time an updated vaultik opens it. A file that -has shipped in a release is never edited. +database is migrated the first time an updated vaultik opens it. A file that has +shipped in a release is never edited. ## Database Tables ### 1. `files` + Stores metadata about files in the filesystem being backed up. **Columns:** + - `id` (TEXT PRIMARY KEY) - UUID for the file record - `path` (TEXT NOT NULL UNIQUE) - Absolute file path -- `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix epoch -- `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to 999999999 +- `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix + epoch +- `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to + 999999999 - `size` (INTEGER NOT NULL) - File size in bytes - `mode` (INTEGER NOT NULL) - Unix file permissions and type - `uid` (INTEGER NOT NULL) - User ID of file owner @@ -45,23 +51,30 @@ Stores metadata about files in the filesystem being backed up. - `link_target` (TEXT) - Symlink target path (NULL for regular files) **Indexes:** + - `idx_files_path` on `path` for efficient lookups -**Purpose:** Tracks file metadata to detect changes between backup runs. Used for incremental backup decisions. The UUID primary key provides stable references that don't change if files are moved. +**Purpose:** Tracks file metadata to detect changes between backup runs. Used +for incremental backup decisions. The UUID primary key provides stable +references that don't change if files are moved. ### 2. `chunks` + Stores information about content-defined chunks created from files. **Columns:** + - `chunk_hash` (TEXT PRIMARY KEY) - SHA256 hash of chunk content - `size` (INTEGER NOT NULL) - Chunk size in bytes **Purpose:** Enables deduplication by tracking unique chunks across all files. ### 3. `file_chunks` + Maps files to their constituent chunks in order. **Columns:** + - `file_id` (TEXT) - File ID (FK to files.id) - `idx` (INTEGER) - Chunk index within file (0-based) - `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash) @@ -70,35 +83,49 @@ Maps files to their constituent chunks in order. **Purpose:** Allows reconstruction of files from chunks during restore. ### 4. `chunk_files` + Reverse mapping showing which files contain each chunk. **Columns:** + - `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash) - `file_id` (TEXT) - File ID (FK to files.id) - `file_offset` (INTEGER) - Byte offset of chunk within file - `length` (INTEGER) - Length of chunk in bytes - PRIMARY KEY (`chunk_hash`, `file_id`) -**Purpose:** Supports efficient queries for chunk usage and deduplication statistics. +**Purpose:** Supports efficient queries for chunk usage and deduplication +statistics. ### 5. `blobs` + Stores information about packed, compressed, and encrypted blob files. **Columns:** + - `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts -- `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before compression and encryption (NULL until finalized); see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs) +- `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`, + computed before compression and encryption (NULL until finalized); see + [REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs) - `created_ts` (INTEGER NOT NULL) - Creation timestamp - `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress) -- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression -- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage: when the blob was already stored, under another compression level or other recipients, this can differ from the size of the local copy +- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before + compression +- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and + encryption. Once uploaded, the size of the object in storage: when the blob + was already stored, under another compression level or other recipients, this + can differ from the size of the local copy - `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded) -**Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs. +**Purpose:** Tracks blob lifecycle from creation through upload. The UUID +primary key allows immediate association of chunks with blobs. ### 6. `blob_chunks` + Maps chunks to the blobs that contain them. **Columns:** + - `blob_id` (TEXT) - Blob ID (FK to blobs.id) - `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash) - `offset` (INTEGER) - Byte offset of chunk within blob (before compression) @@ -108,34 +135,46 @@ Maps chunks to the blobs that contain them. **Purpose:** Enables chunk retrieval from blobs during restore operations. ### 7. `snapshots` + Tracks backup snapshots. **Columns:** -- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g. `server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the snapshot name, and an RFC 3339 UTC timestamp) + +- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g. + `server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the + snapshot name, and an RFC 3339 UTC timestamp) - `hostname` (TEXT) - Hostname where backup was created - `vaultik_version` (TEXT) - Version of Vaultik used - `vaultik_git_revision` (TEXT) - Git revision of Vaultik used - `started_at` (INTEGER) - Start timestamp - `completed_at` (INTEGER) - Completion timestamp (NULL if in progress) - `file_count` (INTEGER) - Number of files in snapshot -- `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not stored before -- `new_chunk_size` (INTEGER) - Total size of those chunks before compression; `snapshot list` shows it as NEW CHUNK SIZE +- `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not + stored before +- `new_chunk_size` (INTEGER) - Total size of those chunks before compression; + `snapshot list` shows it as NEW CHUNK SIZE - `blob_count` (INTEGER) - Number of blobs this snapshot created - `total_size` (INTEGER) - Total size of all files -- `uncompressed_chunk_size` (INTEGER) - Total size before compression of the distinct chunks the snapshot's files reference, recorded when its backup ran; `snapshot list` shows it as UNCOMPRESSED SIZE +- `uncompressed_chunk_size` (INTEGER) - Total size before compression of the + distinct chunks the snapshot's files reference, recorded when its backup ran; + `snapshot list` shows it as UNCOMPRESSED SIZE - `blob_size` (INTEGER) - Total compressed size of all referenced blobs -- `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced blobs +- `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced + blobs - `compression_ratio` (REAL) - Compression ratio achieved - `compression_level` (INTEGER) - Compression level used for this snapshot - `upload_bytes` (INTEGER) - Total bytes uploaded during this snapshot - `upload_duration_ms` (INTEGER) - Total milliseconds spent uploading to S3 -**Purpose:** Provides snapshot metadata and statistics including version tracking for compatibility. +**Purpose:** Provides snapshot metadata and statistics including version +tracking for compatibility. ### 8. `snapshot_files` + Maps snapshots to the files they contain. **Columns:** + - `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) - `file_id` (TEXT) - File ID (FK to files.id) - PRIMARY KEY (`snapshot_id`, `file_id`) @@ -143,114 +182,146 @@ Maps snapshots to the files they contain. **Purpose:** Records which files are included in each snapshot. ### 9. `snapshot_blobs` + Maps snapshots to the blobs they reference. **Columns:** + - `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) - `blob_id` (TEXT) - Blob ID (FK to blobs.id) - `blob_hash` (TEXT) - Denormalized blob hash for manifest generation - PRIMARY KEY (`snapshot_id`, `blob_id`) -**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation. +**Purpose:** Tracks blob dependencies for snapshots and enables manifest +generation. ### 10. `snapshot_mount_points` -The directories at or below a configured path, the configured path itself included, that a snapshot's backup found an entry in and, once it had listed them, found on a different filesystem from their parent directory (a different device number), as a mount point is. An automount point whose volume cannot be mounted is not recorded, because the backup finds no entry in it. + +The directories at or below a configured path, the configured path itself +included, that a snapshot's backup found an entry in and, once it had listed +them, found on a different filesystem from their parent directory (a different +device number), as a mount point is. An automount point whose volume cannot be +mounted is not recorded, because the backup finds no entry in it. **Columns:** + - `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) - `path` (TEXT) - Absolute path of the directory - PRIMARY KEY (`snapshot_id`, `path`) -**Purpose:** Lets `snapshot create --prune` skip the prune when a backup does not find a mount point that an earlier snapshot of the same name found, as when the volume is not mounted. +**Purpose:** Lets `snapshot create --prune` skip the prune when a backup does +not find a mount point that an earlier snapshot of the same name found, as when +the volume is not mounted. ### 11. `uploads` + Tracks blob upload metrics. **Columns:** + - `blob_hash` (TEXT PRIMARY KEY) - Hash of uploaded blob -- `snapshot_id` (TEXT NOT NULL) - The snapshot that triggered this upload (FK to snapshots.id) +- `snapshot_id` (TEXT NOT NULL) - The snapshot that triggered this upload (FK to + snapshots.id) - `uploaded_at` (INTEGER) - Upload timestamp - `size` (INTEGER) - Size of uploaded blob - `duration_ms` (INTEGER) - Upload duration in milliseconds -**Purpose:** Performance monitoring and tracking which blobs were newly created (uploaded) during each snapshot. +**Purpose:** Performance monitoring and tracking which blobs were newly created +(uploaded) during each snapshot. ## Data Flow and Operations ### 1. Backup Process 1. **File Scanning** - - `INSERT OR REPLACE INTO files` - Update file metadata - - `SELECT * FROM files WHERE path = ?` - Check if file has changed - - `INSERT INTO snapshot_files` - Add file to current snapshot - - `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a different filesystem from its parent that the walk found an entry in + - `INSERT OR REPLACE INTO files` - Update file metadata + - `SELECT * FROM files WHERE path = ?` - Check if file has changed + - `INSERT INTO snapshot_files` - Add file to current snapshot + - `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a + different filesystem from its parent that the walk found an entry in 2. **Chunking** (for changed files) - - `INSERT OR IGNORE INTO chunks` - Store new chunks - - `INSERT INTO file_chunks` - Map chunks to file - - `INSERT INTO chunk_files` - Create reverse mapping + - `INSERT OR IGNORE INTO chunks` - Store new chunks + - `INSERT INTO file_chunks` - Map chunks to file + - `INSERT INTO chunk_files` - Create reverse mapping 3. **Blob Packing** - - `INSERT INTO blobs` - Create blob record with UUID (blob_hash NULL) - - `INSERT INTO blob_chunks` - Associate chunks with blob immediately - - `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing + - `INSERT INTO blobs` - Create blob record with UUID (blob_hash NULL) + - `INSERT INTO blob_chunks` - Associate chunks with blob immediately + - `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after + packing 4. **Upload** - - `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (the upload is skipped when the blob is already stored) - - `INSERT INTO uploads` - Record upload metrics with snapshot_id - - `INSERT INTO snapshot_blobs` - Associate blob with snapshot + - `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as + uploaded, with the size of the object in storage (the upload is skipped + when the blob is already stored) + - `INSERT INTO uploads` - Record upload metrics with snapshot_id + - `INSERT INTO snapshot_blobs` - Associate blob with snapshot 5. **Snapshot Completion** - - `UPDATE snapshots SET completed_at = ?, stats...` - Finalize snapshot - - Generate and upload blob manifest from `snapshot_blobs` + - `UPDATE snapshots SET completed_at = ?, stats...` - Finalize snapshot + - Generate and upload blob manifest from `snapshot_blobs` ### 2. Incremental Backup 1. **Change Detection** - - `SELECT * FROM files WHERE path = ?` - Get previous file metadata - - Compare mtime, size, mode to detect changes - - Re-chunk a file that lists a chunk no uploaded blob holds, even when its metadata is unchanged - - Skip unchanged files but still add to `snapshot_files` + - `SELECT * FROM files WHERE path = ?` - Get previous file metadata + - Compare mtime, size, mode to detect changes + - Re-chunk a file that lists a chunk no uploaded blob holds, even when its + metadata is unchanged + - Skip unchanged files but still add to `snapshot_files` 2. **Chunk Reuse** - - `SELECT * FROM blob_chunks WHERE chunk_hash = ?` - Find existing chunks - - `INSERT INTO snapshot_blobs` - Reference existing blobs for unchanged files + - `SELECT * FROM blob_chunks WHERE chunk_hash = ?` - Find existing chunks + - `INSERT INTO snapshot_blobs` - Reference existing blobs for unchanged + files ### 3. Snapshot Metadata Export After a snapshot is completed: + 1. Copy database to temporary file 2. Clean temporary database to contain only current snapshot data 3. VACUUM the trimmed database so deleted rows leave no pages behind 4. Compress with zstd and encrypt with age 5. Upload to S3 as `metadata/{remote-key}/db.zst.age` -6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst` +6. Generate blob manifest and upload as + `metadata/{remote-key}/manifest.json.zst` -The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext. The hash uses no secret, so a guessed hostname and snapshot name can still be confirmed against a listing; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its [Accepted Risks](REPOSTRUCTURE.md#accepted-risks). +The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so +the ID is never written to the store in plaintext. The hash uses no secret, so a +guessed hostname and snapshot name can still be confirmed against a listing; see +[REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its +[Accepted Risks](REPOSTRUCTURE.md#accepted-risks). ### 4. Restore Process The restore process doesn't use the local database. Instead: -1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from S3 -2. Downloads the blobs holding the chunks of the files being restored, found through that database's `blob_chunks` table; the manifest is not read + +1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from + S3 +2. Downloads the blobs holding the chunks of the files being restored, found + through that database's `blob_chunks` table; the manifest is not read 3. Reconstructs files from decrypted and decompressed chunks ### 5. Pruning 1. **Identify Unreferenced Blobs** - - Query blobs not referenced by any remaining snapshot - - Delete from S3, once older than `prune_grace_period`, and from the local - database + - Query blobs not referenced by any remaining snapshot + - Delete from S3, once older than `prune_grace_period`, and from the local + database 2. **Delete Metadata Directories With No Manifest** - - Delete each directory under `metadata/` that has no `manifest.json.zst` - and no local snapshot record, once every object in it is older than - `prune_grace_period` + - Delete each directory under `metadata/` that has no `manifest.json.zst` + and no local snapshot record, once every object in it is older than + `prune_grace_period` ### 6. Incomplete Snapshot Cleanup Before each backup: + 1. Query incomplete snapshots (where `completed_at IS NULL`) -2. Delete each one and all its associations, without checking S3 for its metadata +2. Delete each one and all its associations, without checking S3 for its + metadata 3. Clean up orphaned files, chunks, and blobs ## Repository Pattern @@ -260,18 +331,20 @@ Vaultik uses a repository pattern for database access: - `FileRepository` - CRUD operations for files and file metadata - `ChunkRepository` - CRUD operations for content chunks - `FileChunkRepository` - Manage file-to-chunk mappings -- `ChunkFileRepository` - Manage chunk-to-file reverse mappings +- `ChunkFileRepository` - Manage chunk-to-file reverse mappings - `BlobRepository` - Manage blob lifecycle (creation, finalization, upload) - `BlobChunkRepository` - Manage blob-to-chunk associations - `SnapshotRepository` - Manage snapshots and their relationships - `UploadRepository` - Track blob upload metrics Each repository provides methods like: + - `Create()` - Insert new record - `GetByID()` / `GetByPath()` / `GetByHash()` - Retrieve records - `Update()` - Update existing records - `Delete()` - Remove records -- Specialized queries for each entity type (e.g., `DeleteOrphaned()`, `GetIncompleteByHostname()`) +- Specialized queries for each entity type (e.g., `DeleteOrphaned()`, + `GetIncompleteByHostname()`) ## Transaction Management @@ -284,25 +357,33 @@ err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error { ``` This ensures consistency, especially important for operations like: + - Creating file-chunk mappings - Associating chunks with blobs - Updating snapshot statistics ## Performance Considerations -1. **Indexes**: - - Primary keys are automatically indexed - - `idx_files_path` on `files(path)` for efficient file lookups +1. **Indexes**: + - Primary keys are automatically indexed + - `idx_files_path` on `files(path)` for efficient file lookups -2. **Prepared Statements**: All queries use prepared statements for performance and security +2. **Prepared Statements**: All queries use prepared statements for performance + and security -3. **Batch Operations**: Where possible, operations are batched within transactions +3. **Batch Operations**: Where possible, operations are batched within + transactions -4. **Write-Ahead Logging**: The local index runs in SQLite WAL mode with a 10-second busy timeout, so a read-only command such as `snapshot list` can read it while a backup writes to it. Committed rows can sit in the `-wal` file beside the index until a checkpoint, so the metadata export copies the index through SQLite (`VACUUM INTO`), not as a file +4. **Write-Ahead Logging**: The local index runs in SQLite WAL mode with a + 10-second busy timeout, so a read-only command such as `snapshot list` can + read it while a backup writes to it. Committed rows can sit in the `-wal` + file beside the index until a checkpoint, so the metadata export copies the + index through SQLite (`VACUUM INTO`), not as a file ## Data Integrity -1. **Foreign Keys**: Enforced through CASCADE DELETE and application-level repository methods +1. **Foreign Keys**: Enforced through CASCADE DELETE and application-level + repository methods 2. **Unique Constraints**: Chunk hashes, file paths, and blob hashes are unique 3. **Null Handling**: Nullable fields clearly indicate in-progress operations -4. **Timestamp Tracking**: All major operations record timestamps for auditing \ No newline at end of file +4. **Timestamp Tracking**: All major operations record timestamps for auditing diff --git a/docs/REPOSTRUCTURE.md b/docs/REPOSTRUCTURE.md index d2f003a..78a30f6 100644 --- a/docs/REPOSTRUCTURE.md +++ b/docs/REPOSTRUCTURE.md @@ -1,10 +1,13 @@ # Vaultik S3 Repository Structure -This document describes the structure and organization of data stored in the S3 bucket by Vaultik. +This document describes the structure and organization of data stored in the S3 +bucket by Vaultik. ## Overview -Vaultik stores all backup data in an S3-compatible object store. The repository consists of two main components, plus a marker file: +Vaultik stores all backup data in an S3-compatible object store. The repository +consists of two main components, plus a marker file: + 1. **Blobs** - The actual backup data (content-addressed, encrypted) 2. **Metadata** - Snapshot information and manifests (partially encrypted) @@ -23,120 +26,208 @@ Vaultik stores all backup data in an S3-compatible object store. The repository └── manifest.json.zst ``` -The metadata subdirectory is named with the **remote key**, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See [Remote Key Derivation](#remote-key-derivation). +The metadata subdirectory is named with the **remote key**, a one-way hash of +the snapshot ID, not with the human-readable snapshot ID itself. See +[Remote Key Derivation](#remote-key-derivation). ## Marker File (`vaultik-destination.txt`) -Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. While the local index holds snapshots stored there, a destination that answers a look-up of the marker with not found is treated as not present: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last. +Every backup writes this file at the root before it uploads anything. It holds +one fixed line of plain text and names no host, path or time. While the local +index holds snapshots stored there, a destination that answers a look-up of the +marker with not found is treated as not present: `snapshot create` refuses to +write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` +and `remote nuke` behave as for a destination that cannot be listed. An +unplugged volume leaves an empty directory at its mount point, and without the +marker that directory would read as a destination whose snapshots are all gone. +`remote nuke` deletes the marker last. -When the look-up of the marker fails for any reason other than not found, the check is skipped and the command goes on. S3 refuses it on every run to credentials that allow only PUT, so a host with such credentials does not notice a destination emptied since its last backup, and its next backup would leave out the data its local index records as already stored. After the destination is emptied, run `vaultik database delete` on such a host before it backs up again. +When the look-up of the marker fails for any reason other than not found, the +check is skipped and the command goes on. S3 refuses it on every run to +credentials that allow only PUT, so a host with such credentials does not notice +a destination emptied since its last backup, and its next backup would leave out +the data its local index records as already stored. After the destination is +emptied, run `vaultik database delete` on such a host before it backs up again. ## Blobs Directory (`blobs/`) ### Structure + - **Path format**: `blobs///` -- **Example**: `blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678` -- **Sharding**: The two-level directory structure (using the first 4 characters of the hash) prevents any single directory from containing too many objects +- **Example**: + `blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678` +- **Sharding**: The two-level directory structure (using the first 4 characters + of the hash) prevents any single directory from containing too many objects ### Content + - **What it contains**: Packed collections of content-defined chunks from files - **Format**: Zstandard compressed, then Age encrypted - **Encryption**: Always encrypted with Age using the configured recipients -- **Naming**: Content-addressed. The blob's name is `hex(SHA256(SHA256(uncompressed blob contents)))` — the double SHA-256 of the concatenated chunk data, computed before compression and encryption, not a hash of the stored (compressed, encrypted) bytes. One consequence: only a holder of the age private key can check a stored blob's integrity, because matching a blob to its name means decrypting and decompressing it first — which is what `restore` and `verify --deep` do. Implemented in `internal/blobgen` (`DoubleSHA256`). This is the canonical description of blob naming; other documents and comments point here. +- **Naming**: Content-addressed. The blob's name is + `hex(SHA256(SHA256(uncompressed blob contents)))` — the double SHA-256 of the + concatenated chunk data, computed before compression and encryption, not a + hash of the stored (compressed, encrypted) bytes. One consequence: only a + holder of the age private key can check a stored blob's integrity, because + matching a blob to its name means decrypting and decompressing it first — + which is what `restore` and `verify --deep` do. Implemented in + `internal/blobgen` (`DoubleSHA256`). This is the canonical description of blob + naming; other documents and comments point here. ### Why Encrypted -Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy. + +Blobs contain the actual file data from backups and must be encrypted for +security. The content-addressing ensures deduplication while the encryption +ensures privacy. ## Metadata Directory (`metadata/`) -Each snapshot has its own subdirectory. The directory is **not** named with the human-readable snapshot ID; it is named with the remote key — a one-way hash of that ID. The human ID is never written to the destination store as a directory name (see [Remote Key Derivation](#remote-key-derivation)). +Each snapshot has its own subdirectory. The directory is **not** named with the +human-readable snapshot ID; it is named with the remote key — a one-way hash of +that ID. The human ID is never written to the destination store as a directory +name (see [Remote Key Derivation](#remote-key-derivation)). ### Snapshot ID Format -The human-readable snapshot ID is used in CLI arguments, log lines, and the local database. It is not written to the destination store. -- **Format**: `__` (or `_` if no - name was specified) +The human-readable snapshot ID is used in CLI arguments, log lines, and the +local database. It is not written to the destination store. + +- **Format**: `__` (or `_` + if no name was specified) - **Example**: `laptop_home_2024-01-15T14:30:52Z` - **Components**: - - Short hostname (everything before the first dot is stripped from the FQDN) - - Snapshot name from the configured `snapshots:` map (optional) - - RFC3339 UTC timestamp + - Short hostname (everything before the first dot is stripped from the FQDN) + - Snapshot name from the configured `snapshots:` map (optional) + - RFC3339 UTC timestamp -This ID reveals the hostname, the configured snapshot name, and the backup time, so it is never used as the on-disk directory name — the remote key is used instead. +This ID reveals the hostname, the configured snapshot name, and the backup time, +so it is never used as the on-disk directory name — the remote key is used +instead. ### Remote Key Derivation -The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string. The hash is not reversible, but it uses no secret: an observer who guesses a candidate hostname and snapshot name can hash it the same way and confirm whether that snapshot is present. The remote key keeps names out of a plain listing; it does not hide them from a guess. Implemented in `internal/snapshot/remotekey.go`. +The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double +SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The +result is a 64-character hex string. The hash is not reversible, but it uses no +secret: an observer who guesses a candidate hostname and snapshot name can hash +it the same way and confirm whether that snapshot is present. The remote key +keeps names out of a plain listing; it does not hide them from a guess. +Implemented in `internal/snapshot/remotekey.go`. Worked example: + - Snapshot ID: `server1_home_2025-06-01T12:00:00Z` - Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa` -- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/` +- Directory: + `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/` -A plain listing of the destination store therefore shows only these hashes, not the hostname or snapshot name of any backup — but because the hash uses no secret, a guessed hostname and snapshot name can be hashed and confirmed against the listing. The same remote key is stored in the manifest's `snapshot_id` field. +A plain listing of the destination store therefore shows only these hashes, not +the hostname or snapshot name of any backup — but because the hash uses no +secret, a guessed hostname and snapshot name can be hashed and confirmed against +the listing. The same remote key is stored in the manifest's `snapshot_id` +field. ### Files in Each Snapshot Directory #### `db.zst.age` - Encrypted Database + - **What it contains**: Pruned binary SQLite database for this snapshot - **Format**: Binary SQLite → Zstandard compressed → Age encrypted - **Encryption**: Encrypted with Age -- **Purpose**: Contains full file metadata, chunk mappings, and all relationships -- **Why encrypted**: Contains sensitive metadata like file paths, permissions, and ownership +- **Purpose**: Contains full file metadata, chunk mappings, and all + relationships +- **Why encrypted**: Contains sensitive metadata like file paths, permissions, + and ownership #### `manifest.json.zst` - Unencrypted Blob Manifest + - **What it contains**: JSON list of all blob hashes referenced by this snapshot - **Format**: JSON → Zstandard compressed (NOT encrypted) - **Encryption**: NOT encrypted - **Purpose**: Enables pruning operations without requiring decryption keys - **Structure**: + ```json { - "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa", - "timestamp": "2025-06-01T12:00:00Z", - "blob_count": 42, - "total_compressed_size": 1048576, - "blobs": [ - { "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 24576 }, - { "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 32768 } - ] + "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa", + "timestamp": "2025-06-01T12:00:00Z", + "blob_count": 42, + "total_compressed_size": 1048576, + "blobs": [ + { + "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", + "compressed_size": 24576 + }, + { + "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", + "compressed_size": 32768 + } + ] } ``` -`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is written in the clear. + +`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is +written in the clear. ### Why Manifest is Unencrypted + The manifest must be readable without the private key to enable: + 1. **Pruning operations** - Identifying unreferenced blobs for deletion 2. **Storage analysis** - Understanding space usage without decryption 3. **Verification** - Checking blob existence without decryption -4. **Cross-snapshot deduplication analysis** - Finding shared blobs between snapshots +4. **Cross-snapshot deduplication analysis** - Finding shared blobs between + snapshots -The manifest contains the remote key, the backup timestamp, the blob count and total compressed size, and each blob's hash and compressed size. It contains no file names, paths, or other decrypted metadata. +The manifest contains the remote key, the backup timestamp, the blob count and +total compressed size, and each blob's hash and compressed size. It contains no +file names, paths, or other decrypted metadata. ## Security Considerations ### What's Encrypted + - **All file content** (in blobs) -- **All file metadata** (paths, permissions, timestamps, ownership in db.zst.age) +- **All file metadata** (paths, permissions, timestamps, ownership in + db.zst.age) - **File-to-chunk mappings** (in db.zst.age) ### What's Not Encrypted -- **The remote key** — directory names and the manifest `snapshot_id`, a one-way hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation)) + +- **The remote key** — directory names and the manifest `snapshot_id`, a one-way + hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation)) - **The backup timestamp** (in manifest.json.zst) - **Blob hashes and their compressed sizes** (in manifest.json.zst) - **Blob count and total compressed size per snapshot** (in manifest.json.zst) ### Privacy Implications -From the unencrypted data, an observer of the destination store can determine: -- **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear -- How many blobs each snapshot references, and the total compressed size -- The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns) -- **Whether a guessed hostname and snapshot name are present** — the remote key is an unkeyed hash, so an observer holding candidate names can hash each one and match it against the directory listing. The human ID is never published, so it cannot be read off directly, but it can be confirmed by guessing. -Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless. +From the unencrypted data, an observer of the destination store can determine: + +- **When each backup was taken** — not from the directory name, which is a + one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, + which is published in the clear +- How many blobs each snapshot references, and the total compressed size +- The compressed size of each blob, and which blobs are shared between snapshots + (deduplication patterns) +- **Whether a guessed hostname and snapshot name are present** — the remote key + is an unkeyed hash, so an observer holding candidate names can hash each one + and match it against the directory listing. The human ID is never published, + so it cannot be read off directly, but it can be confirmed by guessing. + +Together these give an observer a timing-and-size profile of every snapshot. +This is an accepted, documented property of the format, not a defect: the +manifest is unencrypted so that pruning can run without the private key, and the +timing channel could not be closed by encrypting it anyway — object creation +times and per-object sizes stay visible at the storage layer on both `s3://` and +`file://` destinations regardless. An observer cannot determine: -- The hostname or snapshot name of any backup by reading it off the store — the directory name and the manifest `snapshot_id` are unkeyed hashes of the human ID, so the text is never published (though a guessed name can be confirmed, as above) + +- The hostname or snapshot name of any backup by reading it off the store — the + directory name and the manifest `snapshot_id` are unkeyed hashes of the human + ID, so the text is never published (though a guessed name can be confirmed, as + above) - File names or paths - File contents - File permissions or ownership @@ -145,40 +236,119 @@ An observer cannot determine: ### Accepted Risks -These are known, deliberate properties of the format and the tooling, recorded so an operator can weigh them rather than discover them. +These are known, deliberate properties of the format and the tooling, recorded +so an operator can weigh them rather than discover them. -1. **No proof of authorship.** Restore and `verify --deep` prove that data decrypts with the age private key and matches its unkeyed content hashes. They do not prove who wrote it: anyone who knows a recipient public key and can replace objects on the destination can substitute a snapshot they built. The recipient string is not stored at the destination, but a compromised backed-up host has it. Defences live on the destination side — bucket versioning or object lock, credentials for the source host that cannot delete or overwrite existing versions, and pruning from a trusted host. Note that S3 `PutObject` overwrites an existing key, so PUT permission alone is not append-only. -2. **Compression reveals sizes.** Blobs and `db.zst.age` are zstd-compressed then age-encrypted; the manifest is compressed only. age does not pad, so an object's size is the exact compressed length of its contents. All new chunks packed into one blob share a single zstd stream (8 MiB window, 4 MiB at compression levels 1-2), and a blob is closed at `blob_size_limit` and at the end of each configured path. Because a stored chunk is never packed again, someone who can write into a backed-up file and watch blob sizes learns something only when their controlled data and a secret land in the same chunk of a file that keeps changing. Advice: back up any outsider-writable directory as its own snapshot. -3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of 2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them. -4. **Decrypted data on local disk.** Several commands stage plaintext under `$TMPDIR`: `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` and removed on success, but a `kill -9` or a power loss leaves them behind — delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` should be trusted to the same degree as the restore target. -5. **Store permissions differ per command.** The backed-up host needs only PUT to run `snapshot create`: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — `snapshot verify`, `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`, `snapshot remove`, and `remote nuke` also delete. The recommended cron line uses `--prune`, which runs `prune` on the backed-up host, so granting that host `--prune` gives it credentials that can delete its own backups. To keep the source host to PUT only, prune from a separate trusted host instead. -6. **Changing recipients does not re-encrypt existing data.** Deduplicated chunks and same-named blobs already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run `vaultik database delete` and take a full backup to a fresh destination or prefix. -7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients. Long-lived ciphertext held by a third party (the destination operator) has no fallback if X25519 is ever broken: there is no second recipient type and no post-quantum option. +1. **No proof of authorship.** Restore and `verify --deep` prove that data + decrypts with the age private key and matches its unkeyed content hashes. + They do not prove who wrote it: anyone who knows a recipient public key and + can replace objects on the destination can substitute a snapshot they built. + The recipient string is not stored at the destination, but a compromised + backed-up host has it. Defences live on the destination side — bucket + versioning or object lock, credentials for the source host that cannot delete + or overwrite existing versions, and pruning from a trusted host. Note that S3 + `PutObject` overwrites an existing key, so PUT permission alone is not + append-only. +2. **Compression reveals sizes.** Blobs and `db.zst.age` are zstd-compressed + then age-encrypted; the manifest is compressed only. age does not pad, so an + object's size is the exact compressed length of its contents. All new chunks + packed into one blob share a single zstd stream (8 MiB window, 4 MiB at + compression levels 1-2), and a blob is closed at `blob_size_limit` and at the + end of each configured path. Because a stored chunk is never packed again, + someone who can write into a backed-up file and watch blob sizes learns + something only when their controlled data and a secret land in the same chunk + of a file that keeps changing. Advice: back up any outsider-writable + directory as its own snapshot. +3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The + default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of + 2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a + blob holds hundreds of chunks, so individual chunk lengths are not visible in + the blob's size; lowering the limit toward the chunk size begins to expose + them. +4. **Decrypted data on local disk.** Several commands stage plaintext under + `$TMPDIR`: `snapshot restore` writes decrypted blobs under + `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at + `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at + `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of + the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` + and removed on success, but a `kill -9` or a power loss leaves them behind — + delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` + should be trusted to the same degree as the restore target. +5. **Store permissions differ per command.** The backed-up host needs only PUT + to run `snapshot create`: it writes blobs and metadata and neither reads nor + deletes them. Other commands need more — `snapshot verify`, + `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`, + `snapshot remove`, and `remote nuke` also delete. The recommended cron line + uses `--prune`, which runs `prune` on the backed-up host, so granting that + host `--prune` gives it credentials that can delete its own backups. To keep + the source host to PUT only, prune from a separate trusted host instead. +6. **Changing recipients does not re-encrypt existing data.** Deduplicated + chunks and same-named blobs already on the destination stay encrypted to the + recipients in force when they were written. A new snapshot that reuses them + cannot be restored with a newly added recipient's key alone, because those + reused objects were never encrypted to it. To make everything readable by a + new key, run `vaultik database delete` and take a full backup to a fresh + destination or prefix. +7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients. + Long-lived ciphertext held by a third party (the destination operator) has no + fallback if X25519 is ever broken: there is no second recipient type and no + post-quantum option. ## Consistency Guarantees 1. **Blobs are immutable** - Once written, a blob is never modified -2. **Blobs are written before metadata** - A snapshot's metadata is only written after all its blobs are successfully uploaded -3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst are written as complete files -4. **A snapshot is marked complete in the local DB only after its metadata is uploaded** - `finalizeSnapshotMetadata` runs `ExportSnapshotMetadata` first and records completion (`MarkSnapshotComplete`) only once the export succeeds (see the backup data flow in [ARCHITECTURE.md](../ARCHITECTURE.md)). A crash during the export therefore leaves the snapshot incomplete, so the next backup's `PruneDatabase` drops it and re-backs-up its data, rather than leaving a completed-looking row in the local index with no matching metadata on the destination store. (A crash in the brief moment after the export succeeds but before completion is recorded leaves a fully-restorable snapshot on the destination that the local index drops as incomplete on the next run; `snapshot list` then reports it honestly as remote-only, which is the safe direction: the destination copy stays restorable.) +2. **Blobs are written before metadata** - A snapshot's metadata is only written + after all its blobs are successfully uploaded +3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst + are written as complete files +4. **A snapshot is marked complete in the local DB only after its metadata is + uploaded** - `finalizeSnapshotMetadata` runs `ExportSnapshotMetadata` first + and records completion (`MarkSnapshotComplete`) only once the export succeeds + (see the backup data flow in [ARCHITECTURE.md](../ARCHITECTURE.md)). A crash + during the export therefore leaves the snapshot incomplete, so the next + backup's `PruneDatabase` drops it and re-backs-up its data, rather than + leaving a completed-looking row in the local index with no matching metadata + on the destination store. (A crash in the brief moment after the export + succeeds but before completion is recorded leaves a fully-restorable snapshot + on the destination that the local index drops as incomplete on the next run; + `snapshot list` then reports it honestly as remote-only, which is the safe + direction: the destination copy stays restorable.) ## Pruning Safety The prune operation is safe because: -1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references. Under `metadata/` it deletes only a directory that has no `manifest.json.zst`, whose name is a remote key, and that no local snapshot record claims + +1. It keeps every blob listed in any snapshot's manifest and deletes only blobs + that no manifest references. Under `metadata/` it deletes only a directory + that has no `manifest.json.zst`, whose name is a remote key, and that no + local snapshot record claims 2. Manifests are unencrypted and can be read without keys -3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced -4. It deletes an unreferenced blob only once the blob is older than `prune_grace_period` (default `24h`), and a directory under `metadata/` with no manifest only once every object in it is. A backup uploads its blobs and its `db.zst.age` before its manifest, so a prune running meanwhile, on this host or another sharing the destination, leaves them as long as no backup run takes longer than the grace period. The grace period does not protect a blob a backup reuses from an earlier run that stopped before its manifest: the backup skips the upload of a blob already on the destination, so that blob keeps the time of its first upload, and a prune before the backup's manifest is written can delete it; the new snapshot then cannot be restored. `remote nuke` has no such check +3. If any manifest cannot be downloaded or decoded, prune deletes nothing and + exits with an error, rather than treating that snapshot's blobs as + unreferenced +4. It deletes an unreferenced blob only once the blob is older than + `prune_grace_period` (default `24h`), and a directory under `metadata/` with + no manifest only once every object in it is. A backup uploads its blobs and + its `db.zst.age` before its manifest, so a prune running meanwhile, on this + host or another sharing the destination, leaves them as long as no backup run + takes longer than the grace period. The grace period does not protect a blob + a backup reuses from an earlier run that stopped before its manifest: the + backup skips the upload of a blob already on the destination, so that blob + keeps the time of its first upload, and a prune before the backup's manifest + is written can delete it; the new snapshot then cannot be restored. + `remote nuke` has no such check ## Restoration Requirements To restore from a backup, you need: + 1. **The Age private key** - To decrypt blobs and database 2. **The snapshot metadata** - Both files from the snapshot's metadata directory 3. **All referenced blobs** - As listed in the manifest The restoration process: + 1. Download and decrypt the database dump to understand file structure 2. Download and decrypt the required blobs 3. Reconstruct files from their chunks -4. Restore file metadata (permissions, timestamps, etc.) \ No newline at end of file +4. Restore file metadata (permissions, timestamps, etc.)