Fix the ten findings of the review on the destination, the marker file and chunk sizes

README.md: the "Getting started" destination advice, the Full Disk Access paragraph moved above the block with the reopen step, the mount-point case, the remote nuke failure sentence, the hostname setting, the upload order under "Sharing a destination", the confirmation list and the dev stamp sentence. configs/config.example.yml: the storage_url and blob_size_limit comments. docs/ARCHITECTURE.md and docs/DATAMODEL.md: the Snapshot ID and Marker file terms, and "marker file" in place of "destination marker". Model: fable-5-1
Fix the six findings of the review on config set, version and blob names
2026-11-11 09:11:51 +00:00 · 2026-11-11 08:28:40 +00:00 · 2026-11-11 07:55:31 +00:00 · 2026-11-11 07:11:05 +00:00 · 2026-11-11 07:01:54 +00:00 · 2026-11-11 07:01:54 +00:00
18 changed files with 2762 additions and 2220 deletions
@@ -1,16 +1,32 @@
 # vaultik configuration file example
-# This file shows all available configuration options with their default values
+# This file shows every configuration option. The values are examples, not
-# Copy this file and uncomment/modify the values you need
+# defaults; where an option has a default, its comment gives it.
 # Copy this file and change the values you need
-# Age recipient public keys for encryption
+# Age recipient public keys for encryption (age: https://age-encryption.org/)
 # Backups are encrypted to these public keys. snapshot create needs at least
 # one; listing, verifying and restoring do not
-# Generate with: age-keygen | grep "public key"
+# Generate a key pair, and show its public key, with:
 #   age-keygen -o vaultik_backup_private_key.txt
 #   grep 'public key' vaultik_backup_private_key.txt
 # Keep the private key file: restoring needs it. Do not pipe age-keygen
 # straight into grep, which would discard the private key
 # Default: none
 age_recipients:
  - age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz
 # Age private key for decryption
 # Only snapshot restore and snapshot verify --deep need it. The
 # VAULTIK_AGE_SECRET_KEY environment variable overrides this setting
 # Do not set it on a machine you back up. Supply the environment variable
 # only on a machine where you run snapshot restore or snapshot verify --deep
 # Default: not set
 #age_secret_key: YOUR_AGE_PRIVATE_KEY
 # Named snapshots - each snapshot can contain multiple paths
 # Each snapshot gets its own ID and can have snapshot-specific excludes
 # Required: at least one snapshot with at least one path. Only the config
 # commands, version and completion run without one
 snapshots:
  testing:
    paths:
@@ -253,11 +269,18 @@ snapshots:
      - "**/vendor"
 # Global patterns to exclude from all backups
 # Default: none
 exclude:
  - "*.tmp"
-# Storage URL - use either this OR the s3 section below
+# Storage URL. When it is not set, the endpoint, bucket and prefix in the s3
 # section below are used instead.
 # Supports: s3://bucket/prefix, file:///path, rclone://remote/path
 # An s3:// URL also needs s3.access_key_id and s3.secret_access_key below
 # A file:// path must be absolute, and the directory must already exist:
 # vaultik never creates it
 # An rclone:// URL names an rclone remote (https://rclone.org/), a storage
 # service set up in rclone's own config (rclone config)
 storage_url: "rclone://myremote/path/to/backups"
 # S3-compatible storage configuration
@@ -283,26 +306,30 @@ storage_url: "rclone://myremote/path/to/backups"
 #  #region: us-east-1
 #
 #  # Use SSL/TLS for S3 connections
 #  # Only read when storage_url is not set and the endpoint has no scheme;
 #  # with an s3:// storage_url, use ?ssl=false in the URL instead
 #  # Default: true
 #  #use_ssl: true
 #
 #  # Part size for multipart uploads
 #  # Minimum 5MiB, maximum 5GiB; affects memory usage during upload
-#  # A blob too large for 10,000 parts of this size gets larger parts
+#  # An object too large for 10,000 parts of this size gets larger parts
 #  # Supports: 10MB, 16MiB, 100MiB, etc. (5MB is below the minimum)
 #  # Default: 5MiB
 #  #part_size: 5MiB
 # Path to local SQLite index database
 # This database tracks file state for incremental backups
-# It must be an absolute path; ~ is expanded
+# The VAULTIK_INDEX_PATH environment variable overrides this setting
 # Both must be absolute paths; a leading ~ is expanded
 # Default: the platform data directory, e.g.
 #   macOS: ~/Library/Application Support/vaultik/index.sqlite
 #   Linux: ~/.local/share/vaultik/index.sqlite
 #index_path: /path/to/index.sqlite
-# Average chunk size for content-defined chunking
+# Average size of a chunk, a piece of a file's content. Chunk boundaries
-# Smaller chunks = better deduplication but more metadata
+# depend on the file's content, not on fixed offsets
 # Smaller chunks find more duplicate content and make the index larger
 # Minimum 1MB
 # Supports: 10MB, 5M, 1GB, 64MiB, etc.
 # Default: 10MB
@@ -312,9 +339,11 @@ storage_url: "rclone://myremote/path/to/backups"
 # Multiple chunks are packed into blobs up to this size
 # Must be at least four times chunk_size (the largest chunk the chunker can
 # emit); a smaller limit would let a single-chunk blob exceed it.
-# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
+# Chunking uses no secret (the parameters of the FastCDC chunking algorithm
-# large limit a blob holds hundreds of chunks, so individual chunk lengths are
+# are fixed and public). A full blob at a large limit holds hundreds of
-# not visible in its size; lowering the limit toward chunk_size exposes them.
+# chunks, so its size does not show any one chunk's length. The last blob of
 # each configured path can hold a single chunk. Lowering the limit toward
 # chunk_size exposes chunk lengths.
 # A backup needs free temporary space, because each blob is written whole to
 # a temporary file before it is uploaded (up to about blob_size_limit; an
 # rclone destination that cannot stream uploads needs about twice that) and
@@ -330,12 +359,14 @@ storage_url: "rclone://myremote/path/to/backups"
 # Default: 3
 compression_level: 5
-# How old a blob that no snapshot lists must be before prune deletes it, and
+# How old a blob that no snapshot lists must be before vaultik prune deletes
-# how old every object in a directory under metadata/ with no manifest must
+# it. Each snapshot has a directory under metadata/ at the destination, and
-# be before prune deletes that directory
+# its manifest there lists the blobs it uses. This is also how old every
-# A backup uploads its blobs and its db.zst.age before its manifest, so on a
+# object in a directory with no manifest must be before vaultik prune deletes
-# destination shared between hosts or users this must be longer than the
+# that directory
-# longest backup run
+# A backup uploads its blobs and its encrypted database (db.zst.age) before
 # its manifest, so on a destination shared between hosts or users this must
 # be longer than the longest backup run
 # It does not protect a blob a backup reuses from an earlier run that stopped
 # before its manifest: that blob keeps the time of its first upload, so a
 # prune can delete it
@@ -346,3 +377,9 @@ compression_level: 5
 # Hostname to use in backup metadata
 # Default: system hostname
 #hostname: myserver
 # Environment variables that vaultik sets in its own process when it loads
 # the config. Its keys are variable names
 # Default: none
 #env:
 #  NAME: value
@@ -1,486 +1,418 @@
-# Vaultik Architecture
+# vaultik architecture
-This document describes the internal architecture of Vaultik, focusing on the
+This document describes how the `vaultik` code is organised. It covers three
-data model, type instantiation, and the relationships between core modules.
+things:
-## Overview
+- the packages
 - how they are wired together
 - what happens during a backup, a restore, a verify and a prune
-Vaultik is a backup system that uses content-defined chunking for deduplication
+Two other documents cover the stored data:
 and packs chunks into large, compressed, encrypted blobs for efficient cloud
 storage. The system is built around dependency injection using
 [uber-go/fx](https://github.com/uber-go/fx).
-## Data Flow
+- [`DATAMODEL.md`](DATAMODEL.md) describes the local index.
 - [`REPOSTRUCTURE.md`](REPOSTRUCTURE.md) describes the layout of the
  destination.
 ## Terms
 - **Snapshot**: one backup of the paths listed under one name in the config.
 - **Snapshot ID**: the name of a snapshot, in the form
  `<hostname>_<snapshot-name>_<timestamp>`.
 - **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
  of its content.
 - **Blob**: one object at the destination. It holds many chunks, compressed and
  then encrypted.
 - **Local index**: the SQLite database on the backed-up machine.
 - **Destination**: the place the backups are stored.
 - **Marker file**: `vaultik-destination.txt` at the destination. See
  [The marker file](REPOSTRUCTURE.md#the-marker-file).
 - **Manifest**: the unencrypted list of the blobs one snapshot uses.
 - **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
  at the destination.
 ## How a backup moves data
 ```
-Source Files
+source files
     │
     ▼
-┌─────────────────┐
+  Scanner    walks the paths and finds the files that changed
-│    Scanner      │  Walks directories, detects changed files
+     │
-└────────┬────────┘
+     ▼
-         │
+  Chunker    cuts each changed file into chunks
-         ▼
+     │
-┌─────────────────┐
+     ▼
-│    Chunker      │  Splits files into variable-size chunks (FastCDC)
+  Packer     packs new chunks into a blob, compressed and encrypted
-└────────┬────────┘
+     │
-         │
+     ▼
-         ▼
+  Storer     uploads the blob to the destination
 ┌─────────────────┐
 │    Packer       │  Accumulates chunks, compresses (zstd), encrypts (age)
 └────────┬────────┘
         │
         ▼
 ┌─────────────────┐
 │   S3 Client     │  Uploads blobs to remote storage
 └─────────────────┘
 ```
-## Data Model
+## Packages
-### Core Entities
+- `cmd/vaultik`: the `main` function. It starts optional profiling and calls
  `cli.Entry`.
 - `internal/cli`: the commands, their flags and their help text. It builds the
  application, takes the lock and handles interrupts.
 - `internal/vaultik`: the code behind each command. `Vaultik` holds every
  dependency a command uses.
 - `internal/snapshot`: the `Scanner`, which backs up one path, and the
  `SnapshotManager`, which creates snapshot records and uploads their metadata.
 - `internal/chunker`: cuts a stream into chunks with the FastCDC algorithm.
 - `internal/blob`: the `Packer`, which collects chunks into blobs.
 - `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
  and the reader that reverses it.
 - `internal/database`: the local index. It holds the schema and the repository
  types that read and write the tables. [`DATAMODEL.md`](DATAMODEL.md#code) says
  which type serves which table.
 - `internal/storage`: the `Storer` interface and its three implementations, for
  S3, a local directory and [rclone](https://rclone.org/).
 - `internal/s3`: the S3 client that the S3 `Storer` uses.
 - `internal/config`: loads and checks the config file.
 - `internal/ui`: formats the status lines, warnings and errors a user sees.
 - `internal/log`: the logger. It writes to stderr.
 - `internal/pidlock`: the lock that `RunWithApp` takes for a command that
  changes data, so that only one runs at a time.
 - `internal/globals`: the version, commit and commit date, set when the binary
  is built.
 - `internal/types`: distinct types for IDs, hashes and paths.
-The database tracks five primary entities and their relationships:
+## Wiring
-```
+The application is assembled with [uber-go/fx](https://github.com/uber-go/fx), a
-┌──────────────┐     ┌──────────────┐     ┌──────────────┐
+dependency injection library. `cli.NewApp` lists the providers:
 │   Snapshot   │────▶│     File     │────▶│    Chunk     │
 └──────────────┘     └──────────────┘     └──────────────┘
       │                                         │
       │                                         │
       ▼                                         ▼
 ┌──────────────┐                          ┌──────────────┐
 │     Blob     │◀─────────────────────────│  BlobChunk   │
 └──────────────┘                          └──────────────┘
 ```
 ### Entity Descriptions
 #### File (`database.File`)
 Represents a file, directory, or symlink in the backup system. Stores metadata
 needed for restoration:
 - Path, mtime
 - Size, mode, ownership (uid, gid)
 - Symlink target (if applicable)
 It also stores `source_path`, the source directory the scan found it under, made
 absolute and with symlinks resolved. Restore does not read it.
 #### Chunk (`database.Chunk`)
 A content-addressed unit of data. Files are split into variable-size chunks
 using the FastCDC algorithm:
 - `ChunkHash`: SHA256 hash of chunk content (primary key)
 - `Size`: Chunk size in bytes
 Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (2.5MB-40MB for
 the 10MB default average).
 #### FileChunk (`database.FileChunk`)
 Maps files to their constituent chunks:
 - `FileID`: Reference to the file
 - `Idx`: Position of this chunk within the file (0-indexed)
 - `ChunkHash`: Reference to the chunk
 #### Blob (`database.Blob`)
 The final storage unit uploaded to S3. Contains many compressed and encrypted
 chunks:
 - `ID`: UUID assigned at creation
 - `Hash`: `hex(SHA256(SHA256(uncompressed blob contents)))`, computed before
  compression and encryption (see
  [docs/REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs))
 - `UncompressedSize`: Total raw chunk data before compression
 - `CompressedSize`: Size after zstd compression and age encryption
 - `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
 Blob creation process:
 1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
 2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
 3. Concurrently, the same bytes are compressed with zstd, then encrypted with
   age (recipients configured in config), and written to a temporary file
 4. On finalize, the blob's name is the double SHA-256 of the uncompressed
   contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed,
   encrypted bytes
 5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and
   then deleted
 A backup needs free temporary space, because each blob is written whole to a
 temporary file before it is uploaded (up to about `blob_size_limit`; an rclone
 destination that cannot stream uploads needs about twice that) and the metadata
 export writes copies of the local index. Temporary files go to `$TMPDIR`
 (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to
 `/var/tmp`.
 #### BlobChunk (`database.BlobChunk`)
 Maps chunks to their position within blobs:
 - `BlobID`: Reference to the blob
 - `ChunkHash`: Reference to the chunk
 - `Offset`: Byte offset within the uncompressed blob
 - `Length`: Chunk size
 #### Snapshot (`database.Snapshot`)
 Represents a point-in-time backup:
 - `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g.
  `server1_home_2025-06-01T12:00:00Z`)
 - Tracks file count, chunk count, blob count, sizes, compression ratio
 - `CompletedAt`: Null until snapshot finishes successfully
 #### SnapshotFile / SnapshotBlob
 Join tables linking snapshots to their files and blobs.
 ### Relationship Summary
 ```
 Snapshot 1──────────▶ N SnapshotFile N ◀────────── 1 File
 Snapshot 1──────────▶ N SnapshotBlob N ◀────────── 1 Blob
 File     1──────────▶ N FileChunk    N ◀────────── 1 Chunk
 Blob     1──────────▶ N BlobChunk    N ◀────────── 1 Chunk
 ```
 ## Type Instantiation
 ### Application Startup
 The CLI uses fx for dependency injection. Here's the instantiation order:
 ```go
-// cli/app.go: NewApp()
+fx.Supply(config.Path(opts.ConfigPath)),
-fx.New(
+fx.Supply(opts.LogOptions),
-    fx.Supply(config.Path(opts.ConfigPath)),         // 1. Config path
+fx.Provide(globals.New),
-    fx.Supply(opts.LogOptions),                      // 2. Log options
+fx.Provide(log.New),
-    fx.Provide(globals.New),                         // 3. Globals
+config.Module,
-    fx.Provide(log.New),                             // 4. Logger config
+database.Module,
-    config.Module,                                   // 5. Config
+log.Module,
-    database.Module,                                 // 6. Database + Repositories
+storage.Module,
-    log.Module,                                      // 7. Logger initialization
+snapshot.Module,
-    storage.Module,                                  // 8. Storage backend (S3/file/rclone)
+fx.Provide(vaultik.New),
-    snapshot.Module,                                 // 9. SnapshotManager + ScannerFactory
+fx.Invoke(setupGlobals),
-    fx.Provide(vaultik.New),                         // 10. Vaultik orchestrator
+fx.NopLogger,
 )
 ```
-### Key Type Instantiation Points
+Each module provides these:
 #### 1. Config (`config.Config`)
 - **Created by**: `config.Module` via `config.LoadConfig()`
 - **When**: Application startup (fx DI)
 - **Contains**: All configuration from YAML file (S3 credentials, encryption
  keys, paths, etc.)
 #### 2. Database (`database.DB`)
- **Created by**: `database.Module` via `database.New()`
+- `config.Module` provides `*config.Config`, from `config.Load`.
- **When**: Application startup (fx DI)
+- `database.Module` provides `*database.DB` and `*database.Repositories`. It
- **Contains**: SQLite connection, path reference
+  creates the index file with mode `0600` when it does not exist.
 - `storage.Module` provides the `storage.Storer` that the config selects.
 - `snapshot.Module` provides the `snapshot.ScannerFactory` and the
  `*snapshot.SnapshotManager`.
 - `vaultik.New` provides `*vaultik.Vaultik`.
 A command runs through `cli.RunOperation`. It works like this:
 - `RunWithApp` takes the lock when the command changes data, and then starts the
  application.
 - The command's function runs in its own goroutine.
 - When the function returns, it asks the application to shut down.
 - On `SIGINT` or `SIGTERM`, the stop hook cancels the `Vaultik` context. It then
  waits for the function to return, so that its temporary files are removed.
 - The wait ends after 30 seconds at most.
 These commands do not build the application:
-#### 3. Repositories (`database.Repositories`)
+- the `config` commands
-
+- `database delete`
- **Created by**: `database.Module` via `database.NewRepositories()`
+- `completion`
- **When**: Application startup (fx DI)
+- `version`
 - **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots,
  etc.)
-#### 4. Vaultik (`vaultik.Vaultik`)
+## Backup
- **Created by**: `vaultik.New(VaultikParams)`
+`Vaultik.CreateSnapshot` runs a backup. It does these steps in order:
 - **When**: Application startup (fx DI)
 - **Contains**: All dependencies for backup operations
-```go
+- `markDestination` checks that the local index is not bound to another
-type Vaultik struct {
+  destination. It then checks the marker file, writes it, and binds the local
-    Globals         *globals.Globals
+  index to the configured destination.
-    Config          *config.Config
+- `PruneDatabase` removes unfinished snapshots and unused records from the local
-    DB              *database.DB
+  index.
-    Repositories    *database.Repositories
+- `createNamedSnapshot` runs once for each snapshot name.
-    Storage         storage.Storer
+- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
    ScannerFactory  snapshot.ScannerFactory
    SnapshotManager *snapshot.SnapshotManager
    Shutdowner      fx.Shutdowner
    Fs              afero.Fs
    ctx             context.Context
    cancel          context.CancelFunc
 }
 ```
-#### 5. SnapshotManager (`snapshot.SnapshotManager`)
+`CreateSnapshot` calls `skipPostBackupPrune` instead of `runPostBackupPrune` in
 these cases:
- **Created by**: `snapshot.Module` via `snapshot.NewSnapshotManager()`
+- `--skip-errors` skipped a file or directory that could not be read.
- **When**: Application startup (fx DI)
+- A configured path held nothing.
- **Responsibility**: Creates/completes snapshots, exports metadata to S3
+- An earlier snapshot of the same name found a mount point that this backup did
  not find.
-#### 6. Scanner (`snapshot.Scanner`)
+`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
 left in the index would make the backup skip files that were never fully stored.
- **Created by**: `ScannerFactory(ScannerParams)`
+### One snapshot
- **When**: Each `CreateSnapshot()` call
+
- **Contains**: Chunker, Packer, progress reporter
+`createNamedSnapshot` does these steps:
-```go
+- `resolveSnapshotPaths` makes each configured path absolute and resolves
-// vaultik/snapshot.go: CreateSnapshot()
+  symlinks.
-scanner := v.ScannerFactory(snapshot.ScannerParams{
+- `SnapshotManager.CreateSnapshotWithName` creates the snapshot record. The
-    EnableProgress: !opts.Cron,
+  record is not yet marked complete.
-    Fs:             v.Fs,
+- `Scanner.Scan` backs up each path in turn.
-})
+- `finalizeSnapshotMetadata` uploads the snapshot's metadata and marks the
-```
+  snapshot complete.
-#### 7. Chunker (`chunker.Chunker`)
+A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a
 `Chunker` and a `Packer`.
- **Created by**: `chunker.NewChunker(avgChunkSize)`
+### Scanning one path
 - **When**: Inside `snapshot.NewScanner()`
 - **Configuration**:
    - `avgChunkSize`: From config (default 10MB)
    - `minChunkSize`: avgChunkSize / 4
    - `maxChunkSize`: avgChunkSize \* 4
-#### 8. Packer (`blob.Packer`)
+`Scanner.Scan` works in these steps:
- **Created by**: `blob.NewPacker(PackerConfig)`
+- `repairInterruptedBlobs` deletes the records of blobs whose upload never
- **When**: Inside `snapshot.NewScanner()`
+  finished. Their data is then stored again.
- **Configuration**:
+- `loadDatabaseState` loads the known files and chunks into memory.
-    - `MaxBlobSize`: Maximum blob size before finalization (typically 10GB)
+- `scanPhase` walks the path. It sorts each regular file and symlink into
-    - `CompressionLevel`: zstd level (1-19)
+  changed or unchanged.
-    - `Recipients`: age public keys for encryption
+- `detectDeletedFiles` counts the files that the previous snapshot of this name
  had and that are now gone.
 - `batchAddFilesToSnapshot` records the unchanged files in the new snapshot.
 - `recordDirectories` records every directory, and the mount points found.
 - `processPhase` reads each changed regular file and stores it. It records each
  changed symlink.
 The walk handles each kind of entry differently:
 - A regular file or a symlink is compared with its record in the index.
 - A directory is recorded with its metadata. It has no content to store.
 - A device node, a named pipe or a socket is skipped.
 - An entry that matches an exclude pattern is skipped. For a directory, so is
  everything in it.
 - The local index, its SQLite side files and the lock file `vaultik.pid` are
  skipped, whatever the exclude patterns say.
-```go
+### Deduplication
-// snapshot/scanner.go: NewScanner()
+
-packerCfg := blob.PackerConfig{
+Data is stored once at two levels.
-    MaxBlobSize:      cfg.MaxBlobSize,
+
-    CompressionLevel: cfg.CompressionLevel,
+- **Files.** A file is not read again when its record in the index still matches
-    Recipients:       cfg.AgeRecipients,
+  it.
-    Repositories:     cfg.Repositories,
+- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk that
-    Fs:               cfg.FS,
+  an uploaded blob already holds, according to the index, is not stored again.
-}
+
-packer, err := blob.NewPacker(packerCfg)
+A file's record matches when all of these are unchanged:
-```
+
 - the size
 - the modification time
 - the mode
 - the owner and group
 - the link target
 One case overrides the file check. A file is read again when it lists a chunk
 that no uploaded blob holds. This happens when the only snapshot that used the
 blob holding that chunk is removed. `loadFilesToRechunk` finds these files.
 ### Chunks
 `chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
 on the content, so an insertion early in a file moves only nearby chunks.
 `chunk_size` sets the average chunk size. No chunk is larger than four times
 this size. Only a file's last chunk can be smaller than a quarter of it. With
 the default of 10 MB, those limits are 40 MB and 2.5 MB.
 `ChunkReaderStreaming` passes each chunk to a callback as it is found. The
 chunker buffers at most twice the largest chunk size.
 ### Blobs
 `blob.Packer` collects new chunks into a blob. For each blob it does this:
 - It gives the blob a UUID and creates a temporary file.
 - It adds each chunk's bytes to a running SHA-256 hash.
 - In the same pass it compresses the bytes with
  [zstd](https://github.com/facebook/zstd) and encrypts them with
  [age](https://age-encryption.org/). The result goes to the temporary file.
 - It records each chunk's offset and length within the blob.
 - It closes the blob when the next chunk would pass `blob_size_limit`, and at
  the end of each path.
-## Module Responsibilities
+The blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. It is computed
 from the chunk data before compression and encryption.
 [`REPOSTRUCTURE.md`](REPOSTRUCTURE.md#blobs) says what follows from that.
-### `internal/cli`
+When a blob is closed, `Scanner.handleBlobReady` stores it:
-Entry point for fx application. Combines all modules and handles signal
+- It looks the blob up at the destination. A blob that is already there is not
-interrupts.
+  uploaded again.
 - Otherwise it uploads the temporary file to
  `blobs/<hash[0:2]>/<hash[2:4]>/<hash>`.
 - It records the upload in the index and links the blob to the snapshot.
 - It deletes the temporary file.
 A file's record is written to the index only after every blob that holds its
 chunks is uploaded.
 ### Snapshot metadata
 `finalizeSnapshotMetadata` does these steps in order:
 - `PopulateSnapshotBlobs` records every blob the snapshot uses.
 - `UpdateSnapshotStatsExtended` stores the snapshot's totals.
 - `ExportSnapshotMetadata` uploads the metadata.
 - `MarkSnapshotComplete` marks the snapshot complete.
-Key functions:
+Completion is recorded last on purpose. A backup that stops during the upload
 leaves the snapshot unfinished, and the next backup removes it and backs up its
 files again.
 `ExportSnapshotMetadata` works on a copy of the index, never on the index
 itself:
 - It copies the index into a temporary directory with `VACUUM INTO`.
 - It deletes the rows of every other snapshot from the copy. It also deletes the
  files and blobs that this snapshot does not use. It then deletes each chunk
  that no remaining file and no remaining blob uses.
 - It keeps the `local_meta` and `schema_migrations` tables as they are. The
  uploaded database therefore names the destination, and lists the migrations
  that have run.
 - It runs `VACUUM`, so that no deleted row is left in the file.
 - It compresses the copy with zstd and encrypts it with age.
 - It builds the manifest from the copy.
 - It uploads `metadata/<remote-key>/db.zst.age`, and then
  `metadata/<remote-key>/manifest.json.zst`.
- `NewApp(AppOptions)` → Creates fx.App with all modules
+The manifest is uploaded last. A directory under `metadata/` with no manifest
- `RunApp(ctx, app)` → Starts app, handles graceful shutdown
+therefore marks a backup that did not finish.
- `RunWithApp(ctx, opts)` → Convenience wrapper
+
 ## Restore
-### `internal/vaultik`
+`Vaultik.Restore` does these steps:
-Main orchestrator containing all dependencies and command implementations.
+- `restoreIdentities` parses the age private key. A missing or malformed key
  stops the restore before anything is downloaded.
 - `downloadSnapshotDB` downloads and decrypts the snapshot's database into a
  temporary directory.
 - `getFilesToRestore` selects the files, from the paths given.
 - `restoreAllFiles` downloads blobs and writes the files.
 - With `--verify`, `verifyRestoredFiles` reads each restored regular file and
  checks its chunk hashes.
-Key methods:
+Given a snapshot ID, the restore looks it up in the local index. When it is not
 there, the restore checks the marker file and lists the snapshot's directory.
 Given a remote key, it reads the local index only to check the marker file. It
 does not read the manifest. It finds each file's chunks and blobs through the
 downloaded database.
- `New(VaultikParams)` → Constructor (fx DI)
+### Checks on the downloaded database
 - `CreateSnapshot(opts)` → Main backup operation
 - `ListSnapshots(jsonOutput)` → List available snapshots
 - `VerifySnapshot(id, deep)` → Verify snapshot integrity
 - `PurgeSnapshots(...)` → Remove old snapshots
-### `internal/chunker`
+Decryption shows that a database is readable. It does not show that the database
 is honest. The restore therefore checks it:
-Content-defined chunking using FastCDC algorithm.
+- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
  trigger, a view or a virtual table, or that lacks an expected table.
 - `verifySnapshotDBIdentity` checks that the snapshot in the database hashes to
  the remote key that was asked for.
 - `containedRestorePath` refuses a stored path that would leave the target
  directory. It also refuses to write through a symlink below the target.
 - A blob is not decompressed beyond the size that the database gives for it.
 - Each downloaded blob is hashed. The hash must match the blob's name.
 ### Order of work
 `restorePlan` orders the files so that few blobs are on disk at once:
 - Downloaded blobs are kept decrypted in a cache directory.
 - A file is restored once every blob it needs is in the cache.
 - When no file is ready, the restore downloads the blobs for the file that needs
  the fewest.
 - `restoreSweeper` deletes a cached blob once no remaining file needs it.
 A regular file is created with mode `0600` and gets its stored mode after its
 content is written. A directory gets its stored mode and modification time after
 the whole restore, and its owner when the command runs as root.
 ## Verify
 Without `--deep`, `Vaultik.VerifySnapshotWithOptions` checks a snapshot without
 the private key:
-Key types:
+- It downloads the manifest.
 - It checks that the snapshot's encrypted database is present.
 - It checks that each listed blob is present with the listed size.
- `Chunk` → Hash, Data, Offset, Size
+`Vaultik.RunDeepVerify` needs the private key. It does these steps:
- `Chunker` → avgChunkSize, minChunkSize, maxChunkSize
+
 - It downloads the manifest and the database. It checks the database as a
  restore does.
 - `verifyManifestAgainstDatabase` checks that both list the same blobs with the
  same sizes.
 - `verifyBlobExistenceFromDB` checks that every blob is present with the
  recorded size.
 - `verifyBlob` downloads each blob, checks the hash of each chunk in it, and
  checks the hash that names the blob.
 ## Prune
 `Vaultik.Prune` calls `checkStorageBinding` and then does three steps:
 - `CleanupLocalSnapshots` drops from the local index each snapshot whose
  manifest is gone from the destination.
 - `PruneDatabase` removes unfinished snapshots and unused records from the local
  index.
 - `PruneBlobs` deletes unused data at the destination.
 `PruneBlobs` reads every manifest at the destination. It then deletes two kinds
 of object, once they are older than `prune_grace_period`:
 - a blob that no manifest lists
 - a directory under `metadata/` that has no manifest, and whose snapshot is not
  in the local index
 It deletes nothing when any manifest cannot be read.
 [`REPOSTRUCTURE.md`](REPOSTRUCTURE.md#pruning) explains why this is safe, and
 where it is not.
 ## The destination
 `storage.Storer` is the only way the code reaches the destination. It has these
 methods:
-Key methods:
+- `Put` and `PutWithProgress` store an object.
 - `Get` reads an object.
 - `Stat` reports an object's size.
 - `Delete` removes an object.
 - `List` and `ListStream` list the objects under a prefix.
 - `DeletePartialUploads` removes what an interrupted upload left behind.
 - `Info` describes the destination.
- `NewChunker(avgChunkSize)` → Constructor
+`storage.NewStorer` picks the implementation from `storage_url`:
- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback
+
-  (preferred)
+- `s3://` gives the S3 implementation.
- `ChunkReader(reader)` → Return all chunks at once (memory-intensive)
+- `file://` gives the local directory implementation.
 - `rclone://` gives the rclone implementation.
 With no `storage_url`, it builds the S3 implementation from the `s3` settings.
-### `internal/blob`
+## The marker file and the destination binding
-Blob packing: accumulates chunks, compresses, encrypts, tracks metadata.
+These checks keep the local index and the destination in step.
-Key types:
+- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when
  the file is reported missing while the local index holds snapshots. It skips
  the check when the local index belongs to another destination.
 - `checkStorageBinding` fails when the local index is bound to a different
  destination from the configured one. It records nothing.
 - `EnsureStorageBinding` binds an unbound local index to the configured
  destination. Only `markDestination` calls it, after the marker file is
  written, so a backup that writes nothing leaves the local index unbound.
- `Packer` → Thread-safe blob accumulator
+Every command that takes the lock calls `checkStorageBinding` first.
- `ChunkRef` → Hash + Data for adding to packer
+
- `FinishedBlob` → Completed blob ready for upload
+## Concurrency
- `BlobWithReader` → FinishedBlob + io.Reader for streaming upload
+
-
+- Each open database uses one connection, so its queries run one at a time.
-Key methods:
+- The index runs in SQLite's write-ahead log (WAL) mode. A second process can
-
+  therefore read it while a backup writes to it.
- `NewPacker(PackerConfig)` → Constructor
+- `Packer` guards its state with a mutex.
- `AddChunk(ChunkRef)` → Add chunk to current blob
+- `Scanner` holds `packerMu` while a chunk is added to the packer and while a
- `FinalizeBlob()` → Compress, encrypt, hash current blob
+  blob is closed.
- `Flush()` → Finalize any in-progress blob
+- `Repositories.WithTx` runs a function in one transaction. It commits when the
- `SetBlobHandler(func)` → Set callback for upload
+  function succeeds and rolls back when it fails.
 ### `internal/snapshot`
 #### Scanner
 Orchestrates the backup process for a directory.
 Key methods:
 - `NewScanner(ScannerConfig)` → Constructor (creates Chunker + Packer)
 - `Scan(ctx, path, snapshotID)` → Main scan operation
 Scan phases:
 1. **Phase 1**: Walk directory, identify files needing processing
 2. **Phase 1b**: Count as deleted the files that the previous snapshot of the
   same name lists, the walk did not find, and are gone from disk; a file that
   is excluded, or inside an excluded directory, is not counted
 3. **Phase 2**: Process files (chunk → pack → upload)
 #### SnapshotManager
 Manages snapshot lifecycle and metadata export.
 Key methods:
 - `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record
 - `PopulateSnapshotBlobs(ctx, snapshotID)` → Record every blob the snapshot
  references
 - `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3
 - `MarkSnapshotComplete(ctx, snapshotID)` → Record completion, only after a
  successful export
 - `CompleteSnapshot(ctx, snapshotID)` → Convenience: populate blobs, then mark
  complete (no export between)
 ### `internal/database`
 SQLite database for local index. Single-writer mode for thread safety.
 Key types:
 - `DB` → Database connection wrapper
 - `Repositories` → Collection of all repository interfaces
 Repository interfaces:
 - `FilesRepository` → CRUD for File records
 - `ChunksRepository` → CRUD for Chunk records
 - `BlobsRepository` → CRUD for Blob records
 - `SnapshotsRepository` → CRUD for Snapshot records
 - Plus join table repositories (FileChunks, BlobChunks, etc.)
 ## Snapshot Creation Flow
 ```
 CreateSnapshot(opts)
    │
    ├─► markDestination()   // Check the marker (see Storage Layout in S3), then write it
    │
    ├─► PruneDatabase()   // Critical: avoid dedup errors
    │
    ├─► SnapshotManager.CreateSnapshot()   // Create DB record
    │
    ├─► For each source directory:
    │       │
    │       ├─► scanner.Scan(ctx, path, snapshotID)
    │       │       │
    │       │       ├─► Phase 1: scanPhase()
    │       │       │       Walk directory
    │       │       │       Check file metadata changes
    │       │       │       Build list of files to process
    │       │       │
    │       │       ├─► Phase 1b: detectDeletedFiles()
    │       │       │
    │       │       └─► Phase 2: processPhase()
    │       │               For each file:
    │       │                   chunker.ChunkReaderStreaming()
    │       │                   For each chunk:
    │       │                       packer.AddChunk()
    │       │                       If blob full → FinalizeBlob()
    │       │                           → handleBlobReady()
    │       │                           → s3Client.PutObjectWithProgress()
    │       │               packer.Flush()  // Final blob
    │       │
    │       └─► Accumulate statistics
    │
    ├─► SnapshotManager.PopulateSnapshotBlobs()   // record referenced blobs
    │
    ├─► SnapshotManager.UpdateSnapshotStatsExtended()
    │
    ├─► SnapshotManager.ExportSnapshotMetadata()
    │       │
    │       ├─► Copy database to temp file
    │       ├─► Clean to only current snapshot data (VACUUM)
    │       ├─► Compress binary SQLite with zstd
    │       ├─► Encrypt with age
    │       ├─► Upload db.zst.age to storage
    │       └─► Upload manifest.json.zst to storage
    │
    └─► SnapshotManager.MarkSnapshotComplete()   // only after the export succeeds
 ```
 ## Deduplication Strategy
 1. **File-level**: Files and symlinks unchanged since last backup are skipped
   (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target),
   unless the file lists a chunk that no uploaded blob holds; such a file is
   re-chunked
 2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash
   already exists in the database, the chunk data is not re-uploaded.
 3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a
   blob are skipped.
 ## Storage Layout in S3
 ```
 bucket/
 ├── vaultik-destination.txt          # Marker written by every backup
 │
 ├── blobs/
 │   └── {hash[0:2]}/
 │       └── {hash[2:4]}/
 │           └── {full-hash}          # Compressed+encrypted blob
 │
 └── metadata/
    └── {remote-key}/
        ├── db.zst.age               # Encrypted binary SQLite database
        └── manifest.json.zst        # Blob list (for pruning/verification)
 ```
 A destination that answers a look-up of `vaultik-destination.txt` with not found
 is treated as not present while the local index holds snapshots stored there
 (`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
 `snapshot create` refuses to write to it, `prune`, `snapshot purge` and
 `remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a
 destination that cannot be listed. This keeps those commands from taking an
 unplugged volume's empty mount point for a destination whose snapshots are all
 gone. When the look-up fails for any reason other than not found, for example
 when S3 refuses it to credentials that allow only PUT, the check is skipped, so
 a host with such credentials does not notice an emptied destination; see
 [docs/REPOSTRUCTURE.md](REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt).
 The `file://` backend never creates its destination directory, so a write to a
 missing one fails.
 The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
 snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
 written to the store as a directory name. See
 [docs/REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) for the
 derivation and a worked example.
 ## Thread Safety
 - `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
 - `Scanner`: Uses `packerMu` mutex to coordinate blob finalization.
 - `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread
  safety.
 - `Repositories.WithTx()`: Handles transaction lifecycle automatically.
@@ -1,389 +1,450 @@
-# Vaultik Data Model
+# vaultik data model
-## Overview
+`vaultik` keeps a SQLite database on the backed-up machine, called the local
 index. This document describes its tables, and how a backup, a restore and
 `prune` use them.
-Vaultik uses a local SQLite database to track file metadata, chunk mappings, and
+The local index has two jobs:
 blob associations during the backup process. This database serves as an index
 for incremental backups and enables efficient deduplication.
-**Important Notes:**
+- It records which files, chunks and blobs are already stored. The next backup
  then does not store them again.
 - It is the source of each snapshot's own database. A backup uploads a trimmed,
  encrypted copy of the index with every snapshot.
- **Version Compatibility**: In rare cases, you may need to use the same version
+A restore reads no file, chunk or blob rows from the local index. It reads the
-  of Vaultik to restore a backup as was used to create it. This ensures
+copy that was uploaded with the snapshot.
  compatibility with the metadata format stored in S3.
-## Schema Migrations
+## Terms
-Vaultik supports schema migrations. They are the numbered SQL files in
+- **Snapshot**: one backup of the paths listed under one name in the config.
-`internal/database/schema/`, embedded in the binary: `000.sql` creates the
+- **Snapshot ID**: the name of a snapshot, in the form
-`schema_migrations` table, which records each migration that has run, and
+  `<hostname>_<snapshot-name>_<timestamp>`.
-`001.sql` creates the application tables. `database.New` opens a database and
+- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
-applies, in order, every migration that database has not yet recorded.
+  content.
 - **Blob**: one object at the destination. It holds many chunks, compressed and
  then encrypted.
 - **Destination**: the place the backups are stored.
 - **Marker file**: `vaultik-destination.txt` at the destination. See
  [The marker file](REPOSTRUCTURE.md#the-marker-file).
 - **Manifest**: the unencrypted list of the blobs one snapshot uses.
 - **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
  at the destination.
-**Before 1.0** no migrations are added, because nothing is installed anywhere
+## Where the index is
 yet. A schema change edits `001.sql` (and the code that uses the affected
 tables) directly. A local database created before the change has already
 recorded `001.sql` as run, so it keeps the old schema and can become unusable;
 `vaultik database delete` followed by a full backup rebuilds it.
-**After 1.0** each schema change is a new numbered file, so an existing local
+The index is the file `index.sqlite` in the platform data directory. By default
-database is migrated the first time an updated vaultik opens it. A file that has
+that is `~/.local/share/vaultik/` on Linux and
-shipped in a release is never edited.
+`~/Library/Application Support/vaultik/` on macOS. The `index_path` setting and
 the `VAULTIK_INDEX_PATH` environment variable change the path. So does
 `XDG_DATA_HOME`.
-## Database Tables
+The file is readable only by its owner, because it lists every backed-up path.
 `vaultik` sets that mode each time it opens the index.
-### 1. `files`
+## Schema migrations
-Stores metadata about files in the filesystem being backed up.
+`vaultik` supports schema migrations. They are the numbered SQL files in
 `internal/database/schema/`, which are built into the binary.
-**Columns:**
+- `000.sql` creates the `schema_migrations` table. That table records each
  migration that has run.
 - `001.sql` creates every other table.
- `id` (TEXT PRIMARY KEY) - UUID for the file record
+`database.New` opens the index. It then applies, in order, every migration that
- `path` (TEXT NOT NULL UNIQUE) - Absolute file path
+the index has not recorded.
 - `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix
  epoch
 - `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to
  999999999
 - `size` (INTEGER NOT NULL) - File size in bytes
 - `mode` (INTEGER NOT NULL) - Unix file permissions and type
 - `uid` (INTEGER NOT NULL) - User ID of file owner
 - `gid` (INTEGER NOT NULL) - Group ID of file owner
 - `link_target` (TEXT) - Symlink target path (NULL for regular files)
-**Indexes:**
+Before version 1.0, no migration is added, because nothing is installed anywhere
 yet. A schema change edits `001.sql` directly, with the code that uses the
 changed tables.
- `idx_files_path` on `path` for efficient lookups
+This has one consequence. An index created before such a change has already
 recorded `001.sql` as run, so it keeps the old schema and can become unusable.
 To rebuild it, run `vaultik database delete` and then a full backup.
-**Purpose:** Tracks file metadata to detect changes between backup runs. Used
+After version 1.0, each schema change is a new numbered file. An existing index
-for incremental backup decisions. The UUID primary key provides stable
+is then migrated the first time an updated `vaultik` opens it. A file that was
-references that don't change if files are moved.
+part of a release is never edited.
-### 2. `chunks`
+Migrations apply to the local index only. A snapshot's database at the
 destination is read as it was written. Each snapshot records the `vaultik`
 version that wrote it.
-Stores information about content-defined chunks created from files.
+## Tables
-**Columns:**
+Times are whole seconds since the Unix epoch, in UTC. The two exceptions are
 noted at their columns.
- `chunk_hash` (TEXT PRIMARY KEY) - SHA256 hash of chunk content
+### `files`
 - `size` (INTEGER NOT NULL) - Chunk size in bytes
-**Purpose:** Enables deduplication by tracking unique chunks across all files.
+One row for each backed-up path: a regular file, a symlink or a directory.
-### 3. `file_chunks`
+- `id`: a UUID. Primary key.
 - `path`: the absolute path. Unique.
 - `source_path`: the configured path under which the scan found this entry, made
  absolute with symlinks resolved. A restore does not read it.
 - `mtime`: the modification time.
 - `mtime_nsec`: the nanoseconds within that second.
 - `size`: the size in bytes.
 - `mode`: the permission bits and the file type.
 - `uid`: the owner's user ID.
 - `gid`: the owner's group ID.
 - `link_target`: the target of a symlink. It is empty for other entries.
-Maps files to their constituent chunks in order.
+`idx_files_path` indexes `path`.
-**Columns:**
+There is one row for each path, and it is updated when the file changes. The row
 is shared by every snapshot in the local index that lists the path. An older
 snapshot's own view of the file is in that snapshot's database at the
 destination.
- `file_id` (TEXT) - File ID (FK to files.id)
+### `chunks`
 - `idx` (INTEGER) - Chunk index within file (0-based)
 - `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
 - PRIMARY KEY (`file_id`, `idx`)
-**Purpose:** Allows reconstruction of files from chunks during restore.
+One row for each distinct chunk.
-### 4. `chunk_files`
+- `chunk_hash`: the SHA-256 hash of the chunk's content, in hex. Primary key.
 - `size`: the size in bytes.
-Reverse mapping showing which files contain each chunk.
+### `file_chunks`
-**Columns:**
+The chunks that make up each file, in order.
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
+- `file_id`: the file. References `files.id`.
- `file_id` (TEXT) - File ID (FK to files.id)
+- `idx`: the position of the chunk within the file, counted from 0.
- `file_offset` (INTEGER) - Byte offset of chunk within file
+- `chunk_hash`: the chunk. References `chunks.chunk_hash`.
 - `length` (INTEGER) - Length of chunk in bytes
 - PRIMARY KEY (`chunk_hash`, `file_id`)
-**Purpose:** Supports efficient queries for chunk usage and deduplication
+The primary key is (`file_id`, `idx`). `idx_file_chunks_chunk_hash` indexes
-statistics.
+`chunk_hash`.
-### 5. `blobs`
+### `chunk_files`
-Stores information about packed, compressed, and encrypted blob files.
+The reverse of `file_chunks`: the files that contain each chunk.
-**Columns:**
+- `chunk_hash`: the chunk. References `chunks.chunk_hash`.
 - `file_id`: the file. References `files.id`.
 - `file_offset`: the byte offset of the chunk within the file.
 - `length`: the length of the chunk in bytes.
- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts
+The primary key is (`chunk_hash`, `file_id`). `idx_chunk_files_file_id` indexes
- `blob_hash` (TEXT UNIQUE) - `hex(SHA256(SHA256(uncompressed blob contents)))`,
+`file_id`.
  computed before compression and encryption (NULL until finalized); see
  [REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs-directory-blobs)
 - `created_ts` (INTEGER NOT NULL) - Creation timestamp
 - `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
 - `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before
  compression
 - `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and
  encryption. Once uploaded, the size of the object in storage: when the blob
  was already stored, under another compression level or other recipients, this
  can differ from the size of the local copy
 - `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded)
-**Purpose:** Tracks blob lifecycle from creation through upload. The UUID
+### `blobs`
 primary key allows immediate association of chunks with blobs.
-### 6. `blob_chunks`
+One row for each blob.
-Maps chunks to the blobs that contain them.
+- `id`: a UUID, given when packing of the blob starts. Primary key.
 - `blob_hash`: the name of the blob at the destination. Unique. Until the blob
  is closed, it holds a placeholder that starts with `temp-placeholder-`.
 - `created_ts`: when packing started.
 - `finished_ts`: when the blob was closed. It is `NULL` before that.
 - `uncompressed_size`: the total size of the blob's chunks.
 - `compressed_size`: the size after compression and encryption.
 - `uploaded_ts`: when the upload finished. It is `NULL` before that.
-**Columns:**
+`blob_hash` is `hex(SHA256(SHA256(uncompressed contents)))`. See
 [`REPOSTRUCTURE.md`](REPOSTRUCTURE.md#blobs).
- `blob_id` (TEXT) - Blob ID (FK to blobs.id)
+Once a blob is uploaded, `compressed_size` is the size of the object at the
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
+destination. A blob that was already there can have a different size from the
- `offset` (INTEGER) - Byte offset of chunk within blob (before compression)
+local copy. That happens when it was written with another compression level or
- `length` (INTEGER) - Length of chunk in bytes
+other recipients.
 - PRIMARY KEY (`blob_id`, `chunk_hash`)
-**Purpose:** Enables chunk retrieval from blobs during restore operations.
+### `blob_chunks`
-### 7. `snapshots`
+The chunks that each blob holds.
-Tracks backup snapshots.
+- `blob_id`: the blob. References `blobs.id`.
 - `chunk_hash`: the chunk. References `chunks.chunk_hash`.
 - `offset`: the byte offset of the chunk within the uncompressed blob.
 - `length`: the length of the chunk in bytes.
-**Columns:**
+The primary key is (`blob_id`, `chunk_hash`). `idx_blob_chunks_chunk_hash`
 indexes `chunk_hash`.
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g.
+A restore finds the blob for each chunk through this table.
  `server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the
  snapshot name, and an RFC 3339 UTC timestamp)
 - `hostname` (TEXT) - Hostname where backup was created
 - `vaultik_version` (TEXT) - Version of Vaultik used
 - `vaultik_git_revision` (TEXT) - Git revision of Vaultik used
 - `started_at` (INTEGER) - Start timestamp
 - `completed_at` (INTEGER) - Completion timestamp (NULL if in progress)
 - `file_count` (INTEGER) - Number of files in snapshot
 - `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not
  stored before
 - `new_chunk_size` (INTEGER) - Total size of those chunks before compression;
  `snapshot list` shows it as NEW CHUNK SIZE
 - `blob_count` (INTEGER) - Number of blobs this snapshot created
 - `total_size` (INTEGER) - Total size of all files
 - `uncompressed_chunk_size` (INTEGER) - Total size before compression of the
  distinct chunks the snapshot's files reference, recorded when its backup ran;
  `snapshot list` shows it as UNCOMPRESSED SIZE
 - `blob_size` (INTEGER) - Total compressed size of all referenced blobs
 - `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced
  blobs
 - `compression_ratio` (REAL) - Compression ratio achieved
 - `compression_level` (INTEGER) - Compression level used for this snapshot
 - `upload_bytes` (INTEGER) - Total bytes uploaded during this snapshot
 - `upload_duration_ms` (INTEGER) - Total milliseconds spent uploading to S3
-**Purpose:** Provides snapshot metadata and statistics including version
+### `snapshots`
 tracking for compatibility.
-### 8. `snapshot_files`
+One row for each snapshot.
-Maps snapshots to the files they contain.
+- `id`: the snapshot ID. Primary key.
 - `hostname`: the `hostname` setting of the config. When that setting is empty,
  it is the machine's hostname.
 - `vaultik_version`: the `vaultik` version that made it.
 - `vaultik_git_revision`: the git commit of that version.
 - `started_at`: when the backup started.
 - `completed_at`: when the backup finished. It is `NULL` until then.
 - `file_count`: the number of files and symlinks in the snapshot. Directories
  are not counted.
 - `chunk_count`: the number of chunks this snapshot stored that were not stored
  before.
 - `new_chunk_size`: the total size of those chunks before compression.
 - `blob_count`: the number of blobs this snapshot created.
 - `total_size`: the total size of all files in the snapshot.
 - `uncompressed_chunk_size`: the total size, before compression, of the distinct
  chunks the snapshot's files use.
 - `blob_size`: the total compressed size of the blobs the snapshot uses.
 - `blob_uncompressed_size`: the total uncompressed size of those blobs.
 - `compression_ratio`: the compression ratio of those blobs.
 - `compression_level`: the compression level the backup used.
 - `upload_bytes`: the bytes uploaded during the backup.
 - `upload_duration_ms`: the time spent uploading, in milliseconds.
-**Columns:**
+The snapshot ID has the form `<hostname>_<snapshot-name>_<timestamp>`. An
 example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
+- the hostname, up to its first `.`
- `file_id` (TEXT) - File ID (FK to files.id)
+- the snapshot name
- PRIMARY KEY (`snapshot_id`, `file_id`)
+- the start time, in UTC
-**Purpose:** Records which files are included in each snapshot.
+The `snapshot list` table shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE`
 and `new_chunk_size` as `NEW CHUNK SIZE`.
-### 9. `snapshot_blobs`
+A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
 shows it only when its manifest is at the destination, and then as remote only.
 The next backup deletes it, and so does `prune`.
-Maps snapshots to the blobs they reference.
+### `snapshot_files`
-**Columns:**
+The files in each snapshot.
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
+- `snapshot_id`: the snapshot. References `snapshots.id`.
- `blob_id` (TEXT) - Blob ID (FK to blobs.id)
+- `file_id`: the file. References `files.id`.
 - `blob_hash` (TEXT) - Denormalized blob hash for manifest generation
 - PRIMARY KEY (`snapshot_id`, `blob_id`)
-**Purpose:** Tracks blob dependencies for snapshots and enables manifest
+The primary key is (`snapshot_id`, `file_id`). `idx_snapshot_files_file_id`
-generation.
+indexes `file_id`.
-### 10. `snapshot_mount_points`
+### `snapshot_blobs`
-The directories at or below a configured path, the configured path itself
+The blobs that each snapshot uses.
 included, that a snapshot's backup found an entry in and, once it had listed
 them, found on a different filesystem from their parent directory (a different
 device number), as a mount point is. An automount point whose volume cannot be
 mounted is not recorded, because the backup finds no entry in it.
 **Columns:**
 - `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
 - `path` (TEXT) - Absolute path of the directory
 - PRIMARY KEY (`snapshot_id`, `path`)
-**Purpose:** Lets `snapshot create --prune` skip the prune when a backup does
+- `snapshot_id`: the snapshot. References `snapshots.id`.
-not find a mount point that an earlier snapshot of the same name found, as when
+- `blob_id`: the blob. References `blobs.id`.
-the volume is not mounted.
+- `blob_hash`: a copy of the blob's hash.
-### 11. `uploads`
+The primary key is (`snapshot_id`, `blob_id`). `idx_snapshot_blobs_blob_id`
 indexes `blob_id`.
-Tracks blob upload metrics.
+The manifest is built from this table.
-**Columns:**
+### `snapshot_mount_points`
- `blob_hash` (TEXT PRIMARY KEY) - Hash of uploaded blob
+The mount points that a snapshot's backup found.
 - `snapshot_id` (TEXT NOT NULL) - The snapshot that triggered this upload (FK to
  snapshots.id)
 - `uploaded_at` (INTEGER) - Upload timestamp
 - `size` (INTEGER) - Size of uploaded blob
 - `duration_ms` (INTEGER) - Upload duration in milliseconds
-**Purpose:** Performance monitoring and tracking which blobs were newly created
+- `snapshot_id`: the snapshot. References `snapshots.id`.
-(uploaded) during each snapshot.
+- `path`: the absolute path of the directory.
-## Data Flow and Operations
+The primary key is (`snapshot_id`, `path`).
-### 1. Backup Process
+A directory is recorded here when all of these hold:
-1. **File Scanning**
+- It is at or below a configured path.
-    - `INSERT OR REPLACE INTO files` - Update file metadata
+- The backup found an entry in it that is not excluded.
-    - `SELECT * FROM files WHERE path = ?` - Check if file has changed
+- It is on a different filesystem from its parent directory.
    - `INSERT INTO snapshot_files` - Add file to current snapshot
    - `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a
      different filesystem from its parent that the walk found an entry in
 2. **Chunking** (for changed files)
    - `INSERT OR IGNORE INTO chunks` - Store new chunks
    - `INSERT INTO file_chunks` - Map chunks to file
    - `INSERT INTO chunk_files` - Create reverse mapping
 3. **Blob Packing**
    - `INSERT INTO blobs` - Create blob record with UUID (blob_hash NULL)
    - `INSERT INTO blob_chunks` - Associate chunks with blob immediately
    - `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after
      packing
-4. **Upload**
+The device number is read after the directory has been listed. An automount
-    - `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as
+point is a directory whose volume is mounted when it is first listed, so the
-      uploaded, with the size of the object in storage (the upload is skipped
+number is that of the volume. An automount point whose volume cannot be mounted
-      when the blob is already stored)
+is not recorded, because the backup finds no entry in it.
    - `INSERT INTO uploads` - Record upload metrics with snapshot_id
    - `INSERT INTO snapshot_blobs` - Associate blob with snapshot
-5. **Snapshot Completion**
+`snapshot create --prune` reads this table. It skips the prune when an earlier
-    - `UPDATE snapshots SET completed_at = ?, stats...` - Finalize snapshot
+snapshot of the same name recorded a mount point that the new backup did not
-    - Generate and upload blob manifest from `snapshot_blobs`
+find. That happens when the volume is not mounted.
-### 2. Incremental Backup
+### `uploads`
-1. **Change Detection**
+One row for each blob that a backup uploaded.
    - `SELECT * FROM files WHERE path = ?` - Get previous file metadata
    - Compare mtime, size, mode to detect changes
    - Re-chunk a file that lists a chunk no uploaded blob holds, even when its
      metadata is unchanged
    - Skip unchanged files but still add to `snapshot_files`
-2. **Chunk Reuse**
+- `blob_hash`: the blob. References `blobs.blob_hash`. Primary key.
-    - `SELECT * FROM blob_chunks WHERE chunk_hash = ?` - Find existing chunks
+- `snapshot_id`: the snapshot whose backup uploaded it. References
-    - `INSERT INTO snapshot_blobs` - Reference existing blobs for unchanged
+  `snapshots.id`.
-      files
+- `uploaded_at`: when the upload started. It is written as a date and time, not
  as seconds.
 - `size`: the size of the uploaded blob.
 - `duration_ms`: the duration of the upload, in milliseconds.
-### 3. Snapshot Metadata Export
+`idx_uploads_snapshot_id` indexes `snapshot_id`.
-After a snapshot is completed:
+A blob that was already at the destination gets no row here.
 ### `local_meta`
 Settings that belong to this copy of the index.
-1. Copy database to temporary file
+- `key`: the name of the setting. Primary key.
-2. Clean temporary database to contain only current snapshot data
+- `value`: its value.
 3. VACUUM the trimmed database so deleted rows leave no pages behind
 4. Compress with zstd and encrypt with age
 5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
 6. Generate blob manifest and upload as
   `metadata/{remote-key}/manifest.json.zst`
-The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so
+The only key is `storage_url`. A backup sets it once it has written the marker
-the ID is never written to the store in plaintext. The hash uses no secret, so a
+file at the destination. No other command sets it. These commands refuse to run
-guessed hostname and snapshot name can still be confirmed against a listing; see
+when the config names a different destination:
 [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation) and its
 [Accepted Risks](REPOSTRUCTURE.md#accepted-risks).
-### 4. Restore Process
+- `snapshot create`
 - `snapshot purge`
 - `snapshot remove`
 - `prune`
 - `remote nuke`
-The restore process doesn't use the local database. Instead:
+### `schema_migrations`
 1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from
   S3
 2. Downloads the blobs holding the chunks of the files being restored, found
   through that database's `blob_chunks` table; the manifest is not read
 3. Reconstructs files from decrypted and decompressed chunks
-### 5. Pruning
+The migrations that have run.
 1. **Identify Unreferenced Blobs**
    - Query blobs not referenced by any remaining snapshot
    - Delete from S3, once older than `prune_grace_period`, and from the local
      database
 2. **Delete Metadata Directories With No Manifest**
    - Delete each directory under `metadata/` that has no `manifest.json.zst`
      and no local snapshot record, once every object in it is older than
      `prune_grace_period`
-### 6. Incomplete Snapshot Cleanup
+- `version`: the number of the migration. Primary key.
-
+- `applied_at`: when it ran. It is written as a date and time, not as seconds.
-Before each backup:
+
-
+## How commands use the tables
-1. Query incomplete snapshots (where `completed_at IS NULL`)
+
-2. Delete each one and all its associations, without checking S3 for its
+### Backup
-   metadata
+
-3. Clean up orphaned files, chunks, and blobs
+A backup works on the tables in this order:
-
+
-## Repository Pattern
+- It saves the destination in `local_meta` when the index has none yet. It does
-
+  so once it has written the marker file at the destination.
-Vaultik uses a repository pattern for database access:
+- It deletes unfinished snapshots, and the rows that no snapshot uses.
-
+- It creates a row in `snapshots`, with `completed_at` set to `NULL`.
- `FileRepository` - CRUD operations for files and file metadata
+- It deletes the `blobs` rows that were never uploaded.
- `ChunkRepository` - CRUD operations for content chunks
+- It walks each path. It compares each regular file and symlink with its row in
- `FileChunkRepository` - Manage file-to-chunk mappings
+  `files`.
- `ChunkFileRepository` - Manage chunk-to-file reverse mappings
+- It links each unchanged file to the snapshot in `snapshot_files`.
- `BlobRepository` - Manage blob lifecycle (creation, finalization, upload)
+- It writes a `files` row and a `snapshot_files` row for each directory.
- `BlobChunkRepository` - Manage blob-to-chunk associations
+- It records each mount point in `snapshot_mount_points`.
- `SnapshotRepository` - Manage snapshots and their relationships
+- It reads each changed regular file and stores it. See below. A changed symlink
- `UploadRepository` - Track blob upload metrics
+  is recorded without being read.
-
+- It fills `snapshot_blobs` with every blob the snapshot uses.
-Each repository provides methods like:
+- It stores the snapshot's totals in `snapshots`.
-
+- It uploads the snapshot's metadata.
- `Create()` - Insert new record
+- It sets `completed_at`.
- `GetByID()` / `GetByPath()` / `GetByHash()` - Retrieve records
+
- `Update()` - Update existing records
+A file counts as changed when its row in `files` differs from the file on disk
- `Delete()` - Remove records
+in any of these:
- Specialized queries for each entity type (e.g., `DeleteOrphaned()`,
+
-  `GetIncompleteByHostname()`)
+- size
-
+- modification time
-## Transaction Management
+- mode
-
+- owner or group
-All database operations that modify multiple tables are wrapped in transactions:
+- link target
 A file is also read again when it lists a chunk that no uploaded blob holds. A
 regular file with no row in `files` is read too.
 Storing a changed file writes these rows:
 - A row in `blobs`, when packing of a blob starts.
 - When the blob is closed, in one transaction:
    - a row in `chunks` for each chunk queued since the last blob was closed.
      When a blob is full, that includes the first chunk of the next blob.
    - the blob's rows in `blob_chunks`
    - the blob's `blob_hash`, sizes and `finished_ts`
 - When the blob is stored at the destination, in one transaction:
    - `uploaded_ts` and `compressed_size` in `blobs`
    - a row in `snapshot_blobs`
    - a row in `uploads`, unless the blob was already at the destination
 - The row in `files`, and its rows in these tables:
    - `file_chunks`
    - `chunk_files`
    - `snapshot_files`
 The row in `files` is written last. It is written only after every blob that
 holds the file's new chunks is stored.
 ### Snapshot metadata
 The metadata of a snapshot is made from a copy of the index:
 - The index is copied into a temporary directory with `VACUUM INTO`.
 - The rows of every other snapshot are deleted from the copy. So are the files
  and blobs that this snapshot does not use. A chunk is then deleted when no
  remaining file and no remaining blob uses it.
 - The `local_meta` and `schema_migrations` tables are kept as they are. The
  uploaded database therefore names the destination, and lists the migrations
  that have run.
 - `VACUUM` rewrites the copy, so that no deleted row is left in the file.
 - The copy is compressed with [zstd](https://github.com/facebook/zstd) and
  encrypted with [age](https://age-encryption.org/).
 - The manifest is built from the copy's `snapshot_blobs` table.
 - The copy is uploaded as `metadata/<remote-key>/db.zst.age`.
 - The manifest is uploaded as `metadata/<remote-key>/manifest.json.zst`.
 The copy is made through SQLite, not by copying the file. The index runs in
 SQLite's write-ahead log (WAL) mode, so rows that are already committed can
 still be in the `-wal` file beside it.
 ### Restore
 A restore reads no file, chunk or blob rows from the local index. It reads
 `snapshots` and `local_meta` there, to look up a snapshot ID and to check the
 marker file.
 - It downloads and decrypts `db.zst.age` into a temporary directory.
 - It opens that file read-only. It applies no migration to it.
 - It reads the snapshot's row in `snapshots`, to check that this is the right
  database.
 - It reads the files to restore from `files`.
 - It finds each file's chunks in `file_chunks`.
 - It finds the blob and offset of each chunk in `blob_chunks`.
 - It finds the name of each blob in `blobs`.
 - With `--verify`, it reads each chunk's size from `chunks`.
 It does not read the manifest.
 ### Prune
 `prune` saves nothing in `local_meta`. It refuses to run when `local_meta` holds
 a different destination from the configured one. It changes the local index in
 two ways:
 - It deletes each snapshot whose manifest is gone from the destination.
 - It deletes unfinished snapshots, and the rows that no snapshot uses.
 The second step removes unused rows in this order:
 - files that are in no snapshot
 - blobs that are in no snapshot
 - `blob_chunks` rows whose chunk is gone
 - chunks that no file and no blob uses
 `prune` then deletes unused data at the destination. That step reads the
 manifests, not the local index. It reads the local index only to keep a
 directory under `metadata/` whose snapshot is still recorded there.
 ## Code
 Each repository type in `internal/database` serves one or more tables:
 - `FileRepository`: `files`
 - `ChunkRepository`: `chunks`
 - `FileChunkRepository`: `file_chunks`
 - `ChunkFileRepository`: `chunk_files`
 - `BlobRepository`: `blobs`
 - `BlobChunkRepository`: `blob_chunks`
 - `SnapshotRepository`:
    - `snapshots`
    - `snapshot_files`
    - `snapshot_blobs`
    - `snapshot_mount_points`
    - `uploads`, where it deletes a snapshot's rows
 - `UploadRepository`: `uploads`
 - `LocalMetaRepository`: `local_meta`
 No repository type serves `schema_migrations`. The code that applies the
 migrations reads and writes it.
 `Repositories` holds one of each type.
 `Repositories.WithTx` runs a function in one transaction:
 ```go
 err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
-    // Multiple repository operations using tx
+    // several repository calls that use tx
 })
 ```
-This ensures consistency, especially important for operations like:
+The transaction commits when the function returns no error. Otherwise it is
 rolled back.
- Creating file-chunk mappings
+## SQLite settings
 - Associating chunks with blobs
 - Updating snapshot statistics
-## Performance Considerations
+- The index runs in WAL mode. A command that only reads, such as
-
+  `snapshot list`, can read the index while a backup writes to it.
-1. **Indexes**:
+- A connection waits up to 10 seconds for a lock before it fails.
-    - Primary keys are automatically indexed
+- Foreign keys are enforced.
-    - `idx_files_path` on `files(path)` for efficient file lookups
+- Each open database uses one connection, so its queries run one at a time.
-
+- Queries pass their values as parameters.
 2. **Prepared Statements**: All queries use prepared statements for performance
   and security
 3. **Batch Operations**: Where possible, operations are batched within
   transactions
 4. **Write-Ahead Logging**: The local index runs in SQLite WAL mode with a
   10-second busy timeout, so a read-only command such as `snapshot list` can
   read it while a backup writes to it. Committed rows can sit in the `-wal`
   file beside the index until a checkpoint, so the metadata export copies the
   index through SQLite (`VACUUM INTO`), not as a file
 ## Data Integrity
 1. **Foreign Keys**: Enforced through CASCADE DELETE and application-level
   repository methods
 2. **Unique Constraints**: Chunk hashes, file paths, and blob hashes are unique
 3. **Null Handling**: Nullable fields clearly indicate in-progress operations
 4. **Timestamp Tracking**: All major operations record timestamps for auditing
@@ -1,158 +1,181 @@
-# Vaultik S3 Repository Structure
+# vaultik destination layout
-This document describes the structure and organization of data stored in the S3
+This document describes what `vaultik` stores at the destination. It covers the
-bucket by Vaultik.
+layout, what the layout protects, and the risks it accepts.
-## Overview
+The destination is the place the backups are stored. It is an S3 bucket and
 prefix, a local directory or an [rclone](https://rclone.org/) remote. A remote
 is a storage service set up in rclone. The layout is the same on all three.
-Vaultik stores all backup data in an S3-compatible object store. The repository
+## Terms
 consists of two main components, plus a marker file:
-1. **Blobs** - The actual backup data (content-addressed, encrypted)
+- **Snapshot**: one backup of the paths listed under one name in the config.
-2. **Metadata** - Snapshot information and manifests (partially encrypted)
+- **Snapshot ID**: the name of a snapshot, in the form
  `<hostname>_<snapshot-name>_<timestamp>`.
 - **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
  under `metadata/`. See [Remote keys](#remote-keys).
 - **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
  content.
 - **Blob**: one object under `blobs/`. It holds many chunks.
 - **Local index**: the SQLite database on the backed-up machine.
 - **Recipient**: an [age](https://age-encryption.org/) public key that backups
  are encrypted to.
-## Directory Structure
+## Layout
 ```
-<bucket>/<prefix>/
+<destination>/
 ├── vaultik-destination.txt
 ├── blobs/
 │   └── <hash[0:2]>/
 │       └── <hash[2:4]>/
-│           └── <full-hash>
+│           └── <hash>
 └── metadata/
    └── <remote-key>/
        ├── db.zst.age
        └── manifest.json.zst
 ```
-The metadata subdirectory is named with the **remote key**, a one-way hash of
+The destination holds three things:
 the snapshot ID, not with the human-readable snapshot ID itself. See
 [Remote Key Derivation](#remote-key-derivation).
-## Marker File (`vaultik-destination.txt`)
+- `vaultik-destination.txt`, the marker file
 - `blobs/`, which holds the file contents
 - `metadata/`, which holds one directory for each snapshot
 ## The marker file
 Every backup writes `vaultik-destination.txt` before it uploads anything else.
 The file holds one fixed line of text. It names no host, path or time.
 The marker tells an emptied or missing destination apart from a new one. An
 unplugged volume leaves an empty directory at its mount point. Without the
 marker, that directory would look like a destination whose snapshots are all
 gone.
 The check applies while the local index holds snapshots and belongs to the
 configured destination. When the destination then answers that the marker does
 not exist, commands treat the destination as not present:
 - `snapshot create` refuses to write to it.
 - These commands fail:
    - `prune`
    - `snapshot purge`
    - `remote nuke`
    - `snapshot verify` and `snapshot restore`, when given a remote key, or a
      snapshot ID that is not in the local index
 - `snapshot list` shows the local index alone.
 - `snapshot remove` deletes nothing at the destination.
 The check is skipped when the look-up fails for any other reason. S3 refuses the
 look-up to credentials that can only write. A machine with such credentials
 therefore does not notice an emptied destination. Its next backup would not
 upload data that its local index says is stored. After a destination is emptied,
 run `vaultik database delete` on such a machine before it backs up again.
 Every backup writes this file at the root before it uploads anything. It holds
 one fixed line of plain text and names no host, path or time. While the local
 index holds snapshots stored there, a destination that answers a look-up of the
 marker with not found is treated as not present: `snapshot create` refuses to
 write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list`
 and `remote nuke` behave as for a destination that cannot be listed. An
 unplugged volume leaves an empty directory at its mount point, and without the
 marker that directory would read as a destination whose snapshots are all gone.
 `remote nuke` deletes the marker last.
-When the look-up of the marker fails for any reason other than not found, the
+## Blobs
 check is skipped and the command goes on. S3 refuses it on every run to
 credentials that allow only PUT, so a host with such credentials does not notice
 a destination emptied since its last backup, and its next backup would leave out
 the data its local index records as already stored. After the destination is
 emptied, run `vaultik database delete` on such a host before it backs up again.
-## Blobs Directory (`blobs/`)
+A blob holds the content of many chunks, one after another.
-### Structure
+- **Path**: `blobs/<first 2 characters>/<next 2 characters>/<full hash>`.
 - **Path format**: `blobs/<first-2-chars>/<next-2-chars>/<full-hash>`
 - **Example**:
-  `blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`
+  `blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`.
- **Sharding**: The two-level directory structure (using the first 4 characters
+- **Format**: the chunk data, compressed with
-  of the hash) prevents any single directory from containing too many objects
+  [zstd](https://github.com/facebook/zstd) and then encrypted with age.
 - **Recipients**: the public keys in the `age_recipients` setting when the blob
  was written.
-### Content
+The two directory levels keep any one directory from holding too many objects.
- **What it contains**: Packed collections of content-defined chunks from files
+### Blob names
 - **Format**: Zstandard compressed, then Age encrypted
 - **Encryption**: Always encrypted with Age using the configured recipients
 - **Naming**: Content-addressed. The blob's name is
  `hex(SHA256(SHA256(uncompressed blob contents)))` — the double SHA-256 of the
  concatenated chunk data, computed before compression and encryption, not a
  hash of the stored (compressed, encrypted) bytes. One consequence: only a
  holder of the age private key can check a stored blob's integrity, because
  matching a blob to its name means decrypting and decompressing it first —
  which is what `restore` and `verify --deep` do. Implemented in
  `internal/blobgen` (`DoubleSHA256`). This is the canonical description of blob
  naming; other documents and comments point here.
-### Why Encrypted
+A blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. The contents are
 the chunk data, joined in order, before compression and encryption. The name is
 not a hash of the stored bytes.
-Blobs contain the actual file data from backups and must be encrypted for
+This has three consequences:
 security. The content-addressing ensures deduplication while the encryption
 ensures privacy.
-## Metadata Directory (`metadata/`)
+- Checking a blob against its name needs the private key. The blob must be
  decrypted and decompressed first. `snapshot restore` and
  `snapshot verify --deep` do this.
 - Without the key, only a blob's presence and size can be checked.
  `snapshot verify` does this.
 - Someone who has the whole contents of a blob can compute its name, and so see
  whether it is stored. This works for a file they have that fills blobs on its
  own.
-Each snapshot has its own subdirectory. The directory is **not** named with the
+`blobgen.DoubleSHA256` in `internal/blobgen` computes the name.
 human-readable snapshot ID; it is named with the remote key — a one-way hash of
 that ID. The human ID is never written to the destination store as a directory
 name (see [Remote Key Derivation](#remote-key-derivation)).
-### Snapshot ID Format
+### Uploads
-The human-readable snapshot ID is used in CLI arguments, log lines, and the
+A backup looks a blob up by name before it uploads it. It does not upload a blob
-local database. It is not written to the destination store.
+that is already there.
- **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>`
+On S3, when the destination refuses the look-up, the backup uploads the blob. An
-  if no name was specified)
+object with that name is then replaced by one that holds the same chunk data.
 - **Example**: `laptop_home_2024-01-15T14:30:52Z`
 - **Components**:
    - Short hostname (everything before the first dot is stripped from the FQDN)
    - Snapshot name from the configured `snapshots:` map (optional)
    - RFC3339 UTC timestamp
-This ID reveals the hostname, the configured snapshot name, and the backup time,
+## Metadata
 so it is never used as the on-disk directory name — the remote key is used
 instead.
-### Remote Key Derivation
+Each snapshot has one directory under `metadata/`. The directory is named with
 the snapshot's remote key, not with its snapshot ID.
-The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double
+### Snapshot IDs
 SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The
 result is a 64-character hex string. The hash is not reversible, but it uses no
 secret: an observer who guesses a candidate hostname and snapshot name can hash
 it the same way and confirm whether that snapshot is present. The remote key
 keeps names out of a plain listing; it does not hide them from a guess.
 Implemented in `internal/snapshot/remotekey.go`.
-Worked example:
+A snapshot ID has the form `<hostname>_<snapshot-name>_<timestamp>`. An example
 is `laptop_home_2024-01-15T14:30:52Z`. It has three parts:
 - the hostname, up to its first `.`
 - the snapshot name from the config
 - the time the backup started, in UTC
 The snapshot ID is used on the command line, in log records and in the local
 index. It reveals the hostname, the snapshot name and the backup time. It is
 therefore never written to the destination without encryption.
 ### Remote keys
 The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot ID)))`. It is 64 hex
 characters long. `internal/snapshot/remotekey.go` computes it.
 For example:
 - Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
 - Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
 - Directory:
  `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
-A plain listing of the destination store therefore shows only these hashes, not
+A listing of the destination therefore shows no hostname and no snapshot name.
 the hostname or snapshot name of any backup — but because the hash uses no
 secret, a guessed hostname and snapshot name can be hashed and confirmed against
 the listing. The same remote key is stored in the manifest's `snapshot_id`
 field.
-### Files in Each Snapshot Directory
+The hash cannot be reversed, but it uses no secret. Someone who guesses a
 snapshot ID can hash it the same way and see whether that snapshot exists. The
 remote key keeps names out of a listing. It does not hide them from a guess.
-#### `db.zst.age` - Encrypted Database
+### `db.zst.age`
- **What it contains**: Pruned binary SQLite database for this snapshot
+The snapshot's database. A restore needs it.
 - **Format**: Binary SQLite → Zstandard compressed → Age encrypted
 - **Encryption**: Encrypted with Age
 - **Purpose**: Contains full file metadata, chunk mappings, and all
  relationships
 - **Why encrypted**: Contains sensitive metadata like file paths, permissions,
  and ownership
-#### `manifest.json.zst` - Unencrypted Blob Manifest
+- **Contents**: a SQLite database. It is a copy of the local index without the
  rows of other snapshots. It lists every file in the snapshot with its
  metadata, its chunks, and the blob that holds each chunk.
 - **Format**: the database file, compressed with zstd and then encrypted with
  age.
- **What it contains**: JSON list of all blob hashes referenced by this snapshot
+It is encrypted because it holds file paths, permissions and owners.
- **Format**: JSON → Zstandard compressed (NOT encrypted)
+[`DATAMODEL.md`](DATAMODEL.md) describes its tables.
- **Encryption**: NOT encrypted
+
- **Purpose**: Enables pruning operations without requiring decryption keys
+### `manifest.json.zst`
- **Structure**:
+
 The list of the blobs the snapshot uses.
 - **Format**: JSON, compressed with zstd. It is not encrypted.
 ```json
 {
    "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
-    "timestamp": "2025-06-01T12:00:00Z",
+    "timestamp": "2025-06-01T12:04:31Z",
-    "blob_count": 42,
+    "blob_count": 2,
-    "total_compressed_size": 1048576,
+    "total_compressed_size": 57344,
    "blobs": [
        {
            "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
@@ -166,189 +189,259 @@ field.
 }
 ```
-`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is
+- `snapshot_id`: the remote key, not the snapshot ID.
-written in the clear.
+- `timestamp`: when the manifest was written, at the end of the backup.
 - `blob_count`: the number of blobs.
 - `total_compressed_size`: the total size of those blobs, in bytes.
 - `blobs`: the hash and the size of each blob.
-### Why Manifest is Unencrypted
+The manifest holds no file name and no path.
-The manifest must be readable without the private key to enable:
+The manifest is not encrypted so that these commands work without the private
 key:
-1. **Pruning operations** - Identifying unreferenced blobs for deletion
+- `prune`, which must know which blobs are still in use
-2. **Storage analysis** - Understanding space usage without decryption
+- `snapshot verify`, which checks that each blob is present
-3. **Verification** - Checking blob existence without decryption
+- `snapshot list` and `remote info`, which report what the destination holds
 4. **Cross-snapshot deduplication analysis** - Finding shared blobs between
   snapshots
-The manifest contains the remote key, the backup timestamp, the blob count and
+## What is protected
 total compressed size, and each blob's hash and compressed size. It contains no
 file names, paths, or other decrypted metadata.
-## Security Considerations
+These are encrypted:
-### What's Encrypted
+- file contents, in the blobs
 - in `db.zst.age`:
    - file names and paths
    - permissions and owners
    - modification times
    - which chunks make up which file
- **All file content** (in blobs)
+These are not encrypted:
 - **All file metadata** (paths, permissions, timestamps, ownership in
  db.zst.age)
 - **File-to-chunk mappings** (in db.zst.age)
-### What's Not Encrypted
+- the remote key of each snapshot
 - the time each backup finished, in the manifest
 - the hash and the compressed size of each blob, in the manifest
 - the number of blobs and their total size for each snapshot, in the manifest
- **The remote key** — directory names and the manifest `snapshot_id`, a one-way
+Someone who can read the destination therefore learns these:
  hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation))
 - **The backup timestamp** (in manifest.json.zst)
 - **Blob hashes and their compressed sizes** (in manifest.json.zst)
 - **Blob count and total compressed size per snapshot** (in manifest.json.zst)
-### Privacy Implications
+- when each backup finished
 - how many blobs each snapshot uses, and how large they are
 - which blobs two snapshots share
 - whether a blob holds data they already have in full
 - whether a snapshot ID they guess exists
-From the unencrypted data, an observer of the destination store can determine:
+They do not learn these:
- **When each backup was taken** — not from the directory name, which is a
+- any hostname or snapshot name, unless they guess it
-  one-way hash, but from the plaintext `timestamp` field in manifest.json.zst,
+- file names or paths
-  which is published in the clear
+- file contents
- How many blobs each snapshot references, and the total compressed size
+- permissions or owners
- The compressed size of each blob, and which blobs are shared between snapshots
+- the directory structure
-  (deduplication patterns)
+- which chunks belong to which file
 - **Whether a guessed hostname and snapshot name are present** — the remote key
  is an unkeyed hash, so an observer holding candidate names can hash each one
  and match it against the directory listing. The human ID is never published,
  so it cannot be read off directly, but it can be confirmed by guessing.
-Together these give an observer a timing-and-size profile of every snapshot.
+The times and sizes could not be hidden by encrypting the manifest. The
-This is an accepted, documented property of the format, not a defect: the
+destination itself shows when each object was written and how large it is.
 manifest is unencrypted so that pruning can run without the private key, and the
 timing channel could not be closed by encrypting it anyway — object creation
 times and per-object sizes stay visible at the storage layer on both `s3://` and
 `file://` destinations regardless.
-An observer cannot determine:
+## Accepted risks
- The hostname or snapshot name of any backup by reading it off the store — the
+These are known properties of the format. They are written down so that you can
-  directory name and the manifest `snapshot_id` are unkeyed hashes of the human
+weigh them.
  ID, so the text is never published (though a guessed name can be confirmed, as
  above)
 - File names or paths
 - File contents
 - File permissions or ownership
 - Directory structure
 - Which chunks belong to which files
-### Accepted Risks
+### No proof of who wrote the data
-These are known, deliberate properties of the format and the tooling, recorded
+A restore and `snapshot verify --deep` prove two things. The data decrypts with
-so an operator can weigh them rather than discover them.
+your private key, and it matches its hashes. They do not prove who wrote it.
-1. **No proof of authorship.** Restore and `verify --deep` prove that data
+The hashes use no secret. Someone who knows a recipient's public key and can
-   decrypts with the age private key and matches its unkeyed content hashes.
+replace objects at the destination can put a snapshot of their own there. The
-   They do not prove who wrote it: anyone who knows a recipient public key and
+public key is not stored at the destination, but a backed-up machine that is
-   can replace objects on the destination can substitute a snapshot they built.
+broken into has it.
   The recipient string is not stored at the destination, but a compromised
   backed-up host has it. Defences live on the destination side — bucket
   versioning or object lock, credentials for the source host that cannot delete
   or overwrite existing versions, and pruning from a trusted host. Note that S3
   `PutObject` overwrites an existing key, so PUT permission alone is not
   append-only.
 2. **Compression reveals sizes.** Blobs and `db.zst.age` are zstd-compressed
   then age-encrypted; the manifest is compressed only. age does not pad, so an
   object's size is the exact compressed length of its contents. All new chunks
   packed into one blob share a single zstd stream (8 MiB window, 4 MiB at
   compression levels 1-2), and a blob is closed at `blob_size_limit` and at the
   end of each configured path. Because a stored chunk is never packed again,
   someone who can write into a backed-up file and watch blob sizes learns
   something only when their controlled data and a secret land in the same chunk
   of a file that keeps changing. Advice: back up any outsider-writable
   directory as its own snapshot.
 3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The
   default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of
   2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a
   blob holds hundreds of chunks, so individual chunk lengths are not visible in
   the blob's size; lowering the limit toward the chunk size begins to expose
   them.
 4. **Decrypted data on local disk.** Several commands stage plaintext under
   `$TMPDIR`: `snapshot restore` writes decrypted blobs under
   `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at
   `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at
   `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of
   the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600`
   and removed on success, but a `kill -9` or a power loss leaves them behind —
   delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR`
   should be trusted to the same degree as the restore target.
 5. **Store permissions differ per command.** The backed-up host needs only PUT
   to run `snapshot create`: it writes blobs and metadata and neither reads nor
   deletes them. Other commands need more — `snapshot verify`,
   `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`,
   `snapshot remove`, and `remote nuke` also delete. The recommended cron line
   uses `--prune`, which runs `prune` on the backed-up host, so granting that
   host `--prune` gives it credentials that can delete its own backups. To keep
   the source host to PUT only, prune from a separate trusted host instead.
 6. **Changing recipients does not re-encrypt existing data.** Deduplicated
   chunks and same-named blobs already on the destination stay encrypted to the
   recipients in force when they were written. A new snapshot that reuses them
   cannot be restored with a newly added recipient's key alone, because those
   reused objects were never encrypted to it. To make everything readable by a
   new key, run `vaultik database delete` and take a full backup to a fresh
   destination or prefix.
 7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients.
   Long-lived ciphertext held by a third party (the destination operator) has no
   fallback if X25519 is ever broken: there is no second recipient type and no
   post-quantum option.
-## Consistency Guarantees
+The defences are on the destination's side:
-1. **Blobs are immutable** - Once written, a blob is never modified
+- object versioning or object lock on the bucket
-2. **Blobs are written before metadata** - A snapshot's metadata is only written
+- credentials for the backed-up machine that cannot delete or overwrite
-   after all its blobs are successfully uploaded
+- running `prune` from a trusted machine
 3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst
   are written as complete files
 4. **A snapshot is marked complete in the local DB only after its metadata is
   uploaded** - `finalizeSnapshotMetadata` runs `ExportSnapshotMetadata` first
   and records completion (`MarkSnapshotComplete`) only once the export succeeds
   (see the backup data flow in [ARCHITECTURE.md](ARCHITECTURE.md)). A crash
   during the export therefore leaves the snapshot incomplete, so the next
   backup's `PruneDatabase` drops it and re-backs-up its data, rather than
   leaving a completed-looking row in the local index with no matching metadata
   on the destination store. (A crash in the brief moment after the export
   succeeds but before completion is recorded leaves a fully-restorable snapshot
   on the destination that the local index drops as incomplete on the next run;
   `snapshot list` then reports it honestly as remote-only, which is the safe
   direction: the destination copy stays restorable.)
-## Pruning Safety
+An S3 `PutObject` replaces an existing object. Permission to write alone
 therefore does not stop overwrites.
-The prune operation is safe because:
+### Sizes are visible
-1. It keeps every blob listed in any snapshot's manifest and deletes only blobs
+An object's size follows exactly from the size of its compressed contents.
   that no manifest references. Under `metadata/` it deletes only a directory
   that has no `manifest.json.zst`, whose name is a remote key, and that no
   local snapshot record claims
 2. Manifests are unencrypted and can be read without keys
 3. If any manifest cannot be downloaded or decoded, prune deletes nothing and
   exits with an error, rather than treating that snapshot's blobs as
   unreferenced
 4. It deletes an unreferenced blob only once the blob is older than
   `prune_grace_period` (default `24h`), and a directory under `metadata/` with
   no manifest only once every object in it is. A backup uploads its blobs and
   its `db.zst.age` before its manifest, so a prune running meanwhile, on this
   host or another sharing the destination, leaves them as long as no backup run
   takes longer than the grace period. The grace period does not protect a blob
   a backup reuses from an earlier run that stopped before its manifest: the
   backup skips the upload of a blob already on the destination, so that blob
   keeps the time of its first upload, and a prune before the backup's manifest
   is written can delete it; the new snapshot then cannot be restored.
   `remote nuke` has no such check
-## Restoration Requirements
+All the new chunks in one blob share one zstd stream. The stream looks back 8
 MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at the
 `blob_size_limit` setting and at the end of each configured path. A chunk that
 the local index records as uploaded is never packed again.
-To restore from a backup, you need:
+Data that an outsider writes compresses smaller when it matches a secret near it
 in the same blob, so the blob gets smaller. The blob's size therefore shows the
 outsider whether their data matched the secret.
-1. **The Age private key** - To decrypt blobs and database
+This works only when all of these hold:
 2. **The snapshot metadata** - Both files from the snapshot's metadata directory
 3. **All referenced blobs** - As listed in the manifest
-The restoration process:
+- The outsider can write into a backed-up file.
 - The outsider can see the blob sizes at the destination.
 - The outsider's data and the secret are in the same blob, no further apart than
  the stream looks back.
 - The file keeps changing.
-1. Download and decrypt the database dump to understand file structure
+Back up a directory that outsiders can write to as a snapshot of its own.
-2. Download and decrypt the required blobs
+
-3. Reconstruct files from their chunks
+### Chunk boundaries use no secret
-4. Restore file metadata (permissions, timestamps, etc.)
+
 `vaultik` finds chunk boundaries with the FastCDC algorithm. Its parameters are
 fixed and public. `chunk_size` sets the average chunk size. No chunk is larger
 than four times this size. Only a file's last chunk can be smaller than a
 quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A
 file of 2.5 MB or less is one chunk. An empty file has no chunk.
 At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of chunks.
 The size of the blob then does not show the size of any one chunk. A limit close
 to `chunk_size` begins to show chunk sizes.
 ### Decrypted data on the local disk
 These commands write unencrypted data to the temporary directory:
 - `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/`. There
  is no limit on their total size.
 - `snapshot restore` writes the decrypted database to
  `vaultik-restore-*/snapshot.db`.
 - `snapshot verify --deep` writes the decrypted database to
  `vaultik-verify-*/snapshot.db`.
 - `snapshot create` writes a copy of the local index to
  `vaultik-snapshot-*/snapshot.db`.
 These files are readable only by their owner, and are removed when the command
 ends. A `kill -9` or a power loss leaves them behind. Delete any leftover
 `vaultik-*` entry in the temporary directory yourself.
 Trust the temporary directory as much as the directory you restore into.
 ### Commands need different permissions
 `snapshot create` without `--prune` writes objects. On S3 it needs no other
 permission. Other commands need more:
 - These commands list and read:
    - `snapshot list`
    - `snapshot verify`
    - `snapshot restore`
    - `remote info`
 - These commands list and delete:
    - `snapshot purge`
    - `snapshot remove`
    - `remote nuke`
 - `prune` lists, reads and deletes.
 An rclone remote that can move objects also needs permission to read and delete.
 An upload there looks up the object it replaces and deletes it. It stops when
 the look-up is refused.
 `snapshot verify` and `snapshot restore` list only when they are given a remote
 key, or a snapshot ID that is not in the local index.
 `snapshot remove --local-only` does not reach the destination.
 `snapshot remove --dry-run` deletes nothing.
 `snapshot create --prune` runs a prune on the backed-up machine. That machine
 then holds credentials that can delete its own backups. To keep it to write-only
 credentials, run `prune` from a separate, trusted machine. `prune` there deletes
 only data that no snapshot uses. It does not remove old snapshots.
 ### New recipients cannot read old data
 Changing `age_recipients` does not re-encrypt what is already stored. A chunk
 that the local index records as uploaded is not written again, so it stays
 encrypted to the earlier recipients.
 A new snapshot that uses such chunks cannot be restored with a new recipient's
 key alone. To make everything readable by a new key, run
 `vaultik database delete` and take a full backup to a new destination.
 ### X25519 recipients only
 `vaultik` accepts age X25519 public keys only. It rejects age SSH and plugin
 recipients.
 The encrypted data is held by a third party for a long time. There is no second
 kind of recipient to fall back on if X25519 is broken, and no kind that resists
 a quantum computer.
 ## Order of writes
 A backup writes in a fixed order:
 - the marker file
 - the blobs
 - `db.zst.age`
 - `manifest.json.zst`
 A snapshot's metadata is written only after all its blobs are uploaded. The
 manifest is written last. A directory under `metadata/` that has no manifest
 therefore belongs to a backup that did not finish.
 The local index marks a snapshot complete only after both metadata files are
 uploaded. A backup that stops during that upload leaves the snapshot unfinished.
 The next backup removes it from the local index and backs up its files again.
 One short gap remains. A backup can stop after the upload and before the
 snapshot is marked complete. The destination then holds a snapshot that can be
 restored, and the local index marks it unfinished. `snapshot list` shows it as
 remote only.
 An upload that is cut off leaves no partial object under the final name on S3 or
 on a local directory. The same holds for an rclone remote that can move objects.
 An rclone remote that cannot move objects can keep a partial object.
 ## Pruning
 `prune` deletes two kinds of object at the destination:
 - a blob that no manifest lists
 - a directory under `metadata/` that has no manifest
 It reads the manifests, which are not encrypted, so it needs no private key.
 These rules keep it from deleting data that is in use:
 - It keeps every blob that any manifest lists.
 - It deletes nothing when any manifest cannot be downloaded or decoded.
 - It deletes a blob only once the blob is older than `prune_grace_period`.
 - It deletes a directory only once every object in it is that old.
 - It keeps a directory whose snapshot is in the local index.
 - It leaves alone a directory whose name is not a remote key.
 `prune_grace_period` defaults to `24h`. It protects a backup that is still
 running. Such a backup has uploaded blobs and perhaps `db.zst.age`, but not yet
 its manifest. A prune from this machine or another keeps those objects, as long
 as no backup takes longer than the grace period.
 The grace period does not protect one case. A backup does not upload a blob that
 is already at the destination, so that blob keeps the time of its first upload.
 The blob can be left over from an earlier backup that stopped before its
 manifest. A prune that runs before the new manifest is written can then delete
 that blob. The new snapshot cannot be restored afterwards.
 To avoid this case, give each machine and each user its own destination.
 `remote nuke` applies none of these rules.
 ## What a restore needs
 - the age private key
 - the snapshot's `db.zst.age`
 - the blobs that hold the files to restore
 A restore does not read the manifest, or the file, chunk and blob rows of the
 local index.
 It works in these steps:
 - It downloads and decrypts `db.zst.age`.
 - It checks that the database is the one for the snapshot that was asked for.
 - It downloads and decrypts the blobs it needs. It checks each blob against its
  name.
 - It writes each file from its chunks.
 - It sets each file's permissions and times, and its owner when it runs as root.
@@ -19,4 +19,62 @@ into `main` is the owner's.
 # Future Steps
-Work planned after 1.0 is listed in the README [roadmap](../README.md#roadmap).
+Work planned after 1.0, loosely ordered by priority.
 ## Correctness and operability
 - **Outside security audit.** Before 1.0 the encryption and blob-generation code
  was reviewed. Every bug the review found was fixed. The risks it accepted are
  listed in [Accepted risks](REPOSTRUCTURE.md#accepted-risks). No outside audit
  has been done. age, zstd and content-defined chunking are mostly off-the-shelf
  pieces. The code that joins them needs an outside read:
    - key handling
    - recipient parsing
    - manifest trust boundary
    - restore-time identity validation
 - **Verify restored content end-to-end in CI.** The current integration test
  does this for a small synthetic snapshot but not at scale. A nightly job
  against a multi-GB representative snapshot would catch silent regressions in
  the chunker, packer, or restore planner.
 ## Performance
 - **Parallel blob downloads during restore.** A restore downloads one blob at a
  time now. With a fast S3 endpoint and a multi-core machine, restore speed is
  bound by the fetch, decryption and decompression of each blob. Fetching,
  decrypting and decompressing several blobs at once would make such a restore
  faster. The change must work with `restorePlan`, which orders the files, and
  with `restoreSweeper`, which deletes a cached blob once no remaining file
  needs it.
 - **Bandwidth limiting (`--bwlimit`).** For both upload and download. It is
  useful for backing up over a shared link. It is tricky to make work correctly
  with parallel downloads.
 - **Restart of interrupted restore.** Today a restore can be started again, and
  it then overwrites the partial output. It does not resume from where it
  stopped, and it does not skip files that are already present. A `--resume`
  mode that checks targets before fetching blobs would matter for very large
  restores.
 - **Daemon mode.** A long-running mode that watches for file changes, so that
  frequent backups, such as hourly ones, skip the full scan. It adds little for
  the usual backups from cron every 12 to 36 hours. See
  [issue #204](https://git.eeqj.de/sneak/vaultik/issues/204).
 ## Usability
 - **Man pages and richer `--help` examples.** Cobra generates basic help. Man
  pages would be a separate target.
 - **`vaultik snapshot diff <a> <b>`.** Show which files changed between two
  snapshots, without restoring either.
 - **Status reporting hook for `--cron`.** When a backup fails silently in cron,
  the user has no idea. A configurable hook on completion, for both success and
  failure, would close the loop. It could be a webhook, an email or
  `notify-send`.
 ## Infrastructure
 - **Schema migrations after 1.0.** Migrations are supported, but none are added
  before 1.0 because there is no installed base yet. After 1.0, each schema
  change is a new migration, so an existing local index is migrated when
  `vaultik` is updated. See [`DATAMODEL.md`](DATAMODEL.md#schema-migrations).
 - **Storage backend coverage tests.** S3, `file://` and `rclone://` all share
  the `Storer` interface, but the rclone path is the least exercised in CI.
@@ -318,12 +318,14 @@ func newConfigInitCommand() *cobra.Command {
 		Use:   "init",
 		Short: "Write a default config file",
 		Long: `Creates a default configuration file with commented explanations
-for every setting. If a config file already exists at the target path,
+for the settings it lists. If a config file already exists at the target
-the command refuses to overwrite it.
+path, the command refuses to overwrite it. The file is created with mode
 0600, because it can hold credentials.
 The config is written to the path from --config, $VAULTIK_CONFIG, or
-the platform default config directory (e.g. ~/Library/Application Support/
+config.yml in the platform default config directory
-on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`,
+(~/Library/Application Support/vaultik/ on macOS, ~/.config/vaultik/ on
 Linux, /etc/vaultik/ as root).`,
 		Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			path := configPathForInit()
@@ -362,7 +364,7 @@ on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`,
 func newConfigEditCommand() *cobra.Command {
 	return &cobra.Command{
 		Use:   "edit",
-		Short: "Open the config file in $EDITOR",
+		Short: "Open the config file in $EDITOR (vi when it is not set)",
 		Args:  cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			path, err := ResolveConfigPath()
@@ -444,9 +446,9 @@ func newConfigSetCommand() *cobra.Command {
 		Use:   "set <key> <value>",
 		Short: "Set a config value by dotted path (e.g. compression_level 5)",
 		Long: `Sets a scalar config value addressed by dotted YAML path and writes
-the file back, preserving comments and formatting. Missing keys along
+the file back. Comments in the file are kept. Blank lines can be
-the path are created: as a list when the next path element is a
+removed. Missing keys along the path are created: as a list when the
-number, otherwise as a map.
+next path element is a number, otherwise as a map.
 If the config loads, a change after which it would not, such as
 compression_level 99, is refused and the file is left unchanged. A
@@ -96,8 +96,8 @@ func TestDefaultConfigOnLinux(t *testing.T) {
 // testRecipient is a valid age public key.
 const testRecipient = "age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj"
-// TestConfigSetRecipientOnFreshConfig follows the README quickstart: on the
+// TestConfigSetRecipientOnFreshConfig follows the README's getting started
-// file `config init` writes, `config set age_recipients.0` and
+// steps: on the file `config init` writes, `config set age_recipients.0` and
 // `config set storage_url` give a config that loads with that recipient.
 // Neither is refused, although the config does not load until both are set;
 // once it loads, a recipient that is not a valid key is.
@@ -17,7 +17,11 @@ func NewPruneCommand() *cobra.Command {
 unreferenced blobs and directories under metadata/ with no manifest
 from the backup destination store.
-Local cleanup drops incomplete snapshots and any files, chunks, or
+Needs permission to list, read and delete objects on the destination
 store. It does not need the age private key.
 Local cleanup drops snapshots whose manifest is gone from the
 destination store, incomplete snapshots, and any files, chunks, or
 blobs no longer referenced by a completed snapshot. Remote cleanup
 scans every snapshot manifest in the destination store, builds the
 set of still-referenced blob hashes, and deletes any blob not in that
@@ -40,11 +40,13 @@ func newRemoteNukeCommand() *cobra.Command {
 	cmd := &cobra.Command{
 		Use:   "nuke",
 		Short: "Delete ALL snapshot metadata and blobs from the backup destination store",
-		Long: `Removes every snapshot's metadata and every blob from remote
+		Long: `Removes every snapshot's metadata, every blob and the destination
-storage. After this command completes successfully the bucket prefix is
+marker from the backup destination store, and each removed snapshot from
-empty and the next backup starts from scratch.
+the local index. After this command completes successfully the backup
 destination store is empty and the next backup starts from scratch.
-This is destructive and irreversible. Requires --force.`,
+This is destructive and irreversible. Requires --force, and permission
 to list and delete objects on the destination store.`,
 		Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			if !force {
@@ -75,7 +77,10 @@ func newRemoteInfoCommand() *cobra.Command {
 - Size of all snapshot metadata (per snapshot and total)
 - Count and total size of all blobs
 - Count and size of referenced blobs (from all manifests)
- Count and size of orphaned blobs (not referenced by any manifest)`,
+- Count and size of orphaned blobs (not referenced by any manifest)
 Needs permission to list and read the destination store. It does not
 need the age private key.`,
 		Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			// Use unified config resolution
@@ -35,9 +35,10 @@ func NewRootCommand() *cobra.Command {
 	cmd := &cobra.Command{
 		Use:   "vaultik",
 		Short: "Secure incremental backup tool with asymmetric encryption",
-		Long: `vaultik is a secure incremental backup tool that encrypts data using age
+		Long: `vaultik is an incremental backup tool that encrypts data to age public
-public keys and uploads to S3-compatible storage. No private keys are needed
+keys and uploads it to an S3-compatible object store, a local directory or
-on the source system.`,
+an rclone remote. The backed-up machine needs no private key: only
 'snapshot restore' and 'snapshot verify --deep' need one.`,
 		SilenceUsage: true,
 		// Bare 'vaultik' (no subcommand): print help. The banner is
 		// printed once at process startup by Entry, before cobra
@@ -50,13 +51,16 @@ on the source system.`,
 	// Add global flags
 	cmd.PersistentFlags().StringVar(&rootFlags.ConfigPath, "config", "",
-		"Path to config file (default: $VAULTIK_CONFIG or platform config dir)")
+		"Path to config file (default: $VAULTIK_CONFIG, then the platform "+
 			"config directory, then /etc/vaultik/config.yml)")
 	cmd.PersistentFlags().BoolVarP(&rootFlags.Verbose, "verbose", "v", false,
 		"Enable verbose output")
 	cmd.PersistentFlags().BoolVar(&rootFlags.Debug, "debug", false,
 		"Enable debug output")
 	cmd.PersistentFlags().BoolVarP(&rootFlags.Quiet, "quiet", "q", false,
-		"Suppress non-error output")
+		"Leave out the banner, status lines, tables and reports (warnings, "+
 			"errors, JSON documents, 'config get' values and confirmation "+
 			"prompts are still printed)")
 	cmd.PersistentFlags().BoolVar(&rootFlags.SkipErrors, "skip-errors", false,
 		"Skip files that cannot be read when creating a snapshot, or "+
 			"that cannot be restored when restoring, instead of aborting "+
@@ -60,6 +60,22 @@ func NewSnapshotCommand() *cobra.Command {
 	return cmd
 }
 // snapshotCreateLong is the long help text of 'snapshot create'.
 const snapshotCreateLong = `Creates new snapshots of the configured directories.
 If snapshot names are provided, only those snapshots are created.
 If no names are provided, all configured snapshots are created.
 Needs at least one public key in age_recipients, and permission to write
 objects at the backup destination store. It does not need the age private
 key. With --prune it also needs permission to list, read and delete objects
 there. An rclone remote that can move objects also needs permission to
 read and delete.
 The config is read from the path given by --config or VAULTIK_CONFIG;
 otherwise from the platform config directory (~/.config/vaultik/config.yml
 on Linux), then /etc/vaultik/config.yml.`
 // newSnapshotCreateCommand creates the 'snapshot create' subcommand
 func newSnapshotCreateCommand() *cobra.Command {
 	opts := &vaultik.SnapshotCreateOptions{}
@@ -67,15 +83,8 @@ func newSnapshotCreateCommand() *cobra.Command {
 	cmd := &cobra.Command{
 		Use:   "create [snapshot-names...]",
 		Short: "Create new snapshots",
-		Long: `Creates new snapshots of the configured directories.
+		Long:  snapshotCreateLong,
-
+		Args:  cobra.ArbitraryArgs,
 If snapshot names are provided, only those snapshots are created.
 If no names are provided, all configured snapshots are created.
 The config is read from the path given by --config or VAULTIK_CONFIG;
 otherwise from the platform config directory (~/.config/vaultik/config.yml
 on Linux), then /etc/vaultik/config.yml.`,
 		Args: cobra.ArbitraryArgs,
 		RunE: func(cmd *cobra.Command, args []string) error {
 			err := opts.Validate()
 			if err != nil {
@@ -156,8 +165,15 @@ func newSnapshotListCommand() *cobra.Command {
 		Use:     "list",
 		Aliases: []string{"ls"},
 		Short:   "List all snapshots",
-		Long:    "Lists all snapshots with their ID, timestamp, and compressed size",
+		Long: `Lists the snapshots in the local index and on the backup destination
-		Args:    cobra.NoArgs,
+store, with their ID, timestamp, and sizes.
 Needs permission to list and read the destination store. It does not need
 the age private key. A snapshot found only on the destination store is
 listed by its remote key, because its ID is stored only in the local index
 and in the snapshot's encrypted database. If the destination store cannot
 be listed, the command warns and lists the local index alone.`,
 		Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			return runVaultikApp(cmd, readOnly, false,
 				"Failed to list snapshots",
@@ -179,11 +195,19 @@ func newSnapshotPurgeCommand() *cobra.Command {
 	cmd := &cobra.Command{
 		Use:   "purge",
 		Short: "Purge old snapshots",
-		Long: `Removes snapshots based on age or count criteria.
+		Long: `Removes snapshots based on age or count criteria. Exactly one of
 --keep-latest and --older-than is required.
-Retention is per-snapshot-name: --keep-latest keeps the latest of each
+Retention is per-snapshot-name: --keep-latest keeps the latest snapshot of
-configured snapshot name, not the latest globally. Use --snapshot to
+each name, not the latest globally. Use --snapshot to restrict the
-restrict the operation to specific snapshot names.`,
+operation to specific snapshot names.
 Only snapshots in the local index are removed. Each one is removed from
 the local index and its metadata is deleted from the backup destination
 store. Blobs are not touched: run 'vaultik prune' afterwards.
 Needs permission to list and delete objects on the destination store. If
 the destination store cannot be listed, nothing is removed.`,
 		Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			// Validate flags
@@ -222,14 +246,30 @@ func newSnapshotVerifyCommand() *cobra.Command {
 	cmd := &cobra.Command{
 		Use:   "verify <snapshot-id>",
 		Short: "Check a snapshot's blobs are present with the listed size",
-		Long: "Checks that every blob the snapshot's manifest lists is present\n" +
+		Long: `Checks that every blob the snapshot's manifest lists is present
-			"in storage with the size the manifest records, and that the\n" +
+in storage with the size the manifest records, and that the
-			"snapshot's encrypted database is present. It does not read blob\n" +
+snapshot's encrypted database is present. It does not read blob
-			"contents; use --deep to download, decrypt, and re-hash every blob\n" +
+contents and does not need the age private key.
-			"to detect corruption -- integrity, not who wrote it.\n\n" +
+
-			"The snapshot may be named by its ID or, on a host with no local\n" +
+Needs permission to read the backup destination store, and to list it
-			"index, by the remote key that 'snapshot list' prints for a\n" +
+when given a remote key, or a snapshot ID that is not in the local
-			"remote-only snapshot (an unambiguous leading part is enough).",
+index.
 Use --deep to download, decrypt, and re-hash every blob to detect
 corruption -- integrity, not who wrote it.
 --deep requires the age private key, in the VAULTIK_AGE_SECRET_KEY
 environment variable or as age_secret_key in the config file; the
 environment variable wins. The variable may hold the whole age-keygen
 file, with its comments and all of its keys. Read the key from the
 file rather than typing it, so that it does not land in your shell
 history:
  export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
 The snapshot may be named by its ID or, on a host with no local
 index, by the remote key that 'snapshot list' prints for a
 remote-only snapshot (an unambiguous leading part is enough).`,
 		Args: requireSnapshotIDArg,
 		RunE: func(cmd *cobra.Command, args []string) error {
 			snapshotID := args[0]
@@ -264,7 +304,10 @@ func newSnapshotVerifyCommand() *cobra.Command {
 		},
 	}
-	cmd.Flags().BoolVar(&opts.Deep, "deep", false, "Download and verify blob hashes")
+	cmd.Flags().BoolVar(&opts.Deep, "deep", false,
 		"Download, decrypt and re-hash every blob; requires the age private "+
 			"key in VAULTIK_AGE_SECRET_KEY, or as age_secret_key in the config "+
 			"file")
 	cmd.Flags().BoolVar(&opts.JSON, "json", false, "Output verification results as JSON")
 	return cmd
@@ -288,8 +331,10 @@ know about), which is what 'vaultik prune' does. The command prints the
 'vaultik prune' invocation to run as a follow-up only when it removed the
 snapshot's metadata from the destination store, and never under --json.
-An ID that names no snapshot in the local index or on the destination
+The argument must be a snapshot ID: a remote key is not accepted. An ID
-store is an error.
+that names no snapshot in the local index or on the destination store is
 an error. Needs permission to list and delete objects on the destination
 store, unless --local-only is given.
 Use --local-only to skip the remote half (e.g. when you want to forget a
 snapshot locally without touching the destination store); the snapshot
@@ -318,7 +363,8 @@ nuke --force' — it is the single supported entry point for that.`,
 	cmd.Flags().BoolVarP(&opts.Force, "force", "f", false, "Skip confirmation prompt")
 	cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false,
-		"Show what would be removed without removing")
+		"Show what would be removed without removing; needs no permission "+
 			"to delete")
 	cmd.Flags().BoolVar(&opts.JSON, "json", false,
 		"Output result as JSON; skips the confirmation prompt, as --force does")
 	cmd.Flags().BoolVar(&opts.LocalOnly, "local-only", false,
@@ -37,13 +37,22 @@ The snapshot may be named by its ID or, when restoring on a host with no
 local index, by the remote key that 'snapshot list' prints for a
 remote-only snapshot (an unambiguous leading part is enough).
-Requires the age private key in the VAULTIK_AGE_SECRET_KEY environment
+Each file is restored under its original absolute path inside the target
-variable. The variable may hold the whole age-keygen file (comments and
+directory. File owners are restored only when the command runs as root.
-all of its identities are accepted); read it from the file rather than
+
-typing the key, so it does not land in your shell history:
+Requires the age private key, in the VAULTIK_AGE_SECRET_KEY environment
 variable or as age_secret_key in the config file; the environment
 variable wins. The variable may hold the whole age-keygen file, with its
 comments and all of its keys. Read the key from the file rather than
 typing it, so that it does not land in your shell history:
  export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
 Also needs permission to read the backup destination store, and to list
 it when given a remote key, or a snapshot ID that is not in the local
 index. It also needs free space in the temporary directory for the
 decrypted blobs.
 Examples:
  # Restore entire snapshot
  vaultik snapshot restore myhost_docs_2025-01-01T12:00:00Z /restore
@@ -396,8 +396,6 @@ func Parse(data []byte) (*Config, error) {
 //     never the value. An empty list is accepted, because only snapshot
 //     create needs a recipient and it checks for one itself
 //   - At least one snapshot must be configured with at least one path
 //   - Every exclude pattern, global or in a snapshot, must compile; errors
 //     name the setting and quote the pattern
 //   - Storage must be configured (either storage_url or s3.* fields)
 //   - The index path must be absolute: a relative one would resolve against
 //     the working directory, so each directory would get its own index
@@ -418,18 +416,18 @@ func (c *Config) Validate() error {
 		}
 	}
-	err := c.validateSnapshots()
+	if len(c.Snapshots) == 0 {
-	if err != nil {
+		return errNoSnapshots
 		return err
 	}
-	err = c.validateExcludes()
+	for name, snap := range c.Snapshots {
-	if err != nil {
+		if len(snap.Paths) == 0 {
-		return err
+			return fmt.Errorf("%w: %q", errSnapshotNoPaths, name)
 		}
 	}
 	// Validate storage configuration
-	err = c.validateStorage()
+	err := c.validateStorage()
 	if err != nil {
 		return err
 	}
@@ -468,22 +466,6 @@ func (c *Config) Validate() error {
 	return nil
 }
 // validateSnapshots refuses a config with no snapshot or with a snapshot
 // that has no path.
 func (c *Config) validateSnapshots() error {
 	if len(c.Snapshots) == 0 {
 		return errNoSnapshots
 	}
 	for name, snap := range c.Snapshots {
 		if len(snap.Paths) == 0 {
 			return fmt.Errorf("%w: %q", errSnapshotNoPaths, name)
 		}
 	}
 	return nil
 }
 // validateAgeRecipient parses one age_recipients entry with the age library
 // and returns a value-free error on failure. A recipient string can be
 // sensitive (an operator may paste a secret key by mistake), so neither the
@@ -1,107 +0,0 @@
 package config
 import (
 	"errors"
 	"fmt"
 	"strings"
 	"github.com/gobwas/glob"
 )
 var errBadExcludePattern = errors.New("is not a valid exclude pattern")
 // CompileExcludePattern compiles one exclude pattern into the globs that
 // match it. anchored reports whether the pattern matches only from the root
 // of the source directory. An empty pattern yields no globs. Config.Validate
 // and the scanner both compile patterns with it, so a pattern that loads is
 // one the scanner can use.
 func CompileExcludePattern(pattern string) ([]glob.Glob, bool, error) {
 	if pattern == "" {
 		return nil, false, nil
 	}
 	// A leading "/" anchors the pattern to the source directory.
 	trimmed, anchored := strings.CutPrefix(pattern, "/")
 	// A trailing "/" marks a directory and is not matched.
 	trimmed = strings.TrimSuffix(trimmed, "/")
 	// A leading "**/" demands a "/" before the name and so misses an
 	// entry directly under the source path. It means "at every depth",
 	// which is what an unanchored pattern already does, so drop the
 	// prefix and match unanchored: as in gitignore, "**/x" and
 	// "/**/x" then also match a top-level "x".
 	if rest, found := strings.CutPrefix(trimmed, "**/"); found {
 		trimmed = rest
 		anchored = false
 	}
 	// Likewise "/**/" inside a pattern demands two slashes, so
 	// "a/**/*.log" misses "a/x.log". As in gitignore it stands for
 	// zero or more directories, so also compile the pattern with it
 	// made a single "/". The glob library's {a,b} is no help: with a
 	// "**" inside it in mid-pattern, it misses paths it should match.
 	var globs []glob.Glob
 	for _, variant := range doubleStarVariants(trimmed) {
 		g, err := glob.Compile(variant, '/')
 		if err != nil {
 			return nil, false, fmt.Errorf("%w: %w", errBadExcludePattern, err)
 		}
 		globs = append(globs, g)
 	}
 	return globs, anchored, nil
 }
 // doubleStarVariants returns pattern once for every way of keeping each
 // "/**/" in it or making it a single "/".
 func doubleStarVariants(pattern string) []string {
 	const doubleStar = "/**/"
 	before, after, found := strings.Cut(pattern, doubleStar)
 	if !found {
 		return []string{pattern}
 	}
 	var variants []string
 	for _, rest := range doubleStarVariants(after) {
 		variants = append(variants, before+doubleStar+rest, before+"/"+rest)
 	}
 	return variants
 }
 // validateExcludes refuses an exclude pattern, global or in a snapshot,
 // that does not compile, naming the setting and quoting the pattern.
 func (c *Config) validateExcludes() error {
 	err := validateExcludeList("exclude", c.Exclude)
 	if err != nil {
 		return err
 	}
 	for _, name := range c.SnapshotNames() {
 		err = validateExcludeList("snapshots."+name+".exclude",
 			c.Snapshots[name].Exclude)
 		if err != nil {
 			return err
 		}
 	}
 	return nil
 }
 // validateExcludeList refuses the first pattern in patterns that does not
 // compile. setting is the name of the list, as in "exclude".
 func validateExcludeList(setting string, patterns []string) error {
 	for i, pattern := range patterns {
 		_, _, err := CompileExcludePattern(pattern)
 		if err != nil {
 			return fmt.Errorf("%s[%d]: %q %w", setting, i, pattern, err)
 		}
 	}
 	return nil
 }
@@ -1,65 +0,0 @@
 package config_test
 import (
 	"os"
 	"path/filepath"
 	"strings"
 	"testing"
 	"sneak.berlin/go/vaultik/internal/config"
 )
 // TestLoadRefusesBadExcludePattern checks that an exclude pattern that does
 // not compile, global or in a snapshot, fails at load with the setting and
 // the pattern named, rather than the backup storing what it meant to leave
 // out.
 func TestLoadRefusesBadExcludePattern(t *testing.T) {
 	t.Parallel()
 	const storage = "storage_url: file:///tmp/vaultik-test-storage\n"
 	tests := []struct {
 		name    string
 		config  string
 		wantErr string
 	}{
 		{
 			name: "global",
 			config: "exclude: ['*.tmp', '.cache', 'build[']\n" +
 				"snapshots:\n" +
 				"  home:\n" +
 				"    paths: [/tmp/vaultik-test-source]\n" + storage,
 			wantErr: `exclude[2]: "build[" is not a valid exclude pattern`,
 		},
 		{
 			name: "snapshot",
 			config: "snapshots:\n" +
 				"  home:\n" +
 				"    paths: [/tmp/vaultik-test-source]\n" +
 				"    exclude: ['/a/**/x[']\n" + storage,
 			wantErr: `snapshots.home.exclude[0]: "/a/**/x[" is not a valid exclude pattern`,
 		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			t.Parallel()
 			path := filepath.Join(t.TempDir(), "config.yml")
 			err := os.WriteFile(path, []byte(tt.config), 0o600)
 			if err != nil {
 				t.Fatalf("write config: %v", err)
 			}
 			_, err = config.Load(path)
 			if err == nil {
 				t.Fatalf("Load() returned nil, want an error containing %q", tt.wantErr)
 			}
 			if !strings.Contains(err.Error(), tt.wantErr) {
 				t.Fatalf("Load() error = %q, want it to contain %q", err, tt.wantErr)
 			}
 		})
 	}
 }
@@ -19,7 +19,6 @@ import (
 	"github.com/spf13/afero"
 	"sneak.berlin/go/vaultik/internal/blob"
 	"sneak.berlin/go/vaultik/internal/chunker"
 	"sneak.berlin/go/vaultik/internal/config"
 	"sneak.berlin/go/vaultik/internal/database"
 	"sneak.berlin/go/vaultik/internal/log"
 	"sneak.berlin/go/vaultik/internal/storage"
@@ -222,12 +221,8 @@ func NewScanner(cfg ScannerConfig) *Scanner {
 		progress = NewProgressReporter()
 	}
-	compiledExclude, err := compileExcludePatterns(cfg.Exclude)
+	// Compile exclude patterns
-	if err != nil {
+	compiledExclude := compileExcludePatterns(cfg.Exclude)
 		log.Error("Failed to compile exclude patterns", "error", err)
 		return nil
 	}
 	uiw := cfg.UI
 	if uiw == nil {
@@ -2323,18 +2318,51 @@ func wrapPermissionError(path string, err error) error {
 		"with --skip-errors to continue past unreadable files)", path, err)
 }
-// compileExcludePatterns compiles the exclude patterns into glob matchers.
+// compileExcludePatterns compiles the exclude patterns into glob matchers
-// Config.Validate has already refused a pattern that does not compile.
+func compileExcludePatterns(patterns []string) []compiledPattern {
 func compileExcludePatterns(patterns []string) ([]compiledPattern, error) {
 	var compiled []compiledPattern
 	for _, p := range patterns {
-		globs, anchored, err := config.CompileExcludePattern(p)
+		if p == "" {
-		if err != nil {
+			continue
 			return nil, fmt.Errorf("%q %w", p, err)
 		}
-		for _, g := range globs {
+		// Check if pattern is anchored (starts with /)
 		anchored := strings.HasPrefix(p, "/")
 		pattern := p
 		if anchored {
 			pattern = p[1:] // Remove leading /
 		}
 		// Remove trailing slash if present (directory indicator)
 		pattern = strings.TrimSuffix(pattern, "/")
 		// A leading "**/" demands a "/" before the name and so misses an
 		// entry directly under the source path. It means "at every depth",
 		// which is what an unanchored pattern already does, so drop the
 		// prefix and match unanchored: as in gitignore, "**/x" and
 		// "/**/x" then also match a top-level "x".
 		if rest, found := strings.CutPrefix(pattern, "**/"); found {
 			pattern = rest
 			anchored = false
 		}
 		// Likewise "/**/" inside a pattern demands two slashes, so
 		// "a/**/*.log" misses "a/x.log". As in gitignore it stands for
 		// zero or more directories, so also compile the pattern with it
 		// made a single "/". The glob library's {a,b} is no help: with a
 		// "**" inside it in mid-pattern, it misses paths it should match.
 		for _, variant := range doubleStarVariants(pattern) {
 			// For patterns without path separators, we need to match them as components
 			// e.g., ".git" should match ".git" anywhere in the path
 			g, err := glob.Compile(variant, '/')
 			if err != nil {
 				log.Warn("Invalid exclude pattern, skipping", "pattern", p, "error", err)
 				break
 			}
 			compiled = append(compiled, compiledPattern{
 				pattern:  g,
 				anchored: anchored,
@@ -2343,7 +2371,26 @@ func compileExcludePatterns(patterns []string) ([]compiledPattern, error) {
 		}
 	}
-	return compiled, nil
+	return compiled
 }
 // doubleStarVariants returns pattern once for every way of keeping each
 // "/**/" in it or making it a single "/".
 func doubleStarVariants(pattern string) []string {
 	const doubleStar = "/**/"
 	before, after, found := strings.Cut(pattern, doubleStar)
 	if !found {
 		return []string{pattern}
 	}
 	var variants []string
 	for _, rest := range doubleStarVariants(after) {
 		variants = append(variants, before+doubleStar+rest, before+"/"+rest)
 	}
 	return variants
 }
 // shouldExclude checks if a path should be excluded based on exclude patterns
@@ -161,7 +161,7 @@ func TestS3ListingCutShortIsAnError(t *testing.T) {
 }
 // TestS3URLPrefixKeyLayout pins the bucket keys an s3:// URL reads and
-// writes: the README's remote storage layout, with the prefix joined to
+// writes: the layout in docs/REPOSTRUCTURE.md, with the prefix joined to
 // each key by one "/". s3://b/p and s3://b/p/ must be the same
 // destination, or a host that writes the URL the other way finds no
 // snapshots. The listed object is put straight into the bucket, as
Author	SHA1	Message	Date
clawbot	6e4e8062a1	Fix the ten findings of the review on the destination, the marker file and chunk sizes check / check (push) Successful in 9m14s Details README.md: the "Getting started" destination advice, the Full Disk Access paragraph moved above the block with the reopen step, the mount-point case, the remote nuke failure sentence, the hostname setting, the upload order under "Sharing a destination", the confirmation list and the dev stamp sentence. configs/config.example.yml: the storage_url and blob_size_limit comments. docs/ARCHITECTURE.md and docs/DATAMODEL.md: the Snapshot ID and Marker file terms, and "marker file" in place of "destination marker". Model: fable-5-1	2026-11-11 09:11:51 +00:00
clawbot	a8b9c57e93	Fix the six findings of the review on config set, version and blob names check / check (push) Successful in 11m16s Details README.md: config set keeps comments and can remove blank lines; a config that does not load yet also has a value that is not a size refused; vaultik version reports a tag and adds a development-build note to every other version. docs/REPOSTRUCTURE.md and README.md: someone who has a blob's whole contents can compute its name and see whether it is stored. docs/DATAMODEL.md: snapshot list shows an unfinished snapshot as remote only once its manifest is at the destination; closing a full blob commits the first chunk of the next one. internal/cli/config.go: config set --help says the same about comments and blank lines. Model: fable-5-1	2026-11-11 08:28:40 +00:00
clawbot	f65fd2870b	Fix three review findings in help text, example config and terms check / check (push) Successful in 12m37s Details The snapshot restore help splits its 40-word permissions sentence in two. The snapshot-ID case leaves the parenthesis, and the temporary space requirement gets its own sentence. configs/config.example.yml gains a commented-out env entry, after hostname, so its first line, which says the file shows every option, is true again. docs/REPOSTRUCTURE.md defines remote key in Terms, before the marker-file section uses it. Model: fable-5-1	2026-11-11 07:55:31 +00:00
clawbot	2bf51d65e3	Fix nine review findings in the documents and help text check / check (push) Failing after 19m7s Details - snapshot verify and snapshot restore also list the destination for a snapshot ID that is not in the local index: README, REPOSTRUCTURE, ARCHITECTURE, DATAMODEL and the two commands' help. - prune on a separate machine deletes only unused data and does not remove old snapshots. - The --quiet usage string names everything the flag leaves out. - A warning written as a log record stays on stderr. - config set on a config that does not load refuses three things. - No test is held back by a build tag. - A plain docker build . adds no -dirty. - The backup's order of work records directories. - TODO.md says what parallel downloads would speed up and names restorePlan and restoreSweeper. Model: fable-5-1	2026-11-11 07:11:05 +00:00
sneak	1fcdfcc1c6	Fix the three review findings on README.md Lists `env` under Settings, so the sentences that say every key names a setting are true. Says how `--skip-errors` reports a file whose blob cannot be read. Counts the commands that refuse a changed destination as five, instead of pointing at the wrong list with "these". Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	a0efb2a5bc	Update the rewritten documents for the fixes merged into next The documents, the example config and the help text are rebased onto the fourteen fixes on next. The sentences those fixes made false now match the code: an unknown config key, a relative index_path and a value that is not a size are refused when the config loads; a backup saves the destination in the local index only after it writes the marker file, and no other command saves it; a backup skips the local index, its side files and the lock file; a binary from go install reports its tag and counts as a release; a declined deletion prompt exits 1 with "deletion not confirmed"; the S3 keys are set before an s3:// storage_url. No document quoted the old messages for an unknown snapshot ID, a wrong private key or an unreadable blob. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	da300fc91c	Reword the bootstrap entry and note the macOS paths README.md, "Entrypoints": the `script/bootstrap` item now says in two sentences that Go is installed when the machine has none, and that prettier is installed for the Markdown, with node and yarn when they are missing. README.md, the paragraph after the getting-started block: it now says the block uses macOS paths and that another system puts the destination in a directory of its own. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	6107f60e55	Update README.md for the four fixes merged into next `config init` writes the apps snapshot, which backs up /Applications, only on macOS, so the advice to edit the snapshots section first on another system goes. `version` now labels the commit date "commit date". `script/fmt` formats the Markdown with prettier as well as the Go code, `script/fmt-check` checks both, and `script/bootstrap` installs node and yarn for prettier. No document said that `snapshot verify` or `remote info` printed outside `internal/ui`, so nothing changes for that fix. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	718c10da10	Point links at the moved documents and config example ARCHITECTURE.md, TODO.md and config.example.yml moved to docs/ and configs/ on next. README.md now links to the new paths. docs/ARCHITECTURE.md and the roadmap in docs/TODO.md link to their sibling documents by file name, as docs/DATAMODEL.md and docs/REPOSTRUCTURE.md already do. The help text and configs/config.example.yml name none of the old paths. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	179af61c68	Format the rewritten documents with prettier Re-wraps README.md, docs/ARCHITECTURE.md, docs/DATAMODEL.md, docs/REPOSTRUCTURE.md and the roadmap in docs/TODO.md with `make fmt`, which now runs prettier on Markdown. Only line wrapping changes; the words are those that passed review at `4da102c`. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	1128b85653	Put the main rule before the exceptions in "What every command needs" The README section opened with the commands that need nothing and the config commands, and reached the requirement that covers most commands third. It now states that rule first, then lists the commands that need less. The facts are unchanged; only the order is. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	7c0a6d90c1	Add snapshot verify and restore to the commands a missing marker stops Given a remote key, `snapshot verify` and `snapshot restore` resolve it through `listAllRemoteSnapshotKeys`, which checks the destination marker, so they fail when the marker is missing. Given a snapshot ID they do not check it. Add them to the "These commands fail" list in `README.md` and `docs/REPOSTRUCTURE.md`. Model: fable-5-1	2026-11-11 07:01:54 +00:00
clawbot	d335e19521	Say that a symlink is recorded, not read A new or changed symlink goes through recordNonRegularFile in the scanner and is never opened. README.md, ARCHITECTURE.md and docs/DATAMODEL.md now say so where they describe what a backup reads, and README.md no longer lists a changed link target as a reason to read a file. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	7743ec621e	Define rclone's remote where it is first used The documents use "remote key" for something else, so "remote" as rclone's noun is now defined on first use: a remote is a storage service set up in rclone. Changed in `README.md` (the destination bullet), `docs/REPOSTRUCTURE.md` (the opening) and `config.example.yml` (the `storage_url` comment). Model: fable-5-1	2026-11-11 07:01:54 +00:00
clawbot	ef34926327	Say what a backup needs on an rclone remote and fix four more findings - `README.md`, `docs/REPOSTRUCTURE.md`, `internal/cli/snapshot.go`: a backup needs only write permission on S3; an rclone remote that can move objects also needs read and delete, and an upload there stops when its look-up is refused. - `docs/DATAMODEL.md`, `docs/REPOSTRUCTURE.md`: a restore reads no file, chunk or blob rows from the local index, but it does open it. - `README.md`: confirmation prompts go to stdout. - `internal/cli/remote.go`: `remote nuke` leaves the backup destination store empty, not a bucket prefix. - `README.md`: a TODO section, as `REPO_POLICIES.md` requires. Model: fable-5-1	2026-11-11 07:01:54 +00:00
clawbot	92928c86d4	Read all five documents against the code and fix what the sixth review found Fixes the four findings of the sixth review: `snapshot remove` with `--local-only`, the two comma-chained permission lists, six undefined terms, and the sentence about an unfinished snapshot in the local index. Then reads every sentence of `README.md`, `ARCHITECTURE.md`, `docs/DATAMODEL.md`, `docs/REPOSTRUCTURE.md` and the comments in `config.example.yml` against the code for four faults: a command's needs or behaviour stated without regard to a flag, a false absolute, three or more things chained with commas, and a technical word not defined on first use. Each sentence changed is listed on the pull request. The help string of `snapshot remove --dry-run` now says it needs no permission to delete. Model: fable-5-1	2026-11-11 07:01:54 +00:00
clawbot	864e9a76f4	Fix five README.md statements from the fifth review Under `--json`, `snapshot list` prints no warning for a snapshot with no manifest at the destination; the README now says so and points to `remote_present`. The stdout list names `snapshot create` as well as `snapshot restore` for the per-file error lines. `snapshot create` stops "before it writes anything at the destination", since the destination binding is saved first. `prune_grace_period` is a Go duration, so the README now says only which units it rejects. The commands that run while a backup is in progress are a bulleted list. Model: fable-5-1	2026-11-11 07:01:54 +00:00
clawbot	2662cf44be	Correct README.md on where warnings go and on which make targets use Docker and script/ Warnings and the per-file errors of `snapshot restore` are written to stdout through `internal/ui`, so the stdout list names them and the redirect example says a file captures them. Only `make test` and `make lint` run inside Docker; `make test-coverage` runs on the host. Some `Makefile` targets (`build`, `clean`, `deps`, `test-coverage`, `local`, `install`) call no `script/` entry, so "Most" replaces "only". Fixes the two findings of the fourth review of #399. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	72ea483312	Say which commands need the private key, in README.md and config.example.yml The last sentence of "The private key" and the matching comment in config.example.yml named only the restore machine. snapshot verify --deep needs the key too. Both now name both commands. The "Everyday use" block gains a comment saying its next three commands run on a machine that holds the private key. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	49af986fd1	Fix the four findings of the second review README.md: the opening no longer says the private key is needed only to restore; a backup never needs it, a deep verify does. ARCHITECTURE.md: the concurrency note says each open database uses one connection, since the metadata export opens a second one. docs/REPOSTRUCTURE.md: the size leak is about a secret near the outsider's data in the same blob, not the same chunk, because all new chunks of a blob share one zstd stream. Every path used as link text in the four documents is in backticks. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	281033239e	Fix other sentences that carry faults the review named A search of the four documents, the help text and config.example.yml for the review's faults found these. - prune help, and snapshot purge and snapshot list in README.md: a snapshot is dropped or reported when its manifest is gone, not its metadata or directory. - README.md, version: the date printed is the commit date. - README.md, Configuration: names the commands that start without a config that loads. - README.md, snapshot purge: "each such snapshot" named nothing. - README.md and docs/DATAMODEL.md: a backup can upload unchanged content again, so "uploads only what changed" was false. - docs/REPOSTRUCTURE.md: db.zst.age holds more than one snapshot's rows. - ARCHITECTURE.md: "database records" read as a noun. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	fb58b9cf21	Fix three sentences that kept faults the review named The review of the rewrite named these faults in other files. The same sentences were still in two documents. - docs/REPOSTRUCTURE.md, "The marker file": "index records" read as a noun. It now says the next backup would not upload data that the local index says is stored. - docs/REPOSTRUCTURE.md, "Chunk boundaries use no secret": a chunk is not always between 2.5 MB and 40 MB. Only a file's last chunk can be smaller than a quarter of chunk_size. - ARCHITECTURE.md, "Snapshot metadata": the export does not delete everything outside the snapshot. It now says what is deleted, and that local_meta and schema_migrations are kept. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	536ee8bb75	Fix the review findings in --help and config.example.yml snapshot list says why a remote-only snapshot is listed by its remote key. snapshot verify says what access to the destination it needs. The note on the age-keygen file is two sentences in snapshot verify and snapshot restore. remote nuke says "backup destination store". config.example.yml names the commands that run without a snapshot. Only help strings and comments change. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	a592a62672	Move the roadmap into TODO.md and fix a test comment The rewrite removed the README's roadmap, which TODO.md linked to. The roadmap items are now under Future Steps in TODO.md, in place of the link. A comment in internal/storage/s3_test.go cited the README for the destination layout. That layout is in docs/REPOSTRUCTURE.md. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	86c8d30def	Fix the review findings in the three design documents ARCHITECTURE.md: the chunk size limits, the commit date, the cases in which a backup skips its prune, and two sentences that read badly. docs/DATAMODEL.md: which repository type serves which table, what the metadata export deletes and keeps, the blobs table in a restore, and when a chunk row is deleted. docs/REPOSTRUCTURE.md: how an object's size relates to its contents, what blob sizes can leak and when, and a sentence moved out of the prune rules. All three now say that prune drops a local snapshot whose manifest is gone, not whose directory is gone. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	c1266f3988	Fix the review findings in README.md The first sentence now names the licence, the kind of program, the language and the author. The README gains a Design section, and its sections carry the names REPO_POLICIES.md asks for: Rationale and Getting started, which now holds the install command. Entrypoints opens with the Scripts to Rule Them All statement. Corrected: when a file is read again, the chunk size limits, and the first prune step, which looks for a snapshot's manifest. Two sentences that misparsed are reworded. A test comment that named the README quickstart follows the rename. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	6a2e2aaecc	Reword README.md sentences that read badly on a second pass Split sentences that carried two or three ideas, turned the cases in which a backup does not find a mount point into a list, and named the figures that remote info reports as unknown. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	871cef1284	Say what each command needs in --help and config.example.yml snapshot verify --help now says --deep needs the age private key, from VAULTIK_AGE_SECRET_KEY or age_secret_key, as snapshot restore does. Both now name the config key as well as the environment variable. The help of snapshot create, list, purge, remove, prune, remote info and remote nuke says which access to the destination it needs. config.example.yml gains age_secret_key, and no longer tells the reader to pipe age-keygen into grep, which discards the private key. Only help strings and comments change. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	f016cf54ac	Turn a comma chain in docs/REPOSTRUCTURE.md into a list Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	c8ad289e46	Rewrite docs/REPOSTRUCTURE.md sentence by sentence Rewritten from scratch against the code on next: the layout of the destination, what it protects, the accepted risks, the order of writes and the rules that keep prune from deleting data in use. Corrected: a restore needs only db.zst.age and the blobs it reads, not the manifest. The snapshot ID form without a name is gone, since no command creates one. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	1d743e2255	Turn two comma chains in ARCHITECTURE.md into lists Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	137ef59cae	Rewrite docs/DATAMODEL.md sentence by sentence Rewritten from scratch against internal/database on next. Every table and column in schema/001.sql is listed, with the indexes. Added what the old text lacked: files.source_path, the local_meta and schema_migrations tables, and five indexes. Corrected what it had wrong: blobs.blob_hash holds a placeholder, not NULL, until the blob is closed; blob_chunks rows are written when the blob is closed; queries are parameterised, not prepared. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	5afcd4f6b5	Rewrite ARCHITECTURE.md sentence by sentence Rewritten from scratch against the code on next: the packages, the fx wiring, and the steps of a backup, a restore, a verify and a prune. Names that no longer exist in the code are gone: the S3 client as the upload step, blob.BlobWithReader, PurgeSnapshots and the repository interfaces. The storage layer is described through storage.Storer. Model: fable-5-1	2026-11-11 07:01:54 +00:00
sneak	8058b775d6	Rewrite README.md sentence by sentence Rewritten from scratch against the code on next: what the program is, how to use it, a command reference that says what each command needs, the configuration reference as nested bullets, then development and releasing. snapshot verify --deep now says it needs the private key, from VAULTIK_AGE_SECRET_KEY or age_secret_key, as snapshot restore does. The roadmap section is removed: it described features that are not built. Model: fable-5-1	2026-11-11 07:01:54 +00:00