Document snapshot files and fix the six findings of the last review

Rebased onto next, which added `snapshot files` and refuses an exclude
pattern that does not compile. The README, docs/REPOSTRUCTURE.md, the
example config and the help text now name `snapshot files` wherever
they list the commands that need the private key, read or list the
destination, run without the lock or write a decrypted database. The
config load conditions name exclude patterns, and docs/ARCHITECTURE.md
lists internal/exclude.

The six findings of
#399 (comment) use the
wording that review gives.

Model: opus-5-5
This commit is contained in:
clawbot authored and sneak committed 2026-11-17 00:21:37 +00:00
1 parent 3d5c35d2c1
commit 45d775709d
7 files changed
+145 -48

No files matched your search

+102 -26
View File
@@ -27,8 +27,8 @@ It was written to meet these requirements together:
- Encryption with [age](https://age-encryption.org/), to one or more public - Encryption with [age](https://age-encryption.org/), to one or more public
keys. age uses X25519 keys and the ChaCha20-Poly1305 cipher. keys. age uses X25519 keys and the ChaCha20-Poly1305 cipher.
- Incremental backups. A file is read again only when it has changed, or when - Incremental backups. An unchanged file is usually not read again. See
part of its content has to be stored again. [`snapshot create`](#snapshot-create) for when it is.
- Deduplication. Identical content is stored once, across files and across - Deduplication. Identical content is stored once, across files and across
snapshots. snapshots.
- Compression with [zstd](https://github.com/facebook/zstd), at a level you - Compression with [zstd](https://github.com/facebook/zstd), at a level you
@@ -46,7 +46,7 @@ It was written to meet these requirements together:
- **Snapshot**: one backup of the paths listed under one name in the config, - **Snapshot**: one backup of the paths listed under one name in the config,
taken at one time. taken at one time.
- **Snapshot ID**: the name of a snapshot, in the form - **Snapshot ID**: the identifier of one snapshot, in the form
`<hostname>_<snapshot-name>_<timestamp>`. An example is `<hostname>_<snapshot-name>_<timestamp>`. An example is
`server1_home_2025-06-01T12:00:00Z`. `server1_home_2025-06-01T12:00:00Z`.
- **Destination**: the place the backups are stored. - **Destination**: the place the backups are stored.
@@ -185,6 +185,7 @@ vaultik config get <key>
vaultik config set <key> <value> vaultik config set <key> <value>
vaultik snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>] vaultik snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>]
vaultik snapshot list [--json] vaultik snapshot list [--json]
vaultik snapshot files <snapshot-id> [path] [--json]
vaultik snapshot verify <snapshot-id> [--deep] [--json] vaultik snapshot verify <snapshot-id> [--deep] [--json]
vaultik snapshot purge (--keep-latest | --older-than <duration>) [--snapshot <name>...] [--force] vaultik snapshot purge (--keep-latest | --older-than <duration>) [--snapshot <name>...] [--force]
vaultik snapshot remove <snapshot-id> [--dry-run] [--force] [--local-only] [--json] vaultik snapshot remove <snapshot-id> [--dry-run] [--force] [--local-only] [--json]
@@ -470,6 +471,55 @@ Under `--json`, every warning above goes to stderr, except the one about a
snapshot with no manifest at the destination. That snapshot has `remote_present` snapshot with no manifest at the destination. That snapshot has `remote_present`
set to `false`. set to `false`.
### `snapshot files <snapshot-id> [path]`
Lists the files, symlinks and directories in a snapshot, without restoring it.
The first argument is a snapshot ID, a remote key or the leading part of a
remote key.
It needs:
- The age private key. See [The private key](#the-private-key).
- Permission to read at the destination, and to list it when you give a remote
key, or a snapshot ID that is not in the local index.
The file list is stored only in the snapshot's encrypted database. The command
downloads and decrypts that database. It writes the decrypted database to a
temporary directory and removes it afterwards. See
[Temporary files](#temporary-files).
With a path, the command lists only that path and what is under it. The path
names a file or a directory by its original absolute path. A path that matches
nothing in the snapshot is an error.
The table is sorted by path. It has these columns:
- `MODE`: the type and the permissions. See below.
- `SIZE`: the size in bytes. It is `0` for a symlink or a directory.
- `MODIFIED`: the modification time, in UTC.
- `PATH`: the original absolute path.
In the table, a path that holds a newline or another non-printable character is
shown in double quotes, with that character escaped, as in `"a\nb"`.
The mode looks like the one `ls -l` shows, as in `-rw-r--r--` or `drwxr-xr-x`.
The letters before the nine permission characters show the type and any special
bit. These letters differ from `ls -l`:
- `L`: a symlink, which `ls -l` shows as `l`.
- `u`: setuid, which `ls -l` shows as `s` in place of the owner's `x`.
- `g`: setgid, which `ls -l` shows as `s` in place of the group's `x`.
- `t`: sticky, which `ls -l` shows as `t` in place of the others' `x`.
A sticky directory, for example, is shown as `dtrwxrwxrwx`.
`--json` prints a JSON array instead. Each entry has these fields:
- `path`
- `size`
- `mode`: written as in the table.
- `mtime`: the modification time.
### `snapshot verify <snapshot-id>` ### `snapshot verify <snapshot-id>`
Checks a snapshot at the destination. The argument is a snapshot ID, a remote Checks a snapshot at the destination. The argument is a snapshot ID, a remote
@@ -487,7 +537,8 @@ snapshot ID that is not in the local index.
Flags: Flags:
- `--deep`: download, decrypt and re-hash every blob. See below. - `--deep`: download, decrypt and re-hash every blob. See below.
- `--json`: print the result as JSON. - `--json`: print the result as JSON. A failed check is reported in that
document, not on stderr.
The command exits non-zero when a check fails or cannot be made. The command exits non-zero when a check fails or cannot be made.
@@ -814,6 +865,8 @@ The config loads when it is valid YAML and all of these hold:
- Every entry in `age_recipients` is an age X25519 public key. - Every entry in `age_recipients` is an age X25519 public key.
- `snapshots` has at least one name, and every name has at least one path. - `snapshots` has at least one name, and every name has at least one path.
- Every exclude pattern is valid. The error for an invalid one names the setting
and quotes the pattern.
- A destination is set. Its `storage_url` has one of the three schemes under - A destination is set. Its `storage_url` has one of the three schemes under
[Destinations](#destinations). An S3 destination has credentials. [Destinations](#destinations). An S3 destination has credentials.
- Every `${ENV:NAME}` names an environment variable with a non-empty value. - Every `${ENV:NAME}` names an environment variable with a non-empty value.
@@ -859,10 +912,10 @@ A command that reads the config fails when the path from `--config` or
- `age_secret_key` - `age_secret_key`
- The age private key. - The age private key.
- Default: not set. - Default: not set.
- Only `snapshot restore` and `snapshot verify --deep` use it. - Only the commands that decrypt data use it. See
- The `VAULTIK_AGE_SECRET_KEY` environment variable overrides it.
- Do not set it on a machine you back up. See
[The private key](#the-private-key). [The private key](#the-private-key).
- The `VAULTIK_AGE_SECRET_KEY` environment variable overrides it.
- Do not set it on a machine you back up.
- `snapshots` - `snapshots`
- The named sets of paths to back up. Required. - The named sets of paths to back up. Required.
- `snapshots.<name>.paths`: the paths to back up under this name. - `snapshots.<name>.paths`: the paths to back up under this name.
@@ -948,8 +1001,13 @@ config loads. The error names the setting and quotes the value.
### The private key ### The private key
`snapshot restore` and `snapshot verify --deep` decrypt data, so they need the These commands decrypt data, so they need the age private key:
age private key. No other command needs it.
- `snapshot files`
- `snapshot restore`
- `snapshot verify --deep`
No other command needs it.
Give the key in one of two ways: Give the key in one of two ways:
@@ -970,7 +1028,7 @@ export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
Do not put the private key on a machine you back up. Anyone who breaks into that Do not put the private key on a machine you back up. Anyone who breaks into that
machine could then decrypt its backups. Supply the key only on the machine where machine could then decrypt its backups. Supply the key only on the machine where
you run `snapshot restore` or `snapshot verify --deep`. you run the commands that need it.
### Exclude patterns ### Exclude patterns
@@ -1053,11 +1111,12 @@ A leftover `.partial` file is ignored. You can delete it.
Commands need these permissions at the destination: Commands need these permissions at the destination:
- `snapshot create` without `--prune` writes objects. On S3 it needs no other - `snapshot create` without `--prune` writes objects. With an `s3://`
permission, so a machine you back up can hold credentials that can neither destination it needs no other permission, so a machine you back up can hold
read nor delete. credentials that can neither read nor delete.
- These commands list and read: - These commands list and read:
- `snapshot list` - `snapshot list`
- `snapshot files`
- `snapshot verify` - `snapshot verify`
- `snapshot restore` - `snapshot restore`
- `remote info` - `remote info`
@@ -1067,13 +1126,20 @@ Commands need these permissions at the destination:
- `remote nuke` - `remote nuke`
- `prune` and `snapshot create --prune` list, read and delete. - `prune` and `snapshot create --prune` list, read and delete.
An rclone remote that can move objects also needs permission to read and delete. On an rclone remote, a backup can also need permission to read and delete. Which
An upload there looks up the object it replaces and deletes it. It stops when ones depends on the storage service. An upload to a remote that can move objects
the look-up is refused. looks up the object it replaces and deletes it. It stops when the look-up is
refused.
`snapshot verify` and `snapshot restore` list only when you give a remote key, These commands list only when you give a remote key, or a snapshot ID that is
or a snapshot ID that is not in the local index. `snapshot remove --local-only` not in the local index:
does not reach the destination. `snapshot remove --dry-run` deletes nothing.
- `snapshot files`
- `snapshot verify`
- `snapshot restore`
`snapshot remove --local-only` does not reach the destination.
`snapshot remove --dry-run` deletes nothing.
A machine that runs `snapshot create --prune` can delete its own backups. To A machine that runs `snapshot create --prune` can delete its own backups. To
keep it to write-only credentials, run `prune` from a separate, trusted machine. keep it to write-only credentials, run `prune` from a separate, trusted machine.
@@ -1102,8 +1168,11 @@ not exist, commands treat the destination as not present:
- `prune` - `prune`
- `snapshot purge` - `snapshot purge`
- `remote nuke` - `remote nuke`
- `snapshot verify` and `snapshot restore`, when given a remote key, or a - These commands fail when given a remote key, or a snapshot ID that is not in
snapshot ID that is not in the local index the local index:
- `snapshot files`
- `snapshot verify`
- `snapshot restore`
- `snapshot list` shows the local index alone. - `snapshot list` shows the local index alone.
- `snapshot remove` deletes nothing at the destination. - `snapshot remove` deletes nothing at the destination.
@@ -1123,9 +1192,8 @@ there.
A backup saves the configured destination in the local index once it has written A backup saves the configured destination in the local index once it has written
the marker file there. A backup that fails earlier saves nothing, so a mistyped the marker file there. A backup that fails earlier saves nothing, so a mistyped
`storage_url` can be corrected. `prune`, `snapshot purge`, `snapshot remove` and `storage_url` can be corrected. No other command saves it. Five commands refuse
`remote nuke` never save it. Five commands refuse to run when the config names a to run when the config names a different destination from the saved one:
different destination from the saved one:
- `snapshot create` - `snapshot create`
- `snapshot purge` - `snapshot purge`
@@ -1192,6 +1260,7 @@ No other command takes the lock. These commands run while a backup is in
progress: progress:
- `snapshot list` - `snapshot list`
- `snapshot files`
- `snapshot verify` - `snapshot verify`
- `snapshot restore` - `snapshot restore`
- `info` - `info`
@@ -1218,6 +1287,8 @@ stderr carries the rest:
- log records, including those from `--verbose` and `--debug` - log records, including those from `--verbose` and `--debug`
- the error that a failed command ends with - the error that a failed command ends with
`snapshot verify --json` puts its error in the JSON document instead.
So `vaultik snapshot list > out.txt` captures the list and the warnings printed So `vaultik snapshot list > out.txt` captures the list and the warnings printed
below it. The banner, log records and a final error stay on your terminal. A below it. The banner, log records and a final error stay on your terminal. A
warning written as a log record, such as the one about a config file others can warning written as a log record, such as the one about a config file others can
@@ -1283,8 +1354,13 @@ A backup needs free temporary space for these:
With `TMPDIR` not set, SQLite writes one of those index copies to `/var/tmp`. With `TMPDIR` not set, SQLite writes one of those index copies to `/var/tmp`.
A restore keeps the downloaded blobs on disk, decrypted, until no remaining file A restore keeps the downloaded blobs on disk, decrypted, until no remaining file
needs them. `snapshot restore` and `snapshot verify --deep` also write the needs them.
decrypted snapshot database there.
These commands write the decrypted snapshot database to the temporary directory:
- `snapshot files`
- `snapshot restore`
- `snapshot verify --deep`
These files are readable only by their owner, and are removed when the command These files are readable only by their owner, and are removed when the command
ends. A `kill -9` or a power loss leaves them behind. Delete any leftover ends. A `kill -9` or a power loss leaves them behind. Delete any leftover
+8 -5
View File
@@ -16,17 +16,20 @@ age_recipients:
- age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz - age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz
# Age private key for decryption # Age private key for decryption
# Only snapshot restore and snapshot verify --deep need it. The # Only these commands need it:
# VAULTIK_AGE_SECRET_KEY environment variable overrides this setting # snapshot files
# snapshot restore
# snapshot verify --deep
# The VAULTIK_AGE_SECRET_KEY environment variable overrides this setting
# Do not set it on a machine you back up. Supply the environment variable # Do not set it on a machine you back up. Supply the environment variable
# only on a machine where you run snapshot restore or snapshot verify --deep # only on a machine where you run one of those commands
# Default: not set # Default: not set
#age_secret_key: YOUR_AGE_PRIVATE_KEY #age_secret_key: YOUR_AGE_PRIVATE_KEY
# Named snapshots - each snapshot can contain multiple paths # Named snapshots - each snapshot can contain multiple paths
# Each snapshot gets its own ID and can have snapshot-specific excludes # Each snapshot gets its own ID and can have snapshot-specific excludes
# Required: at least one snapshot with at least one path. Only the config # Required: at least one snapshot, and every snapshot needs at least one path.
# commands, version and completion run without one # Only the config commands, version and completion run without a snapshot
snapshots: snapshots:
testing: testing:
paths: paths:
+3 -1
View File
@@ -16,7 +16,7 @@ Two other documents cover the stored data:
## Terms ## Terms
- **Snapshot**: one backup of the paths listed under one name in the config. - **Snapshot**: one backup of the paths listed under one name in the config.
- **Snapshot ID**: the name of a snapshot, in the form - **Snapshot ID**: the identifier of one snapshot, in the form
`<hostname>_<snapshot-name>_<timestamp>`. `<hostname>_<snapshot-name>_<timestamp>`.
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash - **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
of its content. of its content.
@@ -69,6 +69,8 @@ source files
S3, a local directory and [rclone](https://rclone.org/). S3, a local directory and [rclone](https://rclone.org/).
- `internal/s3`: the S3 client that the S3 `Storer` uses. - `internal/s3`: the S3 client that the S3 `Storer` uses.
- `internal/config`: loads and checks the config file. - `internal/config`: loads and checks the config file.
- `internal/exclude`: compiles exclude patterns, for the config check and for
the `Scanner`.
- `internal/ui`: formats the status lines, warnings and errors a user sees. - `internal/ui`: formats the status lines, warnings and errors a user sees.
- `internal/log`: the logger. It writes to stderr. - `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that `RunWithApp` takes for a command that - `internal/pidlock`: the lock that `RunWithApp` takes for a command that
+1 -1
View File
@@ -17,7 +17,7 @@ copy that was uploaded with the snapshot.
## Terms ## Terms
- **Snapshot**: one backup of the paths listed under one name in the config. - **Snapshot**: one backup of the paths listed under one name in the config.
- **Snapshot ID**: the name of a snapshot, in the form - **Snapshot ID**: the identifier of one snapshot, in the form
`<hostname>_<snapshot-name>_<timestamp>`. `<hostname>_<snapshot-name>_<timestamp>`.
- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that - **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
content. content.
+21 -10
View File
@@ -10,7 +10,7 @@ is a storage service set up in rclone. The layout is the same on all three.
## Terms ## Terms
- **Snapshot**: one backup of the paths listed under one name in the config. - **Snapshot**: one backup of the paths listed under one name in the config.
- **Snapshot ID**: the name of a snapshot, in the form - **Snapshot ID**: the identifier of one snapshot, in the form
`<hostname>_<snapshot-name>_<timestamp>`. `<hostname>_<snapshot-name>_<timestamp>`.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory - **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
under `metadata/`. See [Remote keys](#remote-keys). under `metadata/`. See [Remote keys](#remote-keys).
@@ -61,8 +61,11 @@ not exist, commands treat the destination as not present:
- `prune` - `prune`
- `snapshot purge` - `snapshot purge`
- `remote nuke` - `remote nuke`
- `snapshot verify` and `snapshot restore`, when given a remote key, or a - These commands fail when given a remote key, or a snapshot ID that is not in
snapshot ID that is not in the local index the local index:
- `snapshot files`
- `snapshot verify`
- `snapshot restore`
- `snapshot list` shows the local index alone. - `snapshot list` shows the local index alone.
- `snapshot remove` deletes nothing at the destination. - `snapshot remove` deletes nothing at the destination.
@@ -309,6 +312,7 @@ These commands write unencrypted data to the temporary directory:
is no limit on their total size. is no limit on their total size.
- `snapshot restore` writes the decrypted database to - `snapshot restore` writes the decrypted database to
`vaultik-restore-*/snapshot.db`. `vaultik-restore-*/snapshot.db`.
- `snapshot files` writes the decrypted database to the same place.
- `snapshot verify --deep` writes the decrypted database to - `snapshot verify --deep` writes the decrypted database to
`vaultik-verify-*/snapshot.db`. `vaultik-verify-*/snapshot.db`.
- `snapshot create` writes a copy of the local index to - `snapshot create` writes a copy of the local index to
@@ -322,11 +326,12 @@ Trust the temporary directory as much as the directory you restore into.
### Commands need different permissions ### Commands need different permissions
`snapshot create` without `--prune` writes objects. On S3 it needs no other `snapshot create` without `--prune` writes objects. With an `s3://` destination
permission. Other commands need more: it needs no other permission. Other commands need more:
- These commands list and read: - These commands list and read:
- `snapshot list` - `snapshot list`
- `snapshot files`
- `snapshot verify` - `snapshot verify`
- `snapshot restore` - `snapshot restore`
- `remote info` - `remote info`
@@ -336,12 +341,18 @@ permission. Other commands need more:
- `remote nuke` - `remote nuke`
- `prune` lists, reads and deletes. - `prune` lists, reads and deletes.
An rclone remote that can move objects also needs permission to read and delete. On an rclone remote, a backup can also need permission to read and delete. Which
An upload there looks up the object it replaces and deletes it. It stops when ones depends on the storage service. An upload to a remote that can move objects
the look-up is refused. looks up the object it replaces and deletes it. It stops when the look-up is
refused.
These commands list only when they are given a remote key, or a snapshot ID that
is not in the local index:
- `snapshot files`
- `snapshot verify`
- `snapshot restore`
`snapshot verify` and `snapshot restore` list only when they are given a remote
key, or a snapshot ID that is not in the local index.
`snapshot remove --local-only` does not reach the destination. `snapshot remove --local-only` does not reach the destination.
`snapshot remove --dry-run` deletes nothing. `snapshot remove --dry-run` deletes nothing.
+1 -1
View File
@@ -38,7 +38,7 @@ func NewRootCommand() *cobra.Command {
Long: `vaultik is an incremental backup tool that encrypts data to age public Long: `vaultik is an incremental backup tool that encrypts data to age public
keys and uploads it to an S3-compatible object store, a local directory or keys and uploads it to an S3-compatible object store, a local directory or
an rclone remote. The backed-up machine needs no private key: only an rclone remote. The backed-up machine needs no private key: only
'snapshot restore' and 'snapshot verify --deep' need one.`, 'snapshot files', 'snapshot restore' and 'snapshot verify --deep' need one.`,
SilenceUsage: true, SilenceUsage: true,
// Bare 'vaultik' (no subcommand): print help. The banner is // Bare 'vaultik' (no subcommand): print help. The banner is
// printed once at process startup by Entry, before cobra // printed once at process startup by Entry, before cobra
+9 -4
View File
@@ -74,8 +74,8 @@ If no names are provided, all configured snapshots are created.
Needs at least one public key in age_recipients, and permission to write Needs at least one public key in age_recipients, and permission to write
objects at the backup destination store. It does not need the age private objects at the backup destination store. It does not need the age private
key. With --prune it also needs permission to list, read and delete objects key. With --prune it also needs permission to list, read and delete objects
there. An rclone remote that can move objects also needs permission to there. On an rclone remote, a backup can also need permission to read and
read and delete. delete. Which ones depends on the storage service.
The config is read from the path given by --config or VAULTIK_CONFIG; The config is read from the path given by --config or VAULTIK_CONFIG;
otherwise from the platform config directory (~/.config/vaultik/config.yml otherwise from the platform config directory (~/.config/vaultik/config.yml
@@ -209,8 +209,13 @@ by the remote key that 'snapshot list' prints for a remote-only snapshot
(an unambiguous leading part is enough). (an unambiguous leading part is enough).
The file list is only in the snapshot's encrypted database, so this The file list is only in the snapshot's encrypted database, so this
downloads and decrypts it, and requires the age private key in the downloads and decrypts it. It requires the age private key, in the
VAULTIK_AGE_SECRET_KEY environment variable, as 'snapshot restore' does.`, VAULTIK_AGE_SECRET_KEY environment variable or as age_secret_key in the
config file; the environment variable wins.
Also needs permission to read the backup destination store, and to list
it when given a remote key, or a snapshot ID that is not in the local
index.`,
Args: cobra.RangeArgs(1, filesMaxArgs), Args: cobra.RangeArgs(1, filesMaxArgs),
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
path := "" path := ""