Document snapshot files and fix the six findings of the last review
Rebased onto next, which added `snapshot files` and refuses an exclude pattern that does not compile. The README, docs/REPOSTRUCTURE.md, the example config and the help text now name `snapshot files` wherever they list the commands that need the private key, read or list the destination, run without the lock or write a decrypted database. The config load conditions name exclude patterns, and docs/ARCHITECTURE.md lists internal/exclude. The six findings of #399 (comment) use the wording that review gives. Model: opus-5-5
This commit is contained in:
1 parent
3d5c35d2c1
commit
45d775709d
7 files changed
+145
-48
No files matched your search
@@ -27,8 +27,8 @@ It was written to meet these requirements together:
|
||||
|
||||
- Encryption with [age](https://age-encryption.org/), to one or more public
|
||||
keys. age uses X25519 keys and the ChaCha20-Poly1305 cipher.
|
||||
- Incremental backups. A file is read again only when it has changed, or when
|
||||
part of its content has to be stored again.
|
||||
- Incremental backups. An unchanged file is usually not read again. See
|
||||
[`snapshot create`](#snapshot-create) for when it is.
|
||||
- Deduplication. Identical content is stored once, across files and across
|
||||
snapshots.
|
||||
- Compression with [zstd](https://github.com/facebook/zstd), at a level you
|
||||
@@ -46,7 +46,7 @@ It was written to meet these requirements together:
|
||||
|
||||
- **Snapshot**: one backup of the paths listed under one name in the config,
|
||||
taken at one time.
|
||||
- **Snapshot ID**: the name of a snapshot, in the form
|
||||
- **Snapshot ID**: the identifier of one snapshot, in the form
|
||||
`<hostname>_<snapshot-name>_<timestamp>`. An example is
|
||||
`server1_home_2025-06-01T12:00:00Z`.
|
||||
- **Destination**: the place the backups are stored.
|
||||
@@ -185,6 +185,7 @@ vaultik config get <key>
|
||||
vaultik config set <key> <value>
|
||||
vaultik snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>]
|
||||
vaultik snapshot list [--json]
|
||||
vaultik snapshot files <snapshot-id> [path] [--json]
|
||||
vaultik snapshot verify <snapshot-id> [--deep] [--json]
|
||||
vaultik snapshot purge (--keep-latest | --older-than <duration>) [--snapshot <name>...] [--force]
|
||||
vaultik snapshot remove <snapshot-id> [--dry-run] [--force] [--local-only] [--json]
|
||||
@@ -470,6 +471,55 @@ Under `--json`, every warning above goes to stderr, except the one about a
|
||||
snapshot with no manifest at the destination. That snapshot has `remote_present`
|
||||
set to `false`.
|
||||
|
||||
### `snapshot files <snapshot-id> [path]`
|
||||
|
||||
Lists the files, symlinks and directories in a snapshot, without restoring it.
|
||||
The first argument is a snapshot ID, a remote key or the leading part of a
|
||||
remote key.
|
||||
|
||||
It needs:
|
||||
|
||||
- The age private key. See [The private key](#the-private-key).
|
||||
- Permission to read at the destination, and to list it when you give a remote
|
||||
key, or a snapshot ID that is not in the local index.
|
||||
|
||||
The file list is stored only in the snapshot's encrypted database. The command
|
||||
downloads and decrypts that database. It writes the decrypted database to a
|
||||
temporary directory and removes it afterwards. See
|
||||
[Temporary files](#temporary-files).
|
||||
|
||||
With a path, the command lists only that path and what is under it. The path
|
||||
names a file or a directory by its original absolute path. A path that matches
|
||||
nothing in the snapshot is an error.
|
||||
|
||||
The table is sorted by path. It has these columns:
|
||||
|
||||
- `MODE`: the type and the permissions. See below.
|
||||
- `SIZE`: the size in bytes. It is `0` for a symlink or a directory.
|
||||
- `MODIFIED`: the modification time, in UTC.
|
||||
- `PATH`: the original absolute path.
|
||||
|
||||
In the table, a path that holds a newline or another non-printable character is
|
||||
shown in double quotes, with that character escaped, as in `"a\nb"`.
|
||||
|
||||
The mode looks like the one `ls -l` shows, as in `-rw-r--r--` or `drwxr-xr-x`.
|
||||
The letters before the nine permission characters show the type and any special
|
||||
bit. These letters differ from `ls -l`:
|
||||
|
||||
- `L`: a symlink, which `ls -l` shows as `l`.
|
||||
- `u`: setuid, which `ls -l` shows as `s` in place of the owner's `x`.
|
||||
- `g`: setgid, which `ls -l` shows as `s` in place of the group's `x`.
|
||||
- `t`: sticky, which `ls -l` shows as `t` in place of the others' `x`.
|
||||
|
||||
A sticky directory, for example, is shown as `dtrwxrwxrwx`.
|
||||
|
||||
`--json` prints a JSON array instead. Each entry has these fields:
|
||||
|
||||
- `path`
|
||||
- `size`
|
||||
- `mode`: written as in the table.
|
||||
- `mtime`: the modification time.
|
||||
|
||||
### `snapshot verify <snapshot-id>`
|
||||
|
||||
Checks a snapshot at the destination. The argument is a snapshot ID, a remote
|
||||
@@ -487,7 +537,8 @@ snapshot ID that is not in the local index.
|
||||
Flags:
|
||||
|
||||
- `--deep`: download, decrypt and re-hash every blob. See below.
|
||||
- `--json`: print the result as JSON.
|
||||
- `--json`: print the result as JSON. A failed check is reported in that
|
||||
document, not on stderr.
|
||||
|
||||
The command exits non-zero when a check fails or cannot be made.
|
||||
|
||||
@@ -814,6 +865,8 @@ The config loads when it is valid YAML and all of these hold:
|
||||
|
||||
- Every entry in `age_recipients` is an age X25519 public key.
|
||||
- `snapshots` has at least one name, and every name has at least one path.
|
||||
- Every exclude pattern is valid. The error for an invalid one names the setting
|
||||
and quotes the pattern.
|
||||
- A destination is set. Its `storage_url` has one of the three schemes under
|
||||
[Destinations](#destinations). An S3 destination has credentials.
|
||||
- Every `${ENV:NAME}` names an environment variable with a non-empty value.
|
||||
@@ -859,10 +912,10 @@ A command that reads the config fails when the path from `--config` or
|
||||
- `age_secret_key`
|
||||
- The age private key.
|
||||
- Default: not set.
|
||||
- Only `snapshot restore` and `snapshot verify --deep` use it.
|
||||
- The `VAULTIK_AGE_SECRET_KEY` environment variable overrides it.
|
||||
- Do not set it on a machine you back up. See
|
||||
- Only the commands that decrypt data use it. See
|
||||
[The private key](#the-private-key).
|
||||
- The `VAULTIK_AGE_SECRET_KEY` environment variable overrides it.
|
||||
- Do not set it on a machine you back up.
|
||||
- `snapshots`
|
||||
- The named sets of paths to back up. Required.
|
||||
- `snapshots.<name>.paths`: the paths to back up under this name.
|
||||
@@ -948,8 +1001,13 @@ config loads. The error names the setting and quotes the value.
|
||||
|
||||
### The private key
|
||||
|
||||
`snapshot restore` and `snapshot verify --deep` decrypt data, so they need the
|
||||
age private key. No other command needs it.
|
||||
These commands decrypt data, so they need the age private key:
|
||||
|
||||
- `snapshot files`
|
||||
- `snapshot restore`
|
||||
- `snapshot verify --deep`
|
||||
|
||||
No other command needs it.
|
||||
|
||||
Give the key in one of two ways:
|
||||
|
||||
@@ -970,7 +1028,7 @@ export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
|
||||
|
||||
Do not put the private key on a machine you back up. Anyone who breaks into that
|
||||
machine could then decrypt its backups. Supply the key only on the machine where
|
||||
you run `snapshot restore` or `snapshot verify --deep`.
|
||||
you run the commands that need it.
|
||||
|
||||
### Exclude patterns
|
||||
|
||||
@@ -1053,11 +1111,12 @@ A leftover `.partial` file is ignored. You can delete it.
|
||||
|
||||
Commands need these permissions at the destination:
|
||||
|
||||
- `snapshot create` without `--prune` writes objects. On S3 it needs no other
|
||||
permission, so a machine you back up can hold credentials that can neither
|
||||
read nor delete.
|
||||
- `snapshot create` without `--prune` writes objects. With an `s3://`
|
||||
destination it needs no other permission, so a machine you back up can hold
|
||||
credentials that can neither read nor delete.
|
||||
- These commands list and read:
|
||||
- `snapshot list`
|
||||
- `snapshot files`
|
||||
- `snapshot verify`
|
||||
- `snapshot restore`
|
||||
- `remote info`
|
||||
@@ -1067,13 +1126,20 @@ Commands need these permissions at the destination:
|
||||
- `remote nuke`
|
||||
- `prune` and `snapshot create --prune` list, read and delete.
|
||||
|
||||
An rclone remote that can move objects also needs permission to read and delete.
|
||||
An upload there looks up the object it replaces and deletes it. It stops when
|
||||
the look-up is refused.
|
||||
On an rclone remote, a backup can also need permission to read and delete. Which
|
||||
ones depends on the storage service. An upload to a remote that can move objects
|
||||
looks up the object it replaces and deletes it. It stops when the look-up is
|
||||
refused.
|
||||
|
||||
`snapshot verify` and `snapshot restore` list only when you give a remote key,
|
||||
or a snapshot ID that is not in the local index. `snapshot remove --local-only`
|
||||
does not reach the destination. `snapshot remove --dry-run` deletes nothing.
|
||||
These commands list only when you give a remote key, or a snapshot ID that is
|
||||
not in the local index:
|
||||
|
||||
- `snapshot files`
|
||||
- `snapshot verify`
|
||||
- `snapshot restore`
|
||||
|
||||
`snapshot remove --local-only` does not reach the destination.
|
||||
`snapshot remove --dry-run` deletes nothing.
|
||||
|
||||
A machine that runs `snapshot create --prune` can delete its own backups. To
|
||||
keep it to write-only credentials, run `prune` from a separate, trusted machine.
|
||||
@@ -1102,8 +1168,11 @@ not exist, commands treat the destination as not present:
|
||||
- `prune`
|
||||
- `snapshot purge`
|
||||
- `remote nuke`
|
||||
- `snapshot verify` and `snapshot restore`, when given a remote key, or a
|
||||
snapshot ID that is not in the local index
|
||||
- These commands fail when given a remote key, or a snapshot ID that is not in
|
||||
the local index:
|
||||
- `snapshot files`
|
||||
- `snapshot verify`
|
||||
- `snapshot restore`
|
||||
- `snapshot list` shows the local index alone.
|
||||
- `snapshot remove` deletes nothing at the destination.
|
||||
|
||||
@@ -1123,9 +1192,8 @@ there.
|
||||
|
||||
A backup saves the configured destination in the local index once it has written
|
||||
the marker file there. A backup that fails earlier saves nothing, so a mistyped
|
||||
`storage_url` can be corrected. `prune`, `snapshot purge`, `snapshot remove` and
|
||||
`remote nuke` never save it. Five commands refuse to run when the config names a
|
||||
different destination from the saved one:
|
||||
`storage_url` can be corrected. No other command saves it. Five commands refuse
|
||||
to run when the config names a different destination from the saved one:
|
||||
|
||||
- `snapshot create`
|
||||
- `snapshot purge`
|
||||
@@ -1192,6 +1260,7 @@ No other command takes the lock. These commands run while a backup is in
|
||||
progress:
|
||||
|
||||
- `snapshot list`
|
||||
- `snapshot files`
|
||||
- `snapshot verify`
|
||||
- `snapshot restore`
|
||||
- `info`
|
||||
@@ -1218,6 +1287,8 @@ stderr carries the rest:
|
||||
- log records, including those from `--verbose` and `--debug`
|
||||
- the error that a failed command ends with
|
||||
|
||||
`snapshot verify --json` puts its error in the JSON document instead.
|
||||
|
||||
So `vaultik snapshot list > out.txt` captures the list and the warnings printed
|
||||
below it. The banner, log records and a final error stay on your terminal. A
|
||||
warning written as a log record, such as the one about a config file others can
|
||||
@@ -1283,8 +1354,13 @@ A backup needs free temporary space for these:
|
||||
With `TMPDIR` not set, SQLite writes one of those index copies to `/var/tmp`.
|
||||
|
||||
A restore keeps the downloaded blobs on disk, decrypted, until no remaining file
|
||||
needs them. `snapshot restore` and `snapshot verify --deep` also write the
|
||||
decrypted snapshot database there.
|
||||
needs them.
|
||||
|
||||
These commands write the decrypted snapshot database to the temporary directory:
|
||||
|
||||
- `snapshot files`
|
||||
- `snapshot restore`
|
||||
- `snapshot verify --deep`
|
||||
|
||||
These files are readable only by their owner, and are removed when the command
|
||||
ends. A `kill -9` or a power loss leaves them behind. Delete any leftover
|
||||
|
||||
@@ -16,17 +16,20 @@ age_recipients:
|
||||
- age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz
|
||||
|
||||
# Age private key for decryption
|
||||
# Only snapshot restore and snapshot verify --deep need it. The
|
||||
# VAULTIK_AGE_SECRET_KEY environment variable overrides this setting
|
||||
# Only these commands need it:
|
||||
# snapshot files
|
||||
# snapshot restore
|
||||
# snapshot verify --deep
|
||||
# The VAULTIK_AGE_SECRET_KEY environment variable overrides this setting
|
||||
# Do not set it on a machine you back up. Supply the environment variable
|
||||
# only on a machine where you run snapshot restore or snapshot verify --deep
|
||||
# only on a machine where you run one of those commands
|
||||
# Default: not set
|
||||
#age_secret_key: YOUR_AGE_PRIVATE_KEY
|
||||
|
||||
# Named snapshots - each snapshot can contain multiple paths
|
||||
# Each snapshot gets its own ID and can have snapshot-specific excludes
|
||||
# Required: at least one snapshot with at least one path. Only the config
|
||||
# commands, version and completion run without one
|
||||
# Required: at least one snapshot, and every snapshot needs at least one path.
|
||||
# Only the config commands, version and completion run without a snapshot
|
||||
snapshots:
|
||||
testing:
|
||||
paths:
|
||||
|
||||
@@ -16,7 +16,7 @@ Two other documents cover the stored data:
|
||||
## Terms
|
||||
|
||||
- **Snapshot**: one backup of the paths listed under one name in the config.
|
||||
- **Snapshot ID**: the name of a snapshot, in the form
|
||||
- **Snapshot ID**: the identifier of one snapshot, in the form
|
||||
`<hostname>_<snapshot-name>_<timestamp>`.
|
||||
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
|
||||
of its content.
|
||||
@@ -69,6 +69,8 @@ source files
|
||||
S3, a local directory and [rclone](https://rclone.org/).
|
||||
- `internal/s3`: the S3 client that the S3 `Storer` uses.
|
||||
- `internal/config`: loads and checks the config file.
|
||||
- `internal/exclude`: compiles exclude patterns, for the config check and for
|
||||
the `Scanner`.
|
||||
- `internal/ui`: formats the status lines, warnings and errors a user sees.
|
||||
- `internal/log`: the logger. It writes to stderr.
|
||||
- `internal/pidlock`: the lock that `RunWithApp` takes for a command that
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ copy that was uploaded with the snapshot.
|
||||
## Terms
|
||||
|
||||
- **Snapshot**: one backup of the paths listed under one name in the config.
|
||||
- **Snapshot ID**: the name of a snapshot, in the form
|
||||
- **Snapshot ID**: the identifier of one snapshot, in the form
|
||||
`<hostname>_<snapshot-name>_<timestamp>`.
|
||||
- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
|
||||
content.
|
||||
|
||||
+21
-10
@@ -10,7 +10,7 @@ is a storage service set up in rclone. The layout is the same on all three.
|
||||
## Terms
|
||||
|
||||
- **Snapshot**: one backup of the paths listed under one name in the config.
|
||||
- **Snapshot ID**: the name of a snapshot, in the form
|
||||
- **Snapshot ID**: the identifier of one snapshot, in the form
|
||||
`<hostname>_<snapshot-name>_<timestamp>`.
|
||||
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
|
||||
under `metadata/`. See [Remote keys](#remote-keys).
|
||||
@@ -61,8 +61,11 @@ not exist, commands treat the destination as not present:
|
||||
- `prune`
|
||||
- `snapshot purge`
|
||||
- `remote nuke`
|
||||
- `snapshot verify` and `snapshot restore`, when given a remote key, or a
|
||||
snapshot ID that is not in the local index
|
||||
- These commands fail when given a remote key, or a snapshot ID that is not in
|
||||
the local index:
|
||||
- `snapshot files`
|
||||
- `snapshot verify`
|
||||
- `snapshot restore`
|
||||
- `snapshot list` shows the local index alone.
|
||||
- `snapshot remove` deletes nothing at the destination.
|
||||
|
||||
@@ -309,6 +312,7 @@ These commands write unencrypted data to the temporary directory:
|
||||
is no limit on their total size.
|
||||
- `snapshot restore` writes the decrypted database to
|
||||
`vaultik-restore-*/snapshot.db`.
|
||||
- `snapshot files` writes the decrypted database to the same place.
|
||||
- `snapshot verify --deep` writes the decrypted database to
|
||||
`vaultik-verify-*/snapshot.db`.
|
||||
- `snapshot create` writes a copy of the local index to
|
||||
@@ -322,11 +326,12 @@ Trust the temporary directory as much as the directory you restore into.
|
||||
|
||||
### Commands need different permissions
|
||||
|
||||
`snapshot create` without `--prune` writes objects. On S3 it needs no other
|
||||
permission. Other commands need more:
|
||||
`snapshot create` without `--prune` writes objects. With an `s3://` destination
|
||||
it needs no other permission. Other commands need more:
|
||||
|
||||
- These commands list and read:
|
||||
- `snapshot list`
|
||||
- `snapshot files`
|
||||
- `snapshot verify`
|
||||
- `snapshot restore`
|
||||
- `remote info`
|
||||
@@ -336,12 +341,18 @@ permission. Other commands need more:
|
||||
- `remote nuke`
|
||||
- `prune` lists, reads and deletes.
|
||||
|
||||
An rclone remote that can move objects also needs permission to read and delete.
|
||||
An upload there looks up the object it replaces and deletes it. It stops when
|
||||
the look-up is refused.
|
||||
On an rclone remote, a backup can also need permission to read and delete. Which
|
||||
ones depends on the storage service. An upload to a remote that can move objects
|
||||
looks up the object it replaces and deletes it. It stops when the look-up is
|
||||
refused.
|
||||
|
||||
These commands list only when they are given a remote key, or a snapshot ID that
|
||||
is not in the local index:
|
||||
|
||||
- `snapshot files`
|
||||
- `snapshot verify`
|
||||
- `snapshot restore`
|
||||
|
||||
`snapshot verify` and `snapshot restore` list only when they are given a remote
|
||||
key, or a snapshot ID that is not in the local index.
|
||||
`snapshot remove --local-only` does not reach the destination.
|
||||
`snapshot remove --dry-run` deletes nothing.
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ func NewRootCommand() *cobra.Command {
|
||||
Long: `vaultik is an incremental backup tool that encrypts data to age public
|
||||
keys and uploads it to an S3-compatible object store, a local directory or
|
||||
an rclone remote. The backed-up machine needs no private key: only
|
||||
'snapshot restore' and 'snapshot verify --deep' need one.`,
|
||||
'snapshot files', 'snapshot restore' and 'snapshot verify --deep' need one.`,
|
||||
SilenceUsage: true,
|
||||
// Bare 'vaultik' (no subcommand): print help. The banner is
|
||||
// printed once at process startup by Entry, before cobra
|
||||
|
||||
@@ -74,8 +74,8 @@ If no names are provided, all configured snapshots are created.
|
||||
Needs at least one public key in age_recipients, and permission to write
|
||||
objects at the backup destination store. It does not need the age private
|
||||
key. With --prune it also needs permission to list, read and delete objects
|
||||
there. An rclone remote that can move objects also needs permission to
|
||||
read and delete.
|
||||
there. On an rclone remote, a backup can also need permission to read and
|
||||
delete. Which ones depends on the storage service.
|
||||
|
||||
The config is read from the path given by --config or VAULTIK_CONFIG;
|
||||
otherwise from the platform config directory (~/.config/vaultik/config.yml
|
||||
@@ -209,8 +209,13 @@ by the remote key that 'snapshot list' prints for a remote-only snapshot
|
||||
(an unambiguous leading part is enough).
|
||||
|
||||
The file list is only in the snapshot's encrypted database, so this
|
||||
downloads and decrypts it, and requires the age private key in the
|
||||
VAULTIK_AGE_SECRET_KEY environment variable, as 'snapshot restore' does.`,
|
||||
downloads and decrypts it. It requires the age private key, in the
|
||||
VAULTIK_AGE_SECRET_KEY environment variable or as age_secret_key in the
|
||||
config file; the environment variable wins.
|
||||
|
||||
Also needs permission to read the backup destination store, and to list
|
||||
it when given a remote key, or a snapshot ID that is not in the local
|
||||
index.`,
|
||||
Args: cobra.RangeArgs(1, filesMaxArgs),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
path := ""
|
||||
|
||||
Reference in new issue
Block a user