diff --git a/README.md b/README.md index b94c7e3..3bc6fa4 100644 --- a/README.md +++ b/README.md @@ -27,8 +27,8 @@ It was written to meet these requirements together: - Encryption with [age](https://age-encryption.org/), to one or more public keys. age uses X25519 keys and the ChaCha20-Poly1305 cipher. -- Incremental backups. A file is read again only when it has changed, or when - part of its content has to be stored again. +- Incremental backups. An unchanged file is usually not read again. See + [`snapshot create`](#snapshot-create) for when it is. - Deduplication. Identical content is stored once, across files and across snapshots. - Compression with [zstd](https://github.com/facebook/zstd), at a level you @@ -46,7 +46,7 @@ It was written to meet these requirements together: - **Snapshot**: one backup of the paths listed under one name in the config, taken at one time. -- **Snapshot ID**: the name of a snapshot, in the form +- **Snapshot ID**: the identifier of one snapshot, in the form `__`. An example is `server1_home_2025-06-01T12:00:00Z`. - **Destination**: the place the backups are stored. @@ -185,6 +185,7 @@ vaultik config get vaultik config set vaultik snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than ] vaultik snapshot list [--json] +vaultik snapshot files [path] [--json] vaultik snapshot verify [--deep] [--json] vaultik snapshot purge (--keep-latest | --older-than ) [--snapshot ...] [--force] vaultik snapshot remove [--dry-run] [--force] [--local-only] [--json] @@ -470,6 +471,55 @@ Under `--json`, every warning above goes to stderr, except the one about a snapshot with no manifest at the destination. That snapshot has `remote_present` set to `false`. +### `snapshot files [path]` + +Lists the files, symlinks and directories in a snapshot, without restoring it. +The first argument is a snapshot ID, a remote key or the leading part of a +remote key. + +It needs: + +- The age private key. See [The private key](#the-private-key). +- Permission to read at the destination, and to list it when you give a remote + key, or a snapshot ID that is not in the local index. + +The file list is stored only in the snapshot's encrypted database. The command +downloads and decrypts that database. It writes the decrypted database to a +temporary directory and removes it afterwards. See +[Temporary files](#temporary-files). + +With a path, the command lists only that path and what is under it. The path +names a file or a directory by its original absolute path. A path that matches +nothing in the snapshot is an error. + +The table is sorted by path. It has these columns: + +- `MODE`: the type and the permissions. See below. +- `SIZE`: the size in bytes. It is `0` for a symlink or a directory. +- `MODIFIED`: the modification time, in UTC. +- `PATH`: the original absolute path. + +In the table, a path that holds a newline or another non-printable character is +shown in double quotes, with that character escaped, as in `"a\nb"`. + +The mode looks like the one `ls -l` shows, as in `-rw-r--r--` or `drwxr-xr-x`. +The letters before the nine permission characters show the type and any special +bit. These letters differ from `ls -l`: + +- `L`: a symlink, which `ls -l` shows as `l`. +- `u`: setuid, which `ls -l` shows as `s` in place of the owner's `x`. +- `g`: setgid, which `ls -l` shows as `s` in place of the group's `x`. +- `t`: sticky, which `ls -l` shows as `t` in place of the others' `x`. + +A sticky directory, for example, is shown as `dtrwxrwxrwx`. + +`--json` prints a JSON array instead. Each entry has these fields: + +- `path` +- `size` +- `mode`: written as in the table. +- `mtime`: the modification time. + ### `snapshot verify ` Checks a snapshot at the destination. The argument is a snapshot ID, a remote @@ -487,7 +537,8 @@ snapshot ID that is not in the local index. Flags: - `--deep`: download, decrypt and re-hash every blob. See below. -- `--json`: print the result as JSON. +- `--json`: print the result as JSON. A failed check is reported in that + document, not on stderr. The command exits non-zero when a check fails or cannot be made. @@ -814,6 +865,8 @@ The config loads when it is valid YAML and all of these hold: - Every entry in `age_recipients` is an age X25519 public key. - `snapshots` has at least one name, and every name has at least one path. +- Every exclude pattern is valid. The error for an invalid one names the setting + and quotes the pattern. - A destination is set. Its `storage_url` has one of the three schemes under [Destinations](#destinations). An S3 destination has credentials. - Every `${ENV:NAME}` names an environment variable with a non-empty value. @@ -859,10 +912,10 @@ A command that reads the config fails when the path from `--config` or - `age_secret_key` - The age private key. - Default: not set. - - Only `snapshot restore` and `snapshot verify --deep` use it. - - The `VAULTIK_AGE_SECRET_KEY` environment variable overrides it. - - Do not set it on a machine you back up. See + - Only the commands that decrypt data use it. See [The private key](#the-private-key). + - The `VAULTIK_AGE_SECRET_KEY` environment variable overrides it. + - Do not set it on a machine you back up. - `snapshots` - The named sets of paths to back up. Required. - `snapshots..paths`: the paths to back up under this name. @@ -948,8 +1001,13 @@ config loads. The error names the setting and quotes the value. ### The private key -`snapshot restore` and `snapshot verify --deep` decrypt data, so they need the -age private key. No other command needs it. +These commands decrypt data, so they need the age private key: + +- `snapshot files` +- `snapshot restore` +- `snapshot verify --deep` + +No other command needs it. Give the key in one of two ways: @@ -970,7 +1028,7 @@ export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" Do not put the private key on a machine you back up. Anyone who breaks into that machine could then decrypt its backups. Supply the key only on the machine where -you run `snapshot restore` or `snapshot verify --deep`. +you run the commands that need it. ### Exclude patterns @@ -1053,11 +1111,12 @@ A leftover `.partial` file is ignored. You can delete it. Commands need these permissions at the destination: -- `snapshot create` without `--prune` writes objects. On S3 it needs no other - permission, so a machine you back up can hold credentials that can neither - read nor delete. +- `snapshot create` without `--prune` writes objects. With an `s3://` + destination it needs no other permission, so a machine you back up can hold + credentials that can neither read nor delete. - These commands list and read: - `snapshot list` + - `snapshot files` - `snapshot verify` - `snapshot restore` - `remote info` @@ -1067,13 +1126,20 @@ Commands need these permissions at the destination: - `remote nuke` - `prune` and `snapshot create --prune` list, read and delete. -An rclone remote that can move objects also needs permission to read and delete. -An upload there looks up the object it replaces and deletes it. It stops when -the look-up is refused. +On an rclone remote, a backup can also need permission to read and delete. Which +ones depends on the storage service. An upload to a remote that can move objects +looks up the object it replaces and deletes it. It stops when the look-up is +refused. -`snapshot verify` and `snapshot restore` list only when you give a remote key, -or a snapshot ID that is not in the local index. `snapshot remove --local-only` -does not reach the destination. `snapshot remove --dry-run` deletes nothing. +These commands list only when you give a remote key, or a snapshot ID that is +not in the local index: + +- `snapshot files` +- `snapshot verify` +- `snapshot restore` + +`snapshot remove --local-only` does not reach the destination. +`snapshot remove --dry-run` deletes nothing. A machine that runs `snapshot create --prune` can delete its own backups. To keep it to write-only credentials, run `prune` from a separate, trusted machine. @@ -1102,8 +1168,11 @@ not exist, commands treat the destination as not present: - `prune` - `snapshot purge` - `remote nuke` - - `snapshot verify` and `snapshot restore`, when given a remote key, or a - snapshot ID that is not in the local index +- These commands fail when given a remote key, or a snapshot ID that is not in + the local index: + - `snapshot files` + - `snapshot verify` + - `snapshot restore` - `snapshot list` shows the local index alone. - `snapshot remove` deletes nothing at the destination. @@ -1123,9 +1192,8 @@ there. A backup saves the configured destination in the local index once it has written the marker file there. A backup that fails earlier saves nothing, so a mistyped -`storage_url` can be corrected. `prune`, `snapshot purge`, `snapshot remove` and -`remote nuke` never save it. Five commands refuse to run when the config names a -different destination from the saved one: +`storage_url` can be corrected. No other command saves it. Five commands refuse +to run when the config names a different destination from the saved one: - `snapshot create` - `snapshot purge` @@ -1192,6 +1260,7 @@ No other command takes the lock. These commands run while a backup is in progress: - `snapshot list` +- `snapshot files` - `snapshot verify` - `snapshot restore` - `info` @@ -1218,6 +1287,8 @@ stderr carries the rest: - log records, including those from `--verbose` and `--debug` - the error that a failed command ends with +`snapshot verify --json` puts its error in the JSON document instead. + So `vaultik snapshot list > out.txt` captures the list and the warnings printed below it. The banner, log records and a final error stay on your terminal. A warning written as a log record, such as the one about a config file others can @@ -1283,8 +1354,13 @@ A backup needs free temporary space for these: With `TMPDIR` not set, SQLite writes one of those index copies to `/var/tmp`. A restore keeps the downloaded blobs on disk, decrypted, until no remaining file -needs them. `snapshot restore` and `snapshot verify --deep` also write the -decrypted snapshot database there. +needs them. + +These commands write the decrypted snapshot database to the temporary directory: + +- `snapshot files` +- `snapshot restore` +- `snapshot verify --deep` These files are readable only by their owner, and are removed when the command ends. A `kill -9` or a power loss leaves them behind. Delete any leftover diff --git a/configs/config.example.yml b/configs/config.example.yml index f183744..67117a2 100644 --- a/configs/config.example.yml +++ b/configs/config.example.yml @@ -16,17 +16,20 @@ age_recipients: - age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz # Age private key for decryption -# Only snapshot restore and snapshot verify --deep need it. The -# VAULTIK_AGE_SECRET_KEY environment variable overrides this setting +# Only these commands need it: +# snapshot files +# snapshot restore +# snapshot verify --deep +# The VAULTIK_AGE_SECRET_KEY environment variable overrides this setting # Do not set it on a machine you back up. Supply the environment variable -# only on a machine where you run snapshot restore or snapshot verify --deep +# only on a machine where you run one of those commands # Default: not set #age_secret_key: YOUR_AGE_PRIVATE_KEY # Named snapshots - each snapshot can contain multiple paths # Each snapshot gets its own ID and can have snapshot-specific excludes -# Required: at least one snapshot with at least one path. Only the config -# commands, version and completion run without one +# Required: at least one snapshot, and every snapshot needs at least one path. +# Only the config commands, version and completion run without a snapshot snapshots: testing: paths: diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 34d4626..24a498d 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -16,7 +16,7 @@ Two other documents cover the stored data: ## Terms - **Snapshot**: one backup of the paths listed under one name in the config. -- **Snapshot ID**: the name of a snapshot, in the form +- **Snapshot ID**: the identifier of one snapshot, in the form `__`. - **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash of its content. @@ -69,6 +69,8 @@ source files S3, a local directory and [rclone](https://rclone.org/). - `internal/s3`: the S3 client that the S3 `Storer` uses. - `internal/config`: loads and checks the config file. +- `internal/exclude`: compiles exclude patterns, for the config check and for + the `Scanner`. - `internal/ui`: formats the status lines, warnings and errors a user sees. - `internal/log`: the logger. It writes to stderr. - `internal/pidlock`: the lock that `RunWithApp` takes for a command that diff --git a/docs/DATAMODEL.md b/docs/DATAMODEL.md index 65a4227..ba6b300 100644 --- a/docs/DATAMODEL.md +++ b/docs/DATAMODEL.md @@ -17,7 +17,7 @@ copy that was uploaded with the snapshot. ## Terms - **Snapshot**: one backup of the paths listed under one name in the config. -- **Snapshot ID**: the name of a snapshot, in the form +- **Snapshot ID**: the identifier of one snapshot, in the form `__`. - **Chunk**: a piece of a file's content, named by the SHA-256 hash of that content. diff --git a/docs/REPOSTRUCTURE.md b/docs/REPOSTRUCTURE.md index 52bddf2..b9bf04a 100644 --- a/docs/REPOSTRUCTURE.md +++ b/docs/REPOSTRUCTURE.md @@ -10,7 +10,7 @@ is a storage service set up in rclone. The layout is the same on all three. ## Terms - **Snapshot**: one backup of the paths listed under one name in the config. -- **Snapshot ID**: the name of a snapshot, in the form +- **Snapshot ID**: the identifier of one snapshot, in the form `__`. - **Remote key**: a hash of the snapshot ID. It names the snapshot's directory under `metadata/`. See [Remote keys](#remote-keys). @@ -61,8 +61,11 @@ not exist, commands treat the destination as not present: - `prune` - `snapshot purge` - `remote nuke` - - `snapshot verify` and `snapshot restore`, when given a remote key, or a - snapshot ID that is not in the local index +- These commands fail when given a remote key, or a snapshot ID that is not in + the local index: + - `snapshot files` + - `snapshot verify` + - `snapshot restore` - `snapshot list` shows the local index alone. - `snapshot remove` deletes nothing at the destination. @@ -309,6 +312,7 @@ These commands write unencrypted data to the temporary directory: is no limit on their total size. - `snapshot restore` writes the decrypted database to `vaultik-restore-*/snapshot.db`. +- `snapshot files` writes the decrypted database to the same place. - `snapshot verify --deep` writes the decrypted database to `vaultik-verify-*/snapshot.db`. - `snapshot create` writes a copy of the local index to @@ -322,11 +326,12 @@ Trust the temporary directory as much as the directory you restore into. ### Commands need different permissions -`snapshot create` without `--prune` writes objects. On S3 it needs no other -permission. Other commands need more: +`snapshot create` without `--prune` writes objects. With an `s3://` destination +it needs no other permission. Other commands need more: - These commands list and read: - `snapshot list` + - `snapshot files` - `snapshot verify` - `snapshot restore` - `remote info` @@ -336,12 +341,18 @@ permission. Other commands need more: - `remote nuke` - `prune` lists, reads and deletes. -An rclone remote that can move objects also needs permission to read and delete. -An upload there looks up the object it replaces and deletes it. It stops when -the look-up is refused. +On an rclone remote, a backup can also need permission to read and delete. Which +ones depends on the storage service. An upload to a remote that can move objects +looks up the object it replaces and deletes it. It stops when the look-up is +refused. + +These commands list only when they are given a remote key, or a snapshot ID that +is not in the local index: + +- `snapshot files` +- `snapshot verify` +- `snapshot restore` -`snapshot verify` and `snapshot restore` list only when they are given a remote -key, or a snapshot ID that is not in the local index. `snapshot remove --local-only` does not reach the destination. `snapshot remove --dry-run` deletes nothing. diff --git a/internal/cli/root.go b/internal/cli/root.go index 940bcac..bcf7a2f 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -38,7 +38,7 @@ func NewRootCommand() *cobra.Command { Long: `vaultik is an incremental backup tool that encrypts data to age public keys and uploads it to an S3-compatible object store, a local directory or an rclone remote. The backed-up machine needs no private key: only -'snapshot restore' and 'snapshot verify --deep' need one.`, +'snapshot files', 'snapshot restore' and 'snapshot verify --deep' need one.`, SilenceUsage: true, // Bare 'vaultik' (no subcommand): print help. The banner is // printed once at process startup by Entry, before cobra diff --git a/internal/cli/snapshot.go b/internal/cli/snapshot.go index 1810750..68b054f 100644 --- a/internal/cli/snapshot.go +++ b/internal/cli/snapshot.go @@ -74,8 +74,8 @@ If no names are provided, all configured snapshots are created. Needs at least one public key in age_recipients, and permission to write objects at the backup destination store. It does not need the age private key. With --prune it also needs permission to list, read and delete objects -there. An rclone remote that can move objects also needs permission to -read and delete. +there. On an rclone remote, a backup can also need permission to read and +delete. Which ones depends on the storage service. The config is read from the path given by --config or VAULTIK_CONFIG; otherwise from the platform config directory (~/.config/vaultik/config.yml @@ -209,8 +209,13 @@ by the remote key that 'snapshot list' prints for a remote-only snapshot (an unambiguous leading part is enough). The file list is only in the snapshot's encrypted database, so this -downloads and decrypts it, and requires the age private key in the -VAULTIK_AGE_SECRET_KEY environment variable, as 'snapshot restore' does.`, +downloads and decrypts it. It requires the age private key, in the +VAULTIK_AGE_SECRET_KEY environment variable or as age_secret_key in the +config file; the environment variable wins. + +Also needs permission to read the backup destination store, and to list +it when given a remote key, or a snapshot ID that is not in the local +index.`, Args: cobra.RangeArgs(1, filesMaxArgs), RunE: func(cmd *cobra.Command, args []string) error { path := ""