diff --git a/README.md b/README.md index 4a53d2c..cfcc356 100644 --- a/README.md +++ b/README.md @@ -308,8 +308,11 @@ is `vaultik config set compression_level 99`. The file is then left unchanged. A config that does not load yet is checked less strictly. An example is the file `config init` writes, which has no `storage_url`. There the command refuses only -two things: an unknown key, and a value of the wrong type, such as a word where -a number belongs. +three things: + +- an unknown key +- a value of the wrong type, such as a word where a number belongs +- a `${ENV:NAME}` value whose variable is unset or empty ### `snapshot create` @@ -477,7 +480,8 @@ Without `--deep`, it checks two things: - The snapshot's encrypted database is present. This reads no blob contents and needs no private key. It needs permission to -read at the destination, and to list it when you give a remote key. +read at the destination, and to list it when you give a remote key, or a +snapshot ID that is not in the local index. Flags: @@ -583,7 +587,7 @@ It needs: - The age private key, as for `snapshot verify --deep`. See [The private key](#the-private-key). - Permission to read at the destination, and to list it when you give a remote - key. + key, or a snapshot ID that is not in the local index. - Write access to ``. The command creates the directory when it does not exist. - Free temporary space for the decrypted blobs. See @@ -1066,12 +1070,14 @@ An rclone remote that can move objects also needs permission to read and delete. An upload there looks up the object it replaces and deletes it. It stops when the look-up is refused. -`snapshot verify` and `snapshot restore` list only when you give a remote key. -`snapshot remove --local-only` does not reach the destination. -`snapshot remove --dry-run` deletes nothing. +`snapshot verify` and `snapshot restore` list only when you give a remote key, +or a snapshot ID that is not in the local index. `snapshot remove --local-only` +does not reach the destination. `snapshot remove --dry-run` deletes nothing. A machine that runs `snapshot create --prune` can delete its own backups. To keep it to write-only credentials, run `prune` from a separate, trusted machine. +`prune` there deletes only data that no snapshot uses. It does not remove old +snapshots. A backup tries to look up the marker file and each blob before it writes. On S3 it goes on when the destination refuses those look-ups. @@ -1095,7 +1101,8 @@ not exist, commands treat the destination as not present: - `prune` - `snapshot purge` - `remote nuke` - - `snapshot verify` and `snapshot restore`, when given a remote key + - `snapshot verify` and `snapshot restore`, when given a remote key, or a + snapshot ID that is not in the local index - `snapshot list` shows the local index alone. - `snapshot remove` deletes nothing at the destination. @@ -1210,8 +1217,10 @@ stderr carries the rest: - log records, including those from `--verbose` and `--debug` - the error that a failed command ends with -So `vaultik snapshot list > out.txt` captures the list and any warnings. The -banner, log records and a final error stay on your terminal. +So `vaultik snapshot list > out.txt` captures the list and the warnings printed +below it. The banner, log records and a final error stay on your terminal. A +warning written as a log record, such as the one about a config file others can +read, stays there too. Log records are colored lines when stderr is a terminal. They are JSON, one object per line, when stderr is redirected. @@ -1409,8 +1418,8 @@ call them. Every `docker build` in these scripts uses `--no-cache`. A cached layer would skip the tests or the linter and still report success. -The test suite has no separate integration target. Every test runs every time. -Each test binary is stopped after 90 seconds. +The test suite has no separate integration target. No test is held back by a +build tag. Each test binary is stopped after 90 seconds. To change the Go version, change these together: @@ -1474,6 +1483,9 @@ The version a binary reports comes from git. It is the output of `script/version` gives this value to the `Makefile`. `script/docker` and `script/cibuild` compute it themselves. +A plain `docker build .` adds no `-dirty`. The `Dockerfile` runs `git describe` +without `--dirty`. + `make` outside a git checkout, and `docker build .` without `.git`, stamp `dev`. An image built by `script/docker` or `script/cibuild` outside a git checkout reports `unknown`. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index c2230b6..7af2993 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -291,10 +291,11 @@ therefore marks a backup that did not finish. - With `--verify`, `verifyRestoredFiles` reads each restored regular file and checks its chunk hashes. -Given a snapshot ID, the restore does not read the local index. Given a remote -key, it reads the local index only to check the destination marker. It does not -read the manifest. It finds each file's chunks and blobs through the downloaded -database. +Given a snapshot ID, the restore looks it up in the local index. When it is not +there, the restore checks the destination marker and lists the snapshot's +directory. Given a remote key, it reads the local index only to check the +destination marker. It does not read the manifest. It finds each file's chunks +and blobs through the downloaded database. ### Checks on the downloaded database diff --git a/docs/DATAMODEL.md b/docs/DATAMODEL.md index 30b3874..25cdd25 100644 --- a/docs/DATAMODEL.md +++ b/docs/DATAMODEL.md @@ -299,6 +299,7 @@ A backup works on the tables in this order: - It walks each path. It compares each regular file and symlink with its row in `files`. - It links each unchanged file to the snapshot in `snapshot_files`. +- It writes a `files` row and a `snapshot_files` row for each directory. - It records each mount point in `snapshot_mount_points`. - It reads each changed regular file and stores it. See below. A changed symlink is recorded without being read. @@ -362,9 +363,9 @@ still be in the `-wal` file beside it. ### Restore -A restore reads no file, chunk or blob rows from the local index. Given a remote -key, it reads `snapshots` and `local_meta` there to check the destination -marker. +A restore reads no file, chunk or blob rows from the local index. It reads +`snapshots` and `local_meta` there, to look up a snapshot ID and to check the +destination marker. - It downloads and decrypts `db.zst.age` into a temporary directory. - It opens that file read-only. It applies no migration to it. diff --git a/docs/REPOSTRUCTURE.md b/docs/REPOSTRUCTURE.md index de246e8..1839e2c 100644 --- a/docs/REPOSTRUCTURE.md +++ b/docs/REPOSTRUCTURE.md @@ -59,7 +59,8 @@ not exist, commands treat the destination as not present: - `prune` - `snapshot purge` - `remote nuke` - - `snapshot verify` and `snapshot restore`, when given a remote key + - `snapshot verify` and `snapshot restore`, when given a remote key, or a + snapshot ID that is not in the local index - `snapshot list` shows the local index alone. - `snapshot remove` deletes nothing at the destination. @@ -334,12 +335,14 @@ An upload there looks up the object it replaces and deletes it. It stops when the look-up is refused. `snapshot verify` and `snapshot restore` list only when they are given a remote -key. `snapshot remove --local-only` does not reach the destination. +key, or a snapshot ID that is not in the local index. +`snapshot remove --local-only` does not reach the destination. `snapshot remove --dry-run` deletes nothing. `snapshot create --prune` runs a prune on the backed-up machine. That machine then holds credentials that can delete its own backups. To keep it to write-only -credentials, run `prune` from a separate, trusted machine. +credentials, run `prune` from a separate, trusted machine. `prune` there deletes +only data that no snapshot uses. It does not remove old snapshots. ### New recipients cannot read old data diff --git a/docs/TODO.md b/docs/TODO.md index 9b5892c..ddbc0ae 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -41,10 +41,11 @@ Work planned after 1.0, loosely ordered by priority. - **Parallel blob downloads during restore.** A restore downloads one blob at a time now. With a fast S3 endpoint and a multi-core machine, restore speed is - bound by the fetch, decryption and decompression of each blob. Running several - of those in parallel against the disk cache would close most of the remaining - gap. This needs to interact correctly with the locality planner and the - sweeper. + bound by the fetch, decryption and decompression of each blob. Fetching, + decrypting and decompressing several blobs at once would make such a restore + faster. The change must work with `restorePlan`, which orders the files, and + with `restoreSweeper`, which deletes a cached blob once no remaining file + needs it. - **Bandwidth limiting (`--bwlimit`).** For both upload and download. It is useful for backing up over a shared link. It is tricky to make work correctly with parallel downloads. diff --git a/internal/cli/root.go b/internal/cli/root.go index 843d3ca..940bcac 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -58,8 +58,9 @@ an rclone remote. The backed-up machine needs no private key: only cmd.PersistentFlags().BoolVar(&rootFlags.Debug, "debug", false, "Enable debug output") cmd.PersistentFlags().BoolVarP(&rootFlags.Quiet, "quiet", "q", false, - "Suppress status output (warnings, errors, JSON documents, "+ - "'config get' values and confirmation prompts are still printed)") + "Leave out the banner, status lines, tables and reports (warnings, "+ + "errors, JSON documents, 'config get' values and confirmation "+ + "prompts are still printed)") cmd.PersistentFlags().BoolVar(&rootFlags.SkipErrors, "skip-errors", false, "Skip files that cannot be read when creating a snapshot, or "+ "that cannot be restored when restoring, instead of aborting "+ diff --git a/internal/cli/snapshot.go b/internal/cli/snapshot.go index 94017ee..ab7d40b 100644 --- a/internal/cli/snapshot.go +++ b/internal/cli/snapshot.go @@ -252,7 +252,8 @@ snapshot's encrypted database is present. It does not read blob contents and does not need the age private key. Needs permission to read the backup destination store, and to list it -when a remote key is given. +when given a remote key, or a snapshot ID that is not in the local +index. Use --deep to download, decrypt, and re-hash every blob to detect corruption -- integrity, not who wrote it. diff --git a/internal/cli/snapshot_restore.go b/internal/cli/snapshot_restore.go index 4f0f3db..91a613c 100644 --- a/internal/cli/snapshot_restore.go +++ b/internal/cli/snapshot_restore.go @@ -49,8 +49,9 @@ typing it, so that it does not land in your shell history: export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" Also needs permission to read the backup destination store (and to list -it when a remote key is given), and free space in the temporary directory -for the decrypted blobs. +it when given a remote key, or a snapshot ID that is not in the local +index), and free space in the temporary directory for the decrypted +blobs. Examples: # Restore entire snapshot