- snapshot verify and snapshot restore also list the destination for a snapshot ID that is not in the local index: README, REPOSTRUCTURE, ARCHITECTURE, DATAMODEL and the two commands' help. - prune on a separate machine deletes only unused data and does not remove old snapshots. - The --quiet usage string names everything the flag leaves out. - A warning written as a log record stays on stderr. - config set on a config that does not load refuses three things. - No test is held back by a build tag. - A plain docker build . adds no -dirty. - The backup's order of work records directories. - TODO.md says what parallel downloads would speed up and names restorePlan and restoreSweeper. Model: fable-5-1
This commit is contained in:
1 parent
1fcdfcc1c6
commit
2bf51d65e3
8 files changed
+52
-31
No files matched your search
@@ -308,8 +308,11 @@ is `vaultik config set compression_level 99`. The file is then left unchanged.
|
||||
|
||||
A config that does not load yet is checked less strictly. An example is the file
|
||||
`config init` writes, which has no `storage_url`. There the command refuses only
|
||||
two things: an unknown key, and a value of the wrong type, such as a word where
|
||||
a number belongs.
|
||||
three things:
|
||||
|
||||
- an unknown key
|
||||
- a value of the wrong type, such as a word where a number belongs
|
||||
- a `${ENV:NAME}` value whose variable is unset or empty
|
||||
|
||||
### `snapshot create`
|
||||
|
||||
@@ -477,7 +480,8 @@ Without `--deep`, it checks two things:
|
||||
- The snapshot's encrypted database is present.
|
||||
|
||||
This reads no blob contents and needs no private key. It needs permission to
|
||||
read at the destination, and to list it when you give a remote key.
|
||||
read at the destination, and to list it when you give a remote key, or a
|
||||
snapshot ID that is not in the local index.
|
||||
|
||||
Flags:
|
||||
|
||||
@@ -583,7 +587,7 @@ It needs:
|
||||
- The age private key, as for `snapshot verify --deep`. See
|
||||
[The private key](#the-private-key).
|
||||
- Permission to read at the destination, and to list it when you give a remote
|
||||
key.
|
||||
key, or a snapshot ID that is not in the local index.
|
||||
- Write access to `<target-dir>`. The command creates the directory when it does
|
||||
not exist.
|
||||
- Free temporary space for the decrypted blobs. See
|
||||
@@ -1066,12 +1070,14 @@ An rclone remote that can move objects also needs permission to read and delete.
|
||||
An upload there looks up the object it replaces and deletes it. It stops when
|
||||
the look-up is refused.
|
||||
|
||||
`snapshot verify` and `snapshot restore` list only when you give a remote key.
|
||||
`snapshot remove --local-only` does not reach the destination.
|
||||
`snapshot remove --dry-run` deletes nothing.
|
||||
`snapshot verify` and `snapshot restore` list only when you give a remote key,
|
||||
or a snapshot ID that is not in the local index. `snapshot remove --local-only`
|
||||
does not reach the destination. `snapshot remove --dry-run` deletes nothing.
|
||||
|
||||
A machine that runs `snapshot create --prune` can delete its own backups. To
|
||||
keep it to write-only credentials, run `prune` from a separate, trusted machine.
|
||||
`prune` there deletes only data that no snapshot uses. It does not remove old
|
||||
snapshots.
|
||||
|
||||
A backup tries to look up the marker file and each blob before it writes. On S3
|
||||
it goes on when the destination refuses those look-ups.
|
||||
@@ -1095,7 +1101,8 @@ not exist, commands treat the destination as not present:
|
||||
- `prune`
|
||||
- `snapshot purge`
|
||||
- `remote nuke`
|
||||
- `snapshot verify` and `snapshot restore`, when given a remote key
|
||||
- `snapshot verify` and `snapshot restore`, when given a remote key, or a
|
||||
snapshot ID that is not in the local index
|
||||
- `snapshot list` shows the local index alone.
|
||||
- `snapshot remove` deletes nothing at the destination.
|
||||
|
||||
@@ -1210,8 +1217,10 @@ stderr carries the rest:
|
||||
- log records, including those from `--verbose` and `--debug`
|
||||
- the error that a failed command ends with
|
||||
|
||||
So `vaultik snapshot list > out.txt` captures the list and any warnings. The
|
||||
banner, log records and a final error stay on your terminal.
|
||||
So `vaultik snapshot list > out.txt` captures the list and the warnings printed
|
||||
below it. The banner, log records and a final error stay on your terminal. A
|
||||
warning written as a log record, such as the one about a config file others can
|
||||
read, stays there too.
|
||||
|
||||
Log records are colored lines when stderr is a terminal. They are JSON, one
|
||||
object per line, when stderr is redirected.
|
||||
@@ -1409,8 +1418,8 @@ call them.
|
||||
Every `docker build` in these scripts uses `--no-cache`. A cached layer would
|
||||
skip the tests or the linter and still report success.
|
||||
|
||||
The test suite has no separate integration target. Every test runs every time.
|
||||
Each test binary is stopped after 90 seconds.
|
||||
The test suite has no separate integration target. No test is held back by a
|
||||
build tag. Each test binary is stopped after 90 seconds.
|
||||
|
||||
To change the Go version, change these together:
|
||||
|
||||
@@ -1474,6 +1483,9 @@ The version a binary reports comes from git. It is the output of
|
||||
`script/version` gives this value to the `Makefile`. `script/docker` and
|
||||
`script/cibuild` compute it themselves.
|
||||
|
||||
A plain `docker build .` adds no `-dirty`. The `Dockerfile` runs `git describe`
|
||||
without `--dirty`.
|
||||
|
||||
`make` outside a git checkout, and `docker build .` without `.git`, stamp `dev`.
|
||||
An image built by `script/docker` or `script/cibuild` outside a git checkout
|
||||
reports `unknown`.
|
||||
|
||||
@@ -291,10 +291,11 @@ therefore marks a backup that did not finish.
|
||||
- With `--verify`, `verifyRestoredFiles` reads each restored regular file and
|
||||
checks its chunk hashes.
|
||||
|
||||
Given a snapshot ID, the restore does not read the local index. Given a remote
|
||||
key, it reads the local index only to check the destination marker. It does not
|
||||
read the manifest. It finds each file's chunks and blobs through the downloaded
|
||||
database.
|
||||
Given a snapshot ID, the restore looks it up in the local index. When it is not
|
||||
there, the restore checks the destination marker and lists the snapshot's
|
||||
directory. Given a remote key, it reads the local index only to check the
|
||||
destination marker. It does not read the manifest. It finds each file's chunks
|
||||
and blobs through the downloaded database.
|
||||
|
||||
### Checks on the downloaded database
|
||||
|
||||
|
||||
+4
-3
@@ -299,6 +299,7 @@ A backup works on the tables in this order:
|
||||
- It walks each path. It compares each regular file and symlink with its row in
|
||||
`files`.
|
||||
- It links each unchanged file to the snapshot in `snapshot_files`.
|
||||
- It writes a `files` row and a `snapshot_files` row for each directory.
|
||||
- It records each mount point in `snapshot_mount_points`.
|
||||
- It reads each changed regular file and stores it. See below. A changed symlink
|
||||
is recorded without being read.
|
||||
@@ -362,9 +363,9 @@ still be in the `-wal` file beside it.
|
||||
|
||||
### Restore
|
||||
|
||||
A restore reads no file, chunk or blob rows from the local index. Given a remote
|
||||
key, it reads `snapshots` and `local_meta` there to check the destination
|
||||
marker.
|
||||
A restore reads no file, chunk or blob rows from the local index. It reads
|
||||
`snapshots` and `local_meta` there, to look up a snapshot ID and to check the
|
||||
destination marker.
|
||||
|
||||
- It downloads and decrypts `db.zst.age` into a temporary directory.
|
||||
- It opens that file read-only. It applies no migration to it.
|
||||
|
||||
@@ -59,7 +59,8 @@ not exist, commands treat the destination as not present:
|
||||
- `prune`
|
||||
- `snapshot purge`
|
||||
- `remote nuke`
|
||||
- `snapshot verify` and `snapshot restore`, when given a remote key
|
||||
- `snapshot verify` and `snapshot restore`, when given a remote key, or a
|
||||
snapshot ID that is not in the local index
|
||||
- `snapshot list` shows the local index alone.
|
||||
- `snapshot remove` deletes nothing at the destination.
|
||||
|
||||
@@ -334,12 +335,14 @@ An upload there looks up the object it replaces and deletes it. It stops when
|
||||
the look-up is refused.
|
||||
|
||||
`snapshot verify` and `snapshot restore` list only when they are given a remote
|
||||
key. `snapshot remove --local-only` does not reach the destination.
|
||||
key, or a snapshot ID that is not in the local index.
|
||||
`snapshot remove --local-only` does not reach the destination.
|
||||
`snapshot remove --dry-run` deletes nothing.
|
||||
|
||||
`snapshot create --prune` runs a prune on the backed-up machine. That machine
|
||||
then holds credentials that can delete its own backups. To keep it to write-only
|
||||
credentials, run `prune` from a separate, trusted machine.
|
||||
credentials, run `prune` from a separate, trusted machine. `prune` there deletes
|
||||
only data that no snapshot uses. It does not remove old snapshots.
|
||||
|
||||
### New recipients cannot read old data
|
||||
|
||||
|
||||
+5
-4
@@ -41,10 +41,11 @@ Work planned after 1.0, loosely ordered by priority.
|
||||
|
||||
- **Parallel blob downloads during restore.** A restore downloads one blob at a
|
||||
time now. With a fast S3 endpoint and a multi-core machine, restore speed is
|
||||
bound by the fetch, decryption and decompression of each blob. Running several
|
||||
of those in parallel against the disk cache would close most of the remaining
|
||||
gap. This needs to interact correctly with the locality planner and the
|
||||
sweeper.
|
||||
bound by the fetch, decryption and decompression of each blob. Fetching,
|
||||
decrypting and decompressing several blobs at once would make such a restore
|
||||
faster. The change must work with `restorePlan`, which orders the files, and
|
||||
with `restoreSweeper`, which deletes a cached blob once no remaining file
|
||||
needs it.
|
||||
- **Bandwidth limiting (`--bwlimit`).** For both upload and download. It is
|
||||
useful for backing up over a shared link. It is tricky to make work correctly
|
||||
with parallel downloads.
|
||||
|
||||
@@ -58,8 +58,9 @@ an rclone remote. The backed-up machine needs no private key: only
|
||||
cmd.PersistentFlags().BoolVar(&rootFlags.Debug, "debug", false,
|
||||
"Enable debug output")
|
||||
cmd.PersistentFlags().BoolVarP(&rootFlags.Quiet, "quiet", "q", false,
|
||||
"Suppress status output (warnings, errors, JSON documents, "+
|
||||
"'config get' values and confirmation prompts are still printed)")
|
||||
"Leave out the banner, status lines, tables and reports (warnings, "+
|
||||
"errors, JSON documents, 'config get' values and confirmation "+
|
||||
"prompts are still printed)")
|
||||
cmd.PersistentFlags().BoolVar(&rootFlags.SkipErrors, "skip-errors", false,
|
||||
"Skip files that cannot be read when creating a snapshot, or "+
|
||||
"that cannot be restored when restoring, instead of aborting "+
|
||||
|
||||
@@ -252,7 +252,8 @@ snapshot's encrypted database is present. It does not read blob
|
||||
contents and does not need the age private key.
|
||||
|
||||
Needs permission to read the backup destination store, and to list it
|
||||
when a remote key is given.
|
||||
when given a remote key, or a snapshot ID that is not in the local
|
||||
index.
|
||||
|
||||
Use --deep to download, decrypt, and re-hash every blob to detect
|
||||
corruption -- integrity, not who wrote it.
|
||||
|
||||
@@ -49,8 +49,9 @@ typing it, so that it does not land in your shell history:
|
||||
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
|
||||
|
||||
Also needs permission to read the backup destination store (and to list
|
||||
it when a remote key is given), and free space in the temporary directory
|
||||
for the decrypted blobs.
|
||||
it when given a remote key, or a snapshot ID that is not in the local
|
||||
index), and free space in the temporary directory for the decrypted
|
||||
blobs.
|
||||
|
||||
Examples:
|
||||
# Restore entire snapshot
|
||||
|
||||
Reference in new issue
Block a user