Fix the ten findings of the review on the destination, the marker file and chunk sizes
check / check (push) Successful in 9m14s
check / check (push) Successful in 9m14s
README.md: the "Getting started" destination advice, the Full Disk Access paragraph moved above the block with the reopen step, the mount-point case, the remote nuke failure sentence, the hostname setting, the upload order under "Sharing a destination", the confirmation list and the dev stamp sentence. configs/config.example.yml: the storage_url and blob_size_limit comments. docs/ARCHITECTURE.md and docs/DATAMODEL.md: the Snapshot ID and Marker file terms, and "marker file" in place of "destination marker". Model: fable-5-1
This commit is contained in:
1 parent
a8b9c57e93
commit
6e4e8062a1
4 files changed
+46
-30
No files matched your search
@@ -67,6 +67,11 @@ Installing needs Go 1.26.1 or later. `go install` builds the newest tagged
|
|||||||
version. `vaultik version` then reports that tag. Generating a key pair needs
|
version. `vaultik version` then reports that tag. Generating a key pair needs
|
||||||
`age-keygen`, which comes with [age](https://age-encryption.org/).
|
`age-keygen`, which comes with [age](https://age-encryption.org/).
|
||||||
|
|
||||||
|
On macOS, grant your terminal app Full Disk Access before the first backup. Then
|
||||||
|
quit and reopen the terminal. The setting is under System Settings, Privacy &
|
||||||
|
Security, Full Disk Access. Without it the backup stops with a permission error
|
||||||
|
on a protected directory, unless you give `--skip-errors`.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# Install vaultik.
|
# Install vaultik.
|
||||||
go install sneak.berlin/go/vaultik/cmd/vaultik@latest
|
go install sneak.berlin/go/vaultik/cmd/vaultik@latest
|
||||||
@@ -92,15 +97,11 @@ vaultik snapshot create
|
|||||||
vaultik snapshot list
|
vaultik snapshot list
|
||||||
```
|
```
|
||||||
|
|
||||||
The block uses macOS paths. On another system, put the destination in a
|
The block uses macOS paths. On another system, put the destination outside every
|
||||||
directory of your own. The default config backs up your home directory, with a
|
path the config backs up, such as on another disk. The default config backs up
|
||||||
list of excludes. On macOS it also backs up `/Applications`. A configured path
|
your home directory, with a list of excludes. On macOS it also backs up
|
||||||
that does not exist stops the backup, unless you give `--skip-errors`.
|
`/Applications`. A configured path that does not exist stops the backup, unless
|
||||||
|
you give `--skip-errors`.
|
||||||
On macOS, grant your terminal app Full Disk Access before the first backup. The
|
|
||||||
setting is under System Settings, Privacy & Security, Full Disk Access. Without
|
|
||||||
it the backup stops with a permission error on a protected directory, unless you
|
|
||||||
give `--skip-errors`.
|
|
||||||
|
|
||||||
## Everyday use
|
## Everyday use
|
||||||
|
|
||||||
@@ -407,7 +408,7 @@ entry in it that is not excluded.
|
|||||||
A backup does not find a mount point in these cases:
|
A backup does not find a mount point in these cases:
|
||||||
|
|
||||||
- The volume is not mounted.
|
- The volume is not mounted.
|
||||||
- The directory holds no entry that the backup reads.
|
- The directory holds no entry, or only excluded ones.
|
||||||
- The directory is now excluded.
|
- The directory is now excluded.
|
||||||
- The directory is no longer under a configured path.
|
- The directory is no longer under a configured path.
|
||||||
|
|
||||||
@@ -738,8 +739,8 @@ directory at the destination is deleted.
|
|||||||
The command fails before it deletes anything when the destination cannot be
|
The command fails before it deletes anything when the destination cannot be
|
||||||
listed, or has no marker file although the local index holds snapshots.
|
listed, or has no marker file although the local index holds snapshots.
|
||||||
|
|
||||||
If objects remain afterwards, the command names each one and exits non-zero.
|
If a delete fails, the command exits non-zero. Running it again deletes what is
|
||||||
Running it again deletes what is left.
|
left.
|
||||||
|
|
||||||
### `database delete`
|
### `database delete`
|
||||||
|
|
||||||
@@ -911,8 +912,8 @@ A command that reads the config fails when the path from `--config` or
|
|||||||
- `0s` deletes such data at once. A negative value is refused.
|
- `0s` deletes such data at once. A negative value is refused.
|
||||||
- See [Sharing a destination](#sharing-a-destination).
|
- See [Sharing a destination](#sharing-a-destination).
|
||||||
- `hostname`
|
- `hostname`
|
||||||
- The hostname used in snapshot IDs. Default: the system hostname.
|
- The hostname used in snapshot IDs, up to its first `.`. Default: the
|
||||||
- Only the part before the first `.` is used.
|
system hostname.
|
||||||
- `index_path`
|
- `index_path`
|
||||||
- The path of the local index.
|
- The path of the local index.
|
||||||
- It must be an absolute path. A leading `~` is expanded first. The same
|
- It must be an absolute path. A leading `~` is expanded first. The same
|
||||||
@@ -1151,8 +1152,8 @@ Give each machine and each user its own destination, such as its own prefix or
|
|||||||
directory. The rest of this section applies only when two of them share one.
|
directory. The rest of this section applies only when two of them share one.
|
||||||
|
|
||||||
A prune from one machine can meet a backup from another that is still running. A
|
A prune from one machine can meet a backup from another that is still running. A
|
||||||
backup uploads its blobs first, then its database, and its manifest last. Until
|
backup uploads a snapshot's blobs first, then its database, and its manifest
|
||||||
the manifest exists, no manifest lists the new blobs.
|
last. Until the manifest exists, no manifest lists the new blobs.
|
||||||
|
|
||||||
`prune_grace_period` protects those blobs. A prune keeps a blob that no manifest
|
`prune_grace_period` protects those blobs. A prune keeps a blob that no manifest
|
||||||
lists until the blob is older than the grace period. It keeps a directory under
|
lists until the blob is older than the grace period. It keeps a directory under
|
||||||
@@ -1257,8 +1258,13 @@ Neither flag hides these:
|
|||||||
- `130`: the command was interrupted by Ctrl-C or `SIGTERM`.
|
- `130`: the command was interrupted by Ctrl-C or `SIGTERM`.
|
||||||
|
|
||||||
A deletion prompt that is declined or left unanswered ends the command with the
|
A deletion prompt that is declined or left unanswered ends the command with the
|
||||||
error `deletion not confirmed` and exit status `1`. `prune`, `snapshot purge`,
|
error `deletion not confirmed` and exit status `1`. These commands ask for
|
||||||
`snapshot remove` and `database delete` ask one.
|
confirmation:
|
||||||
|
|
||||||
|
- `prune`
|
||||||
|
- `snapshot purge`
|
||||||
|
- `snapshot remove`
|
||||||
|
- `database delete`
|
||||||
|
|
||||||
An interrupted command removes its temporary files before it exits. It gives up
|
An interrupted command removes its temporary files before it exits. It gives up
|
||||||
after 30 seconds.
|
after 30 seconds.
|
||||||
@@ -1487,9 +1493,9 @@ The version a binary reports comes from git. It is the output of
|
|||||||
A plain `docker build .` adds no `-dirty`. The `Dockerfile` runs `git describe`
|
A plain `docker build .` adds no `-dirty`. The `Dockerfile` runs `git describe`
|
||||||
without `--dirty`.
|
without `--dirty`.
|
||||||
|
|
||||||
`make` outside a git checkout, and `docker build .` without `.git`, stamp `dev`.
|
Outside a git checkout, `make` stamps `dev`. So does `docker build .` without
|
||||||
An image built by `script/docker` or `script/cibuild` outside a git checkout
|
`.git`. An image built by `script/docker` or `script/cibuild` outside a git
|
||||||
reports `unknown`.
|
checkout reports `unknown`.
|
||||||
|
|
||||||
A binary built by plain `go install` or `go build` is not stamped. It reports
|
A binary built by plain `go install` or `go build` is not stamped. It reports
|
||||||
the version Go recorded in it:
|
the version Go recorded in it:
|
||||||
|
|||||||
@@ -273,7 +273,8 @@ snapshots:
|
|||||||
exclude:
|
exclude:
|
||||||
- "*.tmp"
|
- "*.tmp"
|
||||||
|
|
||||||
# Storage URL - use either this OR the s3 section below
|
# Storage URL. When it is not set, the endpoint, bucket and prefix in the s3
|
||||||
|
# section below are used instead.
|
||||||
# Supports: s3://bucket/prefix, file:///path, rclone://remote/path
|
# Supports: s3://bucket/prefix, file:///path, rclone://remote/path
|
||||||
# An s3:// URL also needs s3.access_key_id and s3.secret_access_key below
|
# An s3:// URL also needs s3.access_key_id and s3.secret_access_key below
|
||||||
# A file:// path must be absolute, and the directory must already exist:
|
# A file:// path must be absolute, and the directory must already exist:
|
||||||
@@ -339,9 +340,10 @@ storage_url: "rclone://myremote/path/to/backups"
|
|||||||
# Must be at least four times chunk_size (the largest chunk the chunker can
|
# Must be at least four times chunk_size (the largest chunk the chunker can
|
||||||
# emit); a smaller limit would let a single-chunk blob exceed it.
|
# emit); a smaller limit would let a single-chunk blob exceed it.
|
||||||
# Chunking uses no secret (the parameters of the FastCDC chunking algorithm
|
# Chunking uses no secret (the parameters of the FastCDC chunking algorithm
|
||||||
# are fixed and public). At a large limit a blob holds hundreds of chunks, so
|
# are fixed and public). A full blob at a large limit holds hundreds of
|
||||||
# individual chunk lengths are not visible in its size; lowering the limit
|
# chunks, so its size does not show any one chunk's length. The last blob of
|
||||||
# toward chunk_size exposes them.
|
# each configured path can hold a single chunk. Lowering the limit toward
|
||||||
|
# chunk_size exposes chunk lengths.
|
||||||
# A backup needs free temporary space, because each blob is written whole to
|
# A backup needs free temporary space, because each blob is written whole to
|
||||||
# a temporary file before it is uploaded (up to about blob_size_limit; an
|
# a temporary file before it is uploaded (up to about blob_size_limit; an
|
||||||
# rclone destination that cannot stream uploads needs about twice that) and
|
# rclone destination that cannot stream uploads needs about twice that) and
|
||||||
|
|||||||
@@ -16,12 +16,16 @@ Two other documents cover the stored data:
|
|||||||
## Terms
|
## Terms
|
||||||
|
|
||||||
- **Snapshot**: one backup of the paths listed under one name in the config.
|
- **Snapshot**: one backup of the paths listed under one name in the config.
|
||||||
|
- **Snapshot ID**: the name of a snapshot, in the form
|
||||||
|
`<hostname>_<snapshot-name>_<timestamp>`.
|
||||||
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
|
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
|
||||||
of its content.
|
of its content.
|
||||||
- **Blob**: one object at the destination. It holds many chunks, compressed and
|
- **Blob**: one object at the destination. It holds many chunks, compressed and
|
||||||
then encrypted.
|
then encrypted.
|
||||||
- **Local index**: the SQLite database on the backed-up machine.
|
- **Local index**: the SQLite database on the backed-up machine.
|
||||||
- **Destination**: the place the backups are stored.
|
- **Destination**: the place the backups are stored.
|
||||||
|
- **Marker file**: `vaultik-destination.txt` at the destination. See
|
||||||
|
[The marker file](REPOSTRUCTURE.md#the-marker-file).
|
||||||
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
|
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
|
||||||
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
|
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
|
||||||
at the destination.
|
at the destination.
|
||||||
@@ -292,10 +296,10 @@ therefore marks a backup that did not finish.
|
|||||||
checks its chunk hashes.
|
checks its chunk hashes.
|
||||||
|
|
||||||
Given a snapshot ID, the restore looks it up in the local index. When it is not
|
Given a snapshot ID, the restore looks it up in the local index. When it is not
|
||||||
there, the restore checks the destination marker and lists the snapshot's
|
there, the restore checks the marker file and lists the snapshot's directory.
|
||||||
directory. Given a remote key, it reads the local index only to check the
|
Given a remote key, it reads the local index only to check the marker file. It
|
||||||
destination marker. It does not read the manifest. It finds each file's chunks
|
does not read the manifest. It finds each file's chunks and blobs through the
|
||||||
and blobs through the downloaded database.
|
downloaded database.
|
||||||
|
|
||||||
### Checks on the downloaded database
|
### Checks on the downloaded database
|
||||||
|
|
||||||
|
|||||||
+5
-1
@@ -17,11 +17,15 @@ copy that was uploaded with the snapshot.
|
|||||||
## Terms
|
## Terms
|
||||||
|
|
||||||
- **Snapshot**: one backup of the paths listed under one name in the config.
|
- **Snapshot**: one backup of the paths listed under one name in the config.
|
||||||
|
- **Snapshot ID**: the name of a snapshot, in the form
|
||||||
|
`<hostname>_<snapshot-name>_<timestamp>`.
|
||||||
- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
|
- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
|
||||||
content.
|
content.
|
||||||
- **Blob**: one object at the destination. It holds many chunks, compressed and
|
- **Blob**: one object at the destination. It holds many chunks, compressed and
|
||||||
then encrypted.
|
then encrypted.
|
||||||
- **Destination**: the place the backups are stored.
|
- **Destination**: the place the backups are stored.
|
||||||
|
- **Marker file**: `vaultik-destination.txt` at the destination. See
|
||||||
|
[The marker file](REPOSTRUCTURE.md#the-marker-file).
|
||||||
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
|
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
|
||||||
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
|
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
|
||||||
at the destination.
|
at the destination.
|
||||||
@@ -367,7 +371,7 @@ still be in the `-wal` file beside it.
|
|||||||
|
|
||||||
A restore reads no file, chunk or blob rows from the local index. It reads
|
A restore reads no file, chunk or blob rows from the local index. It reads
|
||||||
`snapshots` and `local_meta` there, to look up a snapshot ID and to check the
|
`snapshots` and `local_meta` there, to look up a snapshot ID and to check the
|
||||||
destination marker.
|
marker file.
|
||||||
|
|
||||||
- It downloads and decrypts `db.zst.age` into a temporary directory.
|
- It downloads and decrypts `db.zst.age` into a temporary directory.
|
||||||
- It opens that file read-only. It applies no migration to it.
|
- It opens that file read-only. It applies no migration to it.
|
||||||
|
|||||||
Reference in new issue
Block a user