From 6e4e8062a138819aeece1e53ec5112b94bf37724 Mon Sep 17 00:00:00 2001 From: clawbot Date: Wed, 11 Nov 2026 09:11:51 +0000 Subject: [PATCH] Fix the ten findings of the review on the destination, the marker file and chunk sizes README.md: the "Getting started" destination advice, the Full Disk Access paragraph moved above the block with the reopen step, the mount-point case, the remote nuke failure sentence, the hostname setting, the upload order under "Sharing a destination", the confirmation list and the dev stamp sentence. configs/config.example.yml: the storage_url and blob_size_limit comments. docs/ARCHITECTURE.md and docs/DATAMODEL.md: the Snapshot ID and Marker file terms, and "marker file" in place of "destination marker". Model: fable-5-1 --- README.md | 48 +++++++++++++++++++++----------------- configs/config.example.yml | 10 ++++---- docs/ARCHITECTURE.md | 12 ++++++---- docs/DATAMODEL.md | 6 ++++- 4 files changed, 46 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index 63cfa12..b94c7e3 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,11 @@ Installing needs Go 1.26.1 or later. `go install` builds the newest tagged version. `vaultik version` then reports that tag. Generating a key pair needs `age-keygen`, which comes with [age](https://age-encryption.org/). +On macOS, grant your terminal app Full Disk Access before the first backup. Then +quit and reopen the terminal. The setting is under System Settings, Privacy & +Security, Full Disk Access. Without it the backup stops with a permission error +on a protected directory, unless you give `--skip-errors`. + ```sh # Install vaultik. go install sneak.berlin/go/vaultik/cmd/vaultik@latest @@ -92,15 +97,11 @@ vaultik snapshot create vaultik snapshot list ``` -The block uses macOS paths. On another system, put the destination in a -directory of your own. The default config backs up your home directory, with a -list of excludes. On macOS it also backs up `/Applications`. A configured path -that does not exist stops the backup, unless you give `--skip-errors`. - -On macOS, grant your terminal app Full Disk Access before the first backup. The -setting is under System Settings, Privacy & Security, Full Disk Access. Without -it the backup stops with a permission error on a protected directory, unless you -give `--skip-errors`. +The block uses macOS paths. On another system, put the destination outside every +path the config backs up, such as on another disk. The default config backs up +your home directory, with a list of excludes. On macOS it also backs up +`/Applications`. A configured path that does not exist stops the backup, unless +you give `--skip-errors`. ## Everyday use @@ -407,7 +408,7 @@ entry in it that is not excluded. A backup does not find a mount point in these cases: - The volume is not mounted. -- The directory holds no entry that the backup reads. +- The directory holds no entry, or only excluded ones. - The directory is now excluded. - The directory is no longer under a configured path. @@ -738,8 +739,8 @@ directory at the destination is deleted. The command fails before it deletes anything when the destination cannot be listed, or has no marker file although the local index holds snapshots. -If objects remain afterwards, the command names each one and exits non-zero. -Running it again deletes what is left. +If a delete fails, the command exits non-zero. Running it again deletes what is +left. ### `database delete` @@ -911,8 +912,8 @@ A command that reads the config fails when the path from `--config` or - `0s` deletes such data at once. A negative value is refused. - See [Sharing a destination](#sharing-a-destination). - `hostname` - - The hostname used in snapshot IDs. Default: the system hostname. - - Only the part before the first `.` is used. + - The hostname used in snapshot IDs, up to its first `.`. Default: the + system hostname. - `index_path` - The path of the local index. - It must be an absolute path. A leading `~` is expanded first. The same @@ -1151,8 +1152,8 @@ Give each machine and each user its own destination, such as its own prefix or directory. The rest of this section applies only when two of them share one. A prune from one machine can meet a backup from another that is still running. A -backup uploads its blobs first, then its database, and its manifest last. Until -the manifest exists, no manifest lists the new blobs. +backup uploads a snapshot's blobs first, then its database, and its manifest +last. Until the manifest exists, no manifest lists the new blobs. `prune_grace_period` protects those blobs. A prune keeps a blob that no manifest lists until the blob is older than the grace period. It keeps a directory under @@ -1257,8 +1258,13 @@ Neither flag hides these: - `130`: the command was interrupted by Ctrl-C or `SIGTERM`. A deletion prompt that is declined or left unanswered ends the command with the -error `deletion not confirmed` and exit status `1`. `prune`, `snapshot purge`, -`snapshot remove` and `database delete` ask one. +error `deletion not confirmed` and exit status `1`. These commands ask for +confirmation: + +- `prune` +- `snapshot purge` +- `snapshot remove` +- `database delete` An interrupted command removes its temporary files before it exits. It gives up after 30 seconds. @@ -1487,9 +1493,9 @@ The version a binary reports comes from git. It is the output of A plain `docker build .` adds no `-dirty`. The `Dockerfile` runs `git describe` without `--dirty`. -`make` outside a git checkout, and `docker build .` without `.git`, stamp `dev`. -An image built by `script/docker` or `script/cibuild` outside a git checkout -reports `unknown`. +Outside a git checkout, `make` stamps `dev`. So does `docker build .` without +`.git`. An image built by `script/docker` or `script/cibuild` outside a git +checkout reports `unknown`. A binary built by plain `go install` or `go build` is not stamped. It reports the version Go recorded in it: diff --git a/configs/config.example.yml b/configs/config.example.yml index 563d10c..f183744 100644 --- a/configs/config.example.yml +++ b/configs/config.example.yml @@ -273,7 +273,8 @@ snapshots: exclude: - "*.tmp" -# Storage URL - use either this OR the s3 section below +# Storage URL. When it is not set, the endpoint, bucket and prefix in the s3 +# section below are used instead. # Supports: s3://bucket/prefix, file:///path, rclone://remote/path # An s3:// URL also needs s3.access_key_id and s3.secret_access_key below # A file:// path must be absolute, and the directory must already exist: @@ -339,9 +340,10 @@ storage_url: "rclone://myremote/path/to/backups" # Must be at least four times chunk_size (the largest chunk the chunker can # emit); a smaller limit would let a single-chunk blob exceed it. # Chunking uses no secret (the parameters of the FastCDC chunking algorithm -# are fixed and public). At a large limit a blob holds hundreds of chunks, so -# individual chunk lengths are not visible in its size; lowering the limit -# toward chunk_size exposes them. +# are fixed and public). A full blob at a large limit holds hundreds of +# chunks, so its size does not show any one chunk's length. The last blob of +# each configured path can hold a single chunk. Lowering the limit toward +# chunk_size exposes chunk lengths. # A backup needs free temporary space, because each blob is written whole to # a temporary file before it is uploaded (up to about blob_size_limit; an # rclone destination that cannot stream uploads needs about twice that) and diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 7af2993..34d4626 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -16,12 +16,16 @@ Two other documents cover the stored data: ## Terms - **Snapshot**: one backup of the paths listed under one name in the config. +- **Snapshot ID**: the name of a snapshot, in the form + `__`. - **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash of its content. - **Blob**: one object at the destination. It holds many chunks, compressed and then encrypted. - **Local index**: the SQLite database on the backed-up machine. - **Destination**: the place the backups are stored. +- **Marker file**: `vaultik-destination.txt` at the destination. See + [The marker file](REPOSTRUCTURE.md#the-marker-file). - **Manifest**: the unencrypted list of the blobs one snapshot uses. - **Remote key**: a hash of the snapshot ID. It names the snapshot's directory at the destination. @@ -292,10 +296,10 @@ therefore marks a backup that did not finish. checks its chunk hashes. Given a snapshot ID, the restore looks it up in the local index. When it is not -there, the restore checks the destination marker and lists the snapshot's -directory. Given a remote key, it reads the local index only to check the -destination marker. It does not read the manifest. It finds each file's chunks -and blobs through the downloaded database. +there, the restore checks the marker file and lists the snapshot's directory. +Given a remote key, it reads the local index only to check the marker file. It +does not read the manifest. It finds each file's chunks and blobs through the +downloaded database. ### Checks on the downloaded database diff --git a/docs/DATAMODEL.md b/docs/DATAMODEL.md index e40a0b4..65a4227 100644 --- a/docs/DATAMODEL.md +++ b/docs/DATAMODEL.md @@ -17,11 +17,15 @@ copy that was uploaded with the snapshot. ## Terms - **Snapshot**: one backup of the paths listed under one name in the config. +- **Snapshot ID**: the name of a snapshot, in the form + `__`. - **Chunk**: a piece of a file's content, named by the SHA-256 hash of that content. - **Blob**: one object at the destination. It holds many chunks, compressed and then encrypted. - **Destination**: the place the backups are stored. +- **Marker file**: `vaultik-destination.txt` at the destination. See + [The marker file](REPOSTRUCTURE.md#the-marker-file). - **Manifest**: the unencrypted list of the blobs one snapshot uses. - **Remote key**: a hash of the snapshot ID. It names the snapshot's directory at the destination. @@ -367,7 +371,7 @@ still be in the `-wal` file beside it. A restore reads no file, chunk or blob rows from the local index. It reads `snapshots` and `local_meta` there, to look up a snapshot ID and to check the -destination marker. +marker file. - It downloads and decrypts `db.zst.age` into a temporary directory. - It opens that file read-only. It applies no migration to it.