Fix the ten findings of the review on the destination, the marker file and chunk sizes
check / check (push) Successful in 9m14s

README.md: the "Getting started" destination advice, the Full Disk Access
paragraph moved above the block with the reopen step, the mount-point
case, the remote nuke failure sentence, the hostname setting, the upload
order under "Sharing a destination", the confirmation list and the dev
stamp sentence. configs/config.example.yml: the storage_url and
blob_size_limit comments. docs/ARCHITECTURE.md and docs/DATAMODEL.md:
the Snapshot ID and Marker file terms, and "marker file" in place of
"destination marker".

Model: fable-5-1
This commit is contained in:
clawbot committed 2026-11-11 09:11:51 +00:00
1 parent a8b9c57e93
commit 6e4e8062a1
4 files changed
+46 -30

No files matched your search

+27 -21
View File
@@ -67,6 +67,11 @@ Installing needs Go 1.26.1 or later. `go install` builds the newest tagged
version. `vaultik version` then reports that tag. Generating a key pair needs version. `vaultik version` then reports that tag. Generating a key pair needs
`age-keygen`, which comes with [age](https://age-encryption.org/). `age-keygen`, which comes with [age](https://age-encryption.org/).
On macOS, grant your terminal app Full Disk Access before the first backup. Then
quit and reopen the terminal. The setting is under System Settings, Privacy &
Security, Full Disk Access. Without it the backup stops with a permission error
on a protected directory, unless you give `--skip-errors`.
```sh ```sh
# Install vaultik. # Install vaultik.
go install sneak.berlin/go/vaultik/cmd/vaultik@latest go install sneak.berlin/go/vaultik/cmd/vaultik@latest
@@ -92,15 +97,11 @@ vaultik snapshot create
vaultik snapshot list vaultik snapshot list
``` ```
The block uses macOS paths. On another system, put the destination in a The block uses macOS paths. On another system, put the destination outside every
directory of your own. The default config backs up your home directory, with a path the config backs up, such as on another disk. The default config backs up
list of excludes. On macOS it also backs up `/Applications`. A configured path your home directory, with a list of excludes. On macOS it also backs up
that does not exist stops the backup, unless you give `--skip-errors`. `/Applications`. A configured path that does not exist stops the backup, unless
you give `--skip-errors`.
On macOS, grant your terminal app Full Disk Access before the first backup. The
setting is under System Settings, Privacy & Security, Full Disk Access. Without
it the backup stops with a permission error on a protected directory, unless you
give `--skip-errors`.
## Everyday use ## Everyday use
@@ -407,7 +408,7 @@ entry in it that is not excluded.
A backup does not find a mount point in these cases: A backup does not find a mount point in these cases:
- The volume is not mounted. - The volume is not mounted.
- The directory holds no entry that the backup reads. - The directory holds no entry, or only excluded ones.
- The directory is now excluded. - The directory is now excluded.
- The directory is no longer under a configured path. - The directory is no longer under a configured path.
@@ -738,8 +739,8 @@ directory at the destination is deleted.
The command fails before it deletes anything when the destination cannot be The command fails before it deletes anything when the destination cannot be
listed, or has no marker file although the local index holds snapshots. listed, or has no marker file although the local index holds snapshots.
If objects remain afterwards, the command names each one and exits non-zero. If a delete fails, the command exits non-zero. Running it again deletes what is
Running it again deletes what is left. left.
### `database delete` ### `database delete`
@@ -911,8 +912,8 @@ A command that reads the config fails when the path from `--config` or
- `0s` deletes such data at once. A negative value is refused. - `0s` deletes such data at once. A negative value is refused.
- See [Sharing a destination](#sharing-a-destination). - See [Sharing a destination](#sharing-a-destination).
- `hostname` - `hostname`
- The hostname used in snapshot IDs. Default: the system hostname. - The hostname used in snapshot IDs, up to its first `.`. Default: the
- Only the part before the first `.` is used. system hostname.
- `index_path` - `index_path`
- The path of the local index. - The path of the local index.
- It must be an absolute path. A leading `~` is expanded first. The same - It must be an absolute path. A leading `~` is expanded first. The same
@@ -1151,8 +1152,8 @@ Give each machine and each user its own destination, such as its own prefix or
directory. The rest of this section applies only when two of them share one. directory. The rest of this section applies only when two of them share one.
A prune from one machine can meet a backup from another that is still running. A A prune from one machine can meet a backup from another that is still running. A
backup uploads its blobs first, then its database, and its manifest last. Until backup uploads a snapshot's blobs first, then its database, and its manifest
the manifest exists, no manifest lists the new blobs. last. Until the manifest exists, no manifest lists the new blobs.
`prune_grace_period` protects those blobs. A prune keeps a blob that no manifest `prune_grace_period` protects those blobs. A prune keeps a blob that no manifest
lists until the blob is older than the grace period. It keeps a directory under lists until the blob is older than the grace period. It keeps a directory under
@@ -1257,8 +1258,13 @@ Neither flag hides these:
- `130`: the command was interrupted by Ctrl-C or `SIGTERM`. - `130`: the command was interrupted by Ctrl-C or `SIGTERM`.
A deletion prompt that is declined or left unanswered ends the command with the A deletion prompt that is declined or left unanswered ends the command with the
error `deletion not confirmed` and exit status `1`. `prune`, `snapshot purge`, error `deletion not confirmed` and exit status `1`. These commands ask for
`snapshot remove` and `database delete` ask one. confirmation:
- `prune`
- `snapshot purge`
- `snapshot remove`
- `database delete`
An interrupted command removes its temporary files before it exits. It gives up An interrupted command removes its temporary files before it exits. It gives up
after 30 seconds. after 30 seconds.
@@ -1487,9 +1493,9 @@ The version a binary reports comes from git. It is the output of
A plain `docker build .` adds no `-dirty`. The `Dockerfile` runs `git describe` A plain `docker build .` adds no `-dirty`. The `Dockerfile` runs `git describe`
without `--dirty`. without `--dirty`.
`make` outside a git checkout, and `docker build .` without `.git`, stamp `dev`. Outside a git checkout, `make` stamps `dev`. So does `docker build .` without
An image built by `script/docker` or `script/cibuild` outside a git checkout `.git`. An image built by `script/docker` or `script/cibuild` outside a git
reports `unknown`. checkout reports `unknown`.
A binary built by plain `go install` or `go build` is not stamped. It reports A binary built by plain `go install` or `go build` is not stamped. It reports
the version Go recorded in it: the version Go recorded in it:
+6 -4
View File
@@ -273,7 +273,8 @@ snapshots:
exclude: exclude:
- "*.tmp" - "*.tmp"
# Storage URL - use either this OR the s3 section below # Storage URL. When it is not set, the endpoint, bucket and prefix in the s3
# section below are used instead.
# Supports: s3://bucket/prefix, file:///path, rclone://remote/path # Supports: s3://bucket/prefix, file:///path, rclone://remote/path
# An s3:// URL also needs s3.access_key_id and s3.secret_access_key below # An s3:// URL also needs s3.access_key_id and s3.secret_access_key below
# A file:// path must be absolute, and the directory must already exist: # A file:// path must be absolute, and the directory must already exist:
@@ -339,9 +340,10 @@ storage_url: "rclone://myremote/path/to/backups"
# Must be at least four times chunk_size (the largest chunk the chunker can # Must be at least four times chunk_size (the largest chunk the chunker can
# emit); a smaller limit would let a single-chunk blob exceed it. # emit); a smaller limit would let a single-chunk blob exceed it.
# Chunking uses no secret (the parameters of the FastCDC chunking algorithm # Chunking uses no secret (the parameters of the FastCDC chunking algorithm
# are fixed and public). At a large limit a blob holds hundreds of chunks, so # are fixed and public). A full blob at a large limit holds hundreds of
# individual chunk lengths are not visible in its size; lowering the limit # chunks, so its size does not show any one chunk's length. The last blob of
# toward chunk_size exposes them. # each configured path can hold a single chunk. Lowering the limit toward
# chunk_size exposes chunk lengths.
# A backup needs free temporary space, because each blob is written whole to # A backup needs free temporary space, because each blob is written whole to
# a temporary file before it is uploaded (up to about blob_size_limit; an # a temporary file before it is uploaded (up to about blob_size_limit; an
# rclone destination that cannot stream uploads needs about twice that) and # rclone destination that cannot stream uploads needs about twice that) and
+8 -4
View File
@@ -16,12 +16,16 @@ Two other documents cover the stored data:
## Terms ## Terms
- **Snapshot**: one backup of the paths listed under one name in the config. - **Snapshot**: one backup of the paths listed under one name in the config.
- **Snapshot ID**: the name of a snapshot, in the form
`<hostname>_<snapshot-name>_<timestamp>`.
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash - **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
of its content. of its content.
- **Blob**: one object at the destination. It holds many chunks, compressed and - **Blob**: one object at the destination. It holds many chunks, compressed and
then encrypted. then encrypted.
- **Local index**: the SQLite database on the backed-up machine. - **Local index**: the SQLite database on the backed-up machine.
- **Destination**: the place the backups are stored. - **Destination**: the place the backups are stored.
- **Marker file**: `vaultik-destination.txt` at the destination. See
[The marker file](REPOSTRUCTURE.md#the-marker-file).
- **Manifest**: the unencrypted list of the blobs one snapshot uses. - **Manifest**: the unencrypted list of the blobs one snapshot uses.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory - **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
at the destination. at the destination.
@@ -292,10 +296,10 @@ therefore marks a backup that did not finish.
checks its chunk hashes. checks its chunk hashes.
Given a snapshot ID, the restore looks it up in the local index. When it is not Given a snapshot ID, the restore looks it up in the local index. When it is not
there, the restore checks the destination marker and lists the snapshot's there, the restore checks the marker file and lists the snapshot's directory.
directory. Given a remote key, it reads the local index only to check the Given a remote key, it reads the local index only to check the marker file. It
destination marker. It does not read the manifest. It finds each file's chunks does not read the manifest. It finds each file's chunks and blobs through the
and blobs through the downloaded database. downloaded database.
### Checks on the downloaded database ### Checks on the downloaded database
+5 -1
View File
@@ -17,11 +17,15 @@ copy that was uploaded with the snapshot.
## Terms ## Terms
- **Snapshot**: one backup of the paths listed under one name in the config. - **Snapshot**: one backup of the paths listed under one name in the config.
- **Snapshot ID**: the name of a snapshot, in the form
`<hostname>_<snapshot-name>_<timestamp>`.
- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that - **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
content. content.
- **Blob**: one object at the destination. It holds many chunks, compressed and - **Blob**: one object at the destination. It holds many chunks, compressed and
then encrypted. then encrypted.
- **Destination**: the place the backups are stored. - **Destination**: the place the backups are stored.
- **Marker file**: `vaultik-destination.txt` at the destination. See
[The marker file](REPOSTRUCTURE.md#the-marker-file).
- **Manifest**: the unencrypted list of the blobs one snapshot uses. - **Manifest**: the unencrypted list of the blobs one snapshot uses.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory - **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
at the destination. at the destination.
@@ -367,7 +371,7 @@ still be in the `-wal` file beside it.
A restore reads no file, chunk or blob rows from the local index. It reads A restore reads no file, chunk or blob rows from the local index. It reads
`snapshots` and `local_meta` there, to look up a snapshot ID and to check the `snapshots` and `local_meta` there, to look up a snapshot ID and to check the
destination marker. marker file.
- It downloads and decrypts `db.zst.age` into a temporary directory. - It downloads and decrypts `db.zst.age` into a temporary directory.
- It opens that file read-only. It applies no migration to it. - It opens that file read-only. It applies no migration to it.