Read all five documents against the code and fix what the sixth review found

Fixes the four findings of the sixth review: `snapshot remove` with
`--local-only`, the two comma-chained permission lists, six undefined terms,
and the sentence about an unfinished snapshot in the local index.

Then reads every sentence of `README.md`, `ARCHITECTURE.md`,
`docs/DATAMODEL.md`, `docs/REPOSTRUCTURE.md` and the comments in
`config.example.yml` against the code for four faults: a command's needs or
behaviour stated without regard to a flag, a false absolute, three or more
things chained with commas, and a technical word not defined on first use.
Each sentence changed is listed on the pull request.

The help string of `snapshot remove --dry-run` now says it needs no
permission to delete.

Model: fable-5-1
This commit is contained in:
clawbot authored and clawbot committed 2026-11-11 07:01:54 +00:00
1 parent 864e9a76f4
commit 92928c86d4
6 files changed
+341 -193

No files matched your search

+36 -26
View File
@@ -63,13 +63,13 @@ source files
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#code) says which type serves which
table.
- `internal/storage`: the `Storer` interface and its three implementations, for
S3, a local directory and rclone.
S3, a local directory and [rclone](https://rclone.org/).
- `internal/s3`: the S3 client that the S3 `Storer` uses.
- `internal/config`: loads and checks the config file.
- `internal/ui`: formats the status lines, warnings and errors a user sees.
- `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that keeps two data-changing commands from
running at once.
- `internal/pidlock`: the lock that `RunWithApp` takes for a command that
changes data, so that only one runs at a time.
- `internal/globals`: the version, commit and commit date, set when the binary
is built.
- `internal/types`: distinct types for IDs, hashes and paths.
@@ -114,8 +114,12 @@ A command runs through `cli.RunOperation`. It works like this:
waits for the function to return, so that its temporary files are removed.
- The wait ends after 30 seconds at most.
The `config` commands, `database delete`, `completion` and `version` do not
build the application.
These commands do not build the application:
- the `config` commands
- `database delete`
- `completion`
- `version`
## Backup
@@ -163,7 +167,8 @@ A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a
- `repairInterruptedBlobs` deletes the records of blobs whose upload never
finished. Their data is then stored again.
- `loadDatabaseState` loads the known files and chunks into memory.
- `scanPhase` walks the path. It sorts each entry into changed or unchanged.
- `scanPhase` walks the path. It sorts each regular file and symlink into
changed or unchanged.
- `detectDeletedFiles` counts the files that the previous snapshot of this name
had and that are now gone.
- `batchAddFilesToSnapshot` records the unchanged files in the new snapshot.
@@ -184,8 +189,8 @@ Data is stored once at two levels.
- **Files.** A file is not read again when its record in the index still matches
it.
- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk whose
hash is already in the index is not stored again.
- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk that
an uploaded blob already holds, according to the index, is not stored again.
A file's record matches when all of these are unchanged:
@@ -208,8 +213,8 @@ on the content, so an insertion early in a file moves only nearby chunks.
this size. Only a file's last chunk can be smaller than a quarter of it. With
the default of 10 MB, those limits are 40 MB and 2.5 MB.
`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole
file is never held in memory.
`ChunkReaderStreaming` passes each chunk to a callback as it is found. The
chunker buffers at most twice the largest chunk size.
### Blobs
@@ -217,8 +222,9 @@ file is never held in memory.
- It gives the blob a UUID and creates a temporary file.
- It adds each chunk's bytes to a running SHA-256 hash.
- In the same pass it compresses the bytes with zstd and encrypts them with
age. The result goes to the temporary file.
- In the same pass it compresses the bytes with
[zstd](https://github.com/facebook/zstd) and encrypts them with
[age](https://age-encryption.org/). The result goes to the temporary file.
- It records each chunk's offset and length within the blob.
- It closes the blob when the next chunk would pass `blob_size_limit`, and at
the end of each path.
@@ -250,8 +256,8 @@ chunks is uploaded.
- `MarkSnapshotComplete` marks the snapshot complete.
Completion is recorded last on purpose. A backup that stops during the upload
leaves the snapshot unfinished, and the next backup removes it and stores its
data again.
leaves the snapshot unfinished, and the next backup removes it and backs up its
files again.
`ExportSnapshotMetadata` works on a copy of the index, never on the index
itself:
@@ -282,11 +288,13 @@ therefore marks a backup that did not finish.
temporary directory.
- `getFilesToRestore` selects the files, from the paths given.
- `restoreAllFiles` downloads blobs and writes the files.
- With `--verify`, `verifyRestoredFiles` reads each restored file and checks its
chunk hashes.
- With `--verify`, `verifyRestoredFiles` reads each restored regular file and
checks its chunk hashes.
The restore does not read the local index or the manifest. It finds each file's
chunks and blobs through the downloaded database.
Given a snapshot ID, the restore does not read the local index. Given a remote
key, it reads the local index only to check the destination marker. It does not
read the manifest. It finds each file's chunks and blobs through the downloaded
database.
### Checks on the downloaded database
@@ -313,12 +321,13 @@ database is honest. The restore therefore checks it:
- `restoreSweeper` deletes a cached blob once no remaining file needs it.
A regular file is created with mode `0600` and gets its stored mode after its
content is written. A directory gets its stored mode, owner and modification
time after the whole restore.
content is written. A directory gets its stored mode and modification time after
the whole restore, and its owner when the command runs as root.
## Verify
`Vaultik.VerifySnapshotWithOptions` checks a snapshot without the private key:
Without `--deep`, `Vaultik.VerifySnapshotWithOptions` checks a snapshot without
the private key:
- It downloads the manifest.
- It checks that the snapshot's encrypted database is present.
@@ -337,7 +346,7 @@ time after the whole restore.
## Prune
`Vaultik.Prune` does three steps:
`Vaultik.Prune` calls `EnsureStorageBinding` and then does three steps:
- `CleanupLocalSnapshots` drops from the local index each snapshot whose
manifest is gone from the destination.
@@ -382,17 +391,18 @@ With no `storage_url`, it builds the S3 implementation from the `s3` settings.
Two checks keep the local index and the destination in step.
- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when
the file is reported missing while the local index holds snapshots.
the file is reported missing while the local index holds snapshots. It skips
the check when the local index belongs to another destination.
- `EnsureStorageBinding` records the configured destination in the local index
on first use. Later it fails when the config names a different destination.
Every command that changes data calls `EnsureStorageBinding` first.
Every command that takes the lock calls `EnsureStorageBinding` first.
## Concurrency
- Each open database uses one connection, so its queries run one at a time.
- The index runs in SQLite's WAL mode. A second process can therefore read it
while a backup writes to it.
- The index runs in SQLite's write-ahead log (WAL) mode. A second process can
therefore read it while a backup writes to it.
- `Packer` guards its state with a mutex.
- `Scanner` holds `packerMu` while a chunk is added to the packer and while a
blob is closed.
+51 -28
View File
@@ -1,7 +1,8 @@
# vaultik data model
`vaultik` keeps a SQLite database on the backed-up machine, called the local
index. This document describes its tables and how each command uses them.
index. This document describes its tables, and how a backup, a restore and
`prune` use them.
The local index has two jobs:
@@ -27,10 +28,11 @@ with the snapshot.
## Where the index is
The index is the file `index.sqlite` in the platform data directory. That is
`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/`
on macOS. The `index_path` setting and the `VAULTIK_INDEX_PATH` environment
variable change the path.
The index is the file `index.sqlite` in the platform data directory. By default
that is `~/.local/share/vaultik/` on Linux and
`~/Library/Application Support/vaultik/` on macOS. The `index_path` setting and
the `VAULTIK_INDEX_PATH` environment variable change the path. So does
`XDG_DATA_HOME`.
The file is readable only by its owner, because it lists every backed-up path.
`vaultik` sets that mode each time it opens the index.
@@ -74,15 +76,15 @@ One row for each backed-up path: a regular file, a symlink or a directory.
- `id`: a UUID. Primary key.
- `path`: the absolute path. Unique.
- `source_path`: the configured path that the scan found this entry under. A
restore does not read it.
- `source_path`: the configured path under which the scan found this entry,
made absolute with symlinks resolved. A restore does not read it.
- `mtime`: the modification time.
- `mtime_nsec`: the nanoseconds within that second.
- `size`: the size in bytes.
- `mode`: the permission bits and the file type.
- `uid`: the owner's user ID.
- `gid`: the owner's group ID.
- `link_target`: the target of a symlink. It is `NULL` for other entries.
- `link_target`: the target of a symlink. It is empty for other entries.
`idx_files_path` indexes `path`.
@@ -161,7 +163,8 @@ A restore finds the blob for each chunk through this table.
One row for each snapshot.
- `id`: the snapshot ID. Primary key.
- `hostname`: the hostname of the machine that made the backup.
- `hostname`: the `hostname` setting of the config. When that setting is empty,
it is the machine's hostname.
- `vaultik_version`: the `vaultik` version that made it.
- `vaultik_git_revision`: the git commit of that version.
- `started_at`: when the backup started.
@@ -189,8 +192,8 @@ example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
- the snapshot name
- the start time, in UTC
`snapshot list` shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE` and
`new_chunk_size` as `NEW CHUNK SIZE`.
The `snapshot list` table shows `uncompressed_chunk_size` as
`UNCOMPRESSED SIZE` and `new_chunk_size` as `NEW CHUNK SIZE`.
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
does not show it. The next backup deletes it, and so does `prune`.
@@ -233,10 +236,10 @@ A directory is recorded here when all of these hold:
- The backup found an entry in it that is not excluded.
- It is on a different filesystem from its parent directory.
The device number is read after the directory has been listed. Listing an
automount point mounts its volume, so the number is that of the volume. An
automount point whose volume cannot be mounted is not recorded, because the
backup finds no entry in it.
The device number is read after the directory has been listed. An automount
point is a directory whose volume is mounted when it is first listed, so the
number is that of the volume. An automount point whose volume cannot be mounted
is not recorded, because the backup finds no entry in it.
`snapshot create --prune` reads this table. It skips the prune when an earlier
snapshot of the same name recorded a mount point that the new backup did not
@@ -266,8 +269,14 @@ Settings that belong to this copy of the index.
- `value`: its value.
The only key is `storage_url`. It holds the destination that the index was
first used with. A command that changes data refuses to run when the config
names a different destination.
first used with. These commands refuse to run when the config names a different
destination:
- `snapshot create`
- `snapshot purge`
- `snapshot remove`
- `prune`
- `remote nuke`
### `schema_migrations`
@@ -282,13 +291,16 @@ The migrations that have run.
A backup works on the tables in this order:
- It saves the destination in `local_meta` when the index has none yet.
- It deletes unfinished snapshots, and the rows that no snapshot uses.
- It creates a row in `snapshots`, with `completed_at` set to `NULL`.
- It deletes the `blobs` rows that were never uploaded.
- It walks each path and compares every entry with its row in `files`.
- It walks each path. It compares each regular file and symlink with its row in
`files`.
- It links each unchanged file to the snapshot in `snapshot_files`.
- It records each mount point in `snapshot_mount_points`.
- It reads each changed file and stores it. See below.
- It reads each changed regular file and stores it. See below. A changed symlink
is recorded without being read.
- It fills `snapshot_blobs` with every blob the snapshot uses.
- It stores the snapshot's totals in `snapshots`.
- It uploads the snapshot's metadata.
@@ -303,7 +315,8 @@ in any of these:
- owner or group
- link target
A file is also read again when it lists a chunk that no uploaded blob holds.
A file is also read again when it lists a chunk that no uploaded blob holds. A
file with no row in `files` is read too.
Storing a changed file writes these rows:
@@ -316,8 +329,10 @@ Storing a changed file writes these rows:
- `uploaded_ts` and `compressed_size` in `blobs`
- a row in `snapshot_blobs`
- a row in `uploads`, unless the blob was already at the destination
- The row in `files`, with its `file_chunks`, `chunk_files` and
`snapshot_files` rows.
- The row in `files`, and its rows in these tables:
- `file_chunks`
- `chunk_files`
- `snapshot_files`
The row in `files` is written last. It is written only after every blob that
holds the file's new chunks is stored.
@@ -334,31 +349,38 @@ The metadata of a snapshot is made from a copy of the index:
uploaded database therefore names the destination, and lists the migrations
that have run.
- `VACUUM` rewrites the copy, so that no deleted row is left in the file.
- The copy is compressed with zstd and encrypted with age.
- The copy is compressed with [zstd](https://github.com/facebook/zstd) and
encrypted with [age](https://age-encryption.org/).
- The manifest is built from the copy's `snapshot_blobs` table.
- The copy is uploaded as `metadata/<remote-key>/db.zst.age`.
- The manifest is uploaded as `metadata/<remote-key>/manifest.json.zst`.
The copy is made through SQLite, not by copying the file. The index runs in WAL
mode, so rows that are already committed can still be in the `-wal` file beside
it.
The copy is made through SQLite, not by copying the file. The index runs in
SQLite's write-ahead log (WAL) mode, so rows that are already committed can
still be in the `-wal` file beside it.
### Restore
A restore does not use the local index.
A restore reads no file, chunk or blob rows from the local index. Given a
remote key, it reads `snapshots` and `local_meta` there to check the destination
marker.
- It downloads and decrypts `db.zst.age` into a temporary directory.
- It opens that file read-only. It applies no migration to it.
- It reads the snapshot's row in `snapshots`, to check that this is the right
database.
- It reads the files to restore from `files`.
- It finds each file's chunks in `file_chunks`.
- It finds the blob and offset of each chunk in `blob_chunks`.
- It finds the name of each blob in `blobs`.
- With `--verify`, it reads each chunk's size from `chunks`.
It does not read the manifest.
### Prune
`prune` changes the local index in two ways:
`prune` first saves the destination in `local_meta` when the index has none yet.
It then changes the local index in two ways:
- It deletes each snapshot whose manifest is gone from the destination.
- It deletes unfinished snapshots, and the rows that no snapshot uses.
@@ -389,6 +411,7 @@ Each repository type in `internal/database` serves one or more tables:
- `snapshot_files`
- `snapshot_blobs`
- `snapshot_mount_points`
- `uploads`, where it deletes a snapshot's rows
- `UploadRepository`: `uploads`
- `LocalMetaRepository`: `local_meta`
+48 -31
View File
@@ -4,8 +4,8 @@ This document describes what `vaultik` stores at the destination. It covers the
layout, what the layout protects, and the risks it accepts.
The destination is the place the backups are stored. It is an S3 bucket and
prefix, a local directory or an rclone remote. The layout is the same on all
three.
prefix, a local directory or a remote of [rclone](https://rclone.org/). The
layout is the same on all three.
## Terms
@@ -16,7 +16,8 @@ three.
content.
- **Blob**: one object under `blobs/`. It holds many chunks.
- **Local index**: the SQLite database on the backed-up machine.
- **Recipient**: an age public key that backups are encrypted to.
- **Recipient**: an [age](https://age-encryption.org/) public key that backups
are encrypted to.
## Layout
@@ -49,12 +50,15 @@ unplugged volume leaves an empty directory at its mount point. Without the
marker, that directory would look like a destination whose snapshots are all
gone.
The check applies while the local index holds snapshots. When the destination
then answers that the marker does not exist, commands treat the destination as
not present:
The check applies while the local index holds snapshots and belongs to the
configured destination. When the destination then answers that the marker does
not exist, commands treat the destination as not present:
- `snapshot create` refuses to write to it.
- `prune`, `snapshot purge` and `remote nuke` fail.
- These commands fail:
- `prune`
- `snapshot purge`
- `remote nuke`
- `snapshot list` shows the local index alone.
- `snapshot remove` deletes nothing at the destination.
@@ -73,8 +77,10 @@ A blob holds the content of many chunks, one after another.
- **Path**: `blobs/<first 2 characters>/<next 2 characters>/<full hash>`.
- **Example**:
`blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`.
- **Format**: the chunk data, compressed with zstd and then encrypted with age.
- **Recipients**: the public keys in `age_recipients` when the blob was written.
- **Format**: the chunk data, compressed with
[zstd](https://github.com/facebook/zstd) and then encrypted with age.
- **Recipients**: the public keys in the `age_recipients` setting when the blob
was written.
The two directory levels keep any one directory from holding too many objects.
@@ -257,9 +263,9 @@ therefore does not stop overwrites.
An object's size follows exactly from the size of its compressed contents.
All the new chunks in one blob share one zstd stream. The stream looks back 8
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at
`blob_size_limit` and at the end of each configured path. A chunk that is
already stored is never packed again.
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at the
`blob_size_limit` setting and at the end of each configured path. A chunk that
the local index records as uploaded is never packed again.
Data that an outsider writes compresses smaller when it matches a secret near it
in the same blob, so the blob gets smaller. The blob's size therefore shows the
@@ -277,14 +283,15 @@ Back up a directory that outsiders can write to as a snapshot of its own.
### Chunk boundaries use no secret
The FastCDC parameters are fixed and public. `chunk_size` sets the average chunk
size. No chunk is larger than four times this size. Only a file's last chunk can
be smaller than a quarter of it. With the default of 10 MB, those limits are 40
MB and 2.5 MB. A file of 2.5 MB or less is one chunk.
`vaultik` finds chunk boundaries with the FastCDC algorithm. Its parameters are
fixed and public. `chunk_size` sets the average chunk size. No chunk is larger
than four times this size. Only a file's last chunk can be smaller than a
quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A
file of 2.5 MB or less is one chunk. An empty file has no chunk.
At the default `blob_size_limit` of 10 GB, a blob holds hundreds of chunks. The
size of the blob then does not show the size of any one chunk. A limit close to
`chunk_size` begins to show chunk sizes.
At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of
chunks. The size of the blob then does not show the size of any one chunk. A
limit close to `chunk_size` begins to show chunk sizes.
### Decrypted data on the local disk
@@ -307,14 +314,24 @@ Trust the temporary directory as much as the directory you restore into.
### Commands need different permissions
`snapshot create` needs only permission to write objects. Other commands need
more:
`snapshot create` without `--prune` needs only permission to write objects.
Other commands need more:
- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list
and read.
- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete.
- These commands list and read:
- `snapshot list`
- `snapshot verify`
- `snapshot restore`
- `remote info`
- These commands list and delete:
- `snapshot purge`
- `snapshot remove`
- `remote nuke`
- `prune` lists, reads and deletes.
`snapshot verify` and `snapshot restore` list only when they are given a remote
key. `snapshot remove --local-only` does not reach the destination.
`snapshot remove --dry-run` deletes nothing.
`snapshot create --prune` runs a prune on the backed-up machine. That machine
then holds credentials that can delete its own backups. To keep it to
write-only credentials, run `prune` from a separate, trusted machine.
@@ -322,8 +339,8 @@ write-only credentials, run `prune` from a separate, trusted machine.
### New recipients cannot read old data
Changing `age_recipients` does not re-encrypt what is already stored. A chunk
that is already stored is not written again, so it stays encrypted to the
earlier recipients.
that the local index records as uploaded is not written again, so it stays
encrypted to the earlier recipients.
A new snapshot that uses such chunks cannot be restored with a new recipient's
key alone. To make everything readable by a new key, run
@@ -335,8 +352,8 @@ key alone. To make everything readable by a new key, run
recipients.
The encrypted data is held by a third party for a long time. There is no second
kind of recipient to fall back on if X25519 is broken, and no post-quantum
option.
kind of recipient to fall back on if X25519 is broken, and no kind that resists
a quantum computer.
## Order of writes
@@ -353,12 +370,12 @@ therefore belongs to a backup that did not finish.
The local index marks a snapshot complete only after both metadata files are
uploaded. A backup that stops during that upload leaves the snapshot
unfinished. The next backup removes it from the local index and stores its data
again.
unfinished. The next backup removes it from the local index and backs up its
files again.
One short gap remains. A backup can stop after the upload and before the
snapshot is marked complete. The destination then holds a snapshot that can be
restored, and the local index does not list it. `snapshot list` shows it as
restored, and the local index marks it unfinished. `snapshot list` shows it as
remote only.
An upload that is cut off leaves no partial object under the final name on S3