Read all five documents against the code and fix what the sixth review found
check / check (push) Successful in 12m5s
check / check (push) Successful in 12m5s
Fixes the four findings of the sixth review: `snapshot remove` with `--local-only`, the two comma-chained permission lists, six undefined terms, and the sentence about an unfinished snapshot in the local index. Then reads every sentence of `README.md`, `ARCHITECTURE.md`, `docs/DATAMODEL.md`, `docs/REPOSTRUCTURE.md` and the comments in `config.example.yml` against the code for four faults: a command's needs or behaviour stated without regard to a flag, a false absolute, three or more things chained with commas, and a technical word not defined on first use. Each sentence changed is listed on the pull request. The help string of `snapshot remove --dry-run` now says it needs no permission to delete. Model: fable-5-1
This commit is contained in:
1 parent
a133b8838e
commit
46b2a9fc78
6 files changed
+341
-193
No files matched your search
+51
-28
@@ -1,7 +1,8 @@
|
||||
# vaultik data model
|
||||
|
||||
`vaultik` keeps a SQLite database on the backed-up machine, called the local
|
||||
index. This document describes its tables and how each command uses them.
|
||||
index. This document describes its tables, and how a backup, a restore and
|
||||
`prune` use them.
|
||||
|
||||
The local index has two jobs:
|
||||
|
||||
@@ -27,10 +28,11 @@ with the snapshot.
|
||||
|
||||
## Where the index is
|
||||
|
||||
The index is the file `index.sqlite` in the platform data directory. That is
|
||||
`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/`
|
||||
on macOS. The `index_path` setting and the `VAULTIK_INDEX_PATH` environment
|
||||
variable change the path.
|
||||
The index is the file `index.sqlite` in the platform data directory. By default
|
||||
that is `~/.local/share/vaultik/` on Linux and
|
||||
`~/Library/Application Support/vaultik/` on macOS. The `index_path` setting and
|
||||
the `VAULTIK_INDEX_PATH` environment variable change the path. So does
|
||||
`XDG_DATA_HOME`.
|
||||
|
||||
The file is readable only by its owner, because it lists every backed-up path.
|
||||
`vaultik` sets that mode each time it opens the index.
|
||||
@@ -74,15 +76,15 @@ One row for each backed-up path: a regular file, a symlink or a directory.
|
||||
|
||||
- `id`: a UUID. Primary key.
|
||||
- `path`: the absolute path. Unique.
|
||||
- `source_path`: the configured path that the scan found this entry under. A
|
||||
restore does not read it.
|
||||
- `source_path`: the configured path under which the scan found this entry,
|
||||
made absolute with symlinks resolved. A restore does not read it.
|
||||
- `mtime`: the modification time.
|
||||
- `mtime_nsec`: the nanoseconds within that second.
|
||||
- `size`: the size in bytes.
|
||||
- `mode`: the permission bits and the file type.
|
||||
- `uid`: the owner's user ID.
|
||||
- `gid`: the owner's group ID.
|
||||
- `link_target`: the target of a symlink. It is `NULL` for other entries.
|
||||
- `link_target`: the target of a symlink. It is empty for other entries.
|
||||
|
||||
`idx_files_path` indexes `path`.
|
||||
|
||||
@@ -161,7 +163,8 @@ A restore finds the blob for each chunk through this table.
|
||||
One row for each snapshot.
|
||||
|
||||
- `id`: the snapshot ID. Primary key.
|
||||
- `hostname`: the hostname of the machine that made the backup.
|
||||
- `hostname`: the `hostname` setting of the config. When that setting is empty,
|
||||
it is the machine's hostname.
|
||||
- `vaultik_version`: the `vaultik` version that made it.
|
||||
- `vaultik_git_revision`: the git commit of that version.
|
||||
- `started_at`: when the backup started.
|
||||
@@ -189,8 +192,8 @@ example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
|
||||
- the snapshot name
|
||||
- the start time, in UTC
|
||||
|
||||
`snapshot list` shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE` and
|
||||
`new_chunk_size` as `NEW CHUNK SIZE`.
|
||||
The `snapshot list` table shows `uncompressed_chunk_size` as
|
||||
`UNCOMPRESSED SIZE` and `new_chunk_size` as `NEW CHUNK SIZE`.
|
||||
|
||||
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
|
||||
does not show it. The next backup deletes it, and so does `prune`.
|
||||
@@ -233,10 +236,10 @@ A directory is recorded here when all of these hold:
|
||||
- The backup found an entry in it that is not excluded.
|
||||
- It is on a different filesystem from its parent directory.
|
||||
|
||||
The device number is read after the directory has been listed. Listing an
|
||||
automount point mounts its volume, so the number is that of the volume. An
|
||||
automount point whose volume cannot be mounted is not recorded, because the
|
||||
backup finds no entry in it.
|
||||
The device number is read after the directory has been listed. An automount
|
||||
point is a directory whose volume is mounted when it is first listed, so the
|
||||
number is that of the volume. An automount point whose volume cannot be mounted
|
||||
is not recorded, because the backup finds no entry in it.
|
||||
|
||||
`snapshot create --prune` reads this table. It skips the prune when an earlier
|
||||
snapshot of the same name recorded a mount point that the new backup did not
|
||||
@@ -266,8 +269,14 @@ Settings that belong to this copy of the index.
|
||||
- `value`: its value.
|
||||
|
||||
The only key is `storage_url`. It holds the destination that the index was
|
||||
first used with. A command that changes data refuses to run when the config
|
||||
names a different destination.
|
||||
first used with. These commands refuse to run when the config names a different
|
||||
destination:
|
||||
|
||||
- `snapshot create`
|
||||
- `snapshot purge`
|
||||
- `snapshot remove`
|
||||
- `prune`
|
||||
- `remote nuke`
|
||||
|
||||
### `schema_migrations`
|
||||
|
||||
@@ -282,13 +291,16 @@ The migrations that have run.
|
||||
|
||||
A backup works on the tables in this order:
|
||||
|
||||
- It saves the destination in `local_meta` when the index has none yet.
|
||||
- It deletes unfinished snapshots, and the rows that no snapshot uses.
|
||||
- It creates a row in `snapshots`, with `completed_at` set to `NULL`.
|
||||
- It deletes the `blobs` rows that were never uploaded.
|
||||
- It walks each path and compares every entry with its row in `files`.
|
||||
- It walks each path. It compares each regular file and symlink with its row in
|
||||
`files`.
|
||||
- It links each unchanged file to the snapshot in `snapshot_files`.
|
||||
- It records each mount point in `snapshot_mount_points`.
|
||||
- It reads each changed file and stores it. See below.
|
||||
- It reads each changed regular file and stores it. See below. A changed symlink
|
||||
is recorded without being read.
|
||||
- It fills `snapshot_blobs` with every blob the snapshot uses.
|
||||
- It stores the snapshot's totals in `snapshots`.
|
||||
- It uploads the snapshot's metadata.
|
||||
@@ -303,7 +315,8 @@ in any of these:
|
||||
- owner or group
|
||||
- link target
|
||||
|
||||
A file is also read again when it lists a chunk that no uploaded blob holds.
|
||||
A file is also read again when it lists a chunk that no uploaded blob holds. A
|
||||
file with no row in `files` is read too.
|
||||
|
||||
Storing a changed file writes these rows:
|
||||
|
||||
@@ -316,8 +329,10 @@ Storing a changed file writes these rows:
|
||||
- `uploaded_ts` and `compressed_size` in `blobs`
|
||||
- a row in `snapshot_blobs`
|
||||
- a row in `uploads`, unless the blob was already at the destination
|
||||
- The row in `files`, with its `file_chunks`, `chunk_files` and
|
||||
`snapshot_files` rows.
|
||||
- The row in `files`, and its rows in these tables:
|
||||
- `file_chunks`
|
||||
- `chunk_files`
|
||||
- `snapshot_files`
|
||||
|
||||
The row in `files` is written last. It is written only after every blob that
|
||||
holds the file's new chunks is stored.
|
||||
@@ -334,31 +349,38 @@ The metadata of a snapshot is made from a copy of the index:
|
||||
uploaded database therefore names the destination, and lists the migrations
|
||||
that have run.
|
||||
- `VACUUM` rewrites the copy, so that no deleted row is left in the file.
|
||||
- The copy is compressed with zstd and encrypted with age.
|
||||
- The copy is compressed with [zstd](https://github.com/facebook/zstd) and
|
||||
encrypted with [age](https://age-encryption.org/).
|
||||
- The manifest is built from the copy's `snapshot_blobs` table.
|
||||
- The copy is uploaded as `metadata/<remote-key>/db.zst.age`.
|
||||
- The manifest is uploaded as `metadata/<remote-key>/manifest.json.zst`.
|
||||
|
||||
The copy is made through SQLite, not by copying the file. The index runs in WAL
|
||||
mode, so rows that are already committed can still be in the `-wal` file beside
|
||||
it.
|
||||
The copy is made through SQLite, not by copying the file. The index runs in
|
||||
SQLite's write-ahead log (WAL) mode, so rows that are already committed can
|
||||
still be in the `-wal` file beside it.
|
||||
|
||||
### Restore
|
||||
|
||||
A restore does not use the local index.
|
||||
A restore reads no file, chunk or blob rows from the local index. Given a
|
||||
remote key, it reads `snapshots` and `local_meta` there to check the destination
|
||||
marker.
|
||||
|
||||
- It downloads and decrypts `db.zst.age` into a temporary directory.
|
||||
- It opens that file read-only. It applies no migration to it.
|
||||
- It reads the snapshot's row in `snapshots`, to check that this is the right
|
||||
database.
|
||||
- It reads the files to restore from `files`.
|
||||
- It finds each file's chunks in `file_chunks`.
|
||||
- It finds the blob and offset of each chunk in `blob_chunks`.
|
||||
- It finds the name of each blob in `blobs`.
|
||||
- With `--verify`, it reads each chunk's size from `chunks`.
|
||||
|
||||
It does not read the manifest.
|
||||
|
||||
### Prune
|
||||
|
||||
`prune` changes the local index in two ways:
|
||||
`prune` first saves the destination in `local_meta` when the index has none yet.
|
||||
It then changes the local index in two ways:
|
||||
|
||||
- It deletes each snapshot whose manifest is gone from the destination.
|
||||
- It deletes unfinished snapshots, and the rows that no snapshot uses.
|
||||
@@ -389,6 +411,7 @@ Each repository type in `internal/database` serves one or more tables:
|
||||
- `snapshot_files`
|
||||
- `snapshot_blobs`
|
||||
- `snapshot_mount_points`
|
||||
- `uploads`, where it deletes a snapshot's rows
|
||||
- `UploadRepository`: `uploads`
|
||||
- `LocalMetaRepository`: `local_meta`
|
||||
|
||||
|
||||
+48
-31
@@ -4,8 +4,8 @@ This document describes what `vaultik` stores at the destination. It covers the
|
||||
layout, what the layout protects, and the risks it accepts.
|
||||
|
||||
The destination is the place the backups are stored. It is an S3 bucket and
|
||||
prefix, a local directory or an rclone remote. The layout is the same on all
|
||||
three.
|
||||
prefix, a local directory or a remote of [rclone](https://rclone.org/). The
|
||||
layout is the same on all three.
|
||||
|
||||
## Terms
|
||||
|
||||
@@ -16,7 +16,8 @@ three.
|
||||
content.
|
||||
- **Blob**: one object under `blobs/`. It holds many chunks.
|
||||
- **Local index**: the SQLite database on the backed-up machine.
|
||||
- **Recipient**: an age public key that backups are encrypted to.
|
||||
- **Recipient**: an [age](https://age-encryption.org/) public key that backups
|
||||
are encrypted to.
|
||||
|
||||
## Layout
|
||||
|
||||
@@ -49,12 +50,15 @@ unplugged volume leaves an empty directory at its mount point. Without the
|
||||
marker, that directory would look like a destination whose snapshots are all
|
||||
gone.
|
||||
|
||||
The check applies while the local index holds snapshots. When the destination
|
||||
then answers that the marker does not exist, commands treat the destination as
|
||||
not present:
|
||||
The check applies while the local index holds snapshots and belongs to the
|
||||
configured destination. When the destination then answers that the marker does
|
||||
not exist, commands treat the destination as not present:
|
||||
|
||||
- `snapshot create` refuses to write to it.
|
||||
- `prune`, `snapshot purge` and `remote nuke` fail.
|
||||
- These commands fail:
|
||||
- `prune`
|
||||
- `snapshot purge`
|
||||
- `remote nuke`
|
||||
- `snapshot list` shows the local index alone.
|
||||
- `snapshot remove` deletes nothing at the destination.
|
||||
|
||||
@@ -73,8 +77,10 @@ A blob holds the content of many chunks, one after another.
|
||||
- **Path**: `blobs/<first 2 characters>/<next 2 characters>/<full hash>`.
|
||||
- **Example**:
|
||||
`blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`.
|
||||
- **Format**: the chunk data, compressed with zstd and then encrypted with age.
|
||||
- **Recipients**: the public keys in `age_recipients` when the blob was written.
|
||||
- **Format**: the chunk data, compressed with
|
||||
[zstd](https://github.com/facebook/zstd) and then encrypted with age.
|
||||
- **Recipients**: the public keys in the `age_recipients` setting when the blob
|
||||
was written.
|
||||
|
||||
The two directory levels keep any one directory from holding too many objects.
|
||||
|
||||
@@ -257,9 +263,9 @@ therefore does not stop overwrites.
|
||||
An object's size follows exactly from the size of its compressed contents.
|
||||
|
||||
All the new chunks in one blob share one zstd stream. The stream looks back 8
|
||||
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at
|
||||
`blob_size_limit` and at the end of each configured path. A chunk that is
|
||||
already stored is never packed again.
|
||||
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at the
|
||||
`blob_size_limit` setting and at the end of each configured path. A chunk that
|
||||
the local index records as uploaded is never packed again.
|
||||
|
||||
Data that an outsider writes compresses smaller when it matches a secret near it
|
||||
in the same blob, so the blob gets smaller. The blob's size therefore shows the
|
||||
@@ -277,14 +283,15 @@ Back up a directory that outsiders can write to as a snapshot of its own.
|
||||
|
||||
### Chunk boundaries use no secret
|
||||
|
||||
The FastCDC parameters are fixed and public. `chunk_size` sets the average chunk
|
||||
size. No chunk is larger than four times this size. Only a file's last chunk can
|
||||
be smaller than a quarter of it. With the default of 10 MB, those limits are 40
|
||||
MB and 2.5 MB. A file of 2.5 MB or less is one chunk.
|
||||
`vaultik` finds chunk boundaries with the FastCDC algorithm. Its parameters are
|
||||
fixed and public. `chunk_size` sets the average chunk size. No chunk is larger
|
||||
than four times this size. Only a file's last chunk can be smaller than a
|
||||
quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A
|
||||
file of 2.5 MB or less is one chunk. An empty file has no chunk.
|
||||
|
||||
At the default `blob_size_limit` of 10 GB, a blob holds hundreds of chunks. The
|
||||
size of the blob then does not show the size of any one chunk. A limit close to
|
||||
`chunk_size` begins to show chunk sizes.
|
||||
At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of
|
||||
chunks. The size of the blob then does not show the size of any one chunk. A
|
||||
limit close to `chunk_size` begins to show chunk sizes.
|
||||
|
||||
### Decrypted data on the local disk
|
||||
|
||||
@@ -307,14 +314,24 @@ Trust the temporary directory as much as the directory you restore into.
|
||||
|
||||
### Commands need different permissions
|
||||
|
||||
`snapshot create` needs only permission to write objects. Other commands need
|
||||
more:
|
||||
`snapshot create` without `--prune` needs only permission to write objects.
|
||||
Other commands need more:
|
||||
|
||||
- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list
|
||||
and read.
|
||||
- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete.
|
||||
- These commands list and read:
|
||||
- `snapshot list`
|
||||
- `snapshot verify`
|
||||
- `snapshot restore`
|
||||
- `remote info`
|
||||
- These commands list and delete:
|
||||
- `snapshot purge`
|
||||
- `snapshot remove`
|
||||
- `remote nuke`
|
||||
- `prune` lists, reads and deletes.
|
||||
|
||||
`snapshot verify` and `snapshot restore` list only when they are given a remote
|
||||
key. `snapshot remove --local-only` does not reach the destination.
|
||||
`snapshot remove --dry-run` deletes nothing.
|
||||
|
||||
`snapshot create --prune` runs a prune on the backed-up machine. That machine
|
||||
then holds credentials that can delete its own backups. To keep it to
|
||||
write-only credentials, run `prune` from a separate, trusted machine.
|
||||
@@ -322,8 +339,8 @@ write-only credentials, run `prune` from a separate, trusted machine.
|
||||
### New recipients cannot read old data
|
||||
|
||||
Changing `age_recipients` does not re-encrypt what is already stored. A chunk
|
||||
that is already stored is not written again, so it stays encrypted to the
|
||||
earlier recipients.
|
||||
that the local index records as uploaded is not written again, so it stays
|
||||
encrypted to the earlier recipients.
|
||||
|
||||
A new snapshot that uses such chunks cannot be restored with a new recipient's
|
||||
key alone. To make everything readable by a new key, run
|
||||
@@ -335,8 +352,8 @@ key alone. To make everything readable by a new key, run
|
||||
recipients.
|
||||
|
||||
The encrypted data is held by a third party for a long time. There is no second
|
||||
kind of recipient to fall back on if X25519 is broken, and no post-quantum
|
||||
option.
|
||||
kind of recipient to fall back on if X25519 is broken, and no kind that resists
|
||||
a quantum computer.
|
||||
|
||||
## Order of writes
|
||||
|
||||
@@ -353,12 +370,12 @@ therefore belongs to a backup that did not finish.
|
||||
|
||||
The local index marks a snapshot complete only after both metadata files are
|
||||
uploaded. A backup that stops during that upload leaves the snapshot
|
||||
unfinished. The next backup removes it from the local index and stores its data
|
||||
again.
|
||||
unfinished. The next backup removes it from the local index and backs up its
|
||||
files again.
|
||||
|
||||
One short gap remains. A backup can stop after the upload and before the
|
||||
snapshot is marked complete. The destination then holds a snapshot that can be
|
||||
restored, and the local index does not list it. `snapshot list` shows it as
|
||||
restored, and the local index marks it unfinished. `snapshot list` shows it as
|
||||
remote only.
|
||||
|
||||
An upload that is cut off leaves no partial object under the final name on S3
|
||||
|
||||
Reference in new issue
Block a user