Read all five documents against the code and fix what the sixth review found
check / check (push) Successful in 12m5s
check / check (push) Successful in 12m5s
Fixes the four findings of the sixth review: `snapshot remove` with `--local-only`, the two comma-chained permission lists, six undefined terms, and the sentence about an unfinished snapshot in the local index. Then reads every sentence of `README.md`, `ARCHITECTURE.md`, `docs/DATAMODEL.md`, `docs/REPOSTRUCTURE.md` and the comments in `config.example.yml` against the code for four faults: a command's needs or behaviour stated without regard to a flag, a false absolute, three or more things chained with commas, and a technical word not defined on first use. Each sentence changed is listed on the pull request. The help string of `snapshot remove --dry-run` now says it needs no permission to delete. Model: fable-5-1
This commit is contained in:
1 parent
a133b8838e
commit
46b2a9fc78
6 files changed
+341
-193
No files matched your search
+48
-31
@@ -4,8 +4,8 @@ This document describes what `vaultik` stores at the destination. It covers the
|
||||
layout, what the layout protects, and the risks it accepts.
|
||||
|
||||
The destination is the place the backups are stored. It is an S3 bucket and
|
||||
prefix, a local directory or an rclone remote. The layout is the same on all
|
||||
three.
|
||||
prefix, a local directory or a remote of [rclone](https://rclone.org/). The
|
||||
layout is the same on all three.
|
||||
|
||||
## Terms
|
||||
|
||||
@@ -16,7 +16,8 @@ three.
|
||||
content.
|
||||
- **Blob**: one object under `blobs/`. It holds many chunks.
|
||||
- **Local index**: the SQLite database on the backed-up machine.
|
||||
- **Recipient**: an age public key that backups are encrypted to.
|
||||
- **Recipient**: an [age](https://age-encryption.org/) public key that backups
|
||||
are encrypted to.
|
||||
|
||||
## Layout
|
||||
|
||||
@@ -49,12 +50,15 @@ unplugged volume leaves an empty directory at its mount point. Without the
|
||||
marker, that directory would look like a destination whose snapshots are all
|
||||
gone.
|
||||
|
||||
The check applies while the local index holds snapshots. When the destination
|
||||
then answers that the marker does not exist, commands treat the destination as
|
||||
not present:
|
||||
The check applies while the local index holds snapshots and belongs to the
|
||||
configured destination. When the destination then answers that the marker does
|
||||
not exist, commands treat the destination as not present:
|
||||
|
||||
- `snapshot create` refuses to write to it.
|
||||
- `prune`, `snapshot purge` and `remote nuke` fail.
|
||||
- These commands fail:
|
||||
- `prune`
|
||||
- `snapshot purge`
|
||||
- `remote nuke`
|
||||
- `snapshot list` shows the local index alone.
|
||||
- `snapshot remove` deletes nothing at the destination.
|
||||
|
||||
@@ -73,8 +77,10 @@ A blob holds the content of many chunks, one after another.
|
||||
- **Path**: `blobs/<first 2 characters>/<next 2 characters>/<full hash>`.
|
||||
- **Example**:
|
||||
`blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`.
|
||||
- **Format**: the chunk data, compressed with zstd and then encrypted with age.
|
||||
- **Recipients**: the public keys in `age_recipients` when the blob was written.
|
||||
- **Format**: the chunk data, compressed with
|
||||
[zstd](https://github.com/facebook/zstd) and then encrypted with age.
|
||||
- **Recipients**: the public keys in the `age_recipients` setting when the blob
|
||||
was written.
|
||||
|
||||
The two directory levels keep any one directory from holding too many objects.
|
||||
|
||||
@@ -257,9 +263,9 @@ therefore does not stop overwrites.
|
||||
An object's size follows exactly from the size of its compressed contents.
|
||||
|
||||
All the new chunks in one blob share one zstd stream. The stream looks back 8
|
||||
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at
|
||||
`blob_size_limit` and at the end of each configured path. A chunk that is
|
||||
already stored is never packed again.
|
||||
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at the
|
||||
`blob_size_limit` setting and at the end of each configured path. A chunk that
|
||||
the local index records as uploaded is never packed again.
|
||||
|
||||
Data that an outsider writes compresses smaller when it matches a secret near it
|
||||
in the same blob, so the blob gets smaller. The blob's size therefore shows the
|
||||
@@ -277,14 +283,15 @@ Back up a directory that outsiders can write to as a snapshot of its own.
|
||||
|
||||
### Chunk boundaries use no secret
|
||||
|
||||
The FastCDC parameters are fixed and public. `chunk_size` sets the average chunk
|
||||
size. No chunk is larger than four times this size. Only a file's last chunk can
|
||||
be smaller than a quarter of it. With the default of 10 MB, those limits are 40
|
||||
MB and 2.5 MB. A file of 2.5 MB or less is one chunk.
|
||||
`vaultik` finds chunk boundaries with the FastCDC algorithm. Its parameters are
|
||||
fixed and public. `chunk_size` sets the average chunk size. No chunk is larger
|
||||
than four times this size. Only a file's last chunk can be smaller than a
|
||||
quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A
|
||||
file of 2.5 MB or less is one chunk. An empty file has no chunk.
|
||||
|
||||
At the default `blob_size_limit` of 10 GB, a blob holds hundreds of chunks. The
|
||||
size of the blob then does not show the size of any one chunk. A limit close to
|
||||
`chunk_size` begins to show chunk sizes.
|
||||
At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of
|
||||
chunks. The size of the blob then does not show the size of any one chunk. A
|
||||
limit close to `chunk_size` begins to show chunk sizes.
|
||||
|
||||
### Decrypted data on the local disk
|
||||
|
||||
@@ -307,14 +314,24 @@ Trust the temporary directory as much as the directory you restore into.
|
||||
|
||||
### Commands need different permissions
|
||||
|
||||
`snapshot create` needs only permission to write objects. Other commands need
|
||||
more:
|
||||
`snapshot create` without `--prune` needs only permission to write objects.
|
||||
Other commands need more:
|
||||
|
||||
- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list
|
||||
and read.
|
||||
- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete.
|
||||
- These commands list and read:
|
||||
- `snapshot list`
|
||||
- `snapshot verify`
|
||||
- `snapshot restore`
|
||||
- `remote info`
|
||||
- These commands list and delete:
|
||||
- `snapshot purge`
|
||||
- `snapshot remove`
|
||||
- `remote nuke`
|
||||
- `prune` lists, reads and deletes.
|
||||
|
||||
`snapshot verify` and `snapshot restore` list only when they are given a remote
|
||||
key. `snapshot remove --local-only` does not reach the destination.
|
||||
`snapshot remove --dry-run` deletes nothing.
|
||||
|
||||
`snapshot create --prune` runs a prune on the backed-up machine. That machine
|
||||
then holds credentials that can delete its own backups. To keep it to
|
||||
write-only credentials, run `prune` from a separate, trusted machine.
|
||||
@@ -322,8 +339,8 @@ write-only credentials, run `prune` from a separate, trusted machine.
|
||||
### New recipients cannot read old data
|
||||
|
||||
Changing `age_recipients` does not re-encrypt what is already stored. A chunk
|
||||
that is already stored is not written again, so it stays encrypted to the
|
||||
earlier recipients.
|
||||
that the local index records as uploaded is not written again, so it stays
|
||||
encrypted to the earlier recipients.
|
||||
|
||||
A new snapshot that uses such chunks cannot be restored with a new recipient's
|
||||
key alone. To make everything readable by a new key, run
|
||||
@@ -335,8 +352,8 @@ key alone. To make everything readable by a new key, run
|
||||
recipients.
|
||||
|
||||
The encrypted data is held by a third party for a long time. There is no second
|
||||
kind of recipient to fall back on if X25519 is broken, and no post-quantum
|
||||
option.
|
||||
kind of recipient to fall back on if X25519 is broken, and no kind that resists
|
||||
a quantum computer.
|
||||
|
||||
## Order of writes
|
||||
|
||||
@@ -353,12 +370,12 @@ therefore belongs to a backup that did not finish.
|
||||
|
||||
The local index marks a snapshot complete only after both metadata files are
|
||||
uploaded. A backup that stops during that upload leaves the snapshot
|
||||
unfinished. The next backup removes it from the local index and stores its data
|
||||
again.
|
||||
unfinished. The next backup removes it from the local index and backs up its
|
||||
files again.
|
||||
|
||||
One short gap remains. A backup can stop after the upload and before the
|
||||
snapshot is marked complete. The destination then holds a snapshot that can be
|
||||
restored, and the local index does not list it. `snapshot list` shows it as
|
||||
restored, and the local index marks it unfinished. `snapshot list` shows it as
|
||||
remote only.
|
||||
|
||||
An upload that is cut off leaves no partial object under the final name on S3
|
||||
|
||||
Reference in new issue
Block a user