Format the rewritten documents with prettier
Re-wraps README.md, docs/ARCHITECTURE.md, docs/DATAMODEL.md,
docs/REPOSTRUCTURE.md and the roadmap in docs/TODO.md with `make fmt`,
which now runs prettier on Markdown. Only line wrapping changes; the
words are those that passed review at 4da102c.
Model: fable-5-1
This commit is contained in:
1 parent
1128b85653
commit
179af61c68
5 files changed
+104
-110
No files matched your search
@@ -199,12 +199,10 @@ vaultik version
|
||||
|
||||
Most commands need these:
|
||||
|
||||
- A config file that loads. See [Configuration](#configuration) for the
|
||||
checks.
|
||||
- A config file that loads. See [Configuration](#configuration) for the checks.
|
||||
- Write access to the local index. The command creates the file, and its
|
||||
directory, when they do not exist.
|
||||
- For an `rclone://` destination, an rclone config file that defines the
|
||||
remote.
|
||||
- For an `rclone://` destination, an rclone config file that defines the remote.
|
||||
|
||||
Each command below lists what it needs beyond this.
|
||||
|
||||
@@ -212,8 +210,8 @@ These commands need less:
|
||||
|
||||
- `completion` and `version` need nothing.
|
||||
- `config init` needs write access to the config directory.
|
||||
- `database delete` needs a config file that loads. It does not open the
|
||||
local index.
|
||||
- `database delete` needs a config file that loads. It does not open the local
|
||||
index.
|
||||
- These need a config file that exists, even an incomplete one:
|
||||
- `config edit`
|
||||
- `config get`
|
||||
@@ -294,8 +292,8 @@ Sets one value in the config file. The key is a dotted path, as in
|
||||
- A key that does not exist yet is created. A missing parent becomes a list when
|
||||
the next part of the path is a number, and a map otherwise.
|
||||
- In a list, the index one past the end appends an entry.
|
||||
- The command prints the key it set, and not the value. A refusal for a value
|
||||
of the wrong type quotes the beginning of that value.
|
||||
- The command prints the key it set, and not the value. A refusal for a value of
|
||||
the wrong type quotes the beginning of that value.
|
||||
- It sets the file's mode to `0600` when group or others could read it.
|
||||
|
||||
The command refuses a change that would stop the config from loading. An example
|
||||
@@ -453,8 +451,8 @@ marker file although the local index holds snapshots.
|
||||
when the destination could not be listed.
|
||||
|
||||
Under `--json`, every warning above goes to stderr, except the one about a
|
||||
snapshot with no manifest at the destination. That snapshot has
|
||||
`remote_present` set to `false`.
|
||||
snapshot with no manifest at the destination. That snapshot has `remote_present`
|
||||
set to `false`.
|
||||
|
||||
### `snapshot verify <snapshot-id>`
|
||||
|
||||
@@ -786,8 +784,8 @@ It also says when the binary is a development build, not a release.
|
||||
The config file is YAML. `vaultik config init` writes one to start from, and
|
||||
[`config.example.yml`](config.example.yml) is a longer example.
|
||||
|
||||
A command refuses to start when the config does not load. These commands are
|
||||
the exceptions:
|
||||
A command refuses to start when the config does not load. These commands are the
|
||||
exceptions:
|
||||
|
||||
- the `config` commands
|
||||
- `completion`
|
||||
@@ -897,8 +895,8 @@ A command that reads the config fails when the path from `--config` or
|
||||
- Only the part before the first `.` is used.
|
||||
- `index_path`
|
||||
- The path of the local index.
|
||||
- Default: `index.sqlite` in the platform data directory. By default that
|
||||
is `~/.local/share/vaultik/` on Linux and
|
||||
- Default: `index.sqlite` in the platform data directory. By default that is
|
||||
`~/.local/share/vaultik/` on Linux and
|
||||
`~/Library/Application Support/vaultik/` on macOS.
|
||||
|
||||
Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB`
|
||||
@@ -1042,9 +1040,9 @@ Commands need these permissions at the destination:
|
||||
- `remote nuke`
|
||||
- `prune` and `snapshot create --prune` list, read and delete.
|
||||
|
||||
An rclone remote that can move objects also needs permission to read and
|
||||
delete. An upload there looks up the object it replaces and deletes it. It stops
|
||||
when the look-up is refused.
|
||||
An rclone remote that can move objects also needs permission to read and delete.
|
||||
An upload there looks up the object it replaces and deletes it. It stops when
|
||||
the look-up is refused.
|
||||
|
||||
`snapshot verify` and `snapshot restore` list only when you give a remote key.
|
||||
`snapshot remove --local-only` does not reach the destination.
|
||||
@@ -1483,8 +1481,8 @@ Each archive holds three files:
|
||||
- `LICENSE`
|
||||
- `README.md`
|
||||
|
||||
The release files are not signed, have no software bill of materials and are
|
||||
not built reproducibly.
|
||||
The release files are not signed, have no software bill of materials and are not
|
||||
built reproducibly.
|
||||
|
||||
## TODO
|
||||
|
||||
@@ -1496,7 +1494,8 @@ Planned work is listed in [`TODO.md`](TODO.md).
|
||||
|
||||
## author
|
||||
|
||||
Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in Berlin in the summer of 2025.
|
||||
Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in
|
||||
Berlin in the summer of 2025.
|
||||
|
||||
Released as a free software gift to the world, no strings attached.
|
||||
|
||||
|
||||
+9
-10
@@ -128,8 +128,8 @@ These commands do not build the application:
|
||||
- `EnsureStorageBinding` checks that the local index belongs to the configured
|
||||
destination.
|
||||
- `markDestination` checks the marker file and then writes it.
|
||||
- `PruneDatabase` removes unfinished snapshots and unused records from the
|
||||
local index.
|
||||
- `PruneDatabase` removes unfinished snapshots and unused records from the local
|
||||
index.
|
||||
- `createNamedSnapshot` runs once for each snapshot name.
|
||||
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
|
||||
|
||||
@@ -142,8 +142,7 @@ these cases:
|
||||
not find.
|
||||
|
||||
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
|
||||
left in the index would make the backup skip files that were never fully
|
||||
stored.
|
||||
left in the index would make the backup skip files that were never fully stored.
|
||||
|
||||
### One snapshot
|
||||
|
||||
@@ -299,8 +298,8 @@ database.
|
||||
|
||||
### Checks on the downloaded database
|
||||
|
||||
Decryption shows that a database is readable. It does not show that the
|
||||
database is honest. The restore therefore checks it:
|
||||
Decryption shows that a database is readable. It does not show that the database
|
||||
is honest. The restore therefore checks it:
|
||||
|
||||
- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
|
||||
trigger, a view or a virtual table, or that lacks an expected table.
|
||||
@@ -317,8 +316,8 @@ database is honest. The restore therefore checks it:
|
||||
|
||||
- Downloaded blobs are kept decrypted in a cache directory.
|
||||
- A file is restored once every blob it needs is in the cache.
|
||||
- When no file is ready, the restore downloads the blobs for the file that
|
||||
needs the fewest.
|
||||
- When no file is ready, the restore downloads the blobs for the file that needs
|
||||
the fewest.
|
||||
- `restoreSweeper` deletes a cached blob once no remaining file needs it.
|
||||
|
||||
A regular file is created with mode `0600` and gets its stored mode after its
|
||||
@@ -351,8 +350,8 @@ the private key:
|
||||
|
||||
- `CleanupLocalSnapshots` drops from the local index each snapshot whose
|
||||
manifest is gone from the destination.
|
||||
- `PruneDatabase` removes unfinished snapshots and unused records from the
|
||||
local index.
|
||||
- `PruneDatabase` removes unfinished snapshots and unused records from the local
|
||||
index.
|
||||
- `PruneBlobs` deletes unused data at the destination.
|
||||
|
||||
`PruneBlobs` reads every manifest at the destination. It then deletes two kinds
|
||||
|
||||
+15
-15
@@ -76,8 +76,8 @@ One row for each backed-up path: a regular file, a symlink or a directory.
|
||||
|
||||
- `id`: a UUID. Primary key.
|
||||
- `path`: the absolute path. Unique.
|
||||
- `source_path`: the configured path under which the scan found this entry,
|
||||
made absolute with symlinks resolved. A restore does not read it.
|
||||
- `source_path`: the configured path under which the scan found this entry, made
|
||||
absolute with symlinks resolved. A restore does not read it.
|
||||
- `mtime`: the modification time.
|
||||
- `mtime_nsec`: the nanoseconds within that second.
|
||||
- `size`: the size in bytes.
|
||||
@@ -88,9 +88,9 @@ One row for each backed-up path: a regular file, a symlink or a directory.
|
||||
|
||||
`idx_files_path` indexes `path`.
|
||||
|
||||
There is one row for each path, and it is updated when the file changes. The
|
||||
row is shared by every snapshot in the local index that lists the path. An
|
||||
older snapshot's own view of the file is in that snapshot's database at the
|
||||
There is one row for each path, and it is updated when the file changes. The row
|
||||
is shared by every snapshot in the local index that lists the path. An older
|
||||
snapshot's own view of the file is in that snapshot's database at the
|
||||
destination.
|
||||
|
||||
### `chunks`
|
||||
@@ -176,8 +176,8 @@ One row for each snapshot.
|
||||
- `new_chunk_size`: the total size of those chunks before compression.
|
||||
- `blob_count`: the number of blobs this snapshot created.
|
||||
- `total_size`: the total size of all files in the snapshot.
|
||||
- `uncompressed_chunk_size`: the total size, before compression, of the
|
||||
distinct chunks the snapshot's files use.
|
||||
- `uncompressed_chunk_size`: the total size, before compression, of the distinct
|
||||
chunks the snapshot's files use.
|
||||
- `blob_size`: the total compressed size of the blobs the snapshot uses.
|
||||
- `blob_uncompressed_size`: the total uncompressed size of those blobs.
|
||||
- `compression_ratio`: the compression ratio of those blobs.
|
||||
@@ -192,11 +192,11 @@ example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
|
||||
- the snapshot name
|
||||
- the start time, in UTC
|
||||
|
||||
The `snapshot list` table shows `uncompressed_chunk_size` as
|
||||
`UNCOMPRESSED SIZE` and `new_chunk_size` as `NEW CHUNK SIZE`.
|
||||
The `snapshot list` table shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE`
|
||||
and `new_chunk_size` as `NEW CHUNK SIZE`.
|
||||
|
||||
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
|
||||
does not show it. The next backup deletes it, and so does `prune`.
|
||||
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list` does
|
||||
not show it. The next backup deletes it, and so does `prune`.
|
||||
|
||||
### `snapshot_files`
|
||||
|
||||
@@ -268,8 +268,8 @@ Settings that belong to this copy of the index.
|
||||
- `key`: the name of the setting. Primary key.
|
||||
- `value`: its value.
|
||||
|
||||
The only key is `storage_url`. It holds the destination that the index was
|
||||
first used with. These commands refuse to run when the config names a different
|
||||
The only key is `storage_url`. It holds the destination that the index was first
|
||||
used with. These commands refuse to run when the config names a different
|
||||
destination:
|
||||
|
||||
- `snapshot create`
|
||||
@@ -361,8 +361,8 @@ still be in the `-wal` file beside it.
|
||||
|
||||
### Restore
|
||||
|
||||
A restore reads no file, chunk or blob rows from the local index. Given a
|
||||
remote key, it reads `snapshots` and `local_meta` there to check the destination
|
||||
A restore reads no file, chunk or blob rows from the local index. Given a remote
|
||||
key, it reads `snapshots` and `local_meta` there to check the destination
|
||||
marker.
|
||||
|
||||
- It downloads and decrypts `db.zst.age` into a temporary directory.
|
||||
|
||||
+23
-25
@@ -63,8 +63,8 @@ not exist, commands treat the destination as not present:
|
||||
- `snapshot list` shows the local index alone.
|
||||
- `snapshot remove` deletes nothing at the destination.
|
||||
|
||||
The check is skipped when the look-up fails for any other reason. S3 refuses
|
||||
the look-up to credentials that can only write. A machine with such credentials
|
||||
The check is skipped when the look-up fails for any other reason. S3 refuses the
|
||||
look-up to credentials that can only write. A machine with such credentials
|
||||
therefore does not notice an emptied destination. Its next backup would not
|
||||
upload data that its local index says is stored. After a destination is emptied,
|
||||
run `vaultik database delete` on such a machine before it backs up again.
|
||||
@@ -103,8 +103,8 @@ This has two consequences:
|
||||
|
||||
### Uploads
|
||||
|
||||
A backup looks a blob up by name before it uploads it. It does not upload a
|
||||
blob that is already there.
|
||||
A backup looks a blob up by name before it uploads it. It does not upload a blob
|
||||
that is already there.
|
||||
|
||||
On S3, when the destination refuses the look-up, the backup uploads the blob. An
|
||||
object with that name is then replaced by one that holds the same chunk data.
|
||||
@@ -290,9 +290,9 @@ than four times this size. Only a file's last chunk can be smaller than a
|
||||
quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A
|
||||
file of 2.5 MB or less is one chunk. An empty file has no chunk.
|
||||
|
||||
At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of
|
||||
chunks. The size of the blob then does not show the size of any one chunk. A
|
||||
limit close to `chunk_size` begins to show chunk sizes.
|
||||
At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of chunks.
|
||||
The size of the blob then does not show the size of any one chunk. A limit close
|
||||
to `chunk_size` begins to show chunk sizes.
|
||||
|
||||
### Decrypted data on the local disk
|
||||
|
||||
@@ -329,17 +329,17 @@ permission. Other commands need more:
|
||||
- `remote nuke`
|
||||
- `prune` lists, reads and deletes.
|
||||
|
||||
An rclone remote that can move objects also needs permission to read and
|
||||
delete. An upload there looks up the object it replaces and deletes it. It stops
|
||||
when the look-up is refused.
|
||||
An rclone remote that can move objects also needs permission to read and delete.
|
||||
An upload there looks up the object it replaces and deletes it. It stops when
|
||||
the look-up is refused.
|
||||
|
||||
`snapshot verify` and `snapshot restore` list only when they are given a remote
|
||||
key. `snapshot remove --local-only` does not reach the destination.
|
||||
`snapshot remove --dry-run` deletes nothing.
|
||||
|
||||
`snapshot create --prune` runs a prune on the backed-up machine. That machine
|
||||
then holds credentials that can delete its own backups. To keep it to
|
||||
write-only credentials, run `prune` from a separate, trusted machine.
|
||||
then holds credentials that can delete its own backups. To keep it to write-only
|
||||
credentials, run `prune` from a separate, trusted machine.
|
||||
|
||||
### New recipients cannot read old data
|
||||
|
||||
@@ -374,18 +374,17 @@ manifest is written last. A directory under `metadata/` that has no manifest
|
||||
therefore belongs to a backup that did not finish.
|
||||
|
||||
The local index marks a snapshot complete only after both metadata files are
|
||||
uploaded. A backup that stops during that upload leaves the snapshot
|
||||
unfinished. The next backup removes it from the local index and backs up its
|
||||
files again.
|
||||
uploaded. A backup that stops during that upload leaves the snapshot unfinished.
|
||||
The next backup removes it from the local index and backs up its files again.
|
||||
|
||||
One short gap remains. A backup can stop after the upload and before the
|
||||
snapshot is marked complete. The destination then holds a snapshot that can be
|
||||
restored, and the local index marks it unfinished. `snapshot list` shows it as
|
||||
remote only.
|
||||
|
||||
An upload that is cut off leaves no partial object under the final name on S3
|
||||
or on a local directory. The same holds for an rclone remote that can move
|
||||
objects. An rclone remote that cannot move objects can keep a partial object.
|
||||
An upload that is cut off leaves no partial object under the final name on S3 or
|
||||
on a local directory. The same holds for an rclone remote that can move objects.
|
||||
An rclone remote that cannot move objects can keep a partial object.
|
||||
|
||||
## Pruning
|
||||
|
||||
@@ -410,11 +409,11 @@ running. Such a backup has uploaded blobs and perhaps `db.zst.age`, but not yet
|
||||
its manifest. A prune from this machine or another keeps those objects, as long
|
||||
as no backup takes longer than the grace period.
|
||||
|
||||
The grace period does not protect one case. A backup does not upload a blob
|
||||
that is already at the destination, so that blob keeps the time of its first
|
||||
upload. The blob can be left over from an earlier backup that stopped before
|
||||
its manifest. A prune that runs before the new manifest is written can then
|
||||
delete that blob. The new snapshot cannot be restored afterwards.
|
||||
The grace period does not protect one case. A backup does not upload a blob that
|
||||
is already at the destination, so that blob keeps the time of its first upload.
|
||||
The blob can be left over from an earlier backup that stopped before its
|
||||
manifest. A prune that runs before the new manifest is written can then delete
|
||||
that blob. The new snapshot cannot be restored afterwards.
|
||||
|
||||
To avoid this case, give each machine and each user its own destination.
|
||||
|
||||
@@ -436,5 +435,4 @@ It works in these steps:
|
||||
- It downloads and decrypts the blobs it needs. It checks each blob against its
|
||||
name.
|
||||
- It writes each file from its chunks.
|
||||
- It sets each file's permissions and times, and its owner when it runs as
|
||||
root.
|
||||
- It sets each file's permissions and times, and its owner when it runs as root.
|
||||
+38
-40
@@ -23,60 +23,58 @@ Work planned after 1.0, loosely ordered by priority.
|
||||
|
||||
## Correctness and operability
|
||||
|
||||
- **Outside security audit.** Before 1.0 the encryption and
|
||||
blob-generation code was reviewed. Every bug the review found was
|
||||
fixed. The risks it accepted are listed in
|
||||
[Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside
|
||||
- **Outside security audit.** Before 1.0 the encryption and blob-generation code
|
||||
was reviewed. Every bug the review found was fixed. The risks it accepted are
|
||||
listed in [Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside
|
||||
audit has been done. age, zstd and content-defined chunking are mostly
|
||||
off-the-shelf pieces. The code that joins them needs an outside read:
|
||||
- key handling
|
||||
- recipient parsing
|
||||
- manifest trust boundary
|
||||
- restore-time identity validation
|
||||
- **Verify restored content end-to-end in CI.** The current integration
|
||||
test does this for a small synthetic snapshot but not at scale. A
|
||||
nightly job against a multi-GB representative snapshot would catch
|
||||
silent regressions in the chunker, packer, or restore planner.
|
||||
- **Verify restored content end-to-end in CI.** The current integration test
|
||||
does this for a small synthetic snapshot but not at scale. A nightly job
|
||||
against a multi-GB representative snapshot would catch silent regressions in
|
||||
the chunker, packer, or restore planner.
|
||||
|
||||
## Performance
|
||||
|
||||
- **Parallel blob downloads during restore.** A restore downloads one
|
||||
blob at a time now. With a fast S3 endpoint and a multi-core machine,
|
||||
restore speed is bound by the fetch, decryption and decompression of
|
||||
each blob. Running several of those in parallel against the disk
|
||||
cache would close most of the remaining gap. This needs to interact
|
||||
correctly with the locality planner and the sweeper.
|
||||
- **Bandwidth limiting (`--bwlimit`).** For both upload and download.
|
||||
It is useful for backing up over a shared link. It is tricky to make
|
||||
work correctly with parallel downloads.
|
||||
- **Restart of interrupted restore.** Today a restore can be started
|
||||
again, and it then overwrites the partial output. It does not resume
|
||||
from where it stopped, and it does not skip files that are already
|
||||
present. A `--resume` mode that checks targets before fetching blobs
|
||||
would matter for very large restores.
|
||||
- **Daemon mode.** A long-running mode that watches for file changes,
|
||||
so that frequent backups, such as hourly ones, skip the full scan. It
|
||||
adds little for the usual backups from cron every 12 to 36 hours. See
|
||||
- **Parallel blob downloads during restore.** A restore downloads one blob at a
|
||||
time now. With a fast S3 endpoint and a multi-core machine, restore speed is
|
||||
bound by the fetch, decryption and decompression of each blob. Running several
|
||||
of those in parallel against the disk cache would close most of the remaining
|
||||
gap. This needs to interact correctly with the locality planner and the
|
||||
sweeper.
|
||||
- **Bandwidth limiting (`--bwlimit`).** For both upload and download. It is
|
||||
useful for backing up over a shared link. It is tricky to make work correctly
|
||||
with parallel downloads.
|
||||
- **Restart of interrupted restore.** Today a restore can be started again, and
|
||||
it then overwrites the partial output. It does not resume from where it
|
||||
stopped, and it does not skip files that are already present. A `--resume`
|
||||
mode that checks targets before fetching blobs would matter for very large
|
||||
restores.
|
||||
- **Daemon mode.** A long-running mode that watches for file changes, so that
|
||||
frequent backups, such as hourly ones, skip the full scan. It adds little for
|
||||
the usual backups from cron every 12 to 36 hours. See
|
||||
[issue #204](https://git.eeqj.de/sneak/vaultik/issues/204).
|
||||
|
||||
## Usability
|
||||
|
||||
- **Man pages and richer `--help` examples.** Cobra generates basic
|
||||
help. Man pages would be a separate target.
|
||||
- **`vaultik snapshot diff <a> <b>`.** Show which files changed between
|
||||
two snapshots, without restoring either.
|
||||
- **Status reporting hook for `--cron`.** When a backup fails silently
|
||||
in cron, the user has no idea. A configurable hook on completion, for
|
||||
both success and failure, would close the loop. It could be a
|
||||
webhook, an email or `notify-send`.
|
||||
- **Man pages and richer `--help` examples.** Cobra generates basic help. Man
|
||||
pages would be a separate target.
|
||||
- **`vaultik snapshot diff <a> <b>`.** Show which files changed between two
|
||||
snapshots, without restoring either.
|
||||
- **Status reporting hook for `--cron`.** When a backup fails silently in cron,
|
||||
the user has no idea. A configurable hook on completion, for both success and
|
||||
failure, would close the loop. It could be a webhook, an email or
|
||||
`notify-send`.
|
||||
|
||||
## Infrastructure
|
||||
|
||||
- **Schema migrations after 1.0.** Migrations are supported, but none
|
||||
are added before 1.0 because there is no installed base yet. After
|
||||
1.0, each schema change is a new migration, so an existing local
|
||||
index is migrated when `vaultik` is updated. See
|
||||
- **Schema migrations after 1.0.** Migrations are supported, but none are added
|
||||
before 1.0 because there is no installed base yet. After 1.0, each schema
|
||||
change is a new migration, so an existing local index is migrated when
|
||||
`vaultik` is updated. See
|
||||
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
|
||||
- **Storage backend coverage tests.** S3, `file://` and `rclone://` all
|
||||
share the `Storer` interface, but the rclone path is the least
|
||||
exercised in CI.
|
||||
- **Storage backend coverage tests.** S3, `file://` and `rclone://` all share
|
||||
the `Storer` interface, but the rclone path is the least exercised in CI.
|
||||
Reference in new issue
Block a user