Format the rewritten documents with prettier

Re-wraps README.md, docs/ARCHITECTURE.md, docs/DATAMODEL.md,
docs/REPOSTRUCTURE.md and the roadmap in docs/TODO.md with `make fmt`,
which now runs prettier on Markdown. Only line wrapping changes; the
words are those that passed review at 4da102c.

Model: fable-5-1
This commit is contained in:
sneak authored and clawbot committed 2026-11-11 07:01:54 +00:00
1 parent 1128b85653
commit 179af61c68
5 files changed
+104 -110

No files matched your search

+19 -20
View File
@@ -199,12 +199,10 @@ vaultik version
Most commands need these: Most commands need these:
- A config file that loads. See [Configuration](#configuration) for the - A config file that loads. See [Configuration](#configuration) for the checks.
checks.
- Write access to the local index. The command creates the file, and its - Write access to the local index. The command creates the file, and its
directory, when they do not exist. directory, when they do not exist.
- For an `rclone://` destination, an rclone config file that defines the - For an `rclone://` destination, an rclone config file that defines the remote.
remote.
Each command below lists what it needs beyond this. Each command below lists what it needs beyond this.
@@ -212,8 +210,8 @@ These commands need less:
- `completion` and `version` need nothing. - `completion` and `version` need nothing.
- `config init` needs write access to the config directory. - `config init` needs write access to the config directory.
- `database delete` needs a config file that loads. It does not open the - `database delete` needs a config file that loads. It does not open the local
local index. index.
- These need a config file that exists, even an incomplete one: - These need a config file that exists, even an incomplete one:
- `config edit` - `config edit`
- `config get` - `config get`
@@ -294,8 +292,8 @@ Sets one value in the config file. The key is a dotted path, as in
- A key that does not exist yet is created. A missing parent becomes a list when - A key that does not exist yet is created. A missing parent becomes a list when
the next part of the path is a number, and a map otherwise. the next part of the path is a number, and a map otherwise.
- In a list, the index one past the end appends an entry. - In a list, the index one past the end appends an entry.
- The command prints the key it set, and not the value. A refusal for a value - The command prints the key it set, and not the value. A refusal for a value of
of the wrong type quotes the beginning of that value. the wrong type quotes the beginning of that value.
- It sets the file's mode to `0600` when group or others could read it. - It sets the file's mode to `0600` when group or others could read it.
The command refuses a change that would stop the config from loading. An example The command refuses a change that would stop the config from loading. An example
@@ -453,8 +451,8 @@ marker file although the local index holds snapshots.
when the destination could not be listed. when the destination could not be listed.
Under `--json`, every warning above goes to stderr, except the one about a Under `--json`, every warning above goes to stderr, except the one about a
snapshot with no manifest at the destination. That snapshot has snapshot with no manifest at the destination. That snapshot has `remote_present`
`remote_present` set to `false`. set to `false`.
### `snapshot verify <snapshot-id>` ### `snapshot verify <snapshot-id>`
@@ -786,8 +784,8 @@ It also says when the binary is a development build, not a release.
The config file is YAML. `vaultik config init` writes one to start from, and The config file is YAML. `vaultik config init` writes one to start from, and
[`config.example.yml`](config.example.yml) is a longer example. [`config.example.yml`](config.example.yml) is a longer example.
A command refuses to start when the config does not load. These commands are A command refuses to start when the config does not load. These commands are the
the exceptions: exceptions:
- the `config` commands - the `config` commands
- `completion` - `completion`
@@ -897,8 +895,8 @@ A command that reads the config fails when the path from `--config` or
- Only the part before the first `.` is used. - Only the part before the first `.` is used.
- `index_path` - `index_path`
- The path of the local index. - The path of the local index.
- Default: `index.sqlite` in the platform data directory. By default that - Default: `index.sqlite` in the platform data directory. By default that is
is `~/.local/share/vaultik/` on Linux and `~/.local/share/vaultik/` on Linux and
`~/Library/Application Support/vaultik/` on macOS. `~/Library/Application Support/vaultik/` on macOS.
Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB` Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB`
@@ -1042,9 +1040,9 @@ Commands need these permissions at the destination:
- `remote nuke` - `remote nuke`
- `prune` and `snapshot create --prune` list, read and delete. - `prune` and `snapshot create --prune` list, read and delete.
An rclone remote that can move objects also needs permission to read and An rclone remote that can move objects also needs permission to read and delete.
delete. An upload there looks up the object it replaces and deletes it. It stops An upload there looks up the object it replaces and deletes it. It stops when
when the look-up is refused. the look-up is refused.
`snapshot verify` and `snapshot restore` list only when you give a remote key. `snapshot verify` and `snapshot restore` list only when you give a remote key.
`snapshot remove --local-only` does not reach the destination. `snapshot remove --local-only` does not reach the destination.
@@ -1483,8 +1481,8 @@ Each archive holds three files:
- `LICENSE` - `LICENSE`
- `README.md` - `README.md`
The release files are not signed, have no software bill of materials and are The release files are not signed, have no software bill of materials and are not
not built reproducibly. built reproducibly.
## TODO ## TODO
@@ -1496,7 +1494,8 @@ Planned work is listed in [`TODO.md`](TODO.md).
## author ## author
Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in Berlin in the summer of 2025. Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in
Berlin in the summer of 2025.
Released as a free software gift to the world, no strings attached. Released as a free software gift to the world, no strings attached.
+9 -10
View File
@@ -128,8 +128,8 @@ These commands do not build the application:
- `EnsureStorageBinding` checks that the local index belongs to the configured - `EnsureStorageBinding` checks that the local index belongs to the configured
destination. destination.
- `markDestination` checks the marker file and then writes it. - `markDestination` checks the marker file and then writes it.
- `PruneDatabase` removes unfinished snapshots and unused records from the - `PruneDatabase` removes unfinished snapshots and unused records from the local
local index. index.
- `createNamedSnapshot` runs once for each snapshot name. - `createNamedSnapshot` runs once for each snapshot name.
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data. - With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
@@ -142,8 +142,7 @@ these cases:
not find. not find.
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot `PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
left in the index would make the backup skip files that were never fully left in the index would make the backup skip files that were never fully stored.
stored.
### One snapshot ### One snapshot
@@ -299,8 +298,8 @@ database.
### Checks on the downloaded database ### Checks on the downloaded database
Decryption shows that a database is readable. It does not show that the Decryption shows that a database is readable. It does not show that the database
database is honest. The restore therefore checks it: is honest. The restore therefore checks it:
- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a - `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
trigger, a view or a virtual table, or that lacks an expected table. trigger, a view or a virtual table, or that lacks an expected table.
@@ -317,8 +316,8 @@ database is honest. The restore therefore checks it:
- Downloaded blobs are kept decrypted in a cache directory. - Downloaded blobs are kept decrypted in a cache directory.
- A file is restored once every blob it needs is in the cache. - A file is restored once every blob it needs is in the cache.
- When no file is ready, the restore downloads the blobs for the file that - When no file is ready, the restore downloads the blobs for the file that needs
needs the fewest. the fewest.
- `restoreSweeper` deletes a cached blob once no remaining file needs it. - `restoreSweeper` deletes a cached blob once no remaining file needs it.
A regular file is created with mode `0600` and gets its stored mode after its A regular file is created with mode `0600` and gets its stored mode after its
@@ -351,8 +350,8 @@ the private key:
- `CleanupLocalSnapshots` drops from the local index each snapshot whose - `CleanupLocalSnapshots` drops from the local index each snapshot whose
manifest is gone from the destination. manifest is gone from the destination.
- `PruneDatabase` removes unfinished snapshots and unused records from the - `PruneDatabase` removes unfinished snapshots and unused records from the local
local index. index.
- `PruneBlobs` deletes unused data at the destination. - `PruneBlobs` deletes unused data at the destination.
`PruneBlobs` reads every manifest at the destination. It then deletes two kinds `PruneBlobs` reads every manifest at the destination. It then deletes two kinds
+15 -15
View File
@@ -76,8 +76,8 @@ One row for each backed-up path: a regular file, a symlink or a directory.
- `id`: a UUID. Primary key. - `id`: a UUID. Primary key.
- `path`: the absolute path. Unique. - `path`: the absolute path. Unique.
- `source_path`: the configured path under which the scan found this entry, - `source_path`: the configured path under which the scan found this entry, made
made absolute with symlinks resolved. A restore does not read it. absolute with symlinks resolved. A restore does not read it.
- `mtime`: the modification time. - `mtime`: the modification time.
- `mtime_nsec`: the nanoseconds within that second. - `mtime_nsec`: the nanoseconds within that second.
- `size`: the size in bytes. - `size`: the size in bytes.
@@ -88,9 +88,9 @@ One row for each backed-up path: a regular file, a symlink or a directory.
`idx_files_path` indexes `path`. `idx_files_path` indexes `path`.
There is one row for each path, and it is updated when the file changes. The There is one row for each path, and it is updated when the file changes. The row
row is shared by every snapshot in the local index that lists the path. An is shared by every snapshot in the local index that lists the path. An older
older snapshot's own view of the file is in that snapshot's database at the snapshot's own view of the file is in that snapshot's database at the
destination. destination.
### `chunks` ### `chunks`
@@ -176,8 +176,8 @@ One row for each snapshot.
- `new_chunk_size`: the total size of those chunks before compression. - `new_chunk_size`: the total size of those chunks before compression.
- `blob_count`: the number of blobs this snapshot created. - `blob_count`: the number of blobs this snapshot created.
- `total_size`: the total size of all files in the snapshot. - `total_size`: the total size of all files in the snapshot.
- `uncompressed_chunk_size`: the total size, before compression, of the - `uncompressed_chunk_size`: the total size, before compression, of the distinct
distinct chunks the snapshot's files use. chunks the snapshot's files use.
- `blob_size`: the total compressed size of the blobs the snapshot uses. - `blob_size`: the total compressed size of the blobs the snapshot uses.
- `blob_uncompressed_size`: the total uncompressed size of those blobs. - `blob_uncompressed_size`: the total uncompressed size of those blobs.
- `compression_ratio`: the compression ratio of those blobs. - `compression_ratio`: the compression ratio of those blobs.
@@ -192,11 +192,11 @@ example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
- the snapshot name - the snapshot name
- the start time, in UTC - the start time, in UTC
The `snapshot list` table shows `uncompressed_chunk_size` as The `snapshot list` table shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE`
`UNCOMPRESSED SIZE` and `new_chunk_size` as `NEW CHUNK SIZE`. and `new_chunk_size` as `NEW CHUNK SIZE`.
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list` A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list` does
does not show it. The next backup deletes it, and so does `prune`. not show it. The next backup deletes it, and so does `prune`.
### `snapshot_files` ### `snapshot_files`
@@ -268,8 +268,8 @@ Settings that belong to this copy of the index.
- `key`: the name of the setting. Primary key. - `key`: the name of the setting. Primary key.
- `value`: its value. - `value`: its value.
The only key is `storage_url`. It holds the destination that the index was The only key is `storage_url`. It holds the destination that the index was first
first used with. These commands refuse to run when the config names a different used with. These commands refuse to run when the config names a different
destination: destination:
- `snapshot create` - `snapshot create`
@@ -361,8 +361,8 @@ still be in the `-wal` file beside it.
### Restore ### Restore
A restore reads no file, chunk or blob rows from the local index. Given a A restore reads no file, chunk or blob rows from the local index. Given a remote
remote key, it reads `snapshots` and `local_meta` there to check the destination key, it reads `snapshots` and `local_meta` there to check the destination
marker. marker.
- It downloads and decrypts `db.zst.age` into a temporary directory. - It downloads and decrypts `db.zst.age` into a temporary directory.
+23 -25
View File
@@ -63,8 +63,8 @@ not exist, commands treat the destination as not present:
- `snapshot list` shows the local index alone. - `snapshot list` shows the local index alone.
- `snapshot remove` deletes nothing at the destination. - `snapshot remove` deletes nothing at the destination.
The check is skipped when the look-up fails for any other reason. S3 refuses The check is skipped when the look-up fails for any other reason. S3 refuses the
the look-up to credentials that can only write. A machine with such credentials look-up to credentials that can only write. A machine with such credentials
therefore does not notice an emptied destination. Its next backup would not therefore does not notice an emptied destination. Its next backup would not
upload data that its local index says is stored. After a destination is emptied, upload data that its local index says is stored. After a destination is emptied,
run `vaultik database delete` on such a machine before it backs up again. run `vaultik database delete` on such a machine before it backs up again.
@@ -103,8 +103,8 @@ This has two consequences:
### Uploads ### Uploads
A backup looks a blob up by name before it uploads it. It does not upload a A backup looks a blob up by name before it uploads it. It does not upload a blob
blob that is already there. that is already there.
On S3, when the destination refuses the look-up, the backup uploads the blob. An On S3, when the destination refuses the look-up, the backup uploads the blob. An
object with that name is then replaced by one that holds the same chunk data. object with that name is then replaced by one that holds the same chunk data.
@@ -290,9 +290,9 @@ than four times this size. Only a file's last chunk can be smaller than a
quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A
file of 2.5 MB or less is one chunk. An empty file has no chunk. file of 2.5 MB or less is one chunk. An empty file has no chunk.
At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of chunks.
chunks. The size of the blob then does not show the size of any one chunk. A The size of the blob then does not show the size of any one chunk. A limit close
limit close to `chunk_size` begins to show chunk sizes. to `chunk_size` begins to show chunk sizes.
### Decrypted data on the local disk ### Decrypted data on the local disk
@@ -329,17 +329,17 @@ permission. Other commands need more:
- `remote nuke` - `remote nuke`
- `prune` lists, reads and deletes. - `prune` lists, reads and deletes.
An rclone remote that can move objects also needs permission to read and An rclone remote that can move objects also needs permission to read and delete.
delete. An upload there looks up the object it replaces and deletes it. It stops An upload there looks up the object it replaces and deletes it. It stops when
when the look-up is refused. the look-up is refused.
`snapshot verify` and `snapshot restore` list only when they are given a remote `snapshot verify` and `snapshot restore` list only when they are given a remote
key. `snapshot remove --local-only` does not reach the destination. key. `snapshot remove --local-only` does not reach the destination.
`snapshot remove --dry-run` deletes nothing. `snapshot remove --dry-run` deletes nothing.
`snapshot create --prune` runs a prune on the backed-up machine. That machine `snapshot create --prune` runs a prune on the backed-up machine. That machine
then holds credentials that can delete its own backups. To keep it to then holds credentials that can delete its own backups. To keep it to write-only
write-only credentials, run `prune` from a separate, trusted machine. credentials, run `prune` from a separate, trusted machine.
### New recipients cannot read old data ### New recipients cannot read old data
@@ -374,18 +374,17 @@ manifest is written last. A directory under `metadata/` that has no manifest
therefore belongs to a backup that did not finish. therefore belongs to a backup that did not finish.
The local index marks a snapshot complete only after both metadata files are The local index marks a snapshot complete only after both metadata files are
uploaded. A backup that stops during that upload leaves the snapshot uploaded. A backup that stops during that upload leaves the snapshot unfinished.
unfinished. The next backup removes it from the local index and backs up its The next backup removes it from the local index and backs up its files again.
files again.
One short gap remains. A backup can stop after the upload and before the One short gap remains. A backup can stop after the upload and before the
snapshot is marked complete. The destination then holds a snapshot that can be snapshot is marked complete. The destination then holds a snapshot that can be
restored, and the local index marks it unfinished. `snapshot list` shows it as restored, and the local index marks it unfinished. `snapshot list` shows it as
remote only. remote only.
An upload that is cut off leaves no partial object under the final name on S3 An upload that is cut off leaves no partial object under the final name on S3 or
or on a local directory. The same holds for an rclone remote that can move on a local directory. The same holds for an rclone remote that can move objects.
objects. An rclone remote that cannot move objects can keep a partial object. An rclone remote that cannot move objects can keep a partial object.
## Pruning ## Pruning
@@ -410,11 +409,11 @@ running. Such a backup has uploaded blobs and perhaps `db.zst.age`, but not yet
its manifest. A prune from this machine or another keeps those objects, as long its manifest. A prune from this machine or another keeps those objects, as long
as no backup takes longer than the grace period. as no backup takes longer than the grace period.
The grace period does not protect one case. A backup does not upload a blob The grace period does not protect one case. A backup does not upload a blob that
that is already at the destination, so that blob keeps the time of its first is already at the destination, so that blob keeps the time of its first upload.
upload. The blob can be left over from an earlier backup that stopped before The blob can be left over from an earlier backup that stopped before its
its manifest. A prune that runs before the new manifest is written can then manifest. A prune that runs before the new manifest is written can then delete
delete that blob. The new snapshot cannot be restored afterwards. that blob. The new snapshot cannot be restored afterwards.
To avoid this case, give each machine and each user its own destination. To avoid this case, give each machine and each user its own destination.
@@ -436,5 +435,4 @@ It works in these steps:
- It downloads and decrypts the blobs it needs. It checks each blob against its - It downloads and decrypts the blobs it needs. It checks each blob against its
name. name.
- It writes each file from its chunks. - It writes each file from its chunks.
- It sets each file's permissions and times, and its owner when it runs as - It sets each file's permissions and times, and its owner when it runs as root.
root.
+38 -40
View File
@@ -23,60 +23,58 @@ Work planned after 1.0, loosely ordered by priority.
## Correctness and operability ## Correctness and operability
- **Outside security audit.** Before 1.0 the encryption and - **Outside security audit.** Before 1.0 the encryption and blob-generation code
blob-generation code was reviewed. Every bug the review found was was reviewed. Every bug the review found was fixed. The risks it accepted are
fixed. The risks it accepted are listed in listed in [Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside
[Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside
audit has been done. age, zstd and content-defined chunking are mostly audit has been done. age, zstd and content-defined chunking are mostly
off-the-shelf pieces. The code that joins them needs an outside read: off-the-shelf pieces. The code that joins them needs an outside read:
- key handling - key handling
- recipient parsing - recipient parsing
- manifest trust boundary - manifest trust boundary
- restore-time identity validation - restore-time identity validation
- **Verify restored content end-to-end in CI.** The current integration - **Verify restored content end-to-end in CI.** The current integration test
test does this for a small synthetic snapshot but not at scale. A does this for a small synthetic snapshot but not at scale. A nightly job
nightly job against a multi-GB representative snapshot would catch against a multi-GB representative snapshot would catch silent regressions in
silent regressions in the chunker, packer, or restore planner. the chunker, packer, or restore planner.
## Performance ## Performance
- **Parallel blob downloads during restore.** A restore downloads one - **Parallel blob downloads during restore.** A restore downloads one blob at a
blob at a time now. With a fast S3 endpoint and a multi-core machine, time now. With a fast S3 endpoint and a multi-core machine, restore speed is
restore speed is bound by the fetch, decryption and decompression of bound by the fetch, decryption and decompression of each blob. Running several
each blob. Running several of those in parallel against the disk of those in parallel against the disk cache would close most of the remaining
cache would close most of the remaining gap. This needs to interact gap. This needs to interact correctly with the locality planner and the
correctly with the locality planner and the sweeper. sweeper.
- **Bandwidth limiting (`--bwlimit`).** For both upload and download. - **Bandwidth limiting (`--bwlimit`).** For both upload and download. It is
It is useful for backing up over a shared link. It is tricky to make useful for backing up over a shared link. It is tricky to make work correctly
work correctly with parallel downloads. with parallel downloads.
- **Restart of interrupted restore.** Today a restore can be started - **Restart of interrupted restore.** Today a restore can be started again, and
again, and it then overwrites the partial output. It does not resume it then overwrites the partial output. It does not resume from where it
from where it stopped, and it does not skip files that are already stopped, and it does not skip files that are already present. A `--resume`
present. A `--resume` mode that checks targets before fetching blobs mode that checks targets before fetching blobs would matter for very large
would matter for very large restores. restores.
- **Daemon mode.** A long-running mode that watches for file changes, - **Daemon mode.** A long-running mode that watches for file changes, so that
so that frequent backups, such as hourly ones, skip the full scan. It frequent backups, such as hourly ones, skip the full scan. It adds little for
adds little for the usual backups from cron every 12 to 36 hours. See the usual backups from cron every 12 to 36 hours. See
[issue #204](https://git.eeqj.de/sneak/vaultik/issues/204). [issue #204](https://git.eeqj.de/sneak/vaultik/issues/204).
## Usability ## Usability
- **Man pages and richer `--help` examples.** Cobra generates basic - **Man pages and richer `--help` examples.** Cobra generates basic help. Man
help. Man pages would be a separate target. pages would be a separate target.
- **`vaultik snapshot diff <a> <b>`.** Show which files changed between - **`vaultik snapshot diff <a> <b>`.** Show which files changed between two
two snapshots, without restoring either. snapshots, without restoring either.
- **Status reporting hook for `--cron`.** When a backup fails silently - **Status reporting hook for `--cron`.** When a backup fails silently in cron,
in cron, the user has no idea. A configurable hook on completion, for the user has no idea. A configurable hook on completion, for both success and
both success and failure, would close the loop. It could be a failure, would close the loop. It could be a webhook, an email or
webhook, an email or `notify-send`. `notify-send`.
## Infrastructure ## Infrastructure
- **Schema migrations after 1.0.** Migrations are supported, but none - **Schema migrations after 1.0.** Migrations are supported, but none are added
are added before 1.0 because there is no installed base yet. After before 1.0 because there is no installed base yet. After 1.0, each schema
1.0, each schema change is a new migration, so an existing local change is a new migration, so an existing local index is migrated when
index is migrated when `vaultik` is updated. See `vaultik` is updated. See
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
- **Storage backend coverage tests.** S3, `file://` and `rclone://` all - **Storage backend coverage tests.** S3, `file://` and `rclone://` all share
share the `Storer` interface, but the rclone path is the least the `Storer` interface, but the rclone path is the least exercised in CI.
exercised in CI.