Format the rewritten documents with prettier

Re-wraps README.md, docs/ARCHITECTURE.md, docs/DATAMODEL.md,
docs/REPOSTRUCTURE.md and the roadmap in docs/TODO.md with `make fmt`,
which now runs prettier on Markdown. Only line wrapping changes; the
words are those that passed review at 4da102c.

Model: fable-5-1
This commit is contained in:
sneak authored and clawbot committed 2026-11-11 07:01:54 +00:00
1 parent 1128b85653
commit 179af61c68
5 files changed
+104 -110

No files matched your search

+19 -20
View File
@@ -199,12 +199,10 @@ vaultik version
Most commands need these:
- A config file that loads. See [Configuration](#configuration) for the
checks.
- A config file that loads. See [Configuration](#configuration) for the checks.
- Write access to the local index. The command creates the file, and its
directory, when they do not exist.
- For an `rclone://` destination, an rclone config file that defines the
remote.
- For an `rclone://` destination, an rclone config file that defines the remote.
Each command below lists what it needs beyond this.
@@ -212,8 +210,8 @@ These commands need less:
- `completion` and `version` need nothing.
- `config init` needs write access to the config directory.
- `database delete` needs a config file that loads. It does not open the
local index.
- `database delete` needs a config file that loads. It does not open the local
index.
- These need a config file that exists, even an incomplete one:
- `config edit`
- `config get`
@@ -294,8 +292,8 @@ Sets one value in the config file. The key is a dotted path, as in
- A key that does not exist yet is created. A missing parent becomes a list when
the next part of the path is a number, and a map otherwise.
- In a list, the index one past the end appends an entry.
- The command prints the key it set, and not the value. A refusal for a value
of the wrong type quotes the beginning of that value.
- The command prints the key it set, and not the value. A refusal for a value of
the wrong type quotes the beginning of that value.
- It sets the file's mode to `0600` when group or others could read it.
The command refuses a change that would stop the config from loading. An example
@@ -453,8 +451,8 @@ marker file although the local index holds snapshots.
when the destination could not be listed.
Under `--json`, every warning above goes to stderr, except the one about a
snapshot with no manifest at the destination. That snapshot has
`remote_present` set to `false`.
snapshot with no manifest at the destination. That snapshot has `remote_present`
set to `false`.
### `snapshot verify <snapshot-id>`
@@ -786,8 +784,8 @@ It also says when the binary is a development build, not a release.
The config file is YAML. `vaultik config init` writes one to start from, and
[`config.example.yml`](config.example.yml) is a longer example.
A command refuses to start when the config does not load. These commands are
the exceptions:
A command refuses to start when the config does not load. These commands are the
exceptions:
- the `config` commands
- `completion`
@@ -897,8 +895,8 @@ A command that reads the config fails when the path from `--config` or
- Only the part before the first `.` is used.
- `index_path`
- The path of the local index.
- Default: `index.sqlite` in the platform data directory. By default that
is `~/.local/share/vaultik/` on Linux and
- Default: `index.sqlite` in the platform data directory. By default that is
`~/.local/share/vaultik/` on Linux and
`~/Library/Application Support/vaultik/` on macOS.
Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB`
@@ -1042,9 +1040,9 @@ Commands need these permissions at the destination:
- `remote nuke`
- `prune` and `snapshot create --prune` list, read and delete.
An rclone remote that can move objects also needs permission to read and
delete. An upload there looks up the object it replaces and deletes it. It stops
when the look-up is refused.
An rclone remote that can move objects also needs permission to read and delete.
An upload there looks up the object it replaces and deletes it. It stops when
the look-up is refused.
`snapshot verify` and `snapshot restore` list only when you give a remote key.
`snapshot remove --local-only` does not reach the destination.
@@ -1483,8 +1481,8 @@ Each archive holds three files:
- `LICENSE`
- `README.md`
The release files are not signed, have no software bill of materials and are
not built reproducibly.
The release files are not signed, have no software bill of materials and are not
built reproducibly.
## TODO
@@ -1496,7 +1494,8 @@ Planned work is listed in [`TODO.md`](TODO.md).
## author
Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in Berlin in the summer of 2025.
Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in
Berlin in the summer of 2025.
Released as a free software gift to the world, no strings attached.
+9 -10
View File
@@ -128,8 +128,8 @@ These commands do not build the application:
- `EnsureStorageBinding` checks that the local index belongs to the configured
destination.
- `markDestination` checks the marker file and then writes it.
- `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `PruneDatabase` removes unfinished snapshots and unused records from the local
index.
- `createNamedSnapshot` runs once for each snapshot name.
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
@@ -142,8 +142,7 @@ these cases:
not find.
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
left in the index would make the backup skip files that were never fully
stored.
left in the index would make the backup skip files that were never fully stored.
### One snapshot
@@ -299,8 +298,8 @@ database.
### Checks on the downloaded database
Decryption shows that a database is readable. It does not show that the
database is honest. The restore therefore checks it:
Decryption shows that a database is readable. It does not show that the database
is honest. The restore therefore checks it:
- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
trigger, a view or a virtual table, or that lacks an expected table.
@@ -317,8 +316,8 @@ database is honest. The restore therefore checks it:
- Downloaded blobs are kept decrypted in a cache directory.
- A file is restored once every blob it needs is in the cache.
- When no file is ready, the restore downloads the blobs for the file that
needs the fewest.
- When no file is ready, the restore downloads the blobs for the file that needs
the fewest.
- `restoreSweeper` deletes a cached blob once no remaining file needs it.
A regular file is created with mode `0600` and gets its stored mode after its
@@ -351,8 +350,8 @@ the private key:
- `CleanupLocalSnapshots` drops from the local index each snapshot whose
manifest is gone from the destination.
- `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `PruneDatabase` removes unfinished snapshots and unused records from the local
index.
- `PruneBlobs` deletes unused data at the destination.
`PruneBlobs` reads every manifest at the destination. It then deletes two kinds
+15 -15
View File
@@ -76,8 +76,8 @@ One row for each backed-up path: a regular file, a symlink or a directory.
- `id`: a UUID. Primary key.
- `path`: the absolute path. Unique.
- `source_path`: the configured path under which the scan found this entry,
made absolute with symlinks resolved. A restore does not read it.
- `source_path`: the configured path under which the scan found this entry, made
absolute with symlinks resolved. A restore does not read it.
- `mtime`: the modification time.
- `mtime_nsec`: the nanoseconds within that second.
- `size`: the size in bytes.
@@ -88,9 +88,9 @@ One row for each backed-up path: a regular file, a symlink or a directory.
`idx_files_path` indexes `path`.
There is one row for each path, and it is updated when the file changes. The
row is shared by every snapshot in the local index that lists the path. An
older snapshot's own view of the file is in that snapshot's database at the
There is one row for each path, and it is updated when the file changes. The row
is shared by every snapshot in the local index that lists the path. An older
snapshot's own view of the file is in that snapshot's database at the
destination.
### `chunks`
@@ -176,8 +176,8 @@ One row for each snapshot.
- `new_chunk_size`: the total size of those chunks before compression.
- `blob_count`: the number of blobs this snapshot created.
- `total_size`: the total size of all files in the snapshot.
- `uncompressed_chunk_size`: the total size, before compression, of the
distinct chunks the snapshot's files use.
- `uncompressed_chunk_size`: the total size, before compression, of the distinct
chunks the snapshot's files use.
- `blob_size`: the total compressed size of the blobs the snapshot uses.
- `blob_uncompressed_size`: the total uncompressed size of those blobs.
- `compression_ratio`: the compression ratio of those blobs.
@@ -192,11 +192,11 @@ example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
- the snapshot name
- the start time, in UTC
The `snapshot list` table shows `uncompressed_chunk_size` as
`UNCOMPRESSED SIZE` and `new_chunk_size` as `NEW CHUNK SIZE`.
The `snapshot list` table shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE`
and `new_chunk_size` as `NEW CHUNK SIZE`.
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
does not show it. The next backup deletes it, and so does `prune`.
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list` does
not show it. The next backup deletes it, and so does `prune`.
### `snapshot_files`
@@ -268,8 +268,8 @@ Settings that belong to this copy of the index.
- `key`: the name of the setting. Primary key.
- `value`: its value.
The only key is `storage_url`. It holds the destination that the index was
first used with. These commands refuse to run when the config names a different
The only key is `storage_url`. It holds the destination that the index was first
used with. These commands refuse to run when the config names a different
destination:
- `snapshot create`
@@ -361,8 +361,8 @@ still be in the `-wal` file beside it.
### Restore
A restore reads no file, chunk or blob rows from the local index. Given a
remote key, it reads `snapshots` and `local_meta` there to check the destination
A restore reads no file, chunk or blob rows from the local index. Given a remote
key, it reads `snapshots` and `local_meta` there to check the destination
marker.
- It downloads and decrypts `db.zst.age` into a temporary directory.
+23 -25
View File
@@ -63,8 +63,8 @@ not exist, commands treat the destination as not present:
- `snapshot list` shows the local index alone.
- `snapshot remove` deletes nothing at the destination.
The check is skipped when the look-up fails for any other reason. S3 refuses
the look-up to credentials that can only write. A machine with such credentials
The check is skipped when the look-up fails for any other reason. S3 refuses the
look-up to credentials that can only write. A machine with such credentials
therefore does not notice an emptied destination. Its next backup would not
upload data that its local index says is stored. After a destination is emptied,
run `vaultik database delete` on such a machine before it backs up again.
@@ -103,8 +103,8 @@ This has two consequences:
### Uploads
A backup looks a blob up by name before it uploads it. It does not upload a
blob that is already there.
A backup looks a blob up by name before it uploads it. It does not upload a blob
that is already there.
On S3, when the destination refuses the look-up, the backup uploads the blob. An
object with that name is then replaced by one that holds the same chunk data.
@@ -290,9 +290,9 @@ than four times this size. Only a file's last chunk can be smaller than a
quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A
file of 2.5 MB or less is one chunk. An empty file has no chunk.
At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of
chunks. The size of the blob then does not show the size of any one chunk. A
limit close to `chunk_size` begins to show chunk sizes.
At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of chunks.
The size of the blob then does not show the size of any one chunk. A limit close
to `chunk_size` begins to show chunk sizes.
### Decrypted data on the local disk
@@ -329,17 +329,17 @@ permission. Other commands need more:
- `remote nuke`
- `prune` lists, reads and deletes.
An rclone remote that can move objects also needs permission to read and
delete. An upload there looks up the object it replaces and deletes it. It stops
when the look-up is refused.
An rclone remote that can move objects also needs permission to read and delete.
An upload there looks up the object it replaces and deletes it. It stops when
the look-up is refused.
`snapshot verify` and `snapshot restore` list only when they are given a remote
key. `snapshot remove --local-only` does not reach the destination.
`snapshot remove --dry-run` deletes nothing.
`snapshot create --prune` runs a prune on the backed-up machine. That machine
then holds credentials that can delete its own backups. To keep it to
write-only credentials, run `prune` from a separate, trusted machine.
then holds credentials that can delete its own backups. To keep it to write-only
credentials, run `prune` from a separate, trusted machine.
### New recipients cannot read old data
@@ -374,18 +374,17 @@ manifest is written last. A directory under `metadata/` that has no manifest
therefore belongs to a backup that did not finish.
The local index marks a snapshot complete only after both metadata files are
uploaded. A backup that stops during that upload leaves the snapshot
unfinished. The next backup removes it from the local index and backs up its
files again.
uploaded. A backup that stops during that upload leaves the snapshot unfinished.
The next backup removes it from the local index and backs up its files again.
One short gap remains. A backup can stop after the upload and before the
snapshot is marked complete. The destination then holds a snapshot that can be
restored, and the local index marks it unfinished. `snapshot list` shows it as
remote only.
An upload that is cut off leaves no partial object under the final name on S3
or on a local directory. The same holds for an rclone remote that can move
objects. An rclone remote that cannot move objects can keep a partial object.
An upload that is cut off leaves no partial object under the final name on S3 or
on a local directory. The same holds for an rclone remote that can move objects.
An rclone remote that cannot move objects can keep a partial object.
## Pruning
@@ -410,11 +409,11 @@ running. Such a backup has uploaded blobs and perhaps `db.zst.age`, but not yet
its manifest. A prune from this machine or another keeps those objects, as long
as no backup takes longer than the grace period.
The grace period does not protect one case. A backup does not upload a blob
that is already at the destination, so that blob keeps the time of its first
upload. The blob can be left over from an earlier backup that stopped before
its manifest. A prune that runs before the new manifest is written can then
delete that blob. The new snapshot cannot be restored afterwards.
The grace period does not protect one case. A backup does not upload a blob that
is already at the destination, so that blob keeps the time of its first upload.
The blob can be left over from an earlier backup that stopped before its
manifest. A prune that runs before the new manifest is written can then delete
that blob. The new snapshot cannot be restored afterwards.
To avoid this case, give each machine and each user its own destination.
@@ -436,5 +435,4 @@ It works in these steps:
- It downloads and decrypts the blobs it needs. It checks each blob against its
name.
- It writes each file from its chunks.
- It sets each file's permissions and times, and its owner when it runs as
root.
- It sets each file's permissions and times, and its owner when it runs as root.
+38 -40
View File
@@ -23,60 +23,58 @@ Work planned after 1.0, loosely ordered by priority.
## Correctness and operability
- **Outside security audit.** Before 1.0 the encryption and
blob-generation code was reviewed. Every bug the review found was
fixed. The risks it accepted are listed in
[Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside
- **Outside security audit.** Before 1.0 the encryption and blob-generation code
was reviewed. Every bug the review found was fixed. The risks it accepted are
listed in [Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside
audit has been done. age, zstd and content-defined chunking are mostly
off-the-shelf pieces. The code that joins them needs an outside read:
- key handling
- recipient parsing
- manifest trust boundary
- restore-time identity validation
- **Verify restored content end-to-end in CI.** The current integration
test does this for a small synthetic snapshot but not at scale. A
nightly job against a multi-GB representative snapshot would catch
silent regressions in the chunker, packer, or restore planner.
- **Verify restored content end-to-end in CI.** The current integration test
does this for a small synthetic snapshot but not at scale. A nightly job
against a multi-GB representative snapshot would catch silent regressions in
the chunker, packer, or restore planner.
## Performance
- **Parallel blob downloads during restore.** A restore downloads one
blob at a time now. With a fast S3 endpoint and a multi-core machine,
restore speed is bound by the fetch, decryption and decompression of
each blob. Running several of those in parallel against the disk
cache would close most of the remaining gap. This needs to interact
correctly with the locality planner and the sweeper.
- **Bandwidth limiting (`--bwlimit`).** For both upload and download.
It is useful for backing up over a shared link. It is tricky to make
work correctly with parallel downloads.
- **Restart of interrupted restore.** Today a restore can be started
again, and it then overwrites the partial output. It does not resume
from where it stopped, and it does not skip files that are already
present. A `--resume` mode that checks targets before fetching blobs
would matter for very large restores.
- **Daemon mode.** A long-running mode that watches for file changes,
so that frequent backups, such as hourly ones, skip the full scan. It
adds little for the usual backups from cron every 12 to 36 hours. See
- **Parallel blob downloads during restore.** A restore downloads one blob at a
time now. With a fast S3 endpoint and a multi-core machine, restore speed is
bound by the fetch, decryption and decompression of each blob. Running several
of those in parallel against the disk cache would close most of the remaining
gap. This needs to interact correctly with the locality planner and the
sweeper.
- **Bandwidth limiting (`--bwlimit`).** For both upload and download. It is
useful for backing up over a shared link. It is tricky to make work correctly
with parallel downloads.
- **Restart of interrupted restore.** Today a restore can be started again, and
it then overwrites the partial output. It does not resume from where it
stopped, and it does not skip files that are already present. A `--resume`
mode that checks targets before fetching blobs would matter for very large
restores.
- **Daemon mode.** A long-running mode that watches for file changes, so that
frequent backups, such as hourly ones, skip the full scan. It adds little for
the usual backups from cron every 12 to 36 hours. See
[issue #204](https://git.eeqj.de/sneak/vaultik/issues/204).
## Usability
- **Man pages and richer `--help` examples.** Cobra generates basic
help. Man pages would be a separate target.
- **`vaultik snapshot diff <a> <b>`.** Show which files changed between
two snapshots, without restoring either.
- **Status reporting hook for `--cron`.** When a backup fails silently
in cron, the user has no idea. A configurable hook on completion, for
both success and failure, would close the loop. It could be a
webhook, an email or `notify-send`.
- **Man pages and richer `--help` examples.** Cobra generates basic help. Man
pages would be a separate target.
- **`vaultik snapshot diff <a> <b>`.** Show which files changed between two
snapshots, without restoring either.
- **Status reporting hook for `--cron`.** When a backup fails silently in cron,
the user has no idea. A configurable hook on completion, for both success and
failure, would close the loop. It could be a webhook, an email or
`notify-send`.
## Infrastructure
- **Schema migrations after 1.0.** Migrations are supported, but none
are added before 1.0 because there is no installed base yet. After
1.0, each schema change is a new migration, so an existing local
index is migrated when `vaultik` is updated. See
- **Schema migrations after 1.0.** Migrations are supported, but none are added
before 1.0 because there is no installed base yet. After 1.0, each schema
change is a new migration, so an existing local index is migrated when
`vaultik` is updated. See
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
- **Storage backend coverage tests.** S3, `file://` and `rclone://` all
share the `Storer` interface, but the rclone path is the least
exercised in CI.
- **Storage backend coverage tests.** S3, `file://` and `rclone://` all share
the `Storer` interface, but the rclone path is the least exercised in CI.