From 179af61c68e9af8e7dba14d0e085c34269d8f505 Mon Sep 17 00:00:00 2001 From: sneak Date: Tue, 10 Nov 2026 22:30:44 +0000 Subject: [PATCH] Format the rewritten documents with prettier Re-wraps README.md, docs/ARCHITECTURE.md, docs/DATAMODEL.md, docs/REPOSTRUCTURE.md and the roadmap in docs/TODO.md with `make fmt`, which now runs prettier on Markdown. Only line wrapping changes; the words are those that passed review at 4da102c. Model: fable-5-1 --- README.md | 39 +++++++++++----------- docs/ARCHITECTURE.md | 19 +++++------ docs/DATAMODEL.md | 30 ++++++++--------- docs/REPOSTRUCTURE.md | 48 +++++++++++++------------- docs/TODO.md | 78 +++++++++++++++++++++---------------------- 5 files changed, 104 insertions(+), 110 deletions(-) diff --git a/README.md b/README.md index 0eee62c..2a3b26b 100644 --- a/README.md +++ b/README.md @@ -199,12 +199,10 @@ vaultik version Most commands need these: -- A config file that loads. See [Configuration](#configuration) for the - checks. +- A config file that loads. See [Configuration](#configuration) for the checks. - Write access to the local index. The command creates the file, and its directory, when they do not exist. -- For an `rclone://` destination, an rclone config file that defines the - remote. +- For an `rclone://` destination, an rclone config file that defines the remote. Each command below lists what it needs beyond this. @@ -212,8 +210,8 @@ These commands need less: - `completion` and `version` need nothing. - `config init` needs write access to the config directory. -- `database delete` needs a config file that loads. It does not open the - local index. +- `database delete` needs a config file that loads. It does not open the local + index. - These need a config file that exists, even an incomplete one: - `config edit` - `config get` @@ -294,8 +292,8 @@ Sets one value in the config file. The key is a dotted path, as in - A key that does not exist yet is created. A missing parent becomes a list when the next part of the path is a number, and a map otherwise. - In a list, the index one past the end appends an entry. -- The command prints the key it set, and not the value. A refusal for a value - of the wrong type quotes the beginning of that value. +- The command prints the key it set, and not the value. A refusal for a value of + the wrong type quotes the beginning of that value. - It sets the file's mode to `0600` when group or others could read it. The command refuses a change that would stop the config from loading. An example @@ -453,8 +451,8 @@ marker file although the local index holds snapshots. when the destination could not be listed. Under `--json`, every warning above goes to stderr, except the one about a -snapshot with no manifest at the destination. That snapshot has -`remote_present` set to `false`. +snapshot with no manifest at the destination. That snapshot has `remote_present` +set to `false`. ### `snapshot verify ` @@ -786,8 +784,8 @@ It also says when the binary is a development build, not a release. The config file is YAML. `vaultik config init` writes one to start from, and [`config.example.yml`](config.example.yml) is a longer example. -A command refuses to start when the config does not load. These commands are -the exceptions: +A command refuses to start when the config does not load. These commands are the +exceptions: - the `config` commands - `completion` @@ -897,8 +895,8 @@ A command that reads the config fails when the path from `--config` or - Only the part before the first `.` is used. - `index_path` - The path of the local index. - - Default: `index.sqlite` in the platform data directory. By default that - is `~/.local/share/vaultik/` on Linux and + - Default: `index.sqlite` in the platform data directory. By default that is + `~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/` on macOS. Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB` @@ -1042,9 +1040,9 @@ Commands need these permissions at the destination: - `remote nuke` - `prune` and `snapshot create --prune` list, read and delete. -An rclone remote that can move objects also needs permission to read and -delete. An upload there looks up the object it replaces and deletes it. It stops -when the look-up is refused. +An rclone remote that can move objects also needs permission to read and delete. +An upload there looks up the object it replaces and deletes it. It stops when +the look-up is refused. `snapshot verify` and `snapshot restore` list only when you give a remote key. `snapshot remove --local-only` does not reach the destination. @@ -1483,8 +1481,8 @@ Each archive holds three files: - `LICENSE` - `README.md` -The release files are not signed, have no software bill of materials and are -not built reproducibly. +The release files are not signed, have no software bill of materials and are not +built reproducibly. ## TODO @@ -1496,7 +1494,8 @@ Planned work is listed in [`TODO.md`](TODO.md). ## author -Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in Berlin in the summer of 2025. +Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in +Berlin in the summer of 2025. Released as a free software gift to the world, no strings attached. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 910a0d3..e5cf4e3 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -128,8 +128,8 @@ These commands do not build the application: - `EnsureStorageBinding` checks that the local index belongs to the configured destination. - `markDestination` checks the marker file and then writes it. -- `PruneDatabase` removes unfinished snapshots and unused records from the - local index. +- `PruneDatabase` removes unfinished snapshots and unused records from the local + index. - `createNamedSnapshot` runs once for each snapshot name. - With `--prune`, `runPostBackupPrune` removes old snapshots and unused data. @@ -142,8 +142,7 @@ these cases: not find. `PruneDatabase` must succeed before anything is scanned. An unfinished snapshot -left in the index would make the backup skip files that were never fully -stored. +left in the index would make the backup skip files that were never fully stored. ### One snapshot @@ -299,8 +298,8 @@ database. ### Checks on the downloaded database -Decryption shows that a database is readable. It does not show that the -database is honest. The restore therefore checks it: +Decryption shows that a database is readable. It does not show that the database +is honest. The restore therefore checks it: - `database.OpenReadOnly` opens the file read-only. It refuses a file that has a trigger, a view or a virtual table, or that lacks an expected table. @@ -317,8 +316,8 @@ database is honest. The restore therefore checks it: - Downloaded blobs are kept decrypted in a cache directory. - A file is restored once every blob it needs is in the cache. -- When no file is ready, the restore downloads the blobs for the file that - needs the fewest. +- When no file is ready, the restore downloads the blobs for the file that needs + the fewest. - `restoreSweeper` deletes a cached blob once no remaining file needs it. A regular file is created with mode `0600` and gets its stored mode after its @@ -351,8 +350,8 @@ the private key: - `CleanupLocalSnapshots` drops from the local index each snapshot whose manifest is gone from the destination. -- `PruneDatabase` removes unfinished snapshots and unused records from the - local index. +- `PruneDatabase` removes unfinished snapshots and unused records from the local + index. - `PruneBlobs` deletes unused data at the destination. `PruneBlobs` reads every manifest at the destination. It then deletes two kinds diff --git a/docs/DATAMODEL.md b/docs/DATAMODEL.md index f16784d..25e5ad6 100644 --- a/docs/DATAMODEL.md +++ b/docs/DATAMODEL.md @@ -76,8 +76,8 @@ One row for each backed-up path: a regular file, a symlink or a directory. - `id`: a UUID. Primary key. - `path`: the absolute path. Unique. -- `source_path`: the configured path under which the scan found this entry, - made absolute with symlinks resolved. A restore does not read it. +- `source_path`: the configured path under which the scan found this entry, made + absolute with symlinks resolved. A restore does not read it. - `mtime`: the modification time. - `mtime_nsec`: the nanoseconds within that second. - `size`: the size in bytes. @@ -88,9 +88,9 @@ One row for each backed-up path: a regular file, a symlink or a directory. `idx_files_path` indexes `path`. -There is one row for each path, and it is updated when the file changes. The -row is shared by every snapshot in the local index that lists the path. An -older snapshot's own view of the file is in that snapshot's database at the +There is one row for each path, and it is updated when the file changes. The row +is shared by every snapshot in the local index that lists the path. An older +snapshot's own view of the file is in that snapshot's database at the destination. ### `chunks` @@ -176,8 +176,8 @@ One row for each snapshot. - `new_chunk_size`: the total size of those chunks before compression. - `blob_count`: the number of blobs this snapshot created. - `total_size`: the total size of all files in the snapshot. -- `uncompressed_chunk_size`: the total size, before compression, of the - distinct chunks the snapshot's files use. +- `uncompressed_chunk_size`: the total size, before compression, of the distinct + chunks the snapshot's files use. - `blob_size`: the total compressed size of the blobs the snapshot uses. - `blob_uncompressed_size`: the total uncompressed size of those blobs. - `compression_ratio`: the compression ratio of those blobs. @@ -192,11 +192,11 @@ example is `server1_home_2025-06-01T12:00:00Z`. It has three parts: - the snapshot name - the start time, in UTC -The `snapshot list` table shows `uncompressed_chunk_size` as -`UNCOMPRESSED SIZE` and `new_chunk_size` as `NEW CHUNK SIZE`. +The `snapshot list` table shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE` +and `new_chunk_size` as `NEW CHUNK SIZE`. -A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list` -does not show it. The next backup deletes it, and so does `prune`. +A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list` does +not show it. The next backup deletes it, and so does `prune`. ### `snapshot_files` @@ -268,8 +268,8 @@ Settings that belong to this copy of the index. - `key`: the name of the setting. Primary key. - `value`: its value. -The only key is `storage_url`. It holds the destination that the index was -first used with. These commands refuse to run when the config names a different +The only key is `storage_url`. It holds the destination that the index was first +used with. These commands refuse to run when the config names a different destination: - `snapshot create` @@ -361,8 +361,8 @@ still be in the `-wal` file beside it. ### Restore -A restore reads no file, chunk or blob rows from the local index. Given a -remote key, it reads `snapshots` and `local_meta` there to check the destination +A restore reads no file, chunk or blob rows from the local index. Given a remote +key, it reads `snapshots` and `local_meta` there to check the destination marker. - It downloads and decrypts `db.zst.age` into a temporary directory. diff --git a/docs/REPOSTRUCTURE.md b/docs/REPOSTRUCTURE.md index 0e9f99e..de246e8 100644 --- a/docs/REPOSTRUCTURE.md +++ b/docs/REPOSTRUCTURE.md @@ -63,8 +63,8 @@ not exist, commands treat the destination as not present: - `snapshot list` shows the local index alone. - `snapshot remove` deletes nothing at the destination. -The check is skipped when the look-up fails for any other reason. S3 refuses -the look-up to credentials that can only write. A machine with such credentials +The check is skipped when the look-up fails for any other reason. S3 refuses the +look-up to credentials that can only write. A machine with such credentials therefore does not notice an emptied destination. Its next backup would not upload data that its local index says is stored. After a destination is emptied, run `vaultik database delete` on such a machine before it backs up again. @@ -103,8 +103,8 @@ This has two consequences: ### Uploads -A backup looks a blob up by name before it uploads it. It does not upload a -blob that is already there. +A backup looks a blob up by name before it uploads it. It does not upload a blob +that is already there. On S3, when the destination refuses the look-up, the backup uploads the blob. An object with that name is then replaced by one that holds the same chunk data. @@ -290,9 +290,9 @@ than four times this size. Only a file's last chunk can be smaller than a quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A file of 2.5 MB or less is one chunk. An empty file has no chunk. -At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of -chunks. The size of the blob then does not show the size of any one chunk. A -limit close to `chunk_size` begins to show chunk sizes. +At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of chunks. +The size of the blob then does not show the size of any one chunk. A limit close +to `chunk_size` begins to show chunk sizes. ### Decrypted data on the local disk @@ -329,17 +329,17 @@ permission. Other commands need more: - `remote nuke` - `prune` lists, reads and deletes. -An rclone remote that can move objects also needs permission to read and -delete. An upload there looks up the object it replaces and deletes it. It stops -when the look-up is refused. +An rclone remote that can move objects also needs permission to read and delete. +An upload there looks up the object it replaces and deletes it. It stops when +the look-up is refused. `snapshot verify` and `snapshot restore` list only when they are given a remote key. `snapshot remove --local-only` does not reach the destination. `snapshot remove --dry-run` deletes nothing. `snapshot create --prune` runs a prune on the backed-up machine. That machine -then holds credentials that can delete its own backups. To keep it to -write-only credentials, run `prune` from a separate, trusted machine. +then holds credentials that can delete its own backups. To keep it to write-only +credentials, run `prune` from a separate, trusted machine. ### New recipients cannot read old data @@ -374,18 +374,17 @@ manifest is written last. A directory under `metadata/` that has no manifest therefore belongs to a backup that did not finish. The local index marks a snapshot complete only after both metadata files are -uploaded. A backup that stops during that upload leaves the snapshot -unfinished. The next backup removes it from the local index and backs up its -files again. +uploaded. A backup that stops during that upload leaves the snapshot unfinished. +The next backup removes it from the local index and backs up its files again. One short gap remains. A backup can stop after the upload and before the snapshot is marked complete. The destination then holds a snapshot that can be restored, and the local index marks it unfinished. `snapshot list` shows it as remote only. -An upload that is cut off leaves no partial object under the final name on S3 -or on a local directory. The same holds for an rclone remote that can move -objects. An rclone remote that cannot move objects can keep a partial object. +An upload that is cut off leaves no partial object under the final name on S3 or +on a local directory. The same holds for an rclone remote that can move objects. +An rclone remote that cannot move objects can keep a partial object. ## Pruning @@ -410,11 +409,11 @@ running. Such a backup has uploaded blobs and perhaps `db.zst.age`, but not yet its manifest. A prune from this machine or another keeps those objects, as long as no backup takes longer than the grace period. -The grace period does not protect one case. A backup does not upload a blob -that is already at the destination, so that blob keeps the time of its first -upload. The blob can be left over from an earlier backup that stopped before -its manifest. A prune that runs before the new manifest is written can then -delete that blob. The new snapshot cannot be restored afterwards. +The grace period does not protect one case. A backup does not upload a blob that +is already at the destination, so that blob keeps the time of its first upload. +The blob can be left over from an earlier backup that stopped before its +manifest. A prune that runs before the new manifest is written can then delete +that blob. The new snapshot cannot be restored afterwards. To avoid this case, give each machine and each user its own destination. @@ -436,5 +435,4 @@ It works in these steps: - It downloads and decrypts the blobs it needs. It checks each blob against its name. - It writes each file from its chunks. -- It sets each file's permissions and times, and its owner when it runs as - root. +- It sets each file's permissions and times, and its owner when it runs as root. diff --git a/docs/TODO.md b/docs/TODO.md index 47f0177..e332a32 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -23,60 +23,58 @@ Work planned after 1.0, loosely ordered by priority. ## Correctness and operability -- **Outside security audit.** Before 1.0 the encryption and - blob-generation code was reviewed. Every bug the review found was - fixed. The risks it accepted are listed in - [Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside +- **Outside security audit.** Before 1.0 the encryption and blob-generation code + was reviewed. Every bug the review found was fixed. The risks it accepted are + listed in [Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside audit has been done. age, zstd and content-defined chunking are mostly off-the-shelf pieces. The code that joins them needs an outside read: - key handling - recipient parsing - manifest trust boundary - restore-time identity validation -- **Verify restored content end-to-end in CI.** The current integration - test does this for a small synthetic snapshot but not at scale. A - nightly job against a multi-GB representative snapshot would catch - silent regressions in the chunker, packer, or restore planner. +- **Verify restored content end-to-end in CI.** The current integration test + does this for a small synthetic snapshot but not at scale. A nightly job + against a multi-GB representative snapshot would catch silent regressions in + the chunker, packer, or restore planner. ## Performance -- **Parallel blob downloads during restore.** A restore downloads one - blob at a time now. With a fast S3 endpoint and a multi-core machine, - restore speed is bound by the fetch, decryption and decompression of - each blob. Running several of those in parallel against the disk - cache would close most of the remaining gap. This needs to interact - correctly with the locality planner and the sweeper. -- **Bandwidth limiting (`--bwlimit`).** For both upload and download. - It is useful for backing up over a shared link. It is tricky to make - work correctly with parallel downloads. -- **Restart of interrupted restore.** Today a restore can be started - again, and it then overwrites the partial output. It does not resume - from where it stopped, and it does not skip files that are already - present. A `--resume` mode that checks targets before fetching blobs - would matter for very large restores. -- **Daemon mode.** A long-running mode that watches for file changes, - so that frequent backups, such as hourly ones, skip the full scan. It - adds little for the usual backups from cron every 12 to 36 hours. See +- **Parallel blob downloads during restore.** A restore downloads one blob at a + time now. With a fast S3 endpoint and a multi-core machine, restore speed is + bound by the fetch, decryption and decompression of each blob. Running several + of those in parallel against the disk cache would close most of the remaining + gap. This needs to interact correctly with the locality planner and the + sweeper. +- **Bandwidth limiting (`--bwlimit`).** For both upload and download. It is + useful for backing up over a shared link. It is tricky to make work correctly + with parallel downloads. +- **Restart of interrupted restore.** Today a restore can be started again, and + it then overwrites the partial output. It does not resume from where it + stopped, and it does not skip files that are already present. A `--resume` + mode that checks targets before fetching blobs would matter for very large + restores. +- **Daemon mode.** A long-running mode that watches for file changes, so that + frequent backups, such as hourly ones, skip the full scan. It adds little for + the usual backups from cron every 12 to 36 hours. See [issue #204](https://git.eeqj.de/sneak/vaultik/issues/204). ## Usability -- **Man pages and richer `--help` examples.** Cobra generates basic - help. Man pages would be a separate target. -- **`vaultik snapshot diff `.** Show which files changed between - two snapshots, without restoring either. -- **Status reporting hook for `--cron`.** When a backup fails silently - in cron, the user has no idea. A configurable hook on completion, for - both success and failure, would close the loop. It could be a - webhook, an email or `notify-send`. +- **Man pages and richer `--help` examples.** Cobra generates basic help. Man + pages would be a separate target. +- **`vaultik snapshot diff `.** Show which files changed between two + snapshots, without restoring either. +- **Status reporting hook for `--cron`.** When a backup fails silently in cron, + the user has no idea. A configurable hook on completion, for both success and + failure, would close the loop. It could be a webhook, an email or + `notify-send`. ## Infrastructure -- **Schema migrations after 1.0.** Migrations are supported, but none - are added before 1.0 because there is no installed base yet. After - 1.0, each schema change is a new migration, so an existing local - index is migrated when `vaultik` is updated. See +- **Schema migrations after 1.0.** Migrations are supported, but none are added + before 1.0 because there is no installed base yet. After 1.0, each schema + change is a new migration, so an existing local index is migrated when + `vaultik` is updated. See [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). -- **Storage backend coverage tests.** S3, `file://` and `rclone://` all - share the `Storer` interface, but the rclone path is the least - exercised in CI. +- **Storage backend coverage tests.** S3, `file://` and `rclone://` all share + the `Storer` interface, but the rclone path is the least exercised in CI.