Format the rewritten documents with prettier
Re-wraps README.md, docs/ARCHITECTURE.md, docs/DATAMODEL.md,
docs/REPOSTRUCTURE.md and the roadmap in docs/TODO.md with `make fmt`,
which now runs prettier on Markdown. Only line wrapping changes; the
words are those that passed review at 4da102c.
Model: fable-5-1
This commit is contained in:
1 parent
1128b85653
commit
179af61c68
5 files changed
+104
-110
No files matched your search
@@ -199,12 +199,10 @@ vaultik version
|
|||||||
|
|
||||||
Most commands need these:
|
Most commands need these:
|
||||||
|
|
||||||
- A config file that loads. See [Configuration](#configuration) for the
|
- A config file that loads. See [Configuration](#configuration) for the checks.
|
||||||
checks.
|
|
||||||
- Write access to the local index. The command creates the file, and its
|
- Write access to the local index. The command creates the file, and its
|
||||||
directory, when they do not exist.
|
directory, when they do not exist.
|
||||||
- For an `rclone://` destination, an rclone config file that defines the
|
- For an `rclone://` destination, an rclone config file that defines the remote.
|
||||||
remote.
|
|
||||||
|
|
||||||
Each command below lists what it needs beyond this.
|
Each command below lists what it needs beyond this.
|
||||||
|
|
||||||
@@ -212,8 +210,8 @@ These commands need less:
|
|||||||
|
|
||||||
- `completion` and `version` need nothing.
|
- `completion` and `version` need nothing.
|
||||||
- `config init` needs write access to the config directory.
|
- `config init` needs write access to the config directory.
|
||||||
- `database delete` needs a config file that loads. It does not open the
|
- `database delete` needs a config file that loads. It does not open the local
|
||||||
local index.
|
index.
|
||||||
- These need a config file that exists, even an incomplete one:
|
- These need a config file that exists, even an incomplete one:
|
||||||
- `config edit`
|
- `config edit`
|
||||||
- `config get`
|
- `config get`
|
||||||
@@ -294,8 +292,8 @@ Sets one value in the config file. The key is a dotted path, as in
|
|||||||
- A key that does not exist yet is created. A missing parent becomes a list when
|
- A key that does not exist yet is created. A missing parent becomes a list when
|
||||||
the next part of the path is a number, and a map otherwise.
|
the next part of the path is a number, and a map otherwise.
|
||||||
- In a list, the index one past the end appends an entry.
|
- In a list, the index one past the end appends an entry.
|
||||||
- The command prints the key it set, and not the value. A refusal for a value
|
- The command prints the key it set, and not the value. A refusal for a value of
|
||||||
of the wrong type quotes the beginning of that value.
|
the wrong type quotes the beginning of that value.
|
||||||
- It sets the file's mode to `0600` when group or others could read it.
|
- It sets the file's mode to `0600` when group or others could read it.
|
||||||
|
|
||||||
The command refuses a change that would stop the config from loading. An example
|
The command refuses a change that would stop the config from loading. An example
|
||||||
@@ -453,8 +451,8 @@ marker file although the local index holds snapshots.
|
|||||||
when the destination could not be listed.
|
when the destination could not be listed.
|
||||||
|
|
||||||
Under `--json`, every warning above goes to stderr, except the one about a
|
Under `--json`, every warning above goes to stderr, except the one about a
|
||||||
snapshot with no manifest at the destination. That snapshot has
|
snapshot with no manifest at the destination. That snapshot has `remote_present`
|
||||||
`remote_present` set to `false`.
|
set to `false`.
|
||||||
|
|
||||||
### `snapshot verify <snapshot-id>`
|
### `snapshot verify <snapshot-id>`
|
||||||
|
|
||||||
@@ -786,8 +784,8 @@ It also says when the binary is a development build, not a release.
|
|||||||
The config file is YAML. `vaultik config init` writes one to start from, and
|
The config file is YAML. `vaultik config init` writes one to start from, and
|
||||||
[`config.example.yml`](config.example.yml) is a longer example.
|
[`config.example.yml`](config.example.yml) is a longer example.
|
||||||
|
|
||||||
A command refuses to start when the config does not load. These commands are
|
A command refuses to start when the config does not load. These commands are the
|
||||||
the exceptions:
|
exceptions:
|
||||||
|
|
||||||
- the `config` commands
|
- the `config` commands
|
||||||
- `completion`
|
- `completion`
|
||||||
@@ -897,8 +895,8 @@ A command that reads the config fails when the path from `--config` or
|
|||||||
- Only the part before the first `.` is used.
|
- Only the part before the first `.` is used.
|
||||||
- `index_path`
|
- `index_path`
|
||||||
- The path of the local index.
|
- The path of the local index.
|
||||||
- Default: `index.sqlite` in the platform data directory. By default that
|
- Default: `index.sqlite` in the platform data directory. By default that is
|
||||||
is `~/.local/share/vaultik/` on Linux and
|
`~/.local/share/vaultik/` on Linux and
|
||||||
`~/Library/Application Support/vaultik/` on macOS.
|
`~/Library/Application Support/vaultik/` on macOS.
|
||||||
|
|
||||||
Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB`
|
Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB`
|
||||||
@@ -1042,9 +1040,9 @@ Commands need these permissions at the destination:
|
|||||||
- `remote nuke`
|
- `remote nuke`
|
||||||
- `prune` and `snapshot create --prune` list, read and delete.
|
- `prune` and `snapshot create --prune` list, read and delete.
|
||||||
|
|
||||||
An rclone remote that can move objects also needs permission to read and
|
An rclone remote that can move objects also needs permission to read and delete.
|
||||||
delete. An upload there looks up the object it replaces and deletes it. It stops
|
An upload there looks up the object it replaces and deletes it. It stops when
|
||||||
when the look-up is refused.
|
the look-up is refused.
|
||||||
|
|
||||||
`snapshot verify` and `snapshot restore` list only when you give a remote key.
|
`snapshot verify` and `snapshot restore` list only when you give a remote key.
|
||||||
`snapshot remove --local-only` does not reach the destination.
|
`snapshot remove --local-only` does not reach the destination.
|
||||||
@@ -1483,8 +1481,8 @@ Each archive holds three files:
|
|||||||
- `LICENSE`
|
- `LICENSE`
|
||||||
- `README.md`
|
- `README.md`
|
||||||
|
|
||||||
The release files are not signed, have no software bill of materials and are
|
The release files are not signed, have no software bill of materials and are not
|
||||||
not built reproducibly.
|
built reproducibly.
|
||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
@@ -1496,7 +1494,8 @@ Planned work is listed in [`TODO.md`](TODO.md).
|
|||||||
|
|
||||||
## author
|
## author
|
||||||
|
|
||||||
Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in Berlin in the summer of 2025.
|
Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in
|
||||||
|
Berlin in the summer of 2025.
|
||||||
|
|
||||||
Released as a free software gift to the world, no strings attached.
|
Released as a free software gift to the world, no strings attached.
|
||||||
|
|
||||||
|
|||||||
+9
-10
@@ -128,8 +128,8 @@ These commands do not build the application:
|
|||||||
- `EnsureStorageBinding` checks that the local index belongs to the configured
|
- `EnsureStorageBinding` checks that the local index belongs to the configured
|
||||||
destination.
|
destination.
|
||||||
- `markDestination` checks the marker file and then writes it.
|
- `markDestination` checks the marker file and then writes it.
|
||||||
- `PruneDatabase` removes unfinished snapshots and unused records from the
|
- `PruneDatabase` removes unfinished snapshots and unused records from the local
|
||||||
local index.
|
index.
|
||||||
- `createNamedSnapshot` runs once for each snapshot name.
|
- `createNamedSnapshot` runs once for each snapshot name.
|
||||||
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
|
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
|
||||||
|
|
||||||
@@ -142,8 +142,7 @@ these cases:
|
|||||||
not find.
|
not find.
|
||||||
|
|
||||||
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
|
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
|
||||||
left in the index would make the backup skip files that were never fully
|
left in the index would make the backup skip files that were never fully stored.
|
||||||
stored.
|
|
||||||
|
|
||||||
### One snapshot
|
### One snapshot
|
||||||
|
|
||||||
@@ -299,8 +298,8 @@ database.
|
|||||||
|
|
||||||
### Checks on the downloaded database
|
### Checks on the downloaded database
|
||||||
|
|
||||||
Decryption shows that a database is readable. It does not show that the
|
Decryption shows that a database is readable. It does not show that the database
|
||||||
database is honest. The restore therefore checks it:
|
is honest. The restore therefore checks it:
|
||||||
|
|
||||||
- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
|
- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
|
||||||
trigger, a view or a virtual table, or that lacks an expected table.
|
trigger, a view or a virtual table, or that lacks an expected table.
|
||||||
@@ -317,8 +316,8 @@ database is honest. The restore therefore checks it:
|
|||||||
|
|
||||||
- Downloaded blobs are kept decrypted in a cache directory.
|
- Downloaded blobs are kept decrypted in a cache directory.
|
||||||
- A file is restored once every blob it needs is in the cache.
|
- A file is restored once every blob it needs is in the cache.
|
||||||
- When no file is ready, the restore downloads the blobs for the file that
|
- When no file is ready, the restore downloads the blobs for the file that needs
|
||||||
needs the fewest.
|
the fewest.
|
||||||
- `restoreSweeper` deletes a cached blob once no remaining file needs it.
|
- `restoreSweeper` deletes a cached blob once no remaining file needs it.
|
||||||
|
|
||||||
A regular file is created with mode `0600` and gets its stored mode after its
|
A regular file is created with mode `0600` and gets its stored mode after its
|
||||||
@@ -351,8 +350,8 @@ the private key:
|
|||||||
|
|
||||||
- `CleanupLocalSnapshots` drops from the local index each snapshot whose
|
- `CleanupLocalSnapshots` drops from the local index each snapshot whose
|
||||||
manifest is gone from the destination.
|
manifest is gone from the destination.
|
||||||
- `PruneDatabase` removes unfinished snapshots and unused records from the
|
- `PruneDatabase` removes unfinished snapshots and unused records from the local
|
||||||
local index.
|
index.
|
||||||
- `PruneBlobs` deletes unused data at the destination.
|
- `PruneBlobs` deletes unused data at the destination.
|
||||||
|
|
||||||
`PruneBlobs` reads every manifest at the destination. It then deletes two kinds
|
`PruneBlobs` reads every manifest at the destination. It then deletes two kinds
|
||||||
|
|||||||
+15
-15
@@ -76,8 +76,8 @@ One row for each backed-up path: a regular file, a symlink or a directory.
|
|||||||
|
|
||||||
- `id`: a UUID. Primary key.
|
- `id`: a UUID. Primary key.
|
||||||
- `path`: the absolute path. Unique.
|
- `path`: the absolute path. Unique.
|
||||||
- `source_path`: the configured path under which the scan found this entry,
|
- `source_path`: the configured path under which the scan found this entry, made
|
||||||
made absolute with symlinks resolved. A restore does not read it.
|
absolute with symlinks resolved. A restore does not read it.
|
||||||
- `mtime`: the modification time.
|
- `mtime`: the modification time.
|
||||||
- `mtime_nsec`: the nanoseconds within that second.
|
- `mtime_nsec`: the nanoseconds within that second.
|
||||||
- `size`: the size in bytes.
|
- `size`: the size in bytes.
|
||||||
@@ -88,9 +88,9 @@ One row for each backed-up path: a regular file, a symlink or a directory.
|
|||||||
|
|
||||||
`idx_files_path` indexes `path`.
|
`idx_files_path` indexes `path`.
|
||||||
|
|
||||||
There is one row for each path, and it is updated when the file changes. The
|
There is one row for each path, and it is updated when the file changes. The row
|
||||||
row is shared by every snapshot in the local index that lists the path. An
|
is shared by every snapshot in the local index that lists the path. An older
|
||||||
older snapshot's own view of the file is in that snapshot's database at the
|
snapshot's own view of the file is in that snapshot's database at the
|
||||||
destination.
|
destination.
|
||||||
|
|
||||||
### `chunks`
|
### `chunks`
|
||||||
@@ -176,8 +176,8 @@ One row for each snapshot.
|
|||||||
- `new_chunk_size`: the total size of those chunks before compression.
|
- `new_chunk_size`: the total size of those chunks before compression.
|
||||||
- `blob_count`: the number of blobs this snapshot created.
|
- `blob_count`: the number of blobs this snapshot created.
|
||||||
- `total_size`: the total size of all files in the snapshot.
|
- `total_size`: the total size of all files in the snapshot.
|
||||||
- `uncompressed_chunk_size`: the total size, before compression, of the
|
- `uncompressed_chunk_size`: the total size, before compression, of the distinct
|
||||||
distinct chunks the snapshot's files use.
|
chunks the snapshot's files use.
|
||||||
- `blob_size`: the total compressed size of the blobs the snapshot uses.
|
- `blob_size`: the total compressed size of the blobs the snapshot uses.
|
||||||
- `blob_uncompressed_size`: the total uncompressed size of those blobs.
|
- `blob_uncompressed_size`: the total uncompressed size of those blobs.
|
||||||
- `compression_ratio`: the compression ratio of those blobs.
|
- `compression_ratio`: the compression ratio of those blobs.
|
||||||
@@ -192,11 +192,11 @@ example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
|
|||||||
- the snapshot name
|
- the snapshot name
|
||||||
- the start time, in UTC
|
- the start time, in UTC
|
||||||
|
|
||||||
The `snapshot list` table shows `uncompressed_chunk_size` as
|
The `snapshot list` table shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE`
|
||||||
`UNCOMPRESSED SIZE` and `new_chunk_size` as `NEW CHUNK SIZE`.
|
and `new_chunk_size` as `NEW CHUNK SIZE`.
|
||||||
|
|
||||||
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
|
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list` does
|
||||||
does not show it. The next backup deletes it, and so does `prune`.
|
not show it. The next backup deletes it, and so does `prune`.
|
||||||
|
|
||||||
### `snapshot_files`
|
### `snapshot_files`
|
||||||
|
|
||||||
@@ -268,8 +268,8 @@ Settings that belong to this copy of the index.
|
|||||||
- `key`: the name of the setting. Primary key.
|
- `key`: the name of the setting. Primary key.
|
||||||
- `value`: its value.
|
- `value`: its value.
|
||||||
|
|
||||||
The only key is `storage_url`. It holds the destination that the index was
|
The only key is `storage_url`. It holds the destination that the index was first
|
||||||
first used with. These commands refuse to run when the config names a different
|
used with. These commands refuse to run when the config names a different
|
||||||
destination:
|
destination:
|
||||||
|
|
||||||
- `snapshot create`
|
- `snapshot create`
|
||||||
@@ -361,8 +361,8 @@ still be in the `-wal` file beside it.
|
|||||||
|
|
||||||
### Restore
|
### Restore
|
||||||
|
|
||||||
A restore reads no file, chunk or blob rows from the local index. Given a
|
A restore reads no file, chunk or blob rows from the local index. Given a remote
|
||||||
remote key, it reads `snapshots` and `local_meta` there to check the destination
|
key, it reads `snapshots` and `local_meta` there to check the destination
|
||||||
marker.
|
marker.
|
||||||
|
|
||||||
- It downloads and decrypts `db.zst.age` into a temporary directory.
|
- It downloads and decrypts `db.zst.age` into a temporary directory.
|
||||||
|
|||||||
+23
-25
@@ -63,8 +63,8 @@ not exist, commands treat the destination as not present:
|
|||||||
- `snapshot list` shows the local index alone.
|
- `snapshot list` shows the local index alone.
|
||||||
- `snapshot remove` deletes nothing at the destination.
|
- `snapshot remove` deletes nothing at the destination.
|
||||||
|
|
||||||
The check is skipped when the look-up fails for any other reason. S3 refuses
|
The check is skipped when the look-up fails for any other reason. S3 refuses the
|
||||||
the look-up to credentials that can only write. A machine with such credentials
|
look-up to credentials that can only write. A machine with such credentials
|
||||||
therefore does not notice an emptied destination. Its next backup would not
|
therefore does not notice an emptied destination. Its next backup would not
|
||||||
upload data that its local index says is stored. After a destination is emptied,
|
upload data that its local index says is stored. After a destination is emptied,
|
||||||
run `vaultik database delete` on such a machine before it backs up again.
|
run `vaultik database delete` on such a machine before it backs up again.
|
||||||
@@ -103,8 +103,8 @@ This has two consequences:
|
|||||||
|
|
||||||
### Uploads
|
### Uploads
|
||||||
|
|
||||||
A backup looks a blob up by name before it uploads it. It does not upload a
|
A backup looks a blob up by name before it uploads it. It does not upload a blob
|
||||||
blob that is already there.
|
that is already there.
|
||||||
|
|
||||||
On S3, when the destination refuses the look-up, the backup uploads the blob. An
|
On S3, when the destination refuses the look-up, the backup uploads the blob. An
|
||||||
object with that name is then replaced by one that holds the same chunk data.
|
object with that name is then replaced by one that holds the same chunk data.
|
||||||
@@ -290,9 +290,9 @@ than four times this size. Only a file's last chunk can be smaller than a
|
|||||||
quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A
|
quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A
|
||||||
file of 2.5 MB or less is one chunk. An empty file has no chunk.
|
file of 2.5 MB or less is one chunk. An empty file has no chunk.
|
||||||
|
|
||||||
At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of
|
At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of chunks.
|
||||||
chunks. The size of the blob then does not show the size of any one chunk. A
|
The size of the blob then does not show the size of any one chunk. A limit close
|
||||||
limit close to `chunk_size` begins to show chunk sizes.
|
to `chunk_size` begins to show chunk sizes.
|
||||||
|
|
||||||
### Decrypted data on the local disk
|
### Decrypted data on the local disk
|
||||||
|
|
||||||
@@ -329,17 +329,17 @@ permission. Other commands need more:
|
|||||||
- `remote nuke`
|
- `remote nuke`
|
||||||
- `prune` lists, reads and deletes.
|
- `prune` lists, reads and deletes.
|
||||||
|
|
||||||
An rclone remote that can move objects also needs permission to read and
|
An rclone remote that can move objects also needs permission to read and delete.
|
||||||
delete. An upload there looks up the object it replaces and deletes it. It stops
|
An upload there looks up the object it replaces and deletes it. It stops when
|
||||||
when the look-up is refused.
|
the look-up is refused.
|
||||||
|
|
||||||
`snapshot verify` and `snapshot restore` list only when they are given a remote
|
`snapshot verify` and `snapshot restore` list only when they are given a remote
|
||||||
key. `snapshot remove --local-only` does not reach the destination.
|
key. `snapshot remove --local-only` does not reach the destination.
|
||||||
`snapshot remove --dry-run` deletes nothing.
|
`snapshot remove --dry-run` deletes nothing.
|
||||||
|
|
||||||
`snapshot create --prune` runs a prune on the backed-up machine. That machine
|
`snapshot create --prune` runs a prune on the backed-up machine. That machine
|
||||||
then holds credentials that can delete its own backups. To keep it to
|
then holds credentials that can delete its own backups. To keep it to write-only
|
||||||
write-only credentials, run `prune` from a separate, trusted machine.
|
credentials, run `prune` from a separate, trusted machine.
|
||||||
|
|
||||||
### New recipients cannot read old data
|
### New recipients cannot read old data
|
||||||
|
|
||||||
@@ -374,18 +374,17 @@ manifest is written last. A directory under `metadata/` that has no manifest
|
|||||||
therefore belongs to a backup that did not finish.
|
therefore belongs to a backup that did not finish.
|
||||||
|
|
||||||
The local index marks a snapshot complete only after both metadata files are
|
The local index marks a snapshot complete only after both metadata files are
|
||||||
uploaded. A backup that stops during that upload leaves the snapshot
|
uploaded. A backup that stops during that upload leaves the snapshot unfinished.
|
||||||
unfinished. The next backup removes it from the local index and backs up its
|
The next backup removes it from the local index and backs up its files again.
|
||||||
files again.
|
|
||||||
|
|
||||||
One short gap remains. A backup can stop after the upload and before the
|
One short gap remains. A backup can stop after the upload and before the
|
||||||
snapshot is marked complete. The destination then holds a snapshot that can be
|
snapshot is marked complete. The destination then holds a snapshot that can be
|
||||||
restored, and the local index marks it unfinished. `snapshot list` shows it as
|
restored, and the local index marks it unfinished. `snapshot list` shows it as
|
||||||
remote only.
|
remote only.
|
||||||
|
|
||||||
An upload that is cut off leaves no partial object under the final name on S3
|
An upload that is cut off leaves no partial object under the final name on S3 or
|
||||||
or on a local directory. The same holds for an rclone remote that can move
|
on a local directory. The same holds for an rclone remote that can move objects.
|
||||||
objects. An rclone remote that cannot move objects can keep a partial object.
|
An rclone remote that cannot move objects can keep a partial object.
|
||||||
|
|
||||||
## Pruning
|
## Pruning
|
||||||
|
|
||||||
@@ -410,11 +409,11 @@ running. Such a backup has uploaded blobs and perhaps `db.zst.age`, but not yet
|
|||||||
its manifest. A prune from this machine or another keeps those objects, as long
|
its manifest. A prune from this machine or another keeps those objects, as long
|
||||||
as no backup takes longer than the grace period.
|
as no backup takes longer than the grace period.
|
||||||
|
|
||||||
The grace period does not protect one case. A backup does not upload a blob
|
The grace period does not protect one case. A backup does not upload a blob that
|
||||||
that is already at the destination, so that blob keeps the time of its first
|
is already at the destination, so that blob keeps the time of its first upload.
|
||||||
upload. The blob can be left over from an earlier backup that stopped before
|
The blob can be left over from an earlier backup that stopped before its
|
||||||
its manifest. A prune that runs before the new manifest is written can then
|
manifest. A prune that runs before the new manifest is written can then delete
|
||||||
delete that blob. The new snapshot cannot be restored afterwards.
|
that blob. The new snapshot cannot be restored afterwards.
|
||||||
|
|
||||||
To avoid this case, give each machine and each user its own destination.
|
To avoid this case, give each machine and each user its own destination.
|
||||||
|
|
||||||
@@ -436,5 +435,4 @@ It works in these steps:
|
|||||||
- It downloads and decrypts the blobs it needs. It checks each blob against its
|
- It downloads and decrypts the blobs it needs. It checks each blob against its
|
||||||
name.
|
name.
|
||||||
- It writes each file from its chunks.
|
- It writes each file from its chunks.
|
||||||
- It sets each file's permissions and times, and its owner when it runs as
|
- It sets each file's permissions and times, and its owner when it runs as root.
|
||||||
root.
|
|
||||||
+38
-40
@@ -23,60 +23,58 @@ Work planned after 1.0, loosely ordered by priority.
|
|||||||
|
|
||||||
## Correctness and operability
|
## Correctness and operability
|
||||||
|
|
||||||
- **Outside security audit.** Before 1.0 the encryption and
|
- **Outside security audit.** Before 1.0 the encryption and blob-generation code
|
||||||
blob-generation code was reviewed. Every bug the review found was
|
was reviewed. Every bug the review found was fixed. The risks it accepted are
|
||||||
fixed. The risks it accepted are listed in
|
listed in [Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside
|
||||||
[Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside
|
|
||||||
audit has been done. age, zstd and content-defined chunking are mostly
|
audit has been done. age, zstd and content-defined chunking are mostly
|
||||||
off-the-shelf pieces. The code that joins them needs an outside read:
|
off-the-shelf pieces. The code that joins them needs an outside read:
|
||||||
- key handling
|
- key handling
|
||||||
- recipient parsing
|
- recipient parsing
|
||||||
- manifest trust boundary
|
- manifest trust boundary
|
||||||
- restore-time identity validation
|
- restore-time identity validation
|
||||||
- **Verify restored content end-to-end in CI.** The current integration
|
- **Verify restored content end-to-end in CI.** The current integration test
|
||||||
test does this for a small synthetic snapshot but not at scale. A
|
does this for a small synthetic snapshot but not at scale. A nightly job
|
||||||
nightly job against a multi-GB representative snapshot would catch
|
against a multi-GB representative snapshot would catch silent regressions in
|
||||||
silent regressions in the chunker, packer, or restore planner.
|
the chunker, packer, or restore planner.
|
||||||
|
|
||||||
## Performance
|
## Performance
|
||||||
|
|
||||||
- **Parallel blob downloads during restore.** A restore downloads one
|
- **Parallel blob downloads during restore.** A restore downloads one blob at a
|
||||||
blob at a time now. With a fast S3 endpoint and a multi-core machine,
|
time now. With a fast S3 endpoint and a multi-core machine, restore speed is
|
||||||
restore speed is bound by the fetch, decryption and decompression of
|
bound by the fetch, decryption and decompression of each blob. Running several
|
||||||
each blob. Running several of those in parallel against the disk
|
of those in parallel against the disk cache would close most of the remaining
|
||||||
cache would close most of the remaining gap. This needs to interact
|
gap. This needs to interact correctly with the locality planner and the
|
||||||
correctly with the locality planner and the sweeper.
|
sweeper.
|
||||||
- **Bandwidth limiting (`--bwlimit`).** For both upload and download.
|
- **Bandwidth limiting (`--bwlimit`).** For both upload and download. It is
|
||||||
It is useful for backing up over a shared link. It is tricky to make
|
useful for backing up over a shared link. It is tricky to make work correctly
|
||||||
work correctly with parallel downloads.
|
with parallel downloads.
|
||||||
- **Restart of interrupted restore.** Today a restore can be started
|
- **Restart of interrupted restore.** Today a restore can be started again, and
|
||||||
again, and it then overwrites the partial output. It does not resume
|
it then overwrites the partial output. It does not resume from where it
|
||||||
from where it stopped, and it does not skip files that are already
|
stopped, and it does not skip files that are already present. A `--resume`
|
||||||
present. A `--resume` mode that checks targets before fetching blobs
|
mode that checks targets before fetching blobs would matter for very large
|
||||||
would matter for very large restores.
|
restores.
|
||||||
- **Daemon mode.** A long-running mode that watches for file changes,
|
- **Daemon mode.** A long-running mode that watches for file changes, so that
|
||||||
so that frequent backups, such as hourly ones, skip the full scan. It
|
frequent backups, such as hourly ones, skip the full scan. It adds little for
|
||||||
adds little for the usual backups from cron every 12 to 36 hours. See
|
the usual backups from cron every 12 to 36 hours. See
|
||||||
[issue #204](https://git.eeqj.de/sneak/vaultik/issues/204).
|
[issue #204](https://git.eeqj.de/sneak/vaultik/issues/204).
|
||||||
|
|
||||||
## Usability
|
## Usability
|
||||||
|
|
||||||
- **Man pages and richer `--help` examples.** Cobra generates basic
|
- **Man pages and richer `--help` examples.** Cobra generates basic help. Man
|
||||||
help. Man pages would be a separate target.
|
pages would be a separate target.
|
||||||
- **`vaultik snapshot diff <a> <b>`.** Show which files changed between
|
- **`vaultik snapshot diff <a> <b>`.** Show which files changed between two
|
||||||
two snapshots, without restoring either.
|
snapshots, without restoring either.
|
||||||
- **Status reporting hook for `--cron`.** When a backup fails silently
|
- **Status reporting hook for `--cron`.** When a backup fails silently in cron,
|
||||||
in cron, the user has no idea. A configurable hook on completion, for
|
the user has no idea. A configurable hook on completion, for both success and
|
||||||
both success and failure, would close the loop. It could be a
|
failure, would close the loop. It could be a webhook, an email or
|
||||||
webhook, an email or `notify-send`.
|
`notify-send`.
|
||||||
|
|
||||||
## Infrastructure
|
## Infrastructure
|
||||||
|
|
||||||
- **Schema migrations after 1.0.** Migrations are supported, but none
|
- **Schema migrations after 1.0.** Migrations are supported, but none are added
|
||||||
are added before 1.0 because there is no installed base yet. After
|
before 1.0 because there is no installed base yet. After 1.0, each schema
|
||||||
1.0, each schema change is a new migration, so an existing local
|
change is a new migration, so an existing local index is migrated when
|
||||||
index is migrated when `vaultik` is updated. See
|
`vaultik` is updated. See
|
||||||
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
|
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
|
||||||
- **Storage backend coverage tests.** S3, `file://` and `rclone://` all
|
- **Storage backend coverage tests.** S3, `file://` and `rclone://` all share
|
||||||
share the `Storer` interface, but the rclone path is the least
|
the `Storer` interface, but the rclone path is the least exercised in CI.
|
||||||
exercised in CI.
|
|
||||||
Reference in new issue
Block a user