sneak/vaultik
1
0 Fork 0
Code Issues 6 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

Read all five documents against the code and fix what the sixth review found
check / check (push) Successful in 12m5s
Details

Browse Source 
Fixes the four findings of the sixth review: `snapshot remove` with
`--local-only`, the two comma-chained permission lists, six undefined terms,
and the sentence about an unfinished snapshot in the local index.

Then reads every sentence of `README.md`, `ARCHITECTURE.md`,
`docs/DATAMODEL.md`, `docs/REPOSTRUCTURE.md` and the comments in
`config.example.yml` against the code for four faults: a command's needs or
behaviour stated without regard to a flag, a false absolute, three or more
things chained with commas, and a technical word not defined on first use.
Each sentence changed is listed on the pull request.

The help string of `snapshot remove --dry-run` now says it needs no
permission to delete.

Model: fable-5-1
This commit is contained in:
clawbot committed 2026-11-09 01:57:09 +00:00
1 parent a133b8838e
commit 46b2a9fc78
6 files changed
+341 -193
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

ARCHITECTURE.md
+36 -26
View File
@@ -63,13 +63,13 @@ source files
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#code) says which type serves which [`docs/DATAMODEL.md`](docs/DATAMODEL.md#code) says which type serves which
table. table.
- `internal/storage`: the `Storer` interface and its three implementations, for - `internal/storage`: the `Storer` interface and its three implementations, for
S3, a local directory and rclone. S3, a local directory and [rclone](https://rclone.org/).
- `internal/s3`: the S3 client that the S3 `Storer` uses. - `internal/s3`: the S3 client that the S3 `Storer` uses.
- `internal/config`: loads and checks the config file. - `internal/config`: loads and checks the config file.
- `internal/ui`: formats the status lines, warnings and errors a user sees. - `internal/ui`: formats the status lines, warnings and errors a user sees.
- `internal/log`: the logger. It writes to stderr. - `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that keeps two data-changing commands from - `internal/pidlock`: the lock that `RunWithApp` takes for a command that
running at once. changes data, so that only one runs at a time.
- `internal/globals`: the version, commit and commit date, set when the binary - `internal/globals`: the version, commit and commit date, set when the binary
is built. is built.
- `internal/types`: distinct types for IDs, hashes and paths. - `internal/types`: distinct types for IDs, hashes and paths.
@@ -114,8 +114,12 @@ A command runs through `cli.RunOperation`. It works like this:
waits for the function to return, so that its temporary files are removed. waits for the function to return, so that its temporary files are removed.
- The wait ends after 30 seconds at most. - The wait ends after 30 seconds at most.
The `config` commands, `database delete`, `completion` and `version` do not These commands do not build the application:
build the application.
- the `config` commands
- `database delete`
- `completion`
- `version`
## Backup ## Backup
@@ -163,7 +167,8 @@ A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a
- `repairInterruptedBlobs` deletes the records of blobs whose upload never - `repairInterruptedBlobs` deletes the records of blobs whose upload never
finished. Their data is then stored again. finished. Their data is then stored again.
- `loadDatabaseState` loads the known files and chunks into memory. - `loadDatabaseState` loads the known files and chunks into memory.
- `scanPhase` walks the path. It sorts each entry into changed or unchanged. - `scanPhase` walks the path. It sorts each regular file and symlink into
changed or unchanged.
- `detectDeletedFiles` counts the files that the previous snapshot of this name - `detectDeletedFiles` counts the files that the previous snapshot of this name
had and that are now gone. had and that are now gone.
- `batchAddFilesToSnapshot` records the unchanged files in the new snapshot. - `batchAddFilesToSnapshot` records the unchanged files in the new snapshot.
@@ -184,8 +189,8 @@ Data is stored once at two levels.
- **Files.** A file is not read again when its record in the index still matches - **Files.** A file is not read again when its record in the index still matches
it. it.
- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk whose - **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk that
hash is already in the index is not stored again. an uploaded blob already holds, according to the index, is not stored again.
A file's record matches when all of these are unchanged: A file's record matches when all of these are unchanged:
@@ -208,8 +213,8 @@ on the content, so an insertion early in a file moves only nearby chunks.
this size. Only a file's last chunk can be smaller than a quarter of it. With this size. Only a file's last chunk can be smaller than a quarter of it. With
the default of 10 MB, those limits are 40 MB and 2.5 MB. the default of 10 MB, those limits are 40 MB and 2.5 MB.
`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole `ChunkReaderStreaming` passes each chunk to a callback as it is found. The
file is never held in memory. chunker buffers at most twice the largest chunk size.
### Blobs ### Blobs
@@ -217,8 +222,9 @@ file is never held in memory.
- It gives the blob a UUID and creates a temporary file. - It gives the blob a UUID and creates a temporary file.
- It adds each chunk's bytes to a running SHA-256 hash. - It adds each chunk's bytes to a running SHA-256 hash.
- In the same pass it compresses the bytes with zstd and encrypts them with - In the same pass it compresses the bytes with
age. The result goes to the temporary file. [zstd](https://github.com/facebook/zstd) and encrypts them with
[age](https://age-encryption.org/). The result goes to the temporary file.
- It records each chunk's offset and length within the blob. - It records each chunk's offset and length within the blob.
- It closes the blob when the next chunk would pass `blob_size_limit`, and at - It closes the blob when the next chunk would pass `blob_size_limit`, and at
the end of each path. the end of each path.
@@ -250,8 +256,8 @@ chunks is uploaded.
- `MarkSnapshotComplete` marks the snapshot complete. - `MarkSnapshotComplete` marks the snapshot complete.
Completion is recorded last on purpose. A backup that stops during the upload Completion is recorded last on purpose. A backup that stops during the upload
leaves the snapshot unfinished, and the next backup removes it and stores its leaves the snapshot unfinished, and the next backup removes it and backs up its
data again. files again.
`ExportSnapshotMetadata` works on a copy of the index, never on the index `ExportSnapshotMetadata` works on a copy of the index, never on the index
itself: itself:
@@ -282,11 +288,13 @@ therefore marks a backup that did not finish.
temporary directory. temporary directory.
- `getFilesToRestore` selects the files, from the paths given. - `getFilesToRestore` selects the files, from the paths given.
- `restoreAllFiles` downloads blobs and writes the files. - `restoreAllFiles` downloads blobs and writes the files.
- With `--verify`, `verifyRestoredFiles` reads each restored file and checks its - With `--verify`, `verifyRestoredFiles` reads each restored regular file and
chunk hashes. checks its chunk hashes.
The restore does not read the local index or the manifest. It finds each file's Given a snapshot ID, the restore does not read the local index. Given a remote
chunks and blobs through the downloaded database. key, it reads the local index only to check the destination marker. It does not
read the manifest. It finds each file's chunks and blobs through the downloaded
database.
### Checks on the downloaded database ### Checks on the downloaded database
@@ -313,12 +321,13 @@ database is honest. The restore therefore checks it:
- `restoreSweeper` deletes a cached blob once no remaining file needs it. - `restoreSweeper` deletes a cached blob once no remaining file needs it.
A regular file is created with mode `0600` and gets its stored mode after its A regular file is created with mode `0600` and gets its stored mode after its
content is written. A directory gets its stored mode, owner and modification content is written. A directory gets its stored mode and modification time after
time after the whole restore. the whole restore, and its owner when the command runs as root.
## Verify ## Verify
`Vaultik.VerifySnapshotWithOptions` checks a snapshot without the private key: Without `--deep`, `Vaultik.VerifySnapshotWithOptions` checks a snapshot without
the private key:
- It downloads the manifest. - It downloads the manifest.
- It checks that the snapshot's encrypted database is present. - It checks that the snapshot's encrypted database is present.
@@ -337,7 +346,7 @@ time after the whole restore.
## Prune ## Prune
`Vaultik.Prune` does three steps: `Vaultik.Prune` calls `EnsureStorageBinding` and then does three steps:
- `CleanupLocalSnapshots` drops from the local index each snapshot whose - `CleanupLocalSnapshots` drops from the local index each snapshot whose
manifest is gone from the destination. manifest is gone from the destination.
@@ -382,17 +391,18 @@ With no `storage_url`, it builds the S3 implementation from the `s3` settings.
Two checks keep the local index and the destination in step. Two checks keep the local index and the destination in step.
- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when - `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when
the file is reported missing while the local index holds snapshots. the file is reported missing while the local index holds snapshots. It skips
the check when the local index belongs to another destination.
- `EnsureStorageBinding` records the configured destination in the local index - `EnsureStorageBinding` records the configured destination in the local index
on first use. Later it fails when the config names a different destination. on first use. Later it fails when the config names a different destination.
Every command that changes data calls `EnsureStorageBinding` first. Every command that takes the lock calls `EnsureStorageBinding` first.
## Concurrency ## Concurrency
- Each open database uses one connection, so its queries run one at a time. - Each open database uses one connection, so its queries run one at a time.
- The index runs in SQLite's WAL mode. A second process can therefore read it - The index runs in SQLite's write-ahead log (WAL) mode. A second process can
while a backup writes to it. therefore read it while a backup writes to it.
- `Packer` guards its state with a mutex. - `Packer` guards its state with a mutex.
- `Scanner` holds `packerMu` while a chunk is added to the packer and while a - `Scanner` holds `packerMu` while a chunk is added to the packer and while a
blob is closed. blob is closed.
README.md
+183 -92
View File
@@ -3,7 +3,7 @@
`vaultik` is an MIT-licensed command-line tool for incremental, encrypted `vaultik` is an MIT-licensed command-line tool for incremental, encrypted
backups, written in Go by [@sneak](https://sneak.berlin). It encrypts your files backups, written in Go by [@sneak](https://sneak.berlin). It encrypts your files
with a public key and uploads them to an S3-compatible object store, a local with a public key and uploads them to an S3-compatible object store, a local
directory or an rclone remote. directory or any storage service that [rclone](https://rclone.org/) supports.
The machine being backed up holds only the public key. It cannot decrypt its own The machine being backed up holds only the public key. It cannot decrypt its own
backups. You keep the private key somewhere else. A backup never needs it. backups. You keep the private key somewhere else. A backup never needs it.
@@ -25,13 +25,14 @@ It was written to meet these requirements together:
## Features ## Features
- Encryption with [age](https://age-encryption.org/) (X25519 and - Encryption with [age](https://age-encryption.org/), to one or more public
ChaCha20-Poly1305), to one or more public keys. keys. age uses X25519 keys and the ChaCha20-Poly1305 cipher.
- Incremental backups. A file is read again only when it has changed, or when - Incremental backups. A file is read again only when it has changed, or when
part of its content has to be stored again. part of its content has to be stored again.
- Deduplication. Identical content is stored once, across files and across - Deduplication. Identical content is stored once, across files and across
snapshots. snapshots.
- zstd compression at a level you choose. - Compression with [zstd](https://github.com/facebook/zstd), at a level you
choose.
- Small files are packed together into large objects. This keeps the number of - Small files are packed together into large objects. This keeps the number of
S3 requests low. S3 requests low.
- File contents, names, paths and metadata are encrypted at the destination. - File contents, names, paths and metadata are encrypted at the destination.
@@ -91,12 +92,13 @@ vaultik snapshot list
The default config is written for macOS. It backs up your home directory and The default config is written for macOS. It backs up your home directory and
`/Applications`, with a list of excludes. A configured path that does not exist `/Applications`, with a list of excludes. A configured path that does not exist
stops the backup. On another system, run `vaultik config edit` first and change stops the backup, unless you give `--skip-errors`. On another system, run
the `snapshots` section. `vaultik config edit` first and change the `snapshots` section.
On macOS, grant your terminal app Full Disk Access before the first backup. The On macOS, grant your terminal app Full Disk Access before the first backup. The
setting is under System Settings, Privacy & Security, Full Disk Access. Without setting is under System Settings, Privacy & Security, Full Disk Access. Without
it the backup stops with a permission error on a protected directory. it the backup stops with a permission error on a protected directory, unless you
give `--skip-errors`.
## Everyday use ## Everyday use
@@ -155,7 +157,7 @@ Use that remote key in place of the snapshot ID:
```sh ```sh
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
# Restore everything, then check the chunk hashes of every restored file. # Restore everything, then check the chunk hashes of the restored files.
vaultik snapshot restore --verify <remote-key> ~/vaultik-restore vaultik snapshot restore --verify <remote-key> ~/vaultik-restore
# Optionally, also check every blob of the snapshot at the destination. # Optionally, also check every blob of the snapshot at the destination.
@@ -194,10 +196,14 @@ vaultik version
### What every command needs ### What every command needs
`config init`, `completion` and `version` need nothing. `completion` and `version` need nothing. `config init` needs write access to the
config directory.
`config edit`, `config get` and `config set` need a config file that exists. It These commands need a config file that exists, even an incomplete one:
does not have to be complete.
- `config edit`
- `config get`
- `config set`
Every other command needs a config file that loads. See Every other command needs a config file that loads. See
[Configuration](#configuration) for the checks. [Configuration](#configuration) for the checks.
@@ -219,24 +225,34 @@ sets file owners only when it runs as root.
[Where the config file is](#where-the-config-file-is). [Where the config file is](#where-the-config-file-is).
- `--verbose`, `-v`: write informational log records to stderr. - `--verbose`, `-v`: write informational log records to stderr.
- `--debug`: write debug log records to stderr. - `--debug`: write debug log records to stderr.
- `--quiet`, `-q`: print only warnings, errors and the output a script reads. - `--quiet`, `-q`: leave out the banner, the status lines, the tables and the
See [Output](#output). reports. See [Output](#output).
- `--skip-errors`: skip a file that cannot be read during a backup, or cannot be - `--skip-errors`: skip a file that cannot be read during a backup, or cannot be
restored during a restore. Without it the command stops at the first such restored during a restore. Without it the command stops at the first such
file. file.
`--quiet` and `--cron` override `--verbose` and `--debug`.
`--skip-errors` applies to `snapshot create` and `snapshot restore` only. A `--skip-errors` applies to `snapshot create` and `snapshot restore` only. A
command that skipped anything still finishes, and then exits non-zero. command that skipped anything still finishes, and then exits non-zero.
### Durations ### Durations
`--keep-newer-than`, `--older-than` and `prune_grace_period` take a duration. These take a duration:
- `--keep-newer-than`
- `--older-than`
- `prune_grace_period`
These units are accepted:
- `d`: days. - `d`: days.
- `w`: weeks. - `w`: weeks.
- `mo`: months of 30 days. - `mo`: months of 30 days.
- `y`: years of 365 days. - `y`: years of 365 days.
- `h`, `m` and `s`: hours, minutes and seconds, as in `36h` or `90m`. - `h`: hours, as in `36h`.
- `m`: minutes, as in `90m`.
- `s`: seconds.
`m` means minutes and `mo` means months. Days, weeks, months and years take `m` means minutes and `mo` means months. Days, weeks, months and years take
whole numbers and can be combined, as in `2w3d`. whole numbers and can be combined, as in `2w3d`.
@@ -249,9 +265,9 @@ Writes a default config file, with a comment that explains each setting it
lists. lists.
- It writes to the path from `--config`, or else from `VAULTIK_CONFIG`. - It writes to the path from `--config`, or else from `VAULTIK_CONFIG`.
- Otherwise it writes to `config.yml` in the platform config directory. That is - Otherwise it writes to `config.yml` in the platform config directory. By
`~/.config/vaultik/` on Linux and `~/Library/Application Support/vaultik/` on default that is `~/.config/vaultik/` on Linux and
macOS. `~/Library/Application Support/vaultik/` on macOS.
- As root it writes to `/etc/vaultik/config.yml` instead. - As root it writes to `/etc/vaultik/config.yml` instead.
- It creates the file with mode `0600`, because the file can hold credentials. - It creates the file with mode `0600`, because the file can hold credentials.
- It refuses to overwrite a file that exists. - It refuses to overwrite a file that exists.
@@ -275,7 +291,8 @@ Sets one value in the config file. The key is a dotted path, as in
- A key that does not exist yet is created. A missing parent becomes a list when - A key that does not exist yet is created. A missing parent becomes a list when
the next part of the path is a number, and a map otherwise. the next part of the path is a number, and a map otherwise.
- In a list, the index one past the end appends an entry. - In a list, the index one past the end appends an entry.
- The command prints the key it set, and never the value. - The command prints the key it set, and not the value. A refusal for a value
of the wrong type quotes the beginning of that value.
- It sets the file's mode to `0600` when group or others could read it. - It sets the file's mode to `0600` when group or others could read it.
The command refuses a change that would stop the config from loading. An example The command refuses a change that would stop the config from loading. An example
@@ -301,7 +318,8 @@ It needs:
It does not need the private key. It does not need the private key.
A file is read again when any of these holds: A file is read when the local index does not know it, or when any of these
holds:
- Its size has changed since the last backup. - Its size has changed since the last backup.
- Its modification time has changed. - Its modification time has changed.
@@ -383,9 +401,9 @@ config.
### `snapshot list` ### `snapshot list`
Lists the snapshots in the local index and the snapshots at the destination, as Lists the finished snapshots in the local index and the snapshots at the
one table. Before the table, the command prints the snapshot names and paths in destination, as one table. Before the table, the command prints the snapshot
the config. names and paths in the config.
It needs permission to list and read at the destination. It needs no private It needs permission to list and read at the destination. It needs no private
key. key.
@@ -402,7 +420,8 @@ The table has these columns:
A snapshot at the destination that is not in the local index is shown as A snapshot at the destination that is not in the local index is shown as
`<remote only:...>`. The dots stand for the first 12 characters of its remote `<remote only:...>`. The dots stand for the first 12 characters of its remote
key. Its last two columns read `<remote only>`, because those sizes come from key. Its last two columns read `<remote only>`, because those sizes come from
the local index. Its timestamp is the time its backup finished. the local index. Its timestamp is the time its manifest was written, at the end
of its backup.
The command reports these below the table: The command reports these below the table:
@@ -424,7 +443,8 @@ marker file although the local index holds snapshots.
- `remote_key`: the full 64-character remote key. - `remote_key`: the full 64-character remote key.
- `timestamp` - `timestamp`
- `compressed_size` - `compressed_size`
- `uncompressed_size` and `new_chunk_size`: left out for a remote-only snapshot. - `uncompressed_size` and `new_chunk_size`: left out when zero, and for a
remote-only snapshot.
- `locally_tracked`: whether the snapshot is in the local index. - `locally_tracked`: whether the snapshot is in the local index.
- `remote_present`: whether the snapshot is at the destination. It is `null` - `remote_present`: whether the snapshot is at the destination. It is `null`
when the destination could not be listed. when the destination could not be listed.
@@ -469,7 +489,8 @@ variable wins when both are set. See [The private key](#the-private-key).
- It downloads and decrypts every blob. It checks the hash of each chunk, and - It downloads and decrypts every blob. It checks the hash of each chunk, and
the hash that names the blob. the hash that names the blob.
It stops at the first blob that fails. The presence check counts every missing or wrong-size blob. The download check
stops at the first blob that fails.
`--deep` proves that the stored data is intact and decrypts with your key. It `--deep` proves that the stored data is intact and decrypts with your key. It
does not prove who wrote the data. does not prove who wrote the data.
@@ -510,20 +531,22 @@ cannot be listed or has no marker file.
Removes one snapshot. The argument must be a snapshot ID. A remote key is not Removes one snapshot. The argument must be a snapshot ID. A remote key is not
accepted. accepted.
It needs the [lock](#locking), and permission to list and delete at the It needs the [lock](#locking). It needs permission to list and delete at the
destination. It needs no private key. destination, unless you give `--local-only`. It needs no private key.
Flags: Flags:
- `--force`, `-f`: do not ask for confirmation. - `--force`, `-f`: do not ask for confirmation.
- `--dry-run`: say what would be removed, and remove nothing. - `--dry-run`: say what would be removed, and remove nothing. It needs no
permission to delete.
- `--local-only`: remove the snapshot from the local index only. The destination - `--local-only`: remove the snapshot from the local index only. The destination
is not changed. is not changed.
- `--json`: print the result as JSON. This also skips the confirmation. - `--json`: print the result as JSON. This also skips the confirmation.
The command removes the snapshot from the local index. It also deletes the The command removes the snapshot from the local index. Without `--local-only`,
snapshot's directory under `metadata/` at the destination. Its blobs stay. Run it also deletes the snapshot's directory under `metadata/` at the destination.
`vaultik prune` afterwards to delete the blobs that no snapshot uses any more. Its blobs stay. Run `vaultik prune` afterwards to delete the blobs that no
snapshot uses any more.
An ID that is in neither the local index nor the destination is an error. With An ID that is in neither the local index nor the destination is an error. With
`--local-only`, the snapshot must be in the local index. `--local-only`, the snapshot must be in the local index.
@@ -586,8 +609,8 @@ Restore into a new directory that only you can write, not a shared one such as
Flags: Flags:
- `--verify`: after the restore, read every restored file and check its chunk - `--verify`: after the restore, read every restored regular file and check its
hashes. chunk hashes.
With `--skip-errors`, a file that cannot be restored is reported and skipped, With `--skip-errors`, a file that cannot be restored is reported and skipped,
and the command exits non-zero. `--verify` does not check a skipped file. and the command exits non-zero. `--verify` does not check a skipped file.
@@ -666,12 +689,13 @@ Reports what the destination holds:
It needs permission to list and read at the destination. It needs no private It needs permission to list and read at the destination. It needs no private
key. key.
When a manifest cannot be read, the figures for blobs that no manifest lists are
reported as unknown. That snapshot's own blob count and size are unknown too.
The command skips a name under `metadata/` that is not a remote key, with a The command skips a name under `metadata/` that is not a remote key, with a
warning. It does not count a file under `blobs/` whose name is not a blob hash. warning. It does not count a file under `blobs/` whose name is not a blob hash.
When a manifest cannot be read, that snapshot's blob count and size are unknown.
The figures for blobs that no manifest lists are unknown when a manifest cannot
be read or sits under a skipped name.
`--json` prints the report as JSON. Unknown figures are `null`. `--json` prints the report as JSON. Unknown figures are `null`.
`unreadable_manifests` lists the remote key of each manifest that could not be `unreadable_manifests` lists the remote key of each manifest that could not be
read. `skipped_manifest_count` counts the manifests under skipped names. read. `skipped_manifest_count` counts the manifests under skipped names.
@@ -715,7 +739,12 @@ Use this command after you point the config at a different destination. See
### `completion <shell>` ### `completion <shell>`
Prints a completion script for `bash`, `zsh`, `fish` or `powershell`. Prints a completion script for one of these shells:
- `bash`
- `zsh`
- `fish`
- `powershell`
```sh ```sh
# zsh, for the current session # zsh, for the current session
@@ -736,31 +765,51 @@ vaultik completion fish > ~/.config/fish/completions/vaultik.fish
### `version` ### `version`
Prints the version, the commit, the commit date and the platform. It labels the Prints these:
commit date `build date`. It also says when the binary is a development build,
not a release. - the version
- the commit
- the commit date, labelled `build date`
- the Go version
- the platform
- the author
- the homepage
- the license
It also says when the binary is a development build, not a release.
## Configuration ## Configuration
The config file is YAML. `vaultik config init` writes one to start from, and The config file is YAML. `vaultik config init` writes one to start from, and
[`config.example.yml`](config.example.yml) is a longer example. [`config.example.yml`](config.example.yml) is a longer example.
A command refuses to start when the config does not load. The `config` commands, A command refuses to start when the config does not load. These commands are
`completion` and `version` are the exceptions. The config loads when all of the exceptions:
these hold:
- the `config` commands
- `completion`
- `version`
The config loads when it is valid YAML and all of these hold:
- Every entry in `age_recipients` is an age X25519 public key. - Every entry in `age_recipients` is an age X25519 public key.
- `snapshots` has at least one name, and every name has at least one path. - `snapshots` has at least one name, and every name has at least one path.
- A destination is set, with credentials when it is S3. - A destination is set. Its `storage_url` has one of the three schemes under
[Destinations](#destinations). An S3 destination has credentials.
- Every `${ENV:NAME}` names an environment variable with a non-empty value.
- Every other setting below is within its limits. - Every other setting below is within its limits.
`vaultik` warns when group or others can read the config file. Every command that needs the local index warns when group or others can read the
config file.
A value written as `${ENV:NAME}` is replaced by the environment variable `NAME`. A value written as `${ENV:NAME}` is replaced by the environment variable `NAME`.
This keeps a credential out of the file. This keeps a credential out of the file.
A leading `~` is expanded to your home directory in snapshot paths, in A leading `~` is expanded to your home directory in these places:
`index_path` and in a `file://~/...` destination.
- snapshot paths
- `index_path`
- a `file://~/...` destination
### Where the config file is ### Where the config file is
@@ -768,11 +817,13 @@ A leading `~` is expanded to your home directory in snapshot paths, in
- the path given with `--config` - the path given with `--config`
- the path in the `VAULTIK_CONFIG` environment variable - the path in the `VAULTIK_CONFIG` environment variable
- `config.yml` in the platform config directory, which is `~/.config/vaultik/` - `config.yml` in the platform config directory, which by default is
on Linux and `~/Library/Application Support/vaultik/` on macOS `~/.config/vaultik/` on Linux and `~/Library/Application Support/vaultik/` on
macOS
- `/etc/vaultik/config.yml` - `/etc/vaultik/config.yml`
A path from `--config` or `VAULTIK_CONFIG` must exist. A command that reads the config fails when the path from `--config` or
`VAULTIK_CONFIG` does not exist. `config init` creates the file there.
### Settings ### Settings
@@ -806,7 +857,8 @@ A path from `--config` or `VAULTIK_CONFIG` must exist.
- `s3.region` - `s3.region`
- The S3 region. Default: `us-east-1`. - The S3 region. Default: `us-east-1`.
- `s3.part_size` - `s3.part_size`
- The size of each part of a multipart upload. Default: `5MiB`. - A large blob is uploaded to S3 in parts. This is the size of each part.
Default: `5MiB`.
- It must be between `5MiB` and `5GiB`. - It must be between `5MiB` and `5GiB`.
- A blob too large for 10,000 parts of this size is uploaded in larger - A blob too large for 10,000 parts of this size is uploaded in larger
parts. parts.
@@ -826,8 +878,11 @@ A path from `--config` or `VAULTIK_CONFIG` must exist.
- It must be at least four times `chunk_size`. - It must be at least four times `chunk_size`.
- `compression_level` - `compression_level`
- The zstd compression level, from `1` to `19`. Default: `3`. - The zstd compression level, from `1` to `19`. Default: `3`.
- Levels `1` to `2`, `3` to `5`, `6` to `9` and `10` to `19` form four - Levels in one of these groups compress the same way:
groups. Levels in one group compress the same way. - `1` to `2`
- `3` to `5`
- `6` to `9`
- `10` to `19`
- `prune_grace_period` - `prune_grace_period`
- How old unused data must be before `prune` deletes it. Default: `24h`. - How old unused data must be before `prune` deletes it. Default: `24h`.
- It applies to a blob that no manifest lists. It also applies to every - It applies to a blob that no manifest lists. It also applies to every
@@ -839,8 +894,8 @@ A path from `--config` or `VAULTIK_CONFIG` must exist.
- Only the part before the first `.` is used. - Only the part before the first `.` is used.
- `index_path` - `index_path`
- The path of the local index. - The path of the local index.
- Default: `index.sqlite` in the platform data directory. That is - Default: `index.sqlite` in the platform data directory. By default that
`~/.local/share/vaultik/` on Linux and is `~/.local/share/vaultik/` on Linux and
`~/Library/Application Support/vaultik/` on macOS. `~/Library/Application Support/vaultik/` on macOS.
Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB` Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB`
@@ -856,7 +911,9 @@ and `GiB` are powers of 1024.
destination. destination.
- `EDITOR`: the editor that `config edit` starts. - `EDITOR`: the editor that `config edit` starts.
- `TMPDIR`: the directory for temporary files. - `TMPDIR`: the directory for temporary files.
- `NO_COLOR`: when set, status lines have no color. - `NO_COLOR`: a non-empty value turns off color in status lines.
- `XDG_CONFIG_HOME` and `XDG_DATA_HOME`: move the platform config and data
directories.
- `VAULTIK_CPUPROFILE`: write a CPU profile to this path. For debugging. - `VAULTIK_CPUPROFILE`: write a CPU profile to this path. For debugging.
- `VAULTIK_MEMPROFILE`: write a heap profile to this path on exit. For - `VAULTIK_MEMPROFILE`: write a heap profile to this path on exit. For
debugging. debugging.
@@ -966,15 +1023,26 @@ A leftover `.partial` file is ignored. You can delete it.
### Destination permissions ### Destination permissions
Each command needs different permissions at the destination. Commands need these permissions at the destination:
- `snapshot create` needs only permission to write objects. A machine you back - `snapshot create` without `--prune` needs only permission to write objects. A
up can therefore hold credentials that can neither read nor delete. machine you back up can therefore hold credentials that can neither read nor
- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list delete.
and read. - These commands list and read:
- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete. - `snapshot list`
- `snapshot verify`
- `snapshot restore`
- `remote info`
- These commands list and delete:
- `snapshot purge`
- `snapshot remove`
- `remote nuke`
- `prune` and `snapshot create --prune` list, read and delete. - `prune` and `snapshot create --prune` list, read and delete.
`snapshot verify` and `snapshot restore` list only when you give a remote key.
`snapshot remove --local-only` does not reach the destination.
`snapshot remove --dry-run` deletes nothing.
A machine that runs `snapshot create --prune` can delete its own backups. To A machine that runs `snapshot create --prune` can delete its own backups. To
keep it to write-only credentials, run `prune` from a separate, trusted machine. keep it to write-only credentials, run `prune` from a separate, trusted machine.
@@ -991,12 +1059,15 @@ unplugged volume leaves an empty directory at its mount point. Without the
marker, that directory would look like a destination whose snapshots are all marker, that directory would look like a destination whose snapshots are all
gone. gone.
The check applies while the local index holds snapshots. When the destination The check applies while the local index holds snapshots and belongs to the
then answers that the marker does not exist, commands treat the destination as configured destination. When the destination then answers that the marker does
not present: not exist, commands treat the destination as not present:
- `snapshot create` refuses to write to it. - `snapshot create` refuses to write to it.
- `prune`, `snapshot purge` and `remote nuke` fail. - These commands fail:
- `prune`
- `snapshot purge`
- `remote nuke`
- `snapshot list` shows the local index alone. - `snapshot list` shows the local index alone.
- `snapshot remove` deletes nothing at the destination. - `snapshot remove` deletes nothing at the destination.
@@ -1025,7 +1096,11 @@ different destination:
- `remote nuke` - `remote nuke`
The destination that is compared is `storage_url`. When `storage_url` is not The destination that is compared is `storage_url`. When `storage_url` is not
set, it is `s3.endpoint`, `s3.bucket` and `s3.prefix`. set, these three settings are compared instead:
- `s3.endpoint`
- `s3.bucket`
- `s3.prefix`
There are two ways out: There are two ways out:
@@ -1057,8 +1132,7 @@ The new snapshot cannot be restored afterwards.
## Locking ## Locking
Commands that change data take a lock, so that only one of them runs at a time. Five commands take a lock, so that only one of them runs at a time:
These commands take it:
- `snapshot create` - `snapshot create`
- `snapshot purge` - `snapshot purge`
@@ -1068,9 +1142,9 @@ These commands take it:
A second one does not wait. It exits at once with an "already running" error. A second one does not wait. It exits at once with an "already running" error.
The lock is the file `vaultik.pid` in the platform data directory. That is The lock is the file `vaultik.pid` in the platform data directory. By default
`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/` that is `~/.local/share/vaultik/` on Linux and
on macOS. `~/Library/Application Support/vaultik/` on macOS.
The lock covers one user on one machine. It does not stop another user or The lock covers one user on one machine. It does not stop another user or
another machine from writing to the same destination. See another machine from writing to the same destination. See
@@ -1111,7 +1185,8 @@ banner, log records and a final error stay on your terminal.
Log records are colored lines when stderr is a terminal. They are JSON, one Log records are colored lines when stderr is a terminal. They are JSON, one
object per line, when stderr is redirected. object per line, when stderr is redirected.
Status lines are colored only on a terminal, and never when `NO_COLOR` is set. Status lines are colored only on a terminal, and never when `NO_COLOR` has a
non-empty value.
### `--json` ### `--json`
@@ -1183,8 +1258,8 @@ destination, what it protects and the risks it accepts.
## Limitations ## Limitations
- **No extended attributes.** ACLs, macOS Finder metadata, quarantine flags and - **No extended attributes.** Access control lists, macOS Finder metadata,
SELinux labels are not backed up. quarantine flags and SELinux labels are not backed up.
- **No hard links.** Two hard links to one file are backed up as two files. The - **No hard links.** Two hard links to one file are backed up as two files. The
content is stored once, but a restore creates two separate files. content is stored once, but a restore creates two separate files.
- **No sparse files.** A sparse file is backed up at its full size, zeros - **No sparse files.** A sparse file is backed up at its full size, zeros
@@ -1260,25 +1335,33 @@ call them.
- `script/test`: runs the whole test suite, by building the `test` stage of the - `script/test`: runs the whole test suite, by building the `test` stage of the
`Dockerfile`. `Dockerfile`.
- `script/lint`: runs the linter, by building the `lint` stage of the - `script/lint`: runs the linter, by building the `lint` stage of the
`Dockerfile`. The linter never runs on your machine. `Dockerfile`. The linter runs only inside Docker.
- `script/lint-fix`: applies the linter's automatic fixes to your files. It - `script/lint-fix`: applies the linter's automatic fixes to your files. It
needs a local Docker daemon. Run `script/lint` afterwards. needs a local Docker daemon. Run `script/lint` afterwards.
- `script/fmt`: formats the Go code. - `script/fmt`: formats the Go code.
- `script/fmt-check`: checks the formatting and changes nothing. - `script/fmt-check`: checks the formatting and changes nothing.
- `script/check`: runs `script/test`, `script/lint` and `script/fmt-check`. - `script/check`: runs these three scripts:
- `script/precommit`: runs `go mod tidy`, `go fmt` and `script/check`. The git - `script/test`
pre-commit hook calls it. - `script/lint`
- `script/fmt-check`
- `script/precommit`: the git pre-commit hook calls it. It runs these three:
- `go mod tidy`
- `go fmt`
- `script/check`
- `script/install-precommit`: installs that hook. - `script/install-precommit`: installs that hook.
- `script/docker`: builds the Docker image. The build runs the linter and the - `script/docker`: builds the Docker image. The build runs the linter and the
tests too. tests too.
- `script/cibuild`: what CI runs on every push. It runs `script/bootstrap`, - `script/cibuild`: what CI runs on every push. It runs these three:
`script/check` and the image build. - `script/bootstrap`
- `script/check`
- the image build
- `script/version`: prints the version to build into the binary. - `script/version`: prints the version to build into the binary.
- `script/projectname`: prints the project name, which tags the Docker image. - `script/projectname`: prints the project name, which tags the Docker image.
- `script/install-go`: installs the Go version that `go.mod` names into - `script/install-go`: installs the Go version that `go.mod` names into
`.tool/go`. It checks the download against a checksum in the script. `.tool/go`. It checks the download against a checksum in the script.
- `script/install-goreleaser`: installs the pinned `goreleaser` into - `script/install-goreleaser`: installs the pinned
`.tool/bin`, checked the same way. [`goreleaser`](https://goreleaser.com/) into `.tool/bin`, checked the same
way.
- `script/release`: builds and publishes the release files. - `script/release`: builds and publishes the release files.
- `script/release-snapshot`: builds the release files into `./dist` and - `script/release-snapshot`: builds the release files into `./dist` and
publishes nothing. publishes nothing.
@@ -1289,15 +1372,17 @@ skip the tests or the linter and still report success.
The test suite has no separate integration target. Every test runs every time. The test suite has no separate integration target. Every test runs every time.
Each test binary is stopped after 90 seconds. Each test binary is stopped after 90 seconds.
To change the Go version, change these together: `go.mod`, the version and To change the Go version, change these together:
checksums in `script/install-go`, and the two `golang` image digests in the
`Dockerfile`. - `go.mod`
- the version and checksums in `script/install-go`
- the two `golang` image digests in the `Dockerfile`
### Messages to the user ### Messages to the user
Status lines, warnings and errors go through `internal/ui`. The data a command Write status lines, warnings and errors through `internal/ui`. The data a
exists to print is written plain, with no marker and no color. That covers command exists to print is written plain, with no marker and no color. That
tables, reports, JSON documents and `config get` values. covers tables, reports, JSON documents and `config get` values.
`internal/ui` has these kinds of message: `internal/ui` has these kinds of message:
@@ -1383,8 +1468,14 @@ version.
`make release-snapshot` builds the same files into `./dist` and publishes `make release-snapshot` builds the same files into `./dist` and publishes
nothing. nothing.
Each archive holds the binary, `LICENSE` and `README.md`. The release files are Each archive holds three files:
not signed, have no SBOM and are not built reproducibly.
- the binary
- `LICENSE`
- `README.md`
The release files are not signed, have no software bill of materials and are
not built reproducibly.
## license ## license
config.example.yml
+21 -15
View File
@@ -3,7 +3,7 @@
# defaults; where an option has a default, its comment gives it. # defaults; where an option has a default, its comment gives it.
# Copy this file and change the values you need # Copy this file and change the values you need
# Age recipient public keys for encryption # Age recipient public keys for encryption (age: https://age-encryption.org/)
# Backups are encrypted to these public keys. snapshot create needs at least # Backups are encrypted to these public keys. snapshot create needs at least
# one; listing, verifying and restoring do not # one; listing, verifying and restoring do not
# Generate a key pair, and show its public key, with: # Generate a key pair, and show its public key, with:
@@ -11,6 +11,7 @@
# grep 'public key' vaultik_backup_private_key.txt # grep 'public key' vaultik_backup_private_key.txt
# Keep the private key file: restoring needs it. Do not pipe age-keygen # Keep the private key file: restoring needs it. Do not pipe age-keygen
# straight into grep, which would discard the private key # straight into grep, which would discard the private key
# Default: none
age_recipients: age_recipients:
- age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz - age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz
@@ -268,6 +269,7 @@ snapshots:
- "**/vendor" - "**/vendor"
# Global patterns to exclude from all backups # Global patterns to exclude from all backups
# Default: none
exclude: exclude:
- "*.tmp" - "*.tmp"
@@ -276,8 +278,8 @@ exclude:
# An s3:// URL also needs s3.access_key_id and s3.secret_access_key below # An s3:// URL also needs s3.access_key_id and s3.secret_access_key below
# A file:// path must be absolute, and the directory must already exist: # A file:// path must be absolute, and the directory must already exist:
# vaultik never creates it # vaultik never creates it
# An rclone:// URL needs the remote to be defined in rclone's own config # An rclone:// URL names a remote of rclone (https://rclone.org/); the remote
# (rclone config) # must be defined in rclone's own config (rclone config)
storage_url: "rclone://myremote/path/to/backups" storage_url: "rclone://myremote/path/to/backups"
# S3-compatible storage configuration # S3-compatible storage configuration
@@ -310,7 +312,7 @@ storage_url: "rclone://myremote/path/to/backups"
# #
# # Part size for multipart uploads # # Part size for multipart uploads
# # Minimum 5MiB, maximum 5GiB; affects memory usage during upload # # Minimum 5MiB, maximum 5GiB; affects memory usage during upload
# # A blob too large for 10,000 parts of this size gets larger parts # # An object too large for 10,000 parts of this size gets larger parts
# # Supports: 10MB, 16MiB, 100MiB, etc. (5MB is below the minimum) # # Supports: 10MB, 16MiB, 100MiB, etc. (5MB is below the minimum)
# # Default: 5MiB # # Default: 5MiB
# #part_size: 5MiB # #part_size: 5MiB
@@ -323,8 +325,9 @@ storage_url: "rclone://myremote/path/to/backups"
# Linux: ~/.local/share/vaultik/index.sqlite # Linux: ~/.local/share/vaultik/index.sqlite
#index_path: /path/to/index.sqlite #index_path: /path/to/index.sqlite
# Average chunk size for content-defined chunking # Average size of a chunk, a piece of a file's content. Chunk boundaries
# Smaller chunks = better deduplication but more metadata # depend on the file's content, not on fixed offsets
# Smaller chunks find more duplicate content and make the index larger
# Minimum 1MB # Minimum 1MB
# Supports: 10MB, 5M, 1GB, 64MiB, etc. # Supports: 10MB, 5M, 1GB, 64MiB, etc.
# Default: 10MB # Default: 10MB
@@ -334,9 +337,10 @@ storage_url: "rclone://myremote/path/to/backups"
# Multiple chunks are packed into blobs up to this size # Multiple chunks are packed into blobs up to this size
# Must be at least four times chunk_size (the largest chunk the chunker can # Must be at least four times chunk_size (the largest chunk the chunker can
# emit); a smaller limit would let a single-chunk blob exceed it. # emit); a smaller limit would let a single-chunk blob exceed it.
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a # Chunking uses no secret (the parameters of the FastCDC chunking algorithm
# large limit a blob holds hundreds of chunks, so individual chunk lengths are # are fixed and public). At a large limit a blob holds hundreds of chunks, so
# not visible in its size; lowering the limit toward chunk_size exposes them. # individual chunk lengths are not visible in its size; lowering the limit
# toward chunk_size exposes them.
# A backup needs free temporary space, because each blob is written whole to # A backup needs free temporary space, because each blob is written whole to
# a temporary file before it is uploaded (up to about blob_size_limit; an # a temporary file before it is uploaded (up to about blob_size_limit; an
# rclone destination that cannot stream uploads needs about twice that) and # rclone destination that cannot stream uploads needs about twice that) and
@@ -352,12 +356,14 @@ storage_url: "rclone://myremote/path/to/backups"
# Default: 3 # Default: 3
compression_level: 5 compression_level: 5
# How old a blob that no snapshot lists must be before prune deletes it, and # How old a blob that no snapshot lists must be before vaultik prune deletes
# how old every object in a directory under metadata/ with no manifest must # it. Each snapshot has a directory under metadata/ at the destination, and
# be before prune deletes that directory # its manifest there lists the blobs it uses. This is also how old every
# A backup uploads its blobs and its db.zst.age before its manifest, so on a # object in a directory with no manifest must be before vaultik prune deletes
# destination shared between hosts or users this must be longer than the # that directory
# longest backup run # A backup uploads its blobs and its encrypted database (db.zst.age) before
# its manifest, so on a destination shared between hosts or users this must
# be longer than the longest backup run
# It does not protect a blob a backup reuses from an earlier run that stopped # It does not protect a blob a backup reuses from an earlier run that stopped
# before its manifest: that blob keeps the time of its first upload, so a # before its manifest: that blob keeps the time of its first upload, so a
# prune can delete it # prune can delete it
docs/DATAMODEL.md
+51 -28
View File
@@ -1,7 +1,8 @@
# vaultik data model # vaultik data model
`vaultik` keeps a SQLite database on the backed-up machine, called the local `vaultik` keeps a SQLite database on the backed-up machine, called the local
index. This document describes its tables and how each command uses them. index. This document describes its tables, and how a backup, a restore and
`prune` use them.
The local index has two jobs: The local index has two jobs:
@@ -27,10 +28,11 @@ with the snapshot.
## Where the index is ## Where the index is
The index is the file `index.sqlite` in the platform data directory. That is The index is the file `index.sqlite` in the platform data directory. By default
`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/` that is `~/.local/share/vaultik/` on Linux and
on macOS. The `index_path` setting and the `VAULTIK_INDEX_PATH` environment `~/Library/Application Support/vaultik/` on macOS. The `index_path` setting and
variable change the path. the `VAULTIK_INDEX_PATH` environment variable change the path. So does
`XDG_DATA_HOME`.
The file is readable only by its owner, because it lists every backed-up path. The file is readable only by its owner, because it lists every backed-up path.
`vaultik` sets that mode each time it opens the index. `vaultik` sets that mode each time it opens the index.
@@ -74,15 +76,15 @@ One row for each backed-up path: a regular file, a symlink or a directory.
- `id`: a UUID. Primary key. - `id`: a UUID. Primary key.
- `path`: the absolute path. Unique. - `path`: the absolute path. Unique.
- `source_path`: the configured path that the scan found this entry under. A - `source_path`: the configured path under which the scan found this entry,
restore does not read it. made absolute with symlinks resolved. A restore does not read it.
- `mtime`: the modification time. - `mtime`: the modification time.
- `mtime_nsec`: the nanoseconds within that second. - `mtime_nsec`: the nanoseconds within that second.
- `size`: the size in bytes. - `size`: the size in bytes.
- `mode`: the permission bits and the file type. - `mode`: the permission bits and the file type.
- `uid`: the owner's user ID. - `uid`: the owner's user ID.
- `gid`: the owner's group ID. - `gid`: the owner's group ID.
- `link_target`: the target of a symlink. It is `NULL` for other entries. - `link_target`: the target of a symlink. It is empty for other entries.
`idx_files_path` indexes `path`. `idx_files_path` indexes `path`.
@@ -161,7 +163,8 @@ A restore finds the blob for each chunk through this table.
One row for each snapshot. One row for each snapshot.
- `id`: the snapshot ID. Primary key. - `id`: the snapshot ID. Primary key.
- `hostname`: the hostname of the machine that made the backup. - `hostname`: the `hostname` setting of the config. When that setting is empty,
it is the machine's hostname.
- `vaultik_version`: the `vaultik` version that made it. - `vaultik_version`: the `vaultik` version that made it.
- `vaultik_git_revision`: the git commit of that version. - `vaultik_git_revision`: the git commit of that version.
- `started_at`: when the backup started. - `started_at`: when the backup started.
@@ -189,8 +192,8 @@ example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
- the snapshot name - the snapshot name
- the start time, in UTC - the start time, in UTC
`snapshot list` shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE` and The `snapshot list` table shows `uncompressed_chunk_size` as
`new_chunk_size` as `NEW CHUNK SIZE`. `UNCOMPRESSED SIZE` and `new_chunk_size` as `NEW CHUNK SIZE`.
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list` A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
does not show it. The next backup deletes it, and so does `prune`. does not show it. The next backup deletes it, and so does `prune`.
@@ -233,10 +236,10 @@ A directory is recorded here when all of these hold:
- The backup found an entry in it that is not excluded. - The backup found an entry in it that is not excluded.
- It is on a different filesystem from its parent directory. - It is on a different filesystem from its parent directory.
The device number is read after the directory has been listed. Listing an The device number is read after the directory has been listed. An automount
automount point mounts its volume, so the number is that of the volume. An point is a directory whose volume is mounted when it is first listed, so the
automount point whose volume cannot be mounted is not recorded, because the number is that of the volume. An automount point whose volume cannot be mounted
backup finds no entry in it. is not recorded, because the backup finds no entry in it.
`snapshot create --prune` reads this table. It skips the prune when an earlier `snapshot create --prune` reads this table. It skips the prune when an earlier
snapshot of the same name recorded a mount point that the new backup did not snapshot of the same name recorded a mount point that the new backup did not
@@ -266,8 +269,14 @@ Settings that belong to this copy of the index.
- `value`: its value. - `value`: its value.
The only key is `storage_url`. It holds the destination that the index was The only key is `storage_url`. It holds the destination that the index was
first used with. A command that changes data refuses to run when the config first used with. These commands refuse to run when the config names a different
names a different destination. destination:
- `snapshot create`
- `snapshot purge`
- `snapshot remove`
- `prune`
- `remote nuke`
### `schema_migrations` ### `schema_migrations`
@@ -282,13 +291,16 @@ The migrations that have run.
A backup works on the tables in this order: A backup works on the tables in this order:
- It saves the destination in `local_meta` when the index has none yet.
- It deletes unfinished snapshots, and the rows that no snapshot uses. - It deletes unfinished snapshots, and the rows that no snapshot uses.
- It creates a row in `snapshots`, with `completed_at` set to `NULL`. - It creates a row in `snapshots`, with `completed_at` set to `NULL`.
- It deletes the `blobs` rows that were never uploaded. - It deletes the `blobs` rows that were never uploaded.
- It walks each path and compares every entry with its row in `files`. - It walks each path. It compares each regular file and symlink with its row in
`files`.
- It links each unchanged file to the snapshot in `snapshot_files`. - It links each unchanged file to the snapshot in `snapshot_files`.
- It records each mount point in `snapshot_mount_points`. - It records each mount point in `snapshot_mount_points`.
- It reads each changed file and stores it. See below. - It reads each changed regular file and stores it. See below. A changed symlink
is recorded without being read.
- It fills `snapshot_blobs` with every blob the snapshot uses. - It fills `snapshot_blobs` with every blob the snapshot uses.
- It stores the snapshot's totals in `snapshots`. - It stores the snapshot's totals in `snapshots`.
- It uploads the snapshot's metadata. - It uploads the snapshot's metadata.
@@ -303,7 +315,8 @@ in any of these:
- owner or group - owner or group
- link target - link target
A file is also read again when it lists a chunk that no uploaded blob holds. A file is also read again when it lists a chunk that no uploaded blob holds. A
file with no row in `files` is read too.
Storing a changed file writes these rows: Storing a changed file writes these rows:
@@ -316,8 +329,10 @@ Storing a changed file writes these rows:
- `uploaded_ts` and `compressed_size` in `blobs` - `uploaded_ts` and `compressed_size` in `blobs`
- a row in `snapshot_blobs` - a row in `snapshot_blobs`
- a row in `uploads`, unless the blob was already at the destination - a row in `uploads`, unless the blob was already at the destination
- The row in `files`, with its `file_chunks`, `chunk_files` and - The row in `files`, and its rows in these tables:
`snapshot_files` rows. - `file_chunks`
- `chunk_files`
- `snapshot_files`
The row in `files` is written last. It is written only after every blob that The row in `files` is written last. It is written only after every blob that
holds the file's new chunks is stored. holds the file's new chunks is stored.
@@ -334,31 +349,38 @@ The metadata of a snapshot is made from a copy of the index:
uploaded database therefore names the destination, and lists the migrations uploaded database therefore names the destination, and lists the migrations
that have run. that have run.
- `VACUUM` rewrites the copy, so that no deleted row is left in the file. - `VACUUM` rewrites the copy, so that no deleted row is left in the file.
- The copy is compressed with zstd and encrypted with age. - The copy is compressed with [zstd](https://github.com/facebook/zstd) and
encrypted with [age](https://age-encryption.org/).
- The manifest is built from the copy's `snapshot_blobs` table. - The manifest is built from the copy's `snapshot_blobs` table.
- The copy is uploaded as `metadata/<remote-key>/db.zst.age`. - The copy is uploaded as `metadata/<remote-key>/db.zst.age`.
- The manifest is uploaded as `metadata/<remote-key>/manifest.json.zst`. - The manifest is uploaded as `metadata/<remote-key>/manifest.json.zst`.
The copy is made through SQLite, not by copying the file. The index runs in WAL The copy is made through SQLite, not by copying the file. The index runs in
mode, so rows that are already committed can still be in the `-wal` file beside SQLite's write-ahead log (WAL) mode, so rows that are already committed can
it. still be in the `-wal` file beside it.
### Restore ### Restore
A restore does not use the local index. A restore reads no file, chunk or blob rows from the local index. Given a
remote key, it reads `snapshots` and `local_meta` there to check the destination
marker.
- It downloads and decrypts `db.zst.age` into a temporary directory. - It downloads and decrypts `db.zst.age` into a temporary directory.
- It opens that file read-only. It applies no migration to it. - It opens that file read-only. It applies no migration to it.
- It reads the snapshot's row in `snapshots`, to check that this is the right
database.
- It reads the files to restore from `files`. - It reads the files to restore from `files`.
- It finds each file's chunks in `file_chunks`. - It finds each file's chunks in `file_chunks`.
- It finds the blob and offset of each chunk in `blob_chunks`. - It finds the blob and offset of each chunk in `blob_chunks`.
- It finds the name of each blob in `blobs`. - It finds the name of each blob in `blobs`.
- With `--verify`, it reads each chunk's size from `chunks`.
It does not read the manifest. It does not read the manifest.
### Prune ### Prune
`prune` changes the local index in two ways: `prune` first saves the destination in `local_meta` when the index has none yet.
It then changes the local index in two ways:
- It deletes each snapshot whose manifest is gone from the destination. - It deletes each snapshot whose manifest is gone from the destination.
- It deletes unfinished snapshots, and the rows that no snapshot uses. - It deletes unfinished snapshots, and the rows that no snapshot uses.
@@ -389,6 +411,7 @@ Each repository type in `internal/database` serves one or more tables:
- `snapshot_files` - `snapshot_files`
- `snapshot_blobs` - `snapshot_blobs`
- `snapshot_mount_points` - `snapshot_mount_points`
- `uploads`, where it deletes a snapshot's rows
- `UploadRepository`: `uploads` - `UploadRepository`: `uploads`
- `LocalMetaRepository`: `local_meta` - `LocalMetaRepository`: `local_meta`
docs/REPOSTRUCTURE.md
+48 -31
View File
@@ -4,8 +4,8 @@ This document describes what `vaultik` stores at the destination. It covers the
layout, what the layout protects, and the risks it accepts. layout, what the layout protects, and the risks it accepts.
The destination is the place the backups are stored. It is an S3 bucket and The destination is the place the backups are stored. It is an S3 bucket and
prefix, a local directory or an rclone remote. The layout is the same on all prefix, a local directory or a remote of [rclone](https://rclone.org/). The
three. layout is the same on all three.
## Terms ## Terms
@@ -16,7 +16,8 @@ three.
content. content.
- **Blob**: one object under `blobs/`. It holds many chunks. - **Blob**: one object under `blobs/`. It holds many chunks.
- **Local index**: the SQLite database on the backed-up machine. - **Local index**: the SQLite database on the backed-up machine.
- **Recipient**: an age public key that backups are encrypted to. - **Recipient**: an [age](https://age-encryption.org/) public key that backups
are encrypted to.
## Layout ## Layout
@@ -49,12 +50,15 @@ unplugged volume leaves an empty directory at its mount point. Without the
marker, that directory would look like a destination whose snapshots are all marker, that directory would look like a destination whose snapshots are all
gone. gone.
The check applies while the local index holds snapshots. When the destination The check applies while the local index holds snapshots and belongs to the
then answers that the marker does not exist, commands treat the destination as configured destination. When the destination then answers that the marker does
not present: not exist, commands treat the destination as not present:
- `snapshot create` refuses to write to it. - `snapshot create` refuses to write to it.
- `prune`, `snapshot purge` and `remote nuke` fail. - These commands fail:
- `prune`
- `snapshot purge`
- `remote nuke`
- `snapshot list` shows the local index alone. - `snapshot list` shows the local index alone.
- `snapshot remove` deletes nothing at the destination. - `snapshot remove` deletes nothing at the destination.
@@ -73,8 +77,10 @@ A blob holds the content of many chunks, one after another.
- **Path**: `blobs/<first 2 characters>/<next 2 characters>/<full hash>`. - **Path**: `blobs/<first 2 characters>/<next 2 characters>/<full hash>`.
- **Example**: - **Example**:
`blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`. `blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`.
- **Format**: the chunk data, compressed with zstd and then encrypted with age. - **Format**: the chunk data, compressed with
- **Recipients**: the public keys in `age_recipients` when the blob was written. [zstd](https://github.com/facebook/zstd) and then encrypted with age.
- **Recipients**: the public keys in the `age_recipients` setting when the blob
was written.
The two directory levels keep any one directory from holding too many objects. The two directory levels keep any one directory from holding too many objects.
@@ -257,9 +263,9 @@ therefore does not stop overwrites.
An object's size follows exactly from the size of its compressed contents. An object's size follows exactly from the size of its compressed contents.
All the new chunks in one blob share one zstd stream. The stream looks back 8 All the new chunks in one blob share one zstd stream. The stream looks back 8
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at the
`blob_size_limit` and at the end of each configured path. A chunk that is `blob_size_limit` setting and at the end of each configured path. A chunk that
already stored is never packed again. the local index records as uploaded is never packed again.
Data that an outsider writes compresses smaller when it matches a secret near it Data that an outsider writes compresses smaller when it matches a secret near it
in the same blob, so the blob gets smaller. The blob's size therefore shows the in the same blob, so the blob gets smaller. The blob's size therefore shows the
@@ -277,14 +283,15 @@ Back up a directory that outsiders can write to as a snapshot of its own.
### Chunk boundaries use no secret ### Chunk boundaries use no secret
The FastCDC parameters are fixed and public. `chunk_size` sets the average chunk `vaultik` finds chunk boundaries with the FastCDC algorithm. Its parameters are
size. No chunk is larger than four times this size. Only a file's last chunk can fixed and public. `chunk_size` sets the average chunk size. No chunk is larger
be smaller than a quarter of it. With the default of 10 MB, those limits are 40 than four times this size. Only a file's last chunk can be smaller than a
MB and 2.5 MB. A file of 2.5 MB or less is one chunk. quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB. A
file of 2.5 MB or less is one chunk. An empty file has no chunk.
At the default `blob_size_limit` of 10 GB, a blob holds hundreds of chunks. The At the default `blob_size_limit` of 10 GB, a full blob holds hundreds of
size of the blob then does not show the size of any one chunk. A limit close to chunks. The size of the blob then does not show the size of any one chunk. A
`chunk_size` begins to show chunk sizes. limit close to `chunk_size` begins to show chunk sizes.
### Decrypted data on the local disk ### Decrypted data on the local disk
@@ -307,14 +314,24 @@ Trust the temporary directory as much as the directory you restore into.
### Commands need different permissions ### Commands need different permissions
`snapshot create` needs only permission to write objects. Other commands need `snapshot create` without `--prune` needs only permission to write objects.
more: Other commands need more:
- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list - These commands list and read:
and read. - `snapshot list`
- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete. - `snapshot verify`
- `snapshot restore`
- `remote info`
- These commands list and delete:
- `snapshot purge`
- `snapshot remove`
- `remote nuke`
- `prune` lists, reads and deletes. - `prune` lists, reads and deletes.
`snapshot verify` and `snapshot restore` list only when they are given a remote
key. `snapshot remove --local-only` does not reach the destination.
`snapshot remove --dry-run` deletes nothing.
`snapshot create --prune` runs a prune on the backed-up machine. That machine `snapshot create --prune` runs a prune on the backed-up machine. That machine
then holds credentials that can delete its own backups. To keep it to then holds credentials that can delete its own backups. To keep it to
write-only credentials, run `prune` from a separate, trusted machine. write-only credentials, run `prune` from a separate, trusted machine.
@@ -322,8 +339,8 @@ write-only credentials, run `prune` from a separate, trusted machine.
### New recipients cannot read old data ### New recipients cannot read old data
Changing `age_recipients` does not re-encrypt what is already stored. A chunk Changing `age_recipients` does not re-encrypt what is already stored. A chunk
that is already stored is not written again, so it stays encrypted to the that the local index records as uploaded is not written again, so it stays
earlier recipients. encrypted to the earlier recipients.
A new snapshot that uses such chunks cannot be restored with a new recipient's A new snapshot that uses such chunks cannot be restored with a new recipient's
key alone. To make everything readable by a new key, run key alone. To make everything readable by a new key, run
@@ -335,8 +352,8 @@ key alone. To make everything readable by a new key, run
recipients. recipients.
The encrypted data is held by a third party for a long time. There is no second The encrypted data is held by a third party for a long time. There is no second
kind of recipient to fall back on if X25519 is broken, and no post-quantum kind of recipient to fall back on if X25519 is broken, and no kind that resists
option. a quantum computer.
## Order of writes ## Order of writes
@@ -353,12 +370,12 @@ therefore belongs to a backup that did not finish.
The local index marks a snapshot complete only after both metadata files are The local index marks a snapshot complete only after both metadata files are
uploaded. A backup that stops during that upload leaves the snapshot uploaded. A backup that stops during that upload leaves the snapshot
unfinished. The next backup removes it from the local index and stores its data unfinished. The next backup removes it from the local index and backs up its
again. files again.
One short gap remains. A backup can stop after the upload and before the One short gap remains. A backup can stop after the upload and before the
snapshot is marked complete. The destination then holds a snapshot that can be snapshot is marked complete. The destination then holds a snapshot that can be
restored, and the local index does not list it. `snapshot list` shows it as restored, and the local index marks it unfinished. `snapshot list` shows it as
remote only. remote only.
An upload that is cut off leaves no partial object under the final name on S3 An upload that is cut off leaves no partial object under the final name on S3
internal/cli/snapshot.go
+2 -1
View File
@@ -361,7 +361,8 @@ nuke --force' — it is the single supported entry point for that.`,
cmd.Flags().BoolVarP(&opts.Force, "force", "f", false, "Skip confirmation prompt") cmd.Flags().BoolVarP(&opts.Force, "force", "f", false, "Skip confirmation prompt")
cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false, cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false,
"Show what would be removed without removing") "Show what would be removed without removing; needs no permission "+
"to delete")
cmd.Flags().BoolVar(&opts.JSON, "json", false, cmd.Flags().BoolVar(&opts.JSON, "json", false,
"Output result as JSON; skips the confirmation prompt, as --force does") "Output result as JSON; skips the confirmation prompt, as --force does")
cmd.Flags().BoolVar(&opts.LocalOnly, "local-only", false, cmd.Flags().BoolVar(&opts.LocalOnly, "local-only", false,