sneak
de9ef62073
check / check (push) Waiting to run
Rewritten from scratch against the code on next: what the program is, how to use it, a command reference that says what each command needs, the configuration reference as nested bullets, then development and releasing. snapshot verify --deep now says it needs the private key, from VAULTIK_AGE_SECRET_KEY or age_secret_key, as snapshot restore does. The roadmap section is removed: it described features that are not built. Model: fable-5-1
1357 lines
50 KiB
Markdown
1357 lines
50 KiB
Markdown
# vaultik (ваултик)
|
|
|
|
`vaultik` is an incremental backup tool. It encrypts your files with a public
|
|
key and uploads them to an S3-compatible object store, a local directory or an
|
|
rclone remote.
|
|
|
|
The machine being backed up holds only the public key. It cannot decrypt its own
|
|
backups. You keep the private key somewhere else and need it only to restore.
|
|
|
|
## Why
|
|
|
|
Other backup tools, such as `restic`, `borg` and `duplicity`, are designed for
|
|
machines that can hold a decryption key. `vaultik` is for machines that should
|
|
not hold one.
|
|
|
|
It was written to meet these requirements together:
|
|
|
|
- open source
|
|
- no passphrase or private key on the backed-up machine
|
|
- incremental
|
|
- compressed
|
|
- encrypted
|
|
- direct upload to S3-compatible storage, with no intermediate step or tool
|
|
|
|
## Features
|
|
|
|
- Encryption with [age](https://age-encryption.org/) (X25519 and
|
|
ChaCha20-Poly1305), to one or more public keys.
|
|
- Incremental backups. A file is read again only when it has changed.
|
|
- Deduplication. Identical content is stored once, across files and across
|
|
snapshots.
|
|
- zstd compression at a level you choose.
|
|
- Small files are packed together into large objects. This keeps the number of
|
|
S3 requests low.
|
|
- File contents, names, paths and metadata are encrypted at the destination.
|
|
- Three kinds of destination: S3, a local directory and any rclone remote.
|
|
- Regular files, symlinks and directories are backed up, with their permissions,
|
|
owners and modification times.
|
|
- Pure Go, with no C code. It builds for Linux and macOS on `amd64` and `arm64`.
|
|
|
|
## Terms
|
|
|
|
- **Snapshot**: one backup of the paths listed under one name in the config,
|
|
taken at one time.
|
|
- **Snapshot ID**: the name of a snapshot, in the form
|
|
`<hostname>_<snapshot-name>_<timestamp>`. An example is
|
|
`server1_home_2025-06-01T12:00:00Z`.
|
|
- **Destination**: the place the backups are stored.
|
|
- **Chunk**: a piece of a file's content. `vaultik` cuts each file into chunks
|
|
and stores each distinct chunk once.
|
|
- **Blob**: one object at the destination. It holds many chunks, compressed and
|
|
then encrypted.
|
|
- **Local index**: a SQLite database on the backed-up machine. It records what
|
|
is already stored, so the next backup uploads only what changed.
|
|
- **Manifest**: the list of blobs one snapshot uses. It is stored at the
|
|
destination without encryption.
|
|
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
|
|
at the destination, so the destination never shows the snapshot ID.
|
|
|
|
## Install
|
|
|
|
```sh
|
|
go install sneak.berlin/go/vaultik/cmd/vaultik@latest
|
|
```
|
|
|
|
This needs Go 1.26.1 or later. Generating a key pair needs `age-keygen`, which
|
|
comes with [age](https://age-encryption.org/).
|
|
|
|
## Quick start
|
|
|
|
```sh
|
|
# Write a default config file. The command prints the path it wrote.
|
|
vaultik config init
|
|
|
|
# Generate a key pair. Keep this file somewhere safe, away from this machine.
|
|
age-keygen -o vaultik_backup_private_key.txt
|
|
grep 'public key' vaultik_backup_private_key.txt
|
|
|
|
# Set the public key and the destination.
|
|
vaultik config set age_recipients.0 age1YOUR_PUBLIC_KEY_HERE
|
|
vaultik config set storage_url "file:///Volumes/usbstick/mybackup"
|
|
|
|
# Create the destination directory. vaultik never creates it.
|
|
mkdir /Volumes/usbstick/mybackup
|
|
|
|
# Run the first backup.
|
|
vaultik snapshot create
|
|
|
|
# See what is stored.
|
|
vaultik snapshot list
|
|
```
|
|
|
|
The default config is written for macOS. It backs up your home directory and
|
|
`/Applications`, with a list of excludes. A configured path that does not exist
|
|
stops the backup, so on another system run `vaultik config edit` first and
|
|
change the `snapshots` section.
|
|
|
|
On macOS, grant your terminal app Full Disk Access before the first backup. The
|
|
setting is under System Settings, Privacy & Security, Full Disk Access. Without
|
|
it the backup stops with a permission error on a protected directory.
|
|
|
|
## Everyday use
|
|
|
|
```sh
|
|
# Check that a snapshot's blobs are all present, with the listed sizes.
|
|
vaultik snapshot verify <snapshot-id>
|
|
|
|
# Put the private key in the environment. Reading it from the file keeps it
|
|
# out of your shell history.
|
|
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
|
|
|
|
# Download, decrypt and re-hash every blob of a snapshot.
|
|
vaultik snapshot verify --deep <snapshot-id>
|
|
|
|
# Restore a snapshot into a new directory that only you can write.
|
|
vaultik snapshot restore <snapshot-id> ~/vaultik-restore
|
|
|
|
# A daily cron line: back up, then keep four weeks of snapshots.
|
|
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
|
|
```
|
|
|
|
`--prune` deletes data at the destination. Read
|
|
[Sharing a destination](#sharing-a-destination) before you use it on a
|
|
destination that more than one machine or user writes to.
|
|
|
|
## Restoring on another machine
|
|
|
|
You can restore on a machine that never ran the backup, such as a replacement
|
|
for a lost one. That machine needs three things:
|
|
|
|
- the `vaultik` binary
|
|
- the age private key
|
|
- the destination, and credentials that can list and read it
|
|
|
|
It does not need the original local index, config file or hostname.
|
|
|
|
```sh
|
|
# Create a config and point it at the original destination.
|
|
vaultik config init
|
|
vaultik config set storage_url "s3://bucket/prefix?endpoint=https://s3.example.com"
|
|
vaultik config set s3.access_key_id "..."
|
|
vaultik config set s3.secret_access_key "..."
|
|
|
|
# See what the destination holds. This needs no private key.
|
|
vaultik snapshot list
|
|
```
|
|
|
|
On this machine the local index is empty, so the list cannot show snapshot IDs.
|
|
Each snapshot appears as `<remote only:...>`, with the first 12 characters of
|
|
its remote key. The snapshot ID is stored only in the local index and in the
|
|
snapshot's encrypted database.
|
|
|
|
Use that remote key in place of the snapshot ID:
|
|
|
|
```sh
|
|
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
|
|
|
|
# Restore everything, then check the chunk hashes of every restored file.
|
|
vaultik snapshot restore --verify <remote-key> ~/vaultik-restore
|
|
|
|
# Optionally, also check every blob of the snapshot at the destination.
|
|
vaultik snapshot verify --deep <remote-key>
|
|
```
|
|
|
|
Any leading part of the remote key works, as long as it matches one snapshot.
|
|
`snapshot list --json` gives the full 64-character key as `remote_key`.
|
|
|
|
A machine that only restores does not need `age_recipients`.
|
|
|
|
## Commands
|
|
|
|
```sh
|
|
vaultik config init
|
|
vaultik config edit
|
|
vaultik config get <key>
|
|
vaultik config set <key> <value>
|
|
vaultik snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>]
|
|
vaultik snapshot list [--json]
|
|
vaultik snapshot verify <snapshot-id> [--deep] [--json]
|
|
vaultik snapshot purge (--keep-latest | --older-than <duration>) [--snapshot <name>...] [--force]
|
|
vaultik snapshot remove <snapshot-id> [--dry-run] [--force] [--local-only] [--json]
|
|
vaultik snapshot restore <snapshot-id> <target-dir> [paths...] [--verify]
|
|
vaultik prune [--force] [--json]
|
|
vaultik info
|
|
vaultik remote info [--json]
|
|
vaultik remote nuke --force
|
|
vaultik database delete [--force]
|
|
vaultik completion <bash|zsh|fish|powershell>
|
|
vaultik version
|
|
```
|
|
|
|
`snapshot list` has the alias `snapshot ls`. `snapshot remove` has the alias
|
|
`snapshot rm`.
|
|
|
|
### What every command needs
|
|
|
|
`config init`, `completion` and `version` need nothing.
|
|
|
|
`config edit`, `config get` and `config set` need a config file that exists. It
|
|
does not have to be complete.
|
|
|
|
Every other command needs a config file that loads. See
|
|
[Configuration](#configuration) for the checks.
|
|
|
|
Every other command except `database delete` also needs these:
|
|
|
|
- Write access to the local index. The command creates the file, and its
|
|
directory, when they do not exist.
|
|
- For an `rclone://` destination, an rclone config file that defines the remote.
|
|
|
|
Each command below lists what it needs beyond this. No command needs root, but a
|
|
backup can read only what its user can read, and a restore sets file owners only
|
|
as root.
|
|
|
|
### Global flags
|
|
|
|
- `--config <path>`: the config file to use. See
|
|
[Where the config file is](#where-the-config-file-is).
|
|
- `--verbose`, `-v`: write informational log records to stderr.
|
|
- `--debug`: write debug log records to stderr.
|
|
- `--quiet`, `-q`: print only warnings, errors and the output a script reads.
|
|
See [Output](#output).
|
|
- `--skip-errors`: skip a file that cannot be read during a backup, or cannot be
|
|
restored during a restore. Without it the command stops at the first such
|
|
file.
|
|
|
|
`--skip-errors` applies to `snapshot create` and `snapshot restore` only. A
|
|
command that skipped anything still finishes, and then exits non-zero.
|
|
|
|
### Durations
|
|
|
|
`--keep-newer-than`, `--older-than` and `prune_grace_period` take a duration.
|
|
|
|
- `d`: days.
|
|
- `w`: weeks.
|
|
- `mo`: months of 30 days.
|
|
- `y`: years of 365 days.
|
|
- `h`, `m` and `s`: hours, minutes and seconds, as in `36h` or `90m`.
|
|
|
|
`m` means minutes and `mo` means months. Days, weeks, months and years take
|
|
whole numbers and can be combined, as in `2w3d`.
|
|
|
|
`prune_grace_period` accepts only hours, minutes and seconds.
|
|
|
|
### `config init`
|
|
|
|
Writes a default config file, with a comment that explains each setting it
|
|
lists.
|
|
|
|
- It writes to the path from `--config`, or else from `VAULTIK_CONFIG`.
|
|
- Otherwise it writes to `config.yml` in the platform config directory. That is
|
|
`~/.config/vaultik/` on Linux and `~/Library/Application Support/vaultik/` on
|
|
macOS.
|
|
- As root it writes to `/etc/vaultik/config.yml` instead.
|
|
- It creates the file with mode `0600`, because the file can hold credentials.
|
|
- It refuses to overwrite a file that exists.
|
|
|
|
### `config edit`
|
|
|
|
Opens the config file in `$EDITOR`. It uses `vi` when `EDITOR` is not set.
|
|
|
|
### `config get <key>`
|
|
|
|
Prints one value from the config file. The key is a dotted path, as in
|
|
`vaultik config get storage_url` or `vaultik config get age_recipients.0`. A
|
|
list or a map prints as YAML.
|
|
|
|
### `config set <key> <value>`
|
|
|
|
Sets one value in the config file. The key is a dotted path, as in
|
|
`vaultik config set compression_level 9`.
|
|
|
|
- Comments and formatting in the file are kept.
|
|
- A key that does not exist yet is created. A missing parent becomes a list when
|
|
the next part of the path is a number, and a map otherwise.
|
|
- In a list, the index one past the end appends an entry.
|
|
- The command prints the key it set, and never the value.
|
|
- It sets the file's mode to `0600` when group or others could read it.
|
|
|
|
The command refuses a change that would stop the config from loading. An example
|
|
is `vaultik config set compression_level 99`. The file is then left unchanged.
|
|
|
|
A config that does not load yet is checked less strictly. An example is the file
|
|
`config init` writes, which has no `storage_url`. There the command refuses only
|
|
a value of the wrong type, such as a word where a number belongs.
|
|
|
|
### `snapshot create`
|
|
|
|
Backs up the configured snapshots. With no names it backs up all of them. Each
|
|
name gets its own snapshot.
|
|
|
|
It needs:
|
|
|
|
- At least one public key in `age_recipients`.
|
|
- Read access to the configured paths.
|
|
- Permission to write objects at the destination. See
|
|
[Destination permissions](#destination-permissions).
|
|
- Free temporary space. See [Temporary files](#temporary-files).
|
|
- The [lock](#locking).
|
|
|
|
It does not need the private key.
|
|
|
|
A file is read again when any of these has changed since the last backup:
|
|
|
|
- its size
|
|
- its modification time
|
|
- its mode
|
|
- its owner or group
|
|
- its link target
|
|
|
|
Every other file is recorded in the new snapshot without being read.
|
|
|
|
Flags:
|
|
|
|
- `--cron`: print nothing when the backup succeeds. Warnings and errors are
|
|
still printed.
|
|
- `--prune`: delete old snapshots and unused data after the backup. See below.
|
|
- `--keep-newer-than <duration>`: with `--prune`, keep every snapshot newer than
|
|
this.
|
|
|
|
The backup stops before it writes anything in these cases:
|
|
|
|
- A `file://` destination directory does not exist.
|
|
- The destination has no marker file, although the local index holds snapshots.
|
|
See [The marker file](#the-marker-file).
|
|
- The destination in the config is not the one the local index was used with.
|
|
See [Changing the destination](#changing-the-destination).
|
|
|
|
The backup also stops when a configured path is a symlink whose target does not
|
|
exist. It stops when a configured path does not exist, unless `--skip-errors` is
|
|
given.
|
|
|
|
With `--skip-errors`, a file or directory that cannot be read is skipped. The
|
|
snapshot is kept and counts each skipped entry as failed, and the command exits
|
|
non-zero. An error while packing or uploading still stops the backup.
|
|
|
|
A file that is removed during the backup is counted as failed. It does not
|
|
change the exit status.
|
|
|
|
#### `--prune`
|
|
|
|
After a successful backup, `--prune` does two things:
|
|
|
|
- It removes the older snapshots of each name it just backed up. It keeps only
|
|
the newest one, or with `--keep-newer-than` every snapshot newer than that.
|
|
- It then deletes at the destination what [`prune`](#prune) deletes.
|
|
|
|
`--prune` also needs permission to list, read and delete at the destination.
|
|
|
|
`--keep-newer-than` without `--prune` is an error. So is a value that is not a
|
|
duration. Both are reported before the backup starts.
|
|
|
|
The prune does not run after a backup that may have missed data. An older
|
|
snapshot may then hold the only copy of what was missed. This applies when:
|
|
|
|
- `--skip-errors` skipped a file or directory that could not be read.
|
|
- A configured path held nothing, as the mount point of an unmounted volume
|
|
does.
|
|
- An earlier snapshot of the same name found a mount point that this backup did
|
|
not find.
|
|
|
|
In each case every older snapshot is kept. The command says why, names the path
|
|
and exits non-zero. It does so on every backup until the cause is gone.
|
|
|
|
A mount point here is a directory at or below a configured path that is on a
|
|
different filesystem from its parent. It counts only when the backup finds an
|
|
entry in it that is not excluded. A backup does not find it when the volume is
|
|
not mounted, or when the directory is now excluded or no longer configured.
|
|
|
|
If the cause is intended, such as a retired volume, remove the older snapshots
|
|
yourself with `snapshot purge`. Take a path that is empty on purpose out of the
|
|
config.
|
|
|
|
### `snapshot list`
|
|
|
|
Lists the snapshots in the local index and the snapshots at the destination, as
|
|
one table. Before it, the command prints the snapshot names and paths in the
|
|
config.
|
|
|
|
It needs permission to list and read at the destination. It needs no private
|
|
key.
|
|
|
|
The table has these columns:
|
|
|
|
- `SNAPSHOT ID`
|
|
- `TIMESTAMP`: in UTC.
|
|
- `COMPRESSED SIZE`: the total size of the blobs the snapshot uses.
|
|
- `UNCOMPRESSED SIZE`: the total size of its distinct chunks before compression.
|
|
- `NEW CHUNK SIZE`: the size of the chunks this snapshot stored that were not
|
|
stored before.
|
|
|
|
A snapshot at the destination that is not in the local index is shown as
|
|
`<remote only:...>`. The dots stand for the first 12 characters of its remote
|
|
key. Its last two columns read `<remote only>`, because those sizes come from
|
|
the local index. Its timestamp is the time its backup finished.
|
|
|
|
The command reports these below the table:
|
|
|
|
- Snapshots in the local index that are missing at the destination. Run
|
|
`vaultik prune` to drop those records.
|
|
- Remote-only snapshots whose manifest cannot be read. They are left out of the
|
|
table.
|
|
- Remote-only snapshots beyond the first 1000. They are left out of the table.
|
|
- A local index that was used with a different destination. See
|
|
[Changing the destination](#changing-the-destination).
|
|
|
|
When the destination cannot be listed, the command warns and shows the local
|
|
index alone. It still exits zero. It does the same when the destination has no
|
|
marker file although the local index holds snapshots.
|
|
|
|
`--json` prints a JSON array instead. Each entry has these fields:
|
|
|
|
- `id`: the snapshot ID. It is empty for a remote-only snapshot.
|
|
- `remote_key`: the full 64-character remote key.
|
|
- `timestamp`
|
|
- `compressed_size`
|
|
- `uncompressed_size` and `new_chunk_size`: left out for a remote-only snapshot.
|
|
- `locally_tracked`: whether the snapshot is in the local index.
|
|
- `remote_present`: whether the snapshot is at the destination. It is `null`
|
|
when the destination could not be listed.
|
|
|
|
Under `--json`, every warning above goes to stderr.
|
|
|
|
### `snapshot verify <snapshot-id>`
|
|
|
|
Checks a snapshot at the destination. The argument is a snapshot ID, a remote
|
|
key or the leading part of a remote key.
|
|
|
|
Without `--deep`, it checks two things:
|
|
|
|
- Every blob the manifest lists is present, with the size the manifest records.
|
|
- The snapshot's encrypted database is present.
|
|
|
|
This reads no blob contents and needs no private key. It needs permission to
|
|
read at the destination, and to list it when you give a remote key.
|
|
|
|
Flags:
|
|
|
|
- `--deep`: download, decrypt and re-hash every blob. See below.
|
|
- `--json`: print the result as JSON.
|
|
|
|
The command exits non-zero when a check fails or cannot be made.
|
|
|
|
#### `--deep`
|
|
|
|
`--deep` needs the age private key. Set it in the `VAULTIK_AGE_SECRET_KEY`
|
|
environment variable, or as `age_secret_key` in the config file. The environment
|
|
variable wins when both are set. See [The private key](#the-private-key).
|
|
|
|
`--deep` does these checks, in order:
|
|
|
|
- It downloads and decrypts the snapshot's database. It checks that this is the
|
|
database of the snapshot you named.
|
|
- It checks that the manifest and the database list the same blobs, with the
|
|
same sizes.
|
|
- It checks that every blob is present with the recorded size.
|
|
- It downloads and decrypts every blob. It checks the hash of each chunk, and
|
|
the hash that names the blob.
|
|
|
|
It stops at the first blob that fails.
|
|
|
|
`--deep` proves that the stored data is intact and decrypts with your key. It
|
|
does not prove who wrote the data.
|
|
|
|
`--deep` writes the decrypted database to a temporary directory and removes it
|
|
afterwards. See [Temporary files](#temporary-files).
|
|
|
|
### `snapshot purge`
|
|
|
|
Removes old snapshots. You must give exactly one of `--keep-latest` and
|
|
`--older-than`.
|
|
|
|
It needs the [lock](#locking), and permission to list and delete at the
|
|
destination. It needs no private key.
|
|
|
|
Flags:
|
|
|
|
- `--keep-latest`: keep only the newest snapshot of each name.
|
|
- `--older-than <duration>`: remove every snapshot older than this.
|
|
- `--snapshot <name>`: act only on snapshots with this name. Repeat the flag for
|
|
more names.
|
|
- `--force`: do not ask for confirmation.
|
|
|
|
The command acts only on snapshots in the local index. `--keep-latest` keeps the
|
|
newest snapshot of each name, not the newest overall.
|
|
|
|
A removed snapshot leaves the local index, and its directory under `metadata/`
|
|
at the destination is deleted. Its blobs stay. Run `vaultik prune` afterwards to
|
|
delete the blobs that no snapshot uses any more.
|
|
|
|
The command first drops local records of snapshots that are no longer at the
|
|
destination. It fails and removes nothing when the destination cannot be listed
|
|
or has no marker file.
|
|
|
|
### `snapshot remove <snapshot-id>`
|
|
|
|
Removes one snapshot. The argument must be a snapshot ID. A remote key is not
|
|
accepted.
|
|
|
|
It needs the [lock](#locking), and permission to list and delete at the
|
|
destination. It needs no private key.
|
|
|
|
Flags:
|
|
|
|
- `--force`, `-f`: do not ask for confirmation.
|
|
- `--dry-run`: say what would be removed, and remove nothing.
|
|
- `--local-only`: remove the snapshot from the local index only. The destination
|
|
is not changed.
|
|
- `--json`: print the result as JSON. This also skips the confirmation.
|
|
|
|
The command removes the snapshot from the local index. It also deletes the
|
|
snapshot's directory under `metadata/` at the destination. Its blobs stay. Run
|
|
`vaultik prune` afterwards to delete the blobs that no snapshot uses any more.
|
|
|
|
An ID that is in neither the local index nor the destination is an error. With
|
|
`--local-only`, the snapshot must be in the local index.
|
|
|
|
The deletion at the destination can fail, for example when the destination is
|
|
unreachable or has no marker file. What happens then depends on where the
|
|
snapshot was:
|
|
|
|
- A snapshot in the local index is still removed from it, with a warning. Run
|
|
the same command again later to delete its directory at the destination.
|
|
`vaultik prune` does not delete it.
|
|
- For a snapshot that is only at the destination, the failure is an error.
|
|
|
|
### `snapshot restore <snapshot-id> <target-dir> [paths...]`
|
|
|
|
Restores files from a snapshot into `<target-dir>`. The first argument is a
|
|
snapshot ID, a remote key or the leading part of a remote key.
|
|
|
|
It needs:
|
|
|
|
- The age private key, as for `snapshot verify --deep`. See
|
|
[The private key](#the-private-key).
|
|
- Permission to read at the destination, and to list it when you give a remote
|
|
key.
|
|
- Write access to `<target-dir>`. The command creates the directory when it does
|
|
not exist.
|
|
- Free temporary space for the decrypted blobs. See
|
|
[Temporary files](#temporary-files).
|
|
|
|
It does not need `age_recipients`, and it does not take the lock.
|
|
|
|
Each file is restored under its original absolute path inside `<target-dir>`.
|
|
For example, `/home/user/notes.txt` is restored to
|
|
`<target-dir>/home/user/notes.txt`.
|
|
|
|
With no paths, the command restores the whole snapshot. Each path names a file
|
|
or a directory by its original absolute path:
|
|
|
|
```sh
|
|
vaultik snapshot restore <snapshot-id> ~/vaultik-restore /home/user/documents
|
|
```
|
|
|
|
A path that matches nothing is named in a warning. The command restores the
|
|
other paths and then exits non-zero.
|
|
|
|
The command restores these:
|
|
|
|
- regular files, symlinks and directories, empty directories included
|
|
- permissions and modification times
|
|
- owners and groups, when it runs as root
|
|
|
|
Without root, restored files belong to the user who runs the command. The
|
|
command then warns once if any owner or group differs from the snapshot.
|
|
|
|
A file that already exists in `<target-dir>` is replaced. Running the same
|
|
restore again starts from the beginning.
|
|
|
|
Restore into a new directory that only you can write, not a shared one such as
|
|
`/tmp`. No other user can then change the files while they are written.
|
|
|
|
Flags:
|
|
|
|
- `--verify`: after the restore, read every restored file and check its chunk
|
|
hashes.
|
|
|
|
With `--skip-errors`, a file that cannot be restored is reported and skipped,
|
|
and the command exits non-zero. `--verify` does not check a skipped file.
|
|
|
|
### `prune`
|
|
|
|
Removes what is no longer needed, from the local index and from the destination.
|
|
|
|
It needs the [lock](#locking), and permission to list, read and delete at the
|
|
destination. It needs no private key.
|
|
|
|
It works in three steps:
|
|
|
|
- It drops each snapshot from the local index whose directory is gone from the
|
|
destination.
|
|
- It removes unfinished snapshots from the local index. It also removes the file,
|
|
chunk and blob records that no snapshot uses.
|
|
- It reads every manifest at the destination. It then deletes each blob that no
|
|
manifest lists and that is older than `prune_grace_period`.
|
|
|
|
The third step also deletes a directory under `metadata/` that has no manifest.
|
|
A backup leaves one when it stops before it uploads its manifest. Such a
|
|
directory is deleted only when all of these hold:
|
|
|
|
- Every object in it is older than `prune_grace_period`.
|
|
- Its snapshot is not in the local index.
|
|
- Its name has the form of a remote key.
|
|
|
|
`prune_grace_period` defaults to `24h`. A newer blob is kept, because a backup
|
|
that is still running may need it. See
|
|
[Sharing a destination](#sharing-a-destination) for what the grace period does
|
|
not protect.
|
|
|
|
The command deletes nothing at the destination in these cases:
|
|
|
|
- The destination cannot be listed.
|
|
- The destination has no marker file, although the local index holds snapshots.
|
|
- A manifest cannot be read.
|
|
|
|
It exits non-zero when a blob or a directory cannot be deleted.
|
|
|
|
The time to read the manifests grows with the number of snapshots.
|
|
|
|
Flags:
|
|
|
|
- `--force`: do not ask for confirmation before deleting at the destination.
|
|
- `--json`: print the result as JSON. This also skips the confirmation.
|
|
|
|
The JSON document has these fields:
|
|
|
|
- `blobs_found`: the unused blobs old enough to delete.
|
|
- `blobs_deleted`
|
|
- `blobs_failed`: blobs that could not be deleted. Left out when zero.
|
|
- `bytes_freed`
|
|
- `blobs_too_new`: unused blobs that were kept. They are newer than the grace
|
|
period, or the destination gave no time for them.
|
|
- `metadata_dirs_deleted`
|
|
- `metadata_dirs_failed`: left out when zero.
|
|
|
|
### `info`
|
|
|
|
Prints the system details, the settings in use and figures from the local index.
|
|
The figures are the size of the index and its counts of snapshots, blobs and
|
|
files.
|
|
|
|
### `remote info`
|
|
|
|
Reports what the destination holds:
|
|
|
|
- the kind of destination and its location
|
|
- the size of each snapshot's database and manifest
|
|
- the number and size of all blobs
|
|
- the number and size of the blobs that manifests list
|
|
- the number and size of the blobs that no manifest lists
|
|
|
|
It needs permission to list and read at the destination. It needs no private
|
|
key.
|
|
|
|
When a manifest cannot be read, the last figures are reported as unknown. That
|
|
snapshot's own blob count and size are unknown too.
|
|
|
|
The command skips a name under `metadata/` that is not a remote key, with a
|
|
warning. It does not count a file under `blobs/` whose name is not a blob hash.
|
|
|
|
`--json` prints the report as JSON. Unknown figures are `null`.
|
|
`unreadable_manifests` lists the remote key of each manifest that could not be
|
|
read. `skipped_manifest_count` counts the manifests under skipped names.
|
|
|
|
### `remote nuke --force`
|
|
|
|
Deletes everything `vaultik` stored at the destination. This cannot be undone.
|
|
|
|
It needs `--force`, the [lock](#locking), and permission to list and delete at
|
|
the destination.
|
|
|
|
It deletes these, in order:
|
|
|
|
- every object under `metadata/`
|
|
- every object under `blobs/`
|
|
- the marker file
|
|
|
|
It also removes each snapshot from the local index, once that snapshot's
|
|
directory at the destination is deleted.
|
|
|
|
The command fails before it deletes anything when the destination cannot be
|
|
listed, or has no marker file although the local index holds snapshots.
|
|
|
|
If objects remain afterwards, the command names each one and exits non-zero.
|
|
Running it again deletes what is left.
|
|
|
|
### `database delete`
|
|
|
|
Deletes the local index file. The destination is not changed.
|
|
|
|
It needs a config file that loads, because the config gives the path of the
|
|
index.
|
|
|
|
The next backup then reads every file again. It skips the upload of a blob that
|
|
the destination already holds under the same name.
|
|
|
|
Use this command after you point the config at a different destination. See
|
|
[Changing the destination](#changing-the-destination).
|
|
|
|
`--force` skips the confirmation.
|
|
|
|
### `completion <shell>`
|
|
|
|
Prints a completion script for `bash`, `zsh`, `fish` or `powershell`.
|
|
|
|
```sh
|
|
# zsh, for the current session
|
|
source <(vaultik completion zsh)
|
|
|
|
# zsh, permanently
|
|
vaultik completion zsh > "${fpath[1]}/_vaultik"
|
|
|
|
# bash, for the current session
|
|
source <(vaultik completion bash)
|
|
|
|
# bash, permanently, on Linux
|
|
vaultik completion bash > /etc/bash_completion.d/vaultik
|
|
|
|
# fish
|
|
vaultik completion fish > ~/.config/fish/completions/vaultik.fish
|
|
```
|
|
|
|
### `version`
|
|
|
|
Prints the version, the commit, the build date and the platform. It also says
|
|
when the binary is a development build, not a release.
|
|
|
|
## Configuration
|
|
|
|
The config file is YAML. `vaultik config init` writes one to start from, and
|
|
[`config.example.yml`](config.example.yml) is a longer example.
|
|
|
|
A command refuses to start when the config does not load. The config loads when
|
|
all of these hold:
|
|
|
|
- Every entry in `age_recipients` is an age X25519 public key.
|
|
- `snapshots` has at least one name, and every name has at least one path.
|
|
- A destination is set, with credentials when it is S3.
|
|
- Every other setting below is within its limits.
|
|
|
|
`vaultik` warns when group or others can read the config file.
|
|
|
|
A value written as `${ENV:NAME}` is replaced by the environment variable `NAME`.
|
|
This keeps a credential out of the file.
|
|
|
|
A leading `~` is expanded to your home directory in snapshot paths, in
|
|
`index_path` and in a `file://~/...` destination.
|
|
|
|
### Where the config file is
|
|
|
|
`vaultik` uses the first of these that applies:
|
|
|
|
- the path given with `--config`
|
|
- the path in the `VAULTIK_CONFIG` environment variable
|
|
- `config.yml` in the platform config directory, which is `~/.config/vaultik/`
|
|
on Linux and `~/Library/Application Support/vaultik/` on macOS
|
|
- `/etc/vaultik/config.yml`
|
|
|
|
A path from `--config` or `VAULTIK_CONFIG` must exist.
|
|
|
|
### Settings
|
|
|
|
- `age_recipients`
|
|
- The age public keys that backups are encrypted to.
|
|
- Default: none. `snapshot create` needs at least one. No other command
|
|
needs one.
|
|
- Any one of the matching private keys can decrypt.
|
|
- Adding a key later does not re-encrypt what is already stored. See
|
|
[Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks).
|
|
- `age_secret_key`
|
|
- The age private key.
|
|
- Default: not set.
|
|
- Only `snapshot restore` and `snapshot verify --deep` use it.
|
|
- The `VAULTIK_AGE_SECRET_KEY` environment variable overrides it.
|
|
- Do not set it on a machine you back up. See
|
|
[The private key](#the-private-key).
|
|
- `snapshots`
|
|
- The named sets of paths to back up. Required.
|
|
- `snapshots.<name>.paths`: the paths to back up under this name.
|
|
- `snapshots.<name>.exclude`: exclude patterns for this name only.
|
|
- `exclude`
|
|
- Exclude patterns that apply to every snapshot. See
|
|
[Exclude patterns](#exclude-patterns).
|
|
- Default: none.
|
|
- `storage_url`
|
|
- The destination. See [Destinations](#destinations).
|
|
- Required, unless the `s3` settings name the destination.
|
|
- `s3.access_key_id` and `s3.secret_access_key`
|
|
- The S3 credentials. Required for an S3 destination.
|
|
- `s3.region`
|
|
- The S3 region. Default: `us-east-1`.
|
|
- `s3.part_size`
|
|
- The size of each part of a multipart upload. Default: `5MiB`.
|
|
- It must be between `5MiB` and `5GiB`.
|
|
- A blob too large for 10,000 parts of this size is uploaded in larger
|
|
parts.
|
|
- `s3.endpoint`, `s3.bucket`, `s3.prefix` and `s3.use_ssl`
|
|
- An older way to name an S3 destination. They are used only when
|
|
`storage_url` is not set.
|
|
- `s3.endpoint` and `s3.bucket` are then required.
|
|
- `s3.use_ssl: false` selects plain HTTP for an endpoint written without a
|
|
scheme.
|
|
- `chunk_size`
|
|
- The average size of a chunk. Default: `10MB`. Minimum: `1MB`.
|
|
- A chunk is between a quarter of this size and four times this size.
|
|
- A smaller size finds more duplicate content and makes the index larger.
|
|
- `blob_size_limit`
|
|
- The largest size of a blob. Default: `10GB`.
|
|
- It must be at least four times `chunk_size`.
|
|
- `compression_level`
|
|
- The zstd compression level, from `1` to `19`. Default: `3`.
|
|
- Levels `1` to `2`, `3` to `5`, `6` to `9` and `10` to `19` form four
|
|
groups. Levels in one group compress the same way.
|
|
- `prune_grace_period`
|
|
- How old unused data must be before `prune` deletes it. Default: `24h`.
|
|
- It applies to a blob that no manifest lists. It also applies to every
|
|
object in a directory under `metadata/` that has no manifest.
|
|
- `0s` deletes such data at once. A negative value is refused.
|
|
- See [Sharing a destination](#sharing-a-destination).
|
|
- `hostname`
|
|
- The hostname used in snapshot IDs. Default: the system hostname.
|
|
- Only the part before the first `.` is used.
|
|
- `index_path`
|
|
- The path of the local index.
|
|
- Default: `index.sqlite` in the platform data directory. That is
|
|
`~/.local/share/vaultik/` on Linux and
|
|
`~/Library/Application Support/vaultik/` on macOS.
|
|
|
|
Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB`
|
|
and `GiB` are powers of 1024.
|
|
|
|
### Environment variables
|
|
|
|
- `VAULTIK_AGE_SECRET_KEY`: the age private key. See
|
|
[The private key](#the-private-key).
|
|
- `VAULTIK_CONFIG`: the path of the config file. `--config` overrides it.
|
|
- `VAULTIK_INDEX_PATH`: the path of the local index. It overrides `index_path`.
|
|
- `RCLONE_CONFIG`: the path of the rclone config file, for an `rclone://`
|
|
destination.
|
|
- `EDITOR`: the editor that `config edit` starts.
|
|
- `TMPDIR`: the directory for temporary files.
|
|
- `NO_COLOR`: when set, status lines have no color.
|
|
- `VAULTIK_CPUPROFILE`: write a CPU profile to this path. For debugging.
|
|
- `VAULTIK_MEMPROFILE`: write a heap profile to this path on exit. For
|
|
debugging.
|
|
|
|
### The private key
|
|
|
|
`snapshot restore` and `snapshot verify --deep` decrypt data, so they need the
|
|
age private key. No other command needs it.
|
|
|
|
Give the key in one of two ways:
|
|
|
|
- the `VAULTIK_AGE_SECRET_KEY` environment variable
|
|
- `age_secret_key` in the config file
|
|
|
|
The environment variable wins when both are set.
|
|
|
|
The value can be the whole file that `age-keygen` writes. Comments are ignored,
|
|
and every key in the file is used.
|
|
|
|
Set the variable from the file, so that the key is not typed into your shell
|
|
history:
|
|
|
|
```sh
|
|
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
|
|
```
|
|
|
|
Do not put the private key on a machine you back up. Anyone who breaks into that
|
|
machine could then decrypt its backups. Supply the key only on the machine you
|
|
restore on.
|
|
|
|
### Exclude patterns
|
|
|
|
A pattern excludes a file or a whole directory from a backup. Patterns are
|
|
matched against the path below the configured path.
|
|
|
|
- A pattern that starts with `/` matches from the configured path. In a snapshot
|
|
of `~`, `/Library/Caches` matches only `~/Library/Caches`.
|
|
- Any other pattern matches at every depth. `.cache` matches every file or
|
|
directory named `.cache`.
|
|
- `*` and `?` match within one name. `**` matches across directories.
|
|
- A pattern that matches a directory excludes everything in it.
|
|
|
|
`exclude` at the top of the config applies to every snapshot. `exclude` under a
|
|
snapshot name adds patterns for that snapshot.
|
|
|
|
## Destinations
|
|
|
|
`storage_url` selects one of three kinds of destination.
|
|
|
|
### S3
|
|
|
|
```
|
|
s3://bucket/prefix?endpoint=s3.example.com®ion=us-east-1
|
|
```
|
|
|
|
Any S3-compatible object store works.
|
|
|
|
- `endpoint` is the host of the store. The default is `s3.amazonaws.com`.
|
|
- `region` overrides `s3.region`.
|
|
- `ssl=false` selects plain HTTP for an endpoint written without a scheme.
|
|
- No other parameter is accepted.
|
|
|
|
The credentials come from `s3.access_key_id` and `s3.secret_access_key` in the
|
|
config. A URL that carries credentials is refused.
|
|
|
|
### Local directory
|
|
|
|
```
|
|
file:///path/to/backup
|
|
```
|
|
|
|
- The path must be absolute.
|
|
- A percent-escape in the path is decoded, so `my%20stick` names `my stick`.
|
|
- The directory must exist before the first backup. `vaultik` never creates it.
|
|
|
|
A directory on a removable volume is gone while the volume is not mounted. A
|
|
backup then fails. It does not write to the disk under the mount point.
|
|
|
|
This does not hold when the destination is the mount point itself. The empty
|
|
mount point stays behind when the volume is unplugged. There only
|
|
[the marker file](#the-marker-file) stops the backup.
|
|
|
|
### rclone
|
|
|
|
```
|
|
rclone://remote/path
|
|
```
|
|
|
|
This uses any storage provider that rclone supports. Define the remote first
|
|
with `rclone config`. `vaultik` reads rclone's own config file.
|
|
|
|
The URL takes no parameters and no credentials.
|
|
|
|
### Interrupted uploads
|
|
|
|
An upload that is cut off must not leave a partial object under the final name.
|
|
A later backup would take that object for complete.
|
|
|
|
- S3 shows an object only once its upload has finished.
|
|
- The local directory destination writes a temporary file and then renames it.
|
|
- An rclone remote that can move objects is written the same way. The temporary
|
|
name ends in `.partial`.
|
|
- An rclone remote that cannot move objects is written in place. On such a
|
|
remote, a killed upload can leave a partial object under the final name.
|
|
|
|
A leftover `.partial` file is ignored. You can delete it.
|
|
|
|
### Destination permissions
|
|
|
|
Each command needs different permissions at the destination.
|
|
|
|
- `snapshot create` needs only permission to write objects. A machine you back
|
|
up can therefore hold credentials that can neither read nor delete.
|
|
- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list
|
|
and read.
|
|
- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete.
|
|
- `prune` and `snapshot create --prune` list, read and delete.
|
|
|
|
A machine that runs `snapshot create --prune` can delete its own backups. To
|
|
keep it to write-only credentials, run `prune` from a separate, trusted machine.
|
|
|
|
A backup tries to look up the marker file and each blob before it writes. It
|
|
goes on when the destination refuses those look-ups.
|
|
|
|
### The marker file
|
|
|
|
Every backup writes `vaultik-destination.txt` at the top of the destination. The
|
|
file holds one fixed line of text.
|
|
|
|
The marker tells an emptied or missing destination apart from a new one. An
|
|
unplugged volume leaves an empty directory at its mount point. Without the
|
|
marker, that directory would look like a destination whose snapshots are all
|
|
gone.
|
|
|
|
The check applies while the local index holds snapshots. When the destination
|
|
then answers that the marker does not exist, commands treat the destination as
|
|
not present:
|
|
|
|
- `snapshot create` refuses to write to it.
|
|
- `prune`, `snapshot purge` and `remote nuke` fail.
|
|
- `snapshot list` shows the local index alone.
|
|
- `snapshot remove` deletes nothing at the destination.
|
|
|
|
To fix this, mount the volume. If you emptied the destination on purpose, run
|
|
`vaultik database delete` and back up again.
|
|
|
|
The check is skipped when the look-up fails for any other reason. S3 refuses the
|
|
look-up to credentials that can only write. A machine with such credentials
|
|
therefore does not notice an emptied destination. Its next backup would leave
|
|
out the data its local index records as stored. After you empty a destination,
|
|
run `vaultik database delete` on such a machine before it backs up again.
|
|
|
|
### Changing the destination
|
|
|
|
The local index belongs to one destination. It records which blobs are stored
|
|
there.
|
|
|
|
The first command that changes data records the configured destination in the
|
|
local index. Later, these commands refuse to run when the config names a
|
|
different destination:
|
|
|
|
- `snapshot create`
|
|
- `snapshot purge`
|
|
- `snapshot remove`
|
|
- `prune`
|
|
- `remote nuke`
|
|
|
|
The destination is `storage_url`, or else `s3.endpoint`, `s3.bucket` and
|
|
`s3.prefix`.
|
|
|
|
There are two ways out:
|
|
|
|
- Change the config back to the earlier destination.
|
|
- Run `vaultik database delete`. The next backup then starts a new index for the
|
|
new destination.
|
|
|
|
### Sharing a destination
|
|
|
|
Give each machine and each user its own destination, such as its own prefix or
|
|
directory. The rest of this section applies only when two of them share one.
|
|
|
|
A prune from one machine can meet a backup from another that is still running. A
|
|
backup uploads its blobs first, then its database, and its manifest last. Until
|
|
the manifest exists, no manifest lists the new blobs.
|
|
|
|
`prune_grace_period` protects those blobs. A prune keeps a blob that no manifest
|
|
lists until the blob is older than the grace period. It keeps a directory under
|
|
`metadata/` that has no manifest for the same time. Set the grace period longer
|
|
than the longest backup of any machine or user that shares the destination.
|
|
|
|
The grace period does not protect one case. A backup does not upload a blob that
|
|
is already at the destination. Such a blob keeps the time of its first upload.
|
|
It can be left over from an earlier backup that stopped before its manifest. A
|
|
prune from another machine can then delete that blob while the new backup runs.
|
|
The new snapshot cannot be restored afterwards.
|
|
|
|
`remote nuke` ignores the grace period.
|
|
|
|
## Locking
|
|
|
|
Commands that change data take a lock, so that only one of them runs at a time.
|
|
These commands take it:
|
|
|
|
- `snapshot create`
|
|
- `snapshot purge`
|
|
- `snapshot remove`
|
|
- `prune`
|
|
- `remote nuke`
|
|
|
|
A second one does not wait. It exits at once with an "already running" error.
|
|
|
|
The lock is the file `vaultik.pid` in the platform data directory. That is
|
|
`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/`
|
|
on macOS.
|
|
|
|
The lock covers one user on one machine. It does not stop another user or
|
|
another machine from writing to the same destination. See
|
|
[Sharing a destination](#sharing-a-destination).
|
|
|
|
No other command takes the lock. `snapshot list`, `snapshot verify`,
|
|
`snapshot restore`, `info` and `remote info` run while a backup is in progress.
|
|
|
|
## Output
|
|
|
|
### stdout and stderr
|
|
|
|
stdout carries the output you asked for:
|
|
|
|
- status and progress lines
|
|
- tables and reports
|
|
- JSON documents
|
|
- `config get` values
|
|
- completion scripts
|
|
|
|
stderr carries the rest:
|
|
|
|
- the startup banner
|
|
- log records, including those from `--verbose` and `--debug`
|
|
- the error that a failed command ends with
|
|
|
|
So `vaultik snapshot list > out.txt` captures the list and leaves the
|
|
diagnostics on your terminal.
|
|
|
|
Log records are colored lines when stderr is a terminal. They are JSON, one
|
|
object per line, when stderr is redirected.
|
|
|
|
Status lines are colored only on a terminal, and never when `NO_COLOR` is set.
|
|
|
|
### `--json`
|
|
|
|
Under `--json`, stdout holds one JSON document and nothing else. The banner and
|
|
the status lines are left out, and warnings go to stderr. A command can
|
|
therefore be piped as it is:
|
|
|
|
```sh
|
|
vaultik snapshot list --json | jq .
|
|
```
|
|
|
|
### `--quiet` and `--cron`
|
|
|
|
`--quiet` leaves out the banner, the status lines, the tables and the reports.
|
|
`--cron` on `snapshot create` does the same.
|
|
|
|
Neither flag hides these:
|
|
|
|
- warnings and errors
|
|
- `config get` values
|
|
- JSON documents
|
|
- confirmation prompts
|
|
|
|
### Exit status
|
|
|
|
- `0`: the command succeeded.
|
|
- `1`: the command failed.
|
|
- `130`: the command was interrupted by Ctrl-C or `SIGTERM`.
|
|
|
|
An interrupted command removes its temporary files before it exits. It gives up
|
|
after 30 seconds.
|
|
|
|
## Temporary files
|
|
|
|
Temporary files go to `$TMPDIR`, or to `/tmp` when `TMPDIR` is not set.
|
|
|
|
A backup needs free temporary space for these:
|
|
|
|
- One blob at a time. Each blob is written whole to a temporary file before it
|
|
is uploaded, so allow about `blob_size_limit`.
|
|
- About twice that for an rclone remote that cannot stream uploads.
|
|
- Copies of the local index, made when the snapshot's database is prepared.
|
|
|
|
With `TMPDIR` not set, SQLite writes one of those index copies to `/var/tmp`.
|
|
|
|
A restore keeps the downloaded blobs on disk, decrypted, until no remaining file
|
|
needs them. `snapshot restore` and `snapshot verify --deep` also write the
|
|
decrypted snapshot database there.
|
|
|
|
These files are readable only by their owner, and are removed when the command
|
|
ends. A `kill -9` or a power loss leaves them behind. Delete any leftover
|
|
`vaultik-*` entry in the temporary directory yourself.
|
|
|
|
## What the destination reveals
|
|
|
|
File contents, names, paths and metadata are encrypted. Someone who can read the
|
|
destination still learns these:
|
|
|
|
- when each backup finished
|
|
- how many blobs each snapshot uses, and their sizes
|
|
- which blobs two snapshots share
|
|
- whether a hostname and snapshot name they guess exist there
|
|
|
|
The last point holds because a remote key is a plain hash of the snapshot ID,
|
|
made without a secret.
|
|
|
|
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the
|
|
destination, what it protects and the risks it accepts.
|
|
|
|
## Limitations
|
|
|
|
- **No extended attributes.** ACLs, macOS Finder metadata, quarantine flags and
|
|
SELinux labels are not backed up.
|
|
- **No hard links.** Two hard links to one file are backed up as two files. The
|
|
content is stored once, but a restore creates two separate files.
|
|
- **No sparse files.** A sparse file is backed up at its full size, zeros
|
|
included, and restored as a full file.
|
|
- **No bandwidth limit.** Uploads and downloads use all the bandwidth there is.
|
|
- **One download at a time.** A restore downloads blobs one after another.
|
|
- **No device nodes, named pipes or sockets.** They are skipped without a
|
|
message.
|
|
- **No consistent view of changing files.** A file that changes while it is read
|
|
can be stored in a mixed state.
|
|
- **Owners need root.** A restore sets owners and groups only when it runs as
|
|
root.
|
|
- **No resume.** An interrupted restore starts again from the beginning.
|
|
- **The local index can become unusable after an update.** This applies before
|
|
version 1.0. Run `vaultik database delete` and then a full backup. The
|
|
destination is not affected. See
|
|
[Schema migrations](docs/DATAMODEL.md#schema-migrations).
|
|
|
|
## Development
|
|
|
|
[ARCHITECTURE.md](ARCHITECTURE.md) describes how the code is organised.
|
|
[docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index.
|
|
[`AGENTS.md`](AGENTS.md) holds the rules for AI agents.
|
|
[`REPO_POLICIES.md`](REPO_POLICIES.md) holds the repository policies.
|
|
|
|
### Requirements
|
|
|
|
- Go 1.26.1 or later.
|
|
- Docker, with a daemon you can reach. The tests and the linter run only inside
|
|
Docker.
|
|
|
|
### Workflow
|
|
|
|
- Make every change on a feature branch.
|
|
- Write the tests with the change.
|
|
- Update the documentation that the change affects.
|
|
- Run `make fmt`, then `make check`, before each commit.
|
|
|
|
`make check` runs the tests, the linter and the format check. All three must
|
|
pass.
|
|
|
|
### Entrypoints
|
|
|
|
The scripts in `script/` are the entrypoints. The `Makefile` targets only call
|
|
them. The layout follows
|
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all).
|
|
|
|
- `script/bootstrap`: installs what development needs. It installs Go from
|
|
`script/install-go` when the machine has none, and it fails without Docker.
|
|
- `script/setup`: prepares a fresh clone. It runs `script/bootstrap` and
|
|
`script/install-precommit`.
|
|
- `script/test`: runs the whole test suite, by building the `test` stage of the
|
|
`Dockerfile`.
|
|
- `script/lint`: runs the linter, by building the `lint` stage of the
|
|
`Dockerfile`. The linter never runs on your machine.
|
|
- `script/lint-fix`: applies the linter's automatic fixes to your files. It
|
|
needs a local Docker daemon. Run `script/lint` afterwards.
|
|
- `script/fmt`: formats the Go code.
|
|
- `script/fmt-check`: checks the formatting and changes nothing.
|
|
- `script/check`: runs `script/test`, `script/lint` and `script/fmt-check`.
|
|
- `script/precommit`: runs `go mod tidy`, `go fmt` and `script/check`. The git
|
|
pre-commit hook calls it.
|
|
- `script/install-precommit`: installs that hook.
|
|
- `script/docker`: builds the Docker image. The build runs the linter and the
|
|
tests too.
|
|
- `script/cibuild`: what CI runs on every push. It runs `script/bootstrap`,
|
|
`script/check` and the image build.
|
|
- `script/version`: prints the version to build into the binary.
|
|
- `script/projectname`: prints the project name, which tags the Docker image.
|
|
- `script/install-go`: installs the Go version that `go.mod` names into
|
|
`.tool/go`. It checks the download against a checksum in the script.
|
|
- `script/install-goreleaser`: installs the pinned `goreleaser` into
|
|
`.tool/bin`, checked the same way.
|
|
- `script/release`: builds and publishes the release files.
|
|
- `script/release-snapshot`: builds the release files into `./dist` and
|
|
publishes nothing.
|
|
|
|
Every `docker build` in these scripts uses `--no-cache`. A cached layer would
|
|
skip the tests or the linter and still report success.
|
|
|
|
The test suite has no separate integration target. Every test runs every time.
|
|
Each test binary is stopped after 90 seconds.
|
|
|
|
To change the Go version, change these together: `go.mod`, the version and
|
|
checksums in `script/install-go`, and the two `golang` image digests in the
|
|
`Dockerfile`.
|
|
|
|
### Messages to the user
|
|
|
|
Status lines, warnings and errors go through `internal/ui`. The data a command
|
|
exists to print is written plain, with no marker and no color. That covers
|
|
tables, reports, JSON documents and `config get` values.
|
|
|
|
`internal/ui` has these kinds of message:
|
|
|
|
- **Banner**: the startup line. No marker.
|
|
- **Begin**: an operation is starting. Marker `》`.
|
|
- **Complete**: an operation has finished. Marker `》`, in green.
|
|
- **Info**: a neutral status line. Marker `》`.
|
|
- **Notice**: a note the user should see. Marker `》`, in cyan.
|
|
- **Warning**: a problem the command survives. Prefix `⚠️ Warning:`.
|
|
- **Error**: the command stops. Prefix `🛑 ERROR:`.
|
|
- **Progress** and **Detail**: an indented line under another. Marker ` 》`.
|
|
|
|
Messages follow these conventions:
|
|
|
|
- A message is a full sentence that ends with a period.
|
|
- Name things in full. Write "backup destination store", not "storage".
|
|
- An operation that prints a Complete also prints a Begin.
|
|
- Use the tense to show the state: "Uploading" for a Begin, "Uploaded" for a
|
|
Complete.
|
|
- Say what a time belongs to, as in "blob upload elapsed: 30s".
|
|
- "ETA" is a clock time, not a remaining time. Print it with `ui.Time`, and
|
|
print a remaining time with `ui.Duration`.
|
|
|
|
Build a message from the value helpers in `internal/ui`, and never write color
|
|
codes by hand:
|
|
|
|
- `Hex`: a blob or chunk hash, cut to 12 characters.
|
|
- `Snapshot`: a snapshot ID.
|
|
- `Path`: a filesystem path.
|
|
- `Size`: a byte count.
|
|
- `Speed`: a transfer rate, in bits per second.
|
|
- `Duration`: an elapsed or remaining time.
|
|
- `Time`: a clock time, in local time.
|
|
- `Count`: a whole number, with thousands separators.
|
|
- `Percent`: a percentage.
|
|
|
|
## Releasing
|
|
|
|
### Version numbers
|
|
|
|
The version a binary reports comes from git. It is the output of
|
|
`git describe --tags --always --dirty`:
|
|
|
|
- On a tagged commit, it is the tag, such as `v1.0.0`.
|
|
- On a commit after a tag, it is `<tag>-<N>-g<short commit>`.
|
|
- With no tag, it is the short commit.
|
|
- With uncommitted changes to tracked files, `-dirty` is added.
|
|
|
|
`script/version` gives this value to the `Makefile`. `script/docker` and
|
|
`script/cibuild` compute it themselves.
|
|
|
|
A binary built without git reports `dev`. An image built by `script/docker` or
|
|
`script/cibuild` outside a git checkout reports `unknown`.
|
|
|
|
`goreleaser` builds the release binaries. It uses the tag without its leading
|
|
`v`, so the tag `v1.0.0` gives `vaultik 1.0.0`. `script/release-snapshot` stamps
|
|
`dev-` followed by 12 characters of the commit.
|
|
|
|
`vaultik version` calls a build a release only when its version is a plain tag,
|
|
such as `v1.0.0` or `1.0.0`. Every other version is a development build.
|
|
|
|
`make` stops with an error when `script/version` gives no version.
|
|
|
|
### The release workflow
|
|
|
|
`.gitea/workflows/release.yml` runs when a tag that starts with `v` is pushed.
|
|
It does this:
|
|
|
|
- It installs Go with `script/install-go` and `goreleaser` with
|
|
`script/install-goreleaser`.
|
|
- It runs `script/release`.
|
|
- `script/release` builds archives for Linux and macOS on `amd64` and `arm64`,
|
|
and a `checksums.txt`.
|
|
- It publishes them to this repository's releases, as a draft.
|
|
|
|
The workflow needs one repository secret, `RELEASE_TOKEN`. It is a Gitea access
|
|
token with the `write:repository` scope, from an account that can publish
|
|
releases here.
|
|
|
|
`script/release` refuses a `goreleaser` on `PATH` that is not the pinned
|
|
version.
|
|
|
|
`make release-snapshot` builds the same files into `./dist` and publishes
|
|
nothing.
|
|
|
|
Each archive holds the binary, `LICENSE` and `README.md`. The release files are
|
|
not signed, have no SBOM and are not built reproducibly.
|
|
|
|
## license
|
|
|
|
[MIT](https://opensource.org/license/mit/)
|
|
|
|
## author
|
|
|
|
Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in Berlin in the summer of 2025.
|
|
|
|
Released as a free software gift to the world, no strings attached.
|
|
|
|
Contact: [sneak@sneak.berlin](mailto:sneak@sneak.berlin)
|
|
|
|
[https://keys.openpgp.org/vks/v1/by-fingerprint/5539AD00DE4C42F3AFE11575052443F4DF2A55C2](https://keys.openpgp.org/vks/v1/by-fingerprint/5539AD00DE4C42F3AFE11575052443F4DF2A55C2)
|