From de9ef62073f12ef3f5ffe6e101c048c7604e60cb Mon Sep 17 00:00:00 2001 From: sneak Date: Fri, 6 Nov 2026 18:21:23 +0000 Subject: [PATCH] Rewrite README.md sentence by sentence Rewritten from scratch against the code on next: what the program is, how to use it, a command reference that says what each command needs, the configuration reference as nested bullets, then development and releasing. snapshot verify --deep now says it needs the private key, from VAULTIK_AGE_SECRET_KEY or age_secret_key, as snapshot restore does. The roadmap section is removed: it described features that are not built. Model: fable-5-1 --- README.md | 2180 +++++++++++++++++++++++++++++------------------------ 1 file changed, 1201 insertions(+), 979 deletions(-) diff --git a/README.md b/README.md index f83d065..21d55a9 100644 --- a/README.md +++ b/README.md @@ -1,1123 +1,1345 @@ # vaultik (ваултик) -`vaultik` is an incremental backup tool written in Go. It encrypts data -using an `age` public key and uploads each encrypted blob directly to a -remote S3-compatible object store. It requires no private keys, secrets, or -credentials (other than those required to PUT to encrypted object storage, -such as S3 API keys) stored on the backed-up system. +`vaultik` is an incremental backup tool. It encrypts your files with a public +key and uploads them to an S3-compatible object store, a local directory or an +rclone remote. -## quickstart +The machine being backed up holds only the public key. It cannot decrypt its own +backups. You keep the private key somewhere else and need it only to restore. + +## Why + +Other backup tools, such as `restic`, `borg` and `duplicity`, are designed for +machines that can hold a decryption key. `vaultik` is for machines that should +not hold one. + +It was written to meet these requirements together: + +- open source +- no passphrase or private key on the backed-up machine +- incremental +- compressed +- encrypted +- direct upload to S3-compatible storage, with no intermediate step or tool + +## Features + +- Encryption with [age](https://age-encryption.org/) (X25519 and + ChaCha20-Poly1305), to one or more public keys. +- Incremental backups. A file is read again only when it has changed. +- Deduplication. Identical content is stored once, across files and across + snapshots. +- zstd compression at a level you choose. +- Small files are packed together into large objects. This keeps the number of + S3 requests low. +- File contents, names, paths and metadata are encrypted at the destination. +- Three kinds of destination: S3, a local directory and any rclone remote. +- Regular files, symlinks and directories are backed up, with their permissions, + owners and modification times. +- Pure Go, with no C code. It builds for Linux and macOS on `amd64` and `arm64`. + +## Terms + +- **Snapshot**: one backup of the paths listed under one name in the config, + taken at one time. +- **Snapshot ID**: the name of a snapshot, in the form + `__`. An example is + `server1_home_2025-06-01T12:00:00Z`. +- **Destination**: the place the backups are stored. +- **Chunk**: a piece of a file's content. `vaultik` cuts each file into chunks + and stores each distinct chunk once. +- **Blob**: one object at the destination. It holds many chunks, compressed and + then encrypted. +- **Local index**: a SQLite database on the backed-up machine. It records what + is already stored, so the next backup uploads only what changed. +- **Manifest**: the list of blobs one snapshot uses. It is stored at the + destination without encryption. +- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory + at the destination, so the destination never shows the snapshot ID. + +## Install ```sh -# install go install sneak.berlin/go/vaultik/cmd/vaultik@latest +``` -# create a default config file (prints the path it wrote to) +This needs Go 1.26.1 or later. Generating a key pair needs `age-keygen`, which +comes with [age](https://age-encryption.org/). + +## Quick start + +```sh +# Write a default config file. The command prints the path it wrote. vaultik config init -# generate an age keypair; keep the private key file somewhere safe and -# offline — you need it to restore, and the backed-up machine does not need it +# Generate a key pair. Keep this file somewhere safe, away from this machine. age-keygen -o vaultik_backup_private_key.txt grep 'public key' vaultik_backup_private_key.txt -# configure the encryption key and backup destination +# Set the public key and the destination. vaultik config set age_recipients.0 age1YOUR_PUBLIC_KEY_HERE vaultik config set storage_url "file:///Volumes/usbstick/mybackup" -# create the destination directory; vaultik never creates it, so a backup -# to a volume that is not plugged in fails instead of filling the disk +# Create the destination directory. vaultik never creates it. mkdir /Volumes/usbstick/mybackup -# macOS only: grant your terminal app Full Disk Access first -# (System Settings → Privacy & Security → Full Disk Access), otherwise -# the backup will abort with a permission error on protected directories - -# run your first backup (the default config backs up ~ and /Applications -# with sensible excludes) +# Run the first backup. vaultik snapshot create -# see what you have +# See what is stored. vaultik snapshot list ``` -Features: +The default config is written for macOS. It backs up your home directory and +`/Applications`, with a list of excludes. A configured path that does not exist +stops the backup, so on another system run `vaultik config edit` first and +change the `snapshots` section. -* modern encryption ([age](https://age-encryption.org/), X25519 + ChaCha20-Poly1305) -* content-defined chunking with deduplication (FastCDC) -* incremental backups (a file is re-chunked only when it changed or a - chunk it lists is held by no uploaded blob) -* multithreaded zstd compression at configurable levels -* content-addressed immutable storage -* local state tracking in SQLite (enables write-only incremental backups) -* no mutable remote metadata -* no plaintext file paths or metadata in remote storage -* packs small files into large blobs (keeps S3 operation counts down) -* backs up regular files, symlinks, empty directories, and file permissions -* pluggable storage backends: S3, local filesystem, rclone (70+ providers) -* pure Go (no CGO), cross-compiles to linux/darwin × amd64/arm64 +On macOS, grant your terminal app Full Disk Access before the first backup. The +setting is under System Settings, Privacy & Security, Full Disk Access. Without +it the backup stops with a permission error on a protected directory. -## why - -Other backup tools like `restic`, `borg`, and `duplicity` are designed for -environments where the source host can store secrets and has access to -decryption keys. `vaultik` is for environments where you don't want to -store backup decryption keys on your hosts — only public keys for -encryption. - -Requirements that no existing tool meets: - -* open source -* no passphrases or private keys on the source host -* incremental -* compressed -* encrypted -* s3 compatible without an intermediate step or tool - -## daily use +## Everyday use ```sh -# verify a snapshot (shallow: checks all blobs are present with the listed size) +# Check that a snapshot's blobs are all present, with the listed sizes. vaultik snapshot verify -# put the private key file in the environment (reading it from the file -# keeps the key out of your shell history); the whole age-keygen file, -# with one or more identities, is accepted +# Put the private key in the environment. Reading it from the file keeps it +# out of your shell history. export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" -# deep verify (downloads every blob, decrypts it, and re-hashes it to -# detect corruption — this checks integrity, not who wrote the blob) +# Download, decrypt and re-hash every blob of a snapshot. vaultik snapshot verify --deep -# restore (requires the private key). Restore into a new directory you own, -# writable only by you — not a shared location like /tmp +# Restore a snapshot into a new directory that only you can write. vaultik snapshot restore ~/vaultik-restore -# daily cron job: back up, keep a 4-week rolling window of snapshots +# A daily cron line: back up, then keep four weeks of snapshots. # 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w ``` -On a destination shared between hosts or users, the prune in that cron line -relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup -another one is still running, and its directory under `metadata/` that has no -manifest yet, so it must be longer than the longest backup run. It does not -protect a blob that backup reuses from an earlier run that stopped before its -manifest: the prune can delete such a blob, and the new snapshot then cannot -be restored. To not depend on the grace period, give each host its own prefix -or directory. See [locking](#locking). +`--prune` deletes data at the destination. Read +[Sharing a destination](#sharing-a-destination) before you use it on a +destination that more than one machine or user writes to. -## restoring on another machine +## Restoring on another machine -Restoring on a host that never ran the backup — a replacement machine -after the original is gone — is the case vaultik is built for. That host -needs only three things: the `vaultik` binary, the age **private** key, -and the storage credentials for the destination. It does **not** need the -local index, the original config file, or the original hostname. +You can restore on a machine that never ran the backup, such as a replacement +for a lost one. That machine needs three things: + +- the `vaultik` binary +- the age private key +- the destination, and credentials that can list and read it + +It does not need the original local index, config file or hostname. ```sh -# install -go install sneak.berlin/go/vaultik/cmd/vaultik@latest - -# create a config and point it at the ORIGINAL backup destination +# Create a config and point it at the original destination. vaultik config init vaultik config set storage_url "s3://bucket/prefix?endpoint=https://s3.example.com" vaultik config set s3.access_key_id "..." vaultik config set s3.secret_access_key "..." -# see what is on the destination store +# See what the destination holds. This needs no private key. vaultik snapshot list ``` -`snapshot list` reads the destination store without the private key. A -snapshot that is not in this host's (empty) local index is shown as -remote-only: its row is identified by `` rather than by -a `hostname_name_timestamp` name, because the name lives only in the -local index and the encrypted database and cannot be recovered from the -store. Its timestamp and compressed size are real. (See the `snapshot -list` description under [command details](#command-details) for the full -explanation.) +On this machine the local index is empty, so the list cannot show snapshot IDs. +Each snapshot appears as ``, with the first 12 characters of +its remote key. The snapshot ID is stored only in the local index and in the +snapshot's encrypted database. -Use that remote key — the hex printed inside ``, or the -full `remote_key` from `snapshot list --json` — to restore and verify: +Use that remote key in place of the snapshot ID: ```sh -# put the private key file in the environment (reading it from the file -# keeps the key out of your shell history) export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" -# restore everything to a new directory you own (writable only by you, not a -# shared location like /tmp), then check every restored file's chunk hashes +# Restore everything, then check the chunk hashes of every restored file. vaultik snapshot restore --verify ~/vaultik-restore -# optionally, deep-verify the snapshot against the store (downloads every -# blob, decrypts it, and re-hashes it to detect corruption — this checks -# integrity, not who wrote the blob) +# Optionally, also check every blob of the snapshot at the destination. vaultik snapshot verify --deep ``` -`age_recipients` (the public key) is not needed to restore — only the -private key in `VAULTIK_AGE_SECRET_KEY`. Both the abbreviated key printed -in the table and the full 64-character key from `--json` are accepted; a -leading part of the key is enough as long as it is unambiguous. +Any leading part of the remote key works, as long as it matches one snapshot. +`snapshot list --json` gives the full 64-character key as `remote_key`. ---- +A machine that only restores does not need `age_recipients`. -## cli - -### commands +## Commands ```sh -vaultik [--config ] config init -vaultik [--config ] config edit -vaultik [--config ] config get -vaultik [--config ] config set -vaultik [--config ] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than ] -vaultik [--config ] snapshot list [--json] # alias: ls -vaultik [--config ] snapshot verify [--deep] [--json] -vaultik [--config ] snapshot purge [--keep-latest | --older-than ] [--snapshot ...] [--force] -vaultik [--config ] snapshot remove [--dry-run] [--force] [--local-only] [--json] # alias: rm -vaultik [--config ] snapshot restore [paths...] [--verify] -vaultik [--config ] prune [--force] [--json] -vaultik [--config ] info -vaultik [--config ] remote info [--json] -vaultik [--config ] remote nuke --force -vaultik [--config ] database delete [--force] +vaultik config init +vaultik config edit +vaultik config get +vaultik config set +vaultik snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than ] +vaultik snapshot list [--json] +vaultik snapshot verify [--deep] [--json] +vaultik snapshot purge (--keep-latest | --older-than ) [--snapshot ...] [--force] +vaultik snapshot remove [--dry-run] [--force] [--local-only] [--json] +vaultik snapshot restore [paths...] [--verify] +vaultik prune [--force] [--json] +vaultik info +vaultik remote info [--json] +vaultik remote nuke --force +vaultik database delete [--force] vaultik completion vaultik version ``` -### global flags +`snapshot list` has the alias `snapshot ls`. `snapshot remove` has the alias +`snapshot rm`. -* `--config `: Path to config file (default: `$VAULTIK_CONFIG`, then platform config dir, then `/etc/vaultik/config.yml`) -* `--verbose`, `-v`: Enable verbose output (on stderr — see below) -* `--debug`: Enable debug output (on stderr — see below) -* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner) -* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run. A run that skipped anything still finishes but exits non-zero: `snapshot create` keeps the snapshot and counts each file or directory it skipped as failed. A file removed during the backup is also counted as failed but does not change the exit status. +### What every command needs -### locking +`config init`, `completion` and `version` need nothing. -Commands that write persistent state — `snapshot create`, `snapshot -purge`, `snapshot remove`, `prune`, and `remote nuke` — take a -process-wide lock at `$XDG_DATA_HOME/vaultik/vaultik.pid` -(`~/.local/share/vaultik/vaultik.pid` on Linux) for the whole run. Only -one of them runs at a time: a second one exits immediately with an -"already running" error rather than waiting, so two writers can never -corrupt the local index. +`config edit`, `config get` and `config set` need a config file that exists. It +does not have to be complete. -The lock covers one user on one machine. It does not stop another user, -or another host, from writing to the same destination store at the same -time. A backup uploads its blobs, and then its `db.zst.age`, before its -manifest, so a prune from elsewhere at that moment finds blobs no snapshot -lists and a directory under `metadata/` with no manifest. It -deletes such a blob only once it is older than `prune_grace_period` (default -`24h`), and such a directory only once every object in it is, so the grace -period must be longer than the longest backup run of any host or user sharing -the destination. The grace period does not protect a blob a backup reuses -from an earlier run that stopped before its manifest: a backup skips the -upload of a blob already on the destination, so that blob keeps the time of -its first upload, and a prune from elsewhere during the backup can delete it; -the new snapshot then cannot be restored. `remote nuke` deletes everything -regardless. To not depend on the grace period, give each host or user its own -prefix (or directory) on the destination. +Every other command needs a config file that loads. See +[Configuration](#configuration) for the checks. -Read-only commands — `info`, `snapshot list`, `snapshot verify`, and -`remote info` — do not take the lock and are never blocked, so they run -even while a backup is in progress. `snapshot restore` does not take the -lock either: it writes only to the target directory you name, not the -local index or the destination store. `config`, `database delete`, -`completion`, and `version` do not take the lock. +Every other command except `database delete` also needs these: -### stdout and stderr +- Write access to the local index. The command creates the file, and its + directory, when they do not exist. +- For an `rclone://` destination, an rclone config file that defines the remote. -Log output — everything from `--verbose` and `--debug`, and every -warning and error the logger emits — goes to **stderr**, and so does the -startup banner. stdout carries the output you asked for: tables, the -documents produced by `--json`, `config get` values, and completion -scripts. +Each command below lists what it needs beyond this. No command needs root, but a +backup can read only what its user can read, and a restore sets file owners only +as root. -This means `vaultik snapshot list --verbose > out.txt` captures the -listing and leaves the diagnostics on your terminal. To capture both, -redirect stderr as well (`> out.txt 2> log.txt`, or `> out.txt 2>&1` to -interleave them). +### Global flags -The split is what makes `--json` usable from a script. Warnings and -errors are never suppressed — not by `--quiet`, not by `--cron` — so a -logger on stdout would eventually land a log line inside a JSON -document and break the parse. A config file with group- or -world-readable permissions is enough to trigger it. +- `--config `: the config file to use. See + [Where the config file is](#where-the-config-file-is). +- `--verbose`, `-v`: write informational log records to stderr. +- `--debug`: write debug log records to stderr. +- `--quiet`, `-q`: print only warnings, errors and the output a script reads. + See [Output](#output). +- `--skip-errors`: skip a file that cannot be read during a backup, or cannot be + restored during a restore. Without it the command stops at the first such + file. -Format follows the stream: when stderr is a terminal the records are -colorized one-liners, and when it is redirected or piped they are -JSON, one object per line. +`--skip-errors` applies to `snapshot create` and `snapshot restore` only. A +command that skipped anything still finishes, and then exits non-zero. -Under `--json`, stdout holds the document and nothing else. The startup -banner is suppressed, as `--quiet` and `--cron` suppress it, and the -progress narration a command would otherwise print — such as the stale -local records `prune` reconciles away — is suppressed too, so it cannot -land ahead of the document. Every `--json` command therefore pipes on -its own, with no additional flag: `vaultik snapshot list --json | jq .` -and `vaultik prune --json | jq .` both work as written. +### Durations -### environment variables +`--keep-newer-than`, `--older-than` and `prune_grace_period` take a duration. -* `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `snapshot restore` and `snapshot verify --deep`). May hold the whole `age-keygen` file — comments and every identity in it are accepted. Set it from the file, e.g. `export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"`, so the key is not typed into your shell history. -* `VAULTIK_CONFIG`: Path to config file (overridden by `--config`) -* `VAULTIK_INDEX_PATH`: Override local SQLite index path -* `VAULTIK_CPUPROFILE`: Write a CPU profile to this path for the duration of the run (development/debugging) -* `VAULTIK_MEMPROFILE`: Write a heap profile to this path when the run exits (development/debugging) +- `d`: days. +- `w`: weeks. +- `mo`: months of 30 days. +- `y`: years of 365 days. +- `h`, `m` and `s`: hours, minutes and seconds, as in `36h` or `90m`. -### shell completion +`m` means minutes and `mo` means months. Days, weeks, months and years take +whole numbers and can be combined, as in `2w3d`. + +`prune_grace_period` accepts only hours, minutes and seconds. + +### `config init` + +Writes a default config file, with a comment that explains each setting it +lists. + +- It writes to the path from `--config`, or else from `VAULTIK_CONFIG`. +- Otherwise it writes to `config.yml` in the platform config directory. That is + `~/.config/vaultik/` on Linux and `~/Library/Application Support/vaultik/` on + macOS. +- As root it writes to `/etc/vaultik/config.yml` instead. +- It creates the file with mode `0600`, because the file can hold credentials. +- It refuses to overwrite a file that exists. + +### `config edit` + +Opens the config file in `$EDITOR`. It uses `vi` when `EDITOR` is not set. + +### `config get ` + +Prints one value from the config file. The key is a dotted path, as in +`vaultik config get storage_url` or `vaultik config get age_recipients.0`. A +list or a map prints as YAML. + +### `config set ` + +Sets one value in the config file. The key is a dotted path, as in +`vaultik config set compression_level 9`. + +- Comments and formatting in the file are kept. +- A key that does not exist yet is created. A missing parent becomes a list when + the next part of the path is a number, and a map otherwise. +- In a list, the index one past the end appends an entry. +- The command prints the key it set, and never the value. +- It sets the file's mode to `0600` when group or others could read it. + +The command refuses a change that would stop the config from loading. An example +is `vaultik config set compression_level 99`. The file is then left unchanged. + +A config that does not load yet is checked less strictly. An example is the file +`config init` writes, which has no `storage_url`. There the command refuses only +a value of the wrong type, such as a word where a number belongs. + +### `snapshot create` + +Backs up the configured snapshots. With no names it backs up all of them. Each +name gets its own snapshot. + +It needs: + +- At least one public key in `age_recipients`. +- Read access to the configured paths. +- Permission to write objects at the destination. See + [Destination permissions](#destination-permissions). +- Free temporary space. See [Temporary files](#temporary-files). +- The [lock](#locking). + +It does not need the private key. + +A file is read again when any of these has changed since the last backup: + +- its size +- its modification time +- its mode +- its owner or group +- its link target + +Every other file is recorded in the new snapshot without being read. + +Flags: + +- `--cron`: print nothing when the backup succeeds. Warnings and errors are + still printed. +- `--prune`: delete old snapshots and unused data after the backup. See below. +- `--keep-newer-than `: with `--prune`, keep every snapshot newer than + this. + +The backup stops before it writes anything in these cases: + +- A `file://` destination directory does not exist. +- The destination has no marker file, although the local index holds snapshots. + See [The marker file](#the-marker-file). +- The destination in the config is not the one the local index was used with. + See [Changing the destination](#changing-the-destination). + +The backup also stops when a configured path is a symlink whose target does not +exist. It stops when a configured path does not exist, unless `--skip-errors` is +given. + +With `--skip-errors`, a file or directory that cannot be read is skipped. The +snapshot is kept and counts each skipped entry as failed, and the command exits +non-zero. An error while packing or uploading still stops the backup. + +A file that is removed during the backup is counted as failed. It does not +change the exit status. + +#### `--prune` + +After a successful backup, `--prune` does two things: + +- It removes the older snapshots of each name it just backed up. It keeps only + the newest one, or with `--keep-newer-than` every snapshot newer than that. +- It then deletes at the destination what [`prune`](#prune) deletes. + +`--prune` also needs permission to list, read and delete at the destination. + +`--keep-newer-than` without `--prune` is an error. So is a value that is not a +duration. Both are reported before the backup starts. + +The prune does not run after a backup that may have missed data. An older +snapshot may then hold the only copy of what was missed. This applies when: + +- `--skip-errors` skipped a file or directory that could not be read. +- A configured path held nothing, as the mount point of an unmounted volume + does. +- An earlier snapshot of the same name found a mount point that this backup did + not find. + +In each case every older snapshot is kept. The command says why, names the path +and exits non-zero. It does so on every backup until the cause is gone. + +A mount point here is a directory at or below a configured path that is on a +different filesystem from its parent. It counts only when the backup finds an +entry in it that is not excluded. A backup does not find it when the volume is +not mounted, or when the directory is now excluded or no longer configured. + +If the cause is intended, such as a retired volume, remove the older snapshots +yourself with `snapshot purge`. Take a path that is empty on purpose out of the +config. + +### `snapshot list` + +Lists the snapshots in the local index and the snapshots at the destination, as +one table. Before it, the command prints the snapshot names and paths in the +config. + +It needs permission to list and read at the destination. It needs no private +key. + +The table has these columns: + +- `SNAPSHOT ID` +- `TIMESTAMP`: in UTC. +- `COMPRESSED SIZE`: the total size of the blobs the snapshot uses. +- `UNCOMPRESSED SIZE`: the total size of its distinct chunks before compression. +- `NEW CHUNK SIZE`: the size of the chunks this snapshot stored that were not + stored before. + +A snapshot at the destination that is not in the local index is shown as +``. The dots stand for the first 12 characters of its remote +key. Its last two columns read ``, because those sizes come from +the local index. Its timestamp is the time its backup finished. + +The command reports these below the table: + +- Snapshots in the local index that are missing at the destination. Run + `vaultik prune` to drop those records. +- Remote-only snapshots whose manifest cannot be read. They are left out of the + table. +- Remote-only snapshots beyond the first 1000. They are left out of the table. +- A local index that was used with a different destination. See + [Changing the destination](#changing-the-destination). + +When the destination cannot be listed, the command warns and shows the local +index alone. It still exits zero. It does the same when the destination has no +marker file although the local index holds snapshots. + +`--json` prints a JSON array instead. Each entry has these fields: + +- `id`: the snapshot ID. It is empty for a remote-only snapshot. +- `remote_key`: the full 64-character remote key. +- `timestamp` +- `compressed_size` +- `uncompressed_size` and `new_chunk_size`: left out for a remote-only snapshot. +- `locally_tracked`: whether the snapshot is in the local index. +- `remote_present`: whether the snapshot is at the destination. It is `null` + when the destination could not be listed. + +Under `--json`, every warning above goes to stderr. + +### `snapshot verify ` + +Checks a snapshot at the destination. The argument is a snapshot ID, a remote +key or the leading part of a remote key. + +Without `--deep`, it checks two things: + +- Every blob the manifest lists is present, with the size the manifest records. +- The snapshot's encrypted database is present. + +This reads no blob contents and needs no private key. It needs permission to +read at the destination, and to list it when you give a remote key. + +Flags: + +- `--deep`: download, decrypt and re-hash every blob. See below. +- `--json`: print the result as JSON. + +The command exits non-zero when a check fails or cannot be made. + +#### `--deep` + +`--deep` needs the age private key. Set it in the `VAULTIK_AGE_SECRET_KEY` +environment variable, or as `age_secret_key` in the config file. The environment +variable wins when both are set. See [The private key](#the-private-key). + +`--deep` does these checks, in order: + +- It downloads and decrypts the snapshot's database. It checks that this is the + database of the snapshot you named. +- It checks that the manifest and the database list the same blobs, with the + same sizes. +- It checks that every blob is present with the recorded size. +- It downloads and decrypts every blob. It checks the hash of each chunk, and + the hash that names the blob. + +It stops at the first blob that fails. + +`--deep` proves that the stored data is intact and decrypts with your key. It +does not prove who wrote the data. + +`--deep` writes the decrypted database to a temporary directory and removes it +afterwards. See [Temporary files](#temporary-files). + +### `snapshot purge` + +Removes old snapshots. You must give exactly one of `--keep-latest` and +`--older-than`. + +It needs the [lock](#locking), and permission to list and delete at the +destination. It needs no private key. + +Flags: + +- `--keep-latest`: keep only the newest snapshot of each name. +- `--older-than `: remove every snapshot older than this. +- `--snapshot `: act only on snapshots with this name. Repeat the flag for + more names. +- `--force`: do not ask for confirmation. + +The command acts only on snapshots in the local index. `--keep-latest` keeps the +newest snapshot of each name, not the newest overall. + +A removed snapshot leaves the local index, and its directory under `metadata/` +at the destination is deleted. Its blobs stay. Run `vaultik prune` afterwards to +delete the blobs that no snapshot uses any more. + +The command first drops local records of snapshots that are no longer at the +destination. It fails and removes nothing when the destination cannot be listed +or has no marker file. + +### `snapshot remove ` + +Removes one snapshot. The argument must be a snapshot ID. A remote key is not +accepted. + +It needs the [lock](#locking), and permission to list and delete at the +destination. It needs no private key. + +Flags: + +- `--force`, `-f`: do not ask for confirmation. +- `--dry-run`: say what would be removed, and remove nothing. +- `--local-only`: remove the snapshot from the local index only. The destination + is not changed. +- `--json`: print the result as JSON. This also skips the confirmation. + +The command removes the snapshot from the local index. It also deletes the +snapshot's directory under `metadata/` at the destination. Its blobs stay. Run +`vaultik prune` afterwards to delete the blobs that no snapshot uses any more. + +An ID that is in neither the local index nor the destination is an error. With +`--local-only`, the snapshot must be in the local index. + +The deletion at the destination can fail, for example when the destination is +unreachable or has no marker file. What happens then depends on where the +snapshot was: + +- A snapshot in the local index is still removed from it, with a warning. Run + the same command again later to delete its directory at the destination. + `vaultik prune` does not delete it. +- For a snapshot that is only at the destination, the failure is an error. + +### `snapshot restore [paths...]` + +Restores files from a snapshot into ``. The first argument is a +snapshot ID, a remote key or the leading part of a remote key. + +It needs: + +- The age private key, as for `snapshot verify --deep`. See + [The private key](#the-private-key). +- Permission to read at the destination, and to list it when you give a remote + key. +- Write access to ``. The command creates the directory when it does + not exist. +- Free temporary space for the decrypted blobs. See + [Temporary files](#temporary-files). + +It does not need `age_recipients`, and it does not take the lock. + +Each file is restored under its original absolute path inside ``. +For example, `/home/user/notes.txt` is restored to +`/home/user/notes.txt`. + +With no paths, the command restores the whole snapshot. Each path names a file +or a directory by its original absolute path: ```sh -# zsh: load for the current session +vaultik snapshot restore ~/vaultik-restore /home/user/documents +``` + +A path that matches nothing is named in a warning. The command restores the +other paths and then exits non-zero. + +The command restores these: + +- regular files, symlinks and directories, empty directories included +- permissions and modification times +- owners and groups, when it runs as root + +Without root, restored files belong to the user who runs the command. The +command then warns once if any owner or group differs from the snapshot. + +A file that already exists in `` is replaced. Running the same +restore again starts from the beginning. + +Restore into a new directory that only you can write, not a shared one such as +`/tmp`. No other user can then change the files while they are written. + +Flags: + +- `--verify`: after the restore, read every restored file and check its chunk + hashes. + +With `--skip-errors`, a file that cannot be restored is reported and skipped, +and the command exits non-zero. `--verify` does not check a skipped file. + +### `prune` + +Removes what is no longer needed, from the local index and from the destination. + +It needs the [lock](#locking), and permission to list, read and delete at the +destination. It needs no private key. + +It works in three steps: + +- It drops each snapshot from the local index whose directory is gone from the + destination. +- It removes unfinished snapshots from the local index. It also removes the file, + chunk and blob records that no snapshot uses. +- It reads every manifest at the destination. It then deletes each blob that no + manifest lists and that is older than `prune_grace_period`. + +The third step also deletes a directory under `metadata/` that has no manifest. +A backup leaves one when it stops before it uploads its manifest. Such a +directory is deleted only when all of these hold: + +- Every object in it is older than `prune_grace_period`. +- Its snapshot is not in the local index. +- Its name has the form of a remote key. + +`prune_grace_period` defaults to `24h`. A newer blob is kept, because a backup +that is still running may need it. See +[Sharing a destination](#sharing-a-destination) for what the grace period does +not protect. + +The command deletes nothing at the destination in these cases: + +- The destination cannot be listed. +- The destination has no marker file, although the local index holds snapshots. +- A manifest cannot be read. + +It exits non-zero when a blob or a directory cannot be deleted. + +The time to read the manifests grows with the number of snapshots. + +Flags: + +- `--force`: do not ask for confirmation before deleting at the destination. +- `--json`: print the result as JSON. This also skips the confirmation. + +The JSON document has these fields: + +- `blobs_found`: the unused blobs old enough to delete. +- `blobs_deleted` +- `blobs_failed`: blobs that could not be deleted. Left out when zero. +- `bytes_freed` +- `blobs_too_new`: unused blobs that were kept. They are newer than the grace + period, or the destination gave no time for them. +- `metadata_dirs_deleted` +- `metadata_dirs_failed`: left out when zero. + +### `info` + +Prints the system details, the settings in use and figures from the local index. +The figures are the size of the index and its counts of snapshots, blobs and +files. + +### `remote info` + +Reports what the destination holds: + +- the kind of destination and its location +- the size of each snapshot's database and manifest +- the number and size of all blobs +- the number and size of the blobs that manifests list +- the number and size of the blobs that no manifest lists + +It needs permission to list and read at the destination. It needs no private +key. + +When a manifest cannot be read, the last figures are reported as unknown. That +snapshot's own blob count and size are unknown too. + +The command skips a name under `metadata/` that is not a remote key, with a +warning. It does not count a file under `blobs/` whose name is not a blob hash. + +`--json` prints the report as JSON. Unknown figures are `null`. +`unreadable_manifests` lists the remote key of each manifest that could not be +read. `skipped_manifest_count` counts the manifests under skipped names. + +### `remote nuke --force` + +Deletes everything `vaultik` stored at the destination. This cannot be undone. + +It needs `--force`, the [lock](#locking), and permission to list and delete at +the destination. + +It deletes these, in order: + +- every object under `metadata/` +- every object under `blobs/` +- the marker file + +It also removes each snapshot from the local index, once that snapshot's +directory at the destination is deleted. + +The command fails before it deletes anything when the destination cannot be +listed, or has no marker file although the local index holds snapshots. + +If objects remain afterwards, the command names each one and exits non-zero. +Running it again deletes what is left. + +### `database delete` + +Deletes the local index file. The destination is not changed. + +It needs a config file that loads, because the config gives the path of the +index. + +The next backup then reads every file again. It skips the upload of a blob that +the destination already holds under the same name. + +Use this command after you point the config at a different destination. See +[Changing the destination](#changing-the-destination). + +`--force` skips the confirmation. + +### `completion ` + +Prints a completion script for `bash`, `zsh`, `fish` or `powershell`. + +```sh +# zsh, for the current session source <(vaultik completion zsh) -# zsh: install permanently +# zsh, permanently vaultik completion zsh > "${fpath[1]}/_vaultik" -# bash: load for the current session +# bash, for the current session source <(vaultik completion bash) -# bash: install permanently (Linux) +# bash, permanently, on Linux vaultik completion bash > /etc/bash_completion.d/vaultik # fish vaultik completion fish > ~/.config/fish/completions/vaultik.fish ``` -### command details +### `version` -**`config init`**: Write a default config file with commented explanations for -every setting. Writes to the path from `--config`, `$VAULTIK_CONFIG`, or the -platform config directory (`~/Library/Application Support/vaultik/` on macOS, -`~/.config/vaultik/` on Linux, `/etc/vaultik/` as root). Refuses to overwrite an -existing file. Created with mode `0600` since it will contain credentials. +Prints the version, the commit, the build date and the platform. It also says +when the binary is a development build, not a release. -**`config edit`**: Open the config file in `$EDITOR` (falls back to `vi`). +## Configuration -**`config get`**: Print a config value addressed by dotted YAML path -(e.g. `vaultik config get storage_url`). Non-scalar values print as YAML. +The config file is YAML. `vaultik config init` writes one to start from, and +[`config.example.yml`](config.example.yml) is a longer example. -**`config set`**: Set a scalar config value by dotted YAML path -(e.g. `vaultik config set compression_level 9`, -`vaultik config set storage_url "file:///mnt/backups"`). Comments and -formatting in the file are preserved. Missing keys along the path are -created: as a list when the next path element is a number (so -`vaultik config set age_recipients.0 age1...` works on a file without -`age_recipients`), otherwise as a map. If the config loads, a change -after which it would not, such as `vaultik config set compression_level 99`, -is refused and the file is left unchanged. A config that does not load, -such as the one `config init` writes before `storage_url` is set or one -with a `${ENV:...}` reference to a variable not set in this shell, is -checked one setting at a time: only a change after which the changed -setting cannot be read, such as one giving it the wrong type (a single -value where a list belongs or a word where a number belongs), is refused. +A command refuses to start when the config does not load. The config loads when +all of these hold: -**`snapshot create`**: Perform incremental backup of configured snapshots. -* Optional snapshot names argument to create specific snapshots (default: all) -* On macOS, the terminal application running vaultik needs Full Disk Access - (System Settings → Privacy & Security → Full Disk Access) to read - TCC-protected directories; without it the backup aborts with a permission - error that explains how to fix it -* `--cron`: Silent on total success; warnings and errors are still printed - (for crontab) -* `--prune`: After backup, drop older snapshots of each backed-up name, - remove orphaned blobs older than `prune_grace_period` from remote storage, - and delete each directory under `metadata/` that has no - `manifest.json.zst` once every object in it is older than - `prune_grace_period` and its snapshot is not in the local index, as `prune` - does. By default keeps only the latest - snapshot per name; use `--keep-newer-than` for a rolling window. When - `--skip-errors` skipped a file or directory it could not read or list, the - prune does not run and every older snapshot is kept, since one of them may - be the only one that holds what was skipped; the command says so and exits - non-zero. When the backup found nothing in a configured path, as in the - mount point of a volume that is not mounted, the prune does not run - either: every older snapshot is kept, and the command names the path and - exits non-zero, on every run until the path holds something again. Take a - path that is empty on purpose out of the config, or remove the older - snapshots by hand with `snapshot purge`. When an earlier snapshot of the - same name in the local index found a volume mounted at or below a - configured path (a directory on a different filesystem from its parent, - with an entry in it that the backup does not exclude) and this backup does - not, the prune does not run either: every older snapshot is kept, and the - command names the directory and exits non-zero, on every run until the - volume is mounted again. A backup also does not find the volume when it - finds no such entry in its directory, as when an automounted volume cannot - be mounted, or when that directory is no longer backed up, because it is - excluded or its configured path was removed. If any of this is on purpose, - as for a retired volume, remove the older snapshots by hand with - `snapshot purge`. A file removed during the backup does not stop the - prune. -* `--keep-newer-than `: With `--prune`, keep snapshots newer than - this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). - Without `--prune`, or with a value that is not a valid duration, the - command fails before the backup starts. -* Fails before it writes anything when a `file://` destination directory - does not exist, or when the local index holds snapshots and the - destination answers that it has no `vaultik-destination.txt`. When that - look-up fails for any reason other than not found, for example when S3 - refuses it to credentials that allow only PUT, the check is skipped (see - [remote storage layout](#remote-storage-layout)) -* Fails, before anything is pruned, when a configured path is a symlink - whose target does not exist, as one into a volume that is not mounted +- Every entry in `age_recipients` is an age X25519 public key. +- `snapshots` has at least one name, and every name has at least one path. +- A destination is set, with credentials when it is S3. +- Every other setting below is within its limits. -**`snapshot list`**: Show every snapshot known to this host — the union -of the local index and the backup destination store — with timestamps -and three sizes per snapshot (compressed remote size; total -uncompressed chunk size; size of the chunks that snapshot stored that -were not stored before). Both chunk sizes are recorded when the -snapshot's backup runs. +`vaultik` warns when group or others can read the config file. -Listing the destination store does **not** require the age secret key, -so it works in vaultik's intended configuration, where the backed-up -host holds only the public key. A host that has lost its local index -can still see what it has backed up. +A value written as `${ENV:NAME}` is replaced by the environment variable `NAME`. +This keeps a credential out of the file. -What that host cannot see is a remote-only snapshot's name. The -snapshot ID is hashed at the storage boundary and the manifest records -only the hash, so hostname and snapshot name exist solely in the local -index and in the encrypted per-snapshot database. Snapshots found only -on the destination store are therefore listed as -`>` and show `` in -the uncompressed and "new chunk" columns, which come only from the -local index. Their timestamp and compressed size are real, -read from the manifest. +A leading `~` is expanded to your home directory in snapshot paths, in +`index_path` and in a `file://~/...` destination. -Snapshots in the local index with no counterpart on the destination -store are reported below the table as drift, with the `vaultik prune` -invocation that reconciles them. If the local index is bound to a -different destination than the configured one (after a change to -`storage_url`, or `s3.endpoint`, `s3.bucket` or `s3.prefix`), the -command says so instead and gives the same two ways out that `prune` -gives: revert the configuration, or run `vaultik database delete`. +### Where the config file is -If the destination store cannot be listed (unmounted volume, -permission denied, network down), the command warns, falls back to the -local index alone, and still exits zero. It does the same when the -destination answers that it has no `vaultik-destination.txt` while the -local index holds snapshots stored there, as when a volume mounted at -the destination is unplugged and leaves its empty mount point behind. -* `--json`: Output in JSON format. Each entry carries `locally_tracked` - (whether the snapshot is in the local index), `remote_key` (the full - 64-character storage key), and `remote_present` (whether it was seen - on the destination store, or `null` if the destination could not be - listed). Warnings about an unlistable destination, a local index - bound to a different destination (with both destinations and the - same two ways out), unreadable manifests, and a truncated listing all - go to stderr through the logger, so stdout stays a single parseable - document. +`vaultik` uses the first of these that applies: -**`snapshot verify`**: Verify snapshot integrity. -* Default (shallow): checks that every blob the manifest lists is present in - storage with the size the manifest records, and that the encrypted database is - present. It does not read blob contents. -* `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the - encrypted metadata database -* Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a - remote-only snapshot's remote key (or an unambiguous leading part of it) -* `--json`: Output results as JSON +- the path given with `--config` +- the path in the `VAULTIK_CONFIG` environment variable +- `config.yml` in the platform config directory, which is `~/.config/vaultik/` + on Linux and `~/Library/Application Support/vaultik/` on macOS +- `/etc/vaultik/config.yml` -**`snapshot purge`**: Remove old snapshots based on criteria. Retention is -per-snapshot-name (`--keep-latest` keeps the latest of each name, not the -latest globally). -* `--keep-latest`: Keep only the most recent snapshot of each name -* `--older-than `: Remove snapshots older than duration (e.g. `30d`, - `4w`, `6mo`, `1y`; `m` is minutes, `mo` is months) -* `--snapshot `: Restrict to specific snapshot names (repeat for multiple) -* `--force`: Skip confirmation prompt +A path from `--config` or `VAULTIK_CONFIG` must exist. -**`snapshot remove`**: Remove one snapshot. By default this removes the -snapshot from the local index and strips the snapshot's metadata from -the backup destination store. Blobs are NOT touched — deleting blobs -requires reading every remaining remote manifest (the destination store -may hold snapshots this host doesn't know about), which is what -`vaultik prune` does. The command prints the `vaultik prune` invocation -to run as a follow-up only when it removed the snapshot's metadata from -the destination store, and never under `--json`. An ID that names no -snapshot in the local index or on the destination store is an error; -with `--local-only`, the snapshot must be in the local index. Local row -cleanup (files, chunks, blobs the snapshot was the last referrer for) -runs automatically. If removing the snapshot's metadata from the -destination store fails (for example because the store is unreachable, -or the local index holds snapshots and the destination answers that it -has no `vaultik-destination.txt`), the local-DB removal still completes and a -warning is emitted; run `vaultik snapshot remove ` again -once the store is reachable to remove the snapshot's metadata from it -(`vaultik prune` does not). For a snapshot that is only on the -destination store, that failure is an error. To wipe everything on the -destination in one go, use `vaultik remote nuke --force`. -* `--local-only`: Skip remote cleanup; only touch the local index -* `--dry-run`: Show what would be deleted without deleting -* `--force`: Skip confirmation prompt -* `--json`: Output result as JSON. Also skips the confirmation prompt, as - `--force` does. +### Settings -**`snapshot restore`**: Restore files from a backup snapshot. -* Requires `VAULTIK_AGE_SECRET_KEY` environment variable -* Accepts a snapshot ID, or — for a snapshot only on the destination - store — its remote key (or an unambiguous leading part of it) as shown - by `snapshot list`. See - [restoring on another machine](#restoring-on-another-machine). -* Optional path arguments to restore specific files/directories (default: all). - A path that matches nothing in the snapshot is named in a warning, and the - command exits non-zero after restoring the paths that match. -* Preserves file permissions, timestamps, ownership (ownership requires root), - symlinks, and empty directories -* `--verify`: After restoring, verify every restored file's chunk hashes - match. A file skipped under `--skip-errors` is reported as failed to - restore and is not verified. +- `age_recipients` + - The age public keys that backups are encrypted to. + - Default: none. `snapshot create` needs at least one. No other command + needs one. + - Any one of the matching private keys can decrypt. + - Adding a key later does not re-encrypt what is already stored. See + [Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). +- `age_secret_key` + - The age private key. + - Default: not set. + - Only `snapshot restore` and `snapshot verify --deep` use it. + - The `VAULTIK_AGE_SECRET_KEY` environment variable overrides it. + - Do not set it on a machine you back up. See + [The private key](#the-private-key). +- `snapshots` + - The named sets of paths to back up. Required. + - `snapshots..paths`: the paths to back up under this name. + - `snapshots..exclude`: exclude patterns for this name only. +- `exclude` + - Exclude patterns that apply to every snapshot. See + [Exclude patterns](#exclude-patterns). + - Default: none. +- `storage_url` + - The destination. See [Destinations](#destinations). + - Required, unless the `s3` settings name the destination. +- `s3.access_key_id` and `s3.secret_access_key` + - The S3 credentials. Required for an S3 destination. +- `s3.region` + - The S3 region. Default: `us-east-1`. +- `s3.part_size` + - The size of each part of a multipart upload. Default: `5MiB`. + - It must be between `5MiB` and `5GiB`. + - A blob too large for 10,000 parts of this size is uploaded in larger + parts. +- `s3.endpoint`, `s3.bucket`, `s3.prefix` and `s3.use_ssl` + - An older way to name an S3 destination. They are used only when + `storage_url` is not set. + - `s3.endpoint` and `s3.bucket` are then required. + - `s3.use_ssl: false` selects plain HTTP for an endpoint written without a + scheme. +- `chunk_size` + - The average size of a chunk. Default: `10MB`. Minimum: `1MB`. + - A chunk is between a quarter of this size and four times this size. + - A smaller size finds more duplicate content and makes the index larger. +- `blob_size_limit` + - The largest size of a blob. Default: `10GB`. + - It must be at least four times `chunk_size`. +- `compression_level` + - The zstd compression level, from `1` to `19`. Default: `3`. + - Levels `1` to `2`, `3` to `5`, `6` to `9` and `10` to `19` form four + groups. Levels in one group compress the same way. +- `prune_grace_period` + - How old unused data must be before `prune` deletes it. Default: `24h`. + - It applies to a blob that no manifest lists. It also applies to every + object in a directory under `metadata/` that has no manifest. + - `0s` deletes such data at once. A negative value is refused. + - See [Sharing a destination](#sharing-a-destination). +- `hostname` + - The hostname used in snapshot IDs. Default: the system hostname. + - Only the part before the first `.` is used. +- `index_path` + - The path of the local index. + - Default: `index.sqlite` in the platform data directory. That is + `~/.local/share/vaultik/` on Linux and + `~/Library/Application Support/vaultik/` on macOS. -**`prune`**: Tidy up everything that isn't needed. Runs three passes: -(1) reconcile the local index against the destination store — any -local snapshot whose remote metadata is missing is dropped from the -local index; (2) delete orphaned local rows (files, chunks, blobs no -longer referenced by any completed snapshot); (3) list every remote -manifest on the destination store to compute the still-referenced blob -set and delete any blob not in that set that is older than -`prune_grace_period` (default `24h`). A newer one is kept and counted: it -may belong to a backup still running, on this host or another sharing the -destination store, that has not yet uploaded its manifest. The grace period -does not protect a blob such a backup reuses from an earlier run that stopped -before its manifest: that blob keeps the time of its first upload, so it can -be deleted. Step (3) also deletes each directory under `metadata/` that has -no `manifest.json.zst`, as a backup leaves when it stops after uploading -`db.zst.age` but before its manifest, once every object in it is older than -`prune_grace_period` and its snapshot is not in the local index. An object the -listing gives no time for keeps its directory, and a directory whose name is -not a remote key is left alone. Step (3) reads all remote manifests — network -cost scales with the number of snapshots. -`snapshot create --prune` runs the same cleanup automatically; this is the -manual entry point for the same work. If the destination store cannot -be listed, or the local index holds snapshots and the destination answers -that it has no `vaultik-destination.txt` (an unplugged volume's empty mount -point), prune fails and deletes nothing. `snapshot purge` does the same. If a -blob or such a directory cannot be deleted, the command (and -`snapshot create --prune`) exits non-zero. -* `--force`: Skip confirmation prompt -* `--json`: Output stats as JSON. Also skips the confirmation prompt, as - `--force` does. A blob that cannot be deleted is counted in - `blobs_failed`, and the document is printed before the non-zero exit. - A blob kept for being newer than `prune_grace_period`, or because the - listing of the destination gives no time for it, is counted in - `blobs_too_new` and not in `blobs_found`. A directory under `metadata/` - with no manifest is counted in `metadata_dirs_deleted` once deleted, or - in `metadata_dirs_failed` if it cannot be. +Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB` +and `GiB` are powers of 1024. -**`info`**: Display system configuration, storage settings, encryption -recipients, and local database statistics. +### Environment variables -**`remote info`**: Show storage backend type and location plus detailed -remote storage inventory: per-snapshot metadata sizes, blob counts, and -orphaned blob detection. A name under `metadata/` that is not a remote -key is skipped with a warning and is not printed. If a listed -`manifest.json.zst` cannot be read, or sits under a skipped name, the -orphaned blob figures are reported as unknown; `--json` gives them as -`null`, lists the remote key of each unreadable manifest in -`unreadable_manifests` and counts the manifests under skipped names in -`skipped_manifest_count`. An unreadable manifest also leaves its -snapshot's blob count and blob size unknown, `null` in `--json`. A file -under `blobs/` whose name is not a blob hash is not counted as a blob, -since `prune` never deletes it. -* `--json`: Output as JSON +- `VAULTIK_AGE_SECRET_KEY`: the age private key. See + [The private key](#the-private-key). +- `VAULTIK_CONFIG`: the path of the config file. `--config` overrides it. +- `VAULTIK_INDEX_PATH`: the path of the local index. It overrides `index_path`. +- `RCLONE_CONFIG`: the path of the rclone config file, for an `rclone://` + destination. +- `EDITOR`: the editor that `config edit` starts. +- `TMPDIR`: the directory for temporary files. +- `NO_COLOR`: when set, status lines have no color. +- `VAULTIK_CPUPROFILE`: write a CPU profile to this path. For debugging. +- `VAULTIK_MEMPROFILE`: write a heap profile to this path on exit. For + debugging. -**`remote nuke`**: Delete every object under `metadata/` and `blobs/` on -the backup destination store, whatever its name, and then -`vaultik-destination.txt`, leaving the bucket prefix empty. Destructive -and irreversible. This is the single supported way to wipe the entire -destination store. A snapshot is removed from the local index only once -its metadata is deleted from the destination store. If the destination -store cannot be listed, or the local index holds snapshots and the -destination answers that it has no `vaultik-destination.txt` (an unplugged -volume's empty mount point), the command fails before it removes anything. If a -listing afterwards still finds anything under `metadata/` or `blobs/`, -the command names each object left and exits non-zero; running it again -deletes what is left. -* `--force`: Required to confirm destruction. +### The private key -**`database delete`**: Delete the local SQLite state database file -entirely. Remote storage is unaffected; the next backup will do a full -scan and re-deduplicate against existing remote blobs, and the local -index will re-bind to the currently configured storage destination. -Use this after changing `storage_url`, or `s3.endpoint`, `s3.bucket` or -`s3.prefix`, to a different destination. -* `--force`: Skip confirmation prompt +`snapshot restore` and `snapshot verify --deep` decrypt data, so they need the +age private key. No other command needs it. ---- +Give the key in one of two ways: -## storage backends +- the `VAULTIK_AGE_SECRET_KEY` environment variable +- `age_secret_key` in the config file -vaultik supports three storage backends, selected via the `storage_url` config field: +The environment variable wins when both are set. -**S3** (`s3://bucket/prefix?endpoint=host®ion=us-east-1`): Any S3-compatible -object store. Credentials are read from `s3.access_key_id` and -`s3.secret_access_key` in the config file. +The value can be the whole file that `age-keygen` writes. Comments are ignored, +and every key in the file is used. -**Local filesystem** (`file:///path/to/backup`): Stores blobs and metadata on -a local or mounted filesystem. Useful for testing or backing up to a NAS. The -path must be absolute; a relative one is refused when the config loads. -Percent-escapes in the path are decoded, so `my%20stick` and `my stick` name -the same directory. The directory must exist before the first backup: vaultik -never creates it. A directory on a removable volume is gone while the volume is -not mounted, so a backup then fails instead of writing to the disk under the -mount point. The mount point itself stays behind when the volume is unplugged: -there only `vaultik-destination.txt` stops such a backup, and only while the -local index holds snapshots. +Set the variable from the file, so that the key is not typed into your shell +history: -**Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud -providers. Requires rclone to be configured separately (`rclone config`). - -An upload cut off part-way leaves nothing under the object's name on S3, which -shows an object only once its upload has completed, and on the local filesystem -backend, which writes a temporary file and renames it into place. Rclone -remotes with a server-side move (local and sftp among them) are written under a -temporary name ending in `.partial` and moved into place. Rclone remotes without -one are written in place: where such a remote shows a file while it is still -being written, a killed upload can leave a truncated object under its name, -which a later backup takes for complete. A leftover `.partial` file is ignored -and can be deleted. - -A destination shared between hosts or users, each with its own local index, -relies on `prune_grace_period` to keep one's prune from deleting the blobs of -another's backup still running, or its directory under `metadata/` that has -no manifest yet, so it must be longer than the longest backup run. It does not -protect a blob that backup reuses from an earlier run that -stopped before its manifest, which the prune can delete. To not depend on the -grace period, give each host its own prefix (or directory). See -[locking](#locking). - -Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is -still supported for backward compatibility. `storage_url` takes precedence if -both are set. - ---- - -## architecture - -For an implementation-level view of the internals — the data model, the -`fx` dependency-injection wiring, and the scanner — see -[`ARCHITECTURE.md`](ARCHITECTURE.md). - -### remote storage layout - -``` -// -├── vaultik-destination.txt # Marker written by every backup -├── blobs/ -│ └── // -└── metadata/ - └── / - ├── db.zst.age # Encrypted binary SQLite database - └── manifest.json.zst # Unencrypted blob list (for pruning) +```sh +export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" ``` -* `vaultik-destination.txt` holds one fixed line of text, with no host - name, path or time. While the local index holds snapshots stored - there, a destination that answers a look-up of it with not found is - treated as not present: `snapshot create` refuses to write to it, - `prune`, `snapshot purge` and `remote nuke` fail, and `snapshot list` - and `snapshot remove` behave as for a destination that cannot be - listed. This keeps an unplugged volume's empty mount point from being - taken for a destination whose snapshots are all gone. Mount - the volume if it is removable; if the destination was emptied on - purpose, run `vaultik database delete` and back up again. - When the look-up fails for any reason other than not found, the check - is skipped and the command goes on. S3 refuses it on every run to - credentials that allow only PUT, so a host with such credentials does - not notice a destination emptied since its last backup, and its next - backup would leave out the data its local index records as already - stored. After the destination is emptied, run - `vaultik database delete` on such a host before it backs up again. -* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash -* `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted) - containing all file metadata, chunk mappings, and relationships for the snapshot -* `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling - pruning without the private key +Do not put the private key on a machine you back up. Anyone who breaks into that +machine could then decrypt its backups. Supply the key only on the machine you +restore on. -Snapshot IDs follow the human-readable format -`__` (e.g. -`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the -destination store in plaintext. Each snapshot's metadata directory is named -with its ``, a one-way double SHA-256 hash of the ID, so a plain -listing of the store shows no hostname or snapshot name. The hash uses no -secret, though, so an observer who guesses a candidate hostname and snapshot -name can hash it and confirm the snapshot is present; the remote key keeps -names out of a listing but does not hide them from a guess. The backup time is -not hidden either: manifest.json.zst carries a plaintext timestamp, and object -modification times are visible at the storage layer regardless. For example, -`server1_home_2025-06-01T12:00:00Z` is stored under -`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`. -See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the -derivation, and [Security Considerations](docs/REPOSTRUCTURE.md#security-considerations) -(including [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks)) for what the -format does and does not protect. +### Exclude patterns -### data flow +A pattern excludes a file or a whole directory from a backup. Patterns are +matched against the path below the configured path. -**backup:** +- A pattern that starts with `/` matches from the configured path. In a snapshot + of `~`, `/Library/Caches` matches only `~/Library/Caches`. +- Any other pattern matches at every depth. `.cache` matches every file or + directory named `.cache`. +- `*` and `?` match within one name. `**` matches across directories. +- A pattern that matches a directory excludes everything in it. -1. Open local SQLite index, load known files and chunks into memory -2. Walk source directories, compare mtime/size/mode against index -3. For changed/new files: chunk using content-defined chunking (FastCDC) -4. For symlinks and directories: record metadata (no chunking) -5. For each chunk: hash, check dedup, add to blob packer -6. When blob reaches size threshold: compress (zstd), encrypt (age), upload -7. Build snapshot metadata database, compress, encrypt, upload -8. Create unencrypted blob manifest for pruning support +`exclude` at the top of the config applies to every snapshot. `exclude` under a +snapshot name adds patterns for that snapshot. -**restore:** +## Destinations -1. Download and decrypt `metadata//db.zst.age` -2. Open the binary SQLite database -3. Query files (optionally filtered by paths) -4. Download and decrypt required blobs -5. Extract chunks, reconstruct files -6. Restore permissions, timestamps, ownership, symlinks +`storage_url` selects one of three kinds of destination. -**prune:** - -1. List all snapshot manifests -2. Build set of all referenced blob hashes -3. List all blobs in storage -4. Delete any blob not in the referenced set that is older than - `prune_grace_period` -5. Delete each directory under `metadata/` with no manifest whose objects - are all older than `prune_grace_period`, unless its snapshot is in the - local index - -### chunking and deduplication - -* Content-defined chunking using the FastCDC algorithm -* Average chunk size: configurable (default 10MB) -* Deduplication at file level (unchanged files skipped, unless a chunk - the file lists is held by no uploaded blob) and chunk level (identical - chunks across files stored once) -* Multiple chunks packed into blobs to reduce object count - -### encryption - -* Asymmetric encryption using age (X25519 + ChaCha20-Poly1305) -* Only the public key is needed on the source host -* Each blob and each metadata database is encrypted independently -* Multiple recipients supported (encrypt to multiple keys) - -### compression - -* zstd compression at configurable level (1-19, default 3). The level is - accepted as 1-19 but maps onto zstd's four internal speed presets: - 1-2 fastest, 3-5 default, 6-9 better, 10-19 best. Levels within the - same band compress identically. -* Applied before encryption at the blob level - ---- - -## configuration reference - -Run `vaultik config init` to generate a fully commented config file; a -complete annotated example also lives in -[`config.example.yml`](config.example.yml). Key fields: - -| Field | Default | Description | -|-------|---------|-------------| -| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty | -| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. | -| `snapshots` | (required) | Named snapshot definitions with paths and excludes | -| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) | -| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) | -| `exclude` | | Global exclude patterns (applied to all snapshots) | -| `chunk_size` | `10MB` | Average chunk size for content-defined chunking | -| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` | -| `compression_level` | `3` | zstd compression level (1-19) | -| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once | -| `hostname` | system hostname | Hostname used in snapshot IDs | -| `index_path` | platform data dir | Local SQLite index path | - ---- - -## limitations - -* **No extended attributes (xattrs).** ACLs, macOS Finder metadata, - quarantine flags, SELinux labels, and other extended attributes are not - backed up or restored. -* **No hard link detection.** Two hard links to the same inode are backed - up as independent files. Content deduplication means the data is stored - once, but the hard link relationship is lost on restore. -* **No sparse file support.** Sparse files are fully materialized during - backup. A 100 GB sparse VM disk that is mostly zeros will consume the - full (compressed) size in storage. -* **No bandwidth limiting.** Uploads and downloads use whatever bandwidth - is available. There is no `--bwlimit` flag yet. -* **No parallel blob downloads during restore.** Blobs are fetched - sequentially. Restore speed is bound by single-stream throughput. -* **Device nodes, named pipes, and sockets are silently skipped.** Only - regular files, directories, and symlinks are backed up. -* **Before 1.0, an update can make the local index unusable.** Vaultik - supports schema migrations, but none are added before 1.0 because - there is no installed base yet. If an update leaves your local index - unusable, run `vaultik database delete` and then a full backup; remote - storage is unaffected. After 1.0, the local index is migrated when - vaultik is updated. See - [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). -* **Files that change during backup may be inconsistent.** There is no - filesystem snapshot or freeze. If a file is modified between the scan - and chunk phases, the backed-up copy may reflect a partial write. -* **Ownership restoration requires root.** File uid/gid are recorded - and restored, but `chown` requires elevated privileges. Without root, - files are restored with the current user's ownership. - ---- - -## roadmap - -Work planned after 1.0. Loosely ordered by priority. - -### correctness and operability - -* **Outside security audit.** Before 1.0 the encryption and - blob-generation code was reviewed: every bug the review found was - fixed, and the risks it accepted are listed in - [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks). No outside - audit has been done. age + zstd + content-defined chunking - is mostly off-the-shelf pieces, but the seams (key handling, - recipient parsing, manifest trust boundary, restore-time identity - validation) need an outside read. -* **Verify restored content end-to-end in CI.** The current - integration test does this for a small synthetic snapshot but - not at scale. A nightly job against a multi-GB representative - snapshot would catch silent regressions in the chunker, packer, - or restore planner. - -### performance - -* **Parallel blob downloads during restore.** Single-stream right - now. With a fast S3 endpoint and a multi-core machine restore is - bound by per-blob fetch + decrypt + decompress; running N of - those in parallel against the disk cache would close most of the - remaining gap. Needs to interact correctly with the locality - planner and sweeper. -* **Bandwidth limiting (`--bwlimit`).** Both upload and download. - Useful for backing up over a shared link. Tricky to make work - correctly with the parallel-download story. -* **Restart of interrupted restore.** Today restore is restartable - in the sense that re-running it overwrites partial output; it - doesn't resume from where it stopped or skip already-present - files. A `--resume` mode that checks targets before fetching - blobs would matter for very large restores. -* **Daemon mode.** A long-running mode that watches for file - changes so frequent backups, such as hourly, skip the full scan. - It adds little for the usual runs from cron every 12 to 36 hours. - See [issue #204](https://git.eeqj.de/sneak/vaultik/issues/204). - -### usability - -* **Man pages and richer `--help` examples.** Cobra generates - basic help; man pages would be a separate target. -* **`vaultik snapshot diff `** — show which files changed - between two snapshots without restoring either. -* **Status reporting hook for `--cron`.** When a backup fails - silently in cron, the user has no idea. A configurable - webhook / email / `notify-send` hook on completion (success and - failure) would close the loop. - -### infrastructure - -* **Schema migrations after 1.0.** Migrations are supported, but none - are added before 1.0 because there is no installed base yet. After - 1.0, each schema change is a new migration, so an existing local - index is migrated when vaultik is updated (see - [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations)). -* **Storage backend coverage tests.** S3, file://, and rclone:// - all share the Storer interface but the rclone path is the least - exercised in CI. - ---- - -## output style - -Every command's user-facing output is governed by `internal/ui`, in one -of two ways. Color is enabled when the stream written to is a TTY and -the `NO_COLOR` environment variable is unset (https://no-color.org/). - -* **Status, progress, warnings, and errors** go through the `internal/ui` - message methods below: marker-prefixed, colored on a TTY, and — except - warnings and errors — silenced by `--quiet`. This is the operational - narration of the long-running commands (`snapshot create`, `prune`, - `snapshot restore`, and the like) and the confirmations of - `config init`, `config set`, and `database delete`. -* **The data a command exists to produce** is written plain, with no - marker and no color, because a marker would corrupt a table or a - parsed document. This covers the `version`, `info`, `remote info` and - `snapshot verify` reports, the `snapshot list` table, `config get` - values, and every `--json` document. `--quiet` silences the human - reports and tables (`version`, `info`, `remote info`, `snapshot - verify`, `snapshot list`) but never the machine-consumed `config get` - value or the `--json` documents, which a script depends on. The - confirmation prompts of `database delete`, `prune`, `snapshot purge` - and `snapshot remove`, each with what it asks to delete, are also - written this way and always shown: they are an interactive exchange - the operator must see. - -`internal/ui` writes to stdout; it is the output the user asked for. The -exceptions are the startup banner and the error a failed command ends -with, which go to stderr. Structured log records are a different thing -and go through `internal/log`, which writes to stderr (see "stdout and -stderr" above). - -Message classes: - -| Class | Marker | Alignment | Use for | -|-------|--------|-----------|---------| -| Banner | none | column 0 | The startup line printed once per invocation, on stderr | -| Begin | `》` (white) | column 0 | An operation is about to start (present-continuous verb) | -| Complete | `》` (green) | column 0 | An operation just finished (past-tense verb) | -| Info | `》` (white) | column 0 | Neutral status update | -| Notice | `》` (cyan) | column 0 | Important note that is not a warning | -| Warning | `⚠️ Warning:` (orange/yellow) | column 0 | Recoverable problem | -| Error | `🛑 ERROR:` (red) | column 0 | Operation aborted | -| Progress | ` 》` (white) | column 2 | Heartbeat or per-item status during a long-running operation | -| Detail | ` 》` (white) | column 2 | Continuation/sub-line of a preceding Complete (visually identical to Progress) | - -Conventions: - -* Messages are complete English sentences ending with a period. -* Fully qualify terms — say "backup destination store" instead of - "storage", "snapshot source files enumeration" instead of "scan", - "local index database" instead of "database". -* Every operation that emits a Complete also emits a corresponding - Begin. Operations that print only a Begin (because completion is - obvious from a later Begin) should be rare and intentional. -* Use natural verb tense to signal state: "Uploading" for Begin, - "Uploaded" for Complete. Never write the words "begin" or "complete" - in the body — the marker color already conveys that. -* All elapsed and remaining-time fields are explicitly scoped to their - subject: write "blob upload elapsed: 30s, blob upload ETA: 03:15:00 - (est remain 14s)", never just "elapsed 30s, ETA 14s". -* "ETA" means an absolute clock time (when the operation will finish), - not a remaining-duration. Use `ui.Time()` for the former and - `ui.Duration()` for the latter, and label both. -* `ui.Time` formats same-day times as `HH:MM:SS` and other-day times as - `YYYY-MM-DD HH:MM:SS`. No timezone — local time is implied. - -Value colorizers in `internal/ui` colorize specific value types -consistently. Compose messages from these helpers rather than embedding -ANSI escapes inline: - -| Helper | Color | Use for | -|--------|-------|---------| -| `Hex` | cyan | Blob hashes, chunk hashes (truncated to 12 chars + `...`) | -| `Snapshot` | bold cyan | Snapshot IDs (untruncated) | -| `Path` | blue | Filesystem paths | -| `Size` | magenta | Byte counts (human-readable) | -| `Speed` | magenta | Bytes-per-second rates | -| `Duration` | yellow | Elapsed or remaining time | -| `Time` | yellow | Absolute clock times | -| `Count` | magenta | Integer counts with thousands separators | -| `Percent` | magenta | Percentages | - -When `NO_COLOR` is set or output is not a TTY, all helpers return plain -text and the marker prefixes (`》`, `Warning:`, `ERROR:`) emit without -ANSI escapes. The emoji prefixes on Warning and Error are always emitted -regardless of color setting (emoji are not color). - -## requirements - -* Go 1.26 or later -* Docker, with a reachable daemon, to test, lint, check, or commit: - `script/test` and `script/lint` build the `test` and `lint` phases of - the `Dockerfile`, and `make check` and the pre-commit hook both run - them. A `golangci-lint` installed on `PATH` is not a substitute and is - never used on a host, whatever its version. -* S3-compatible object storage (or local filesystem, or rclone remote) - -## development workflow - -All changes follow this workflow. No exceptions. - -1. Create a feature branch off `main`. -2. Write tests. -3. Write the implementation. -4. Fix implementation errors until it compiles and tests pass. -5. Fix linting errors (`make lint`). -6. Update documentation and README as required by the change. -7. Format code (`make fmt`). -8. Run `make check` (lint + fmt-check + test). Fix any issues. Repeat until clean. -9. Commit on the branch. -10. Merge to `main`. -11. Push. - -Do not commit directly to `main`. Do not skip steps. - -Repository policies for AI agents are in [`AGENTS.md`](AGENTS.md). - -## Entrypoints - -This repository adheres to the -[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) -standard: normalized scripts in `script/` are the entrypoints for the -development workflow, and the Makefile targets are thin shims that call -them. We provide: - -* `script/bootstrap` — install all development dependencies (Go, Go - module download). A host without Go gets the `go.mod` version through - `script/install-go`, in `.tool/go`, which `script/bootstrap` itself, - the `Makefile`, `script/fmt`, `script/fmt-check`, `script/precommit` - and `script/release` add to their `PATH`. It - deliberately does not install `golangci-lint`; see `script/lint` - below. -* `script/setup` — make a fresh clone ready for development: runs - `script/bootstrap`, then `script/install-precommit` -* `script/projectname` — print the project name (used for the Docker - image tag) -* `script/version` — print the version string to bake into the binary. - The `Makefile`'s `LDFLAGS` call this. See [releasing](#releasing) for - the rules. -* `script/install-goreleaser` — install the pinned `goreleaser` into - `.tool/bin` from a sha256-verified release archive. Idempotent, and - called by `script/bootstrap`; the release workflow calls it directly - because it needs `goreleaser` but not the Docker daemon - `script/bootstrap` insists on. -* `script/install-go` — install the Go toolchain named by `go.mod`'s - `go` directive into `.tool/go` from a sha256-verified `go.dev` - archive, for Linux or macOS on amd64 or arm64. Idempotent. On a CI - runner it also puts `.tool/go/bin` on `PATH` for the steps that - follow. Called by the release workflow, which needs a host Go for - `goreleaser` to shell out to, and by `script/bootstrap` on a host - without Go. `actions/setup-go` is not used because it verifies the - downloaded toolchain against no value in this repo. Bumping Go edits - `go.mod`, the checksums in this script, and the two `golang` digests - in the `Dockerfile` together. -* `script/release` — cross-compile and publish the release artifacts - with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose - version is not the pinned one, because a different version would build - a different release from the same tag. -* `script/release-snapshot` — the same build with no publishing and no - tagging, into `./dist` -* `script/test` — run the test suite by building the `test` phase of - the `Dockerfile` (verbose rerun on failure). This runs *everything*: - there is no separate integration target and no build-tagged subset - held back, so the full round-trip tests in - `internal/vaultik/integration_test.go` run on every invocation. The - 90s `-timeout` is a hang backstop rather than a performance budget; it - applies per test binary, to test execution only. -* `script/lint` — lint by building the `lint` phase of the - `Dockerfile`, which runs `golangci-lint config verify` and then - `golangci-lint run --config .golangci.yml ./...` as build steps in - the digest-pinned `golangci-lint` image. Nothing lints on the host. - That `FROM` line is the only pin of the linter version, and it changes - in the same commit as a re-vendored `.golangci.yml`. -* `script/lint-fix` — apply the linter's autofixes (rewrites files), - using the same pinned image, parsed out of the `lint` phase's `FROM` - line. It cannot be a build step, because fixes have to land in the - worktree, so it bind-mounts the tree into a `docker run` and therefore - needs a *local* daemon. It is a developer convenience and never a - gate: no gate reads its exit status. Run `make lint` afterwards to - find out whether the tree is clean. -* `script/fmt` — format all code (writes) -* `script/fmt-check` — check formatting (read-only). It runs `gofmt` on - the host, over every Go file outside `.tool`. -* `script/check` — run `script/test`, `script/lint`, and - `script/fmt-check`. -* `script/docker` — build the Docker image tagged via - `script/projectname`, stamped with the version - `git describe --tags --always --dirty` gives on the host, or `unknown` - outside a git checkout. The image's build stage depends on the `lint` - and `test` phases, so this lints and tests too. -* `script/cibuild` — CI entrypoint: runs `script/bootstrap`, - `script/check`, and then the same image build as `script/docker`. - `.gitea/workflows/check.yml` runs it on every push. - - Every `docker build` in these scripts passes `--no-cache`, because on - an unchanged tree a cached check layer is replayed without running and - the build still exits 0. A plain `docker build .` is therefore no - evidence that the checks ran. The cost is that `script/cibuild` runs - the `lint` and `test` phases twice: once in `script/check` and again - in the image build. -* `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must - not change files), then `script/check` -* `script/install-precommit` — install the git pre-commit hook that - runs `script/precommit` - -## releasing - -### version numbers - -The version a binary reports comes from git, not from a constant in a -file. It is `git describe --tags --always --dirty`, which -`script/version` runs for the `Makefile`, and `script/docker` and -`script/cibuild` run themselves: - -* `HEAD` is exactly on a tag → that tag, such as `v1.0.0`. -* a commit after a tag → `--g`. -* no tag reachable → the short commit sha. -* any of these, with uncommitted changes to tracked files → a `-dirty` - suffix, because a modified checkout of a tag is not that tag. - -A `docker build .` of a clone, with no build arguments, runs the same -`git describe` (without `--dirty`) on the `.git` in its build context, -so it stamps the same value for a clean commit; the build fails if the -context carries `.git` and no version, commit or commit date comes out. -A binary built without -git metadata reports `dev`, or `unknown` when `script/docker` or -`script/cibuild` built it outside a git checkout. - -`goreleaser` stamps a release binary with the tag minus its leading -`v`, so the tag `v1.0.0` produces `vaultik 1.0.0`, matching the archive -name `vaultik_1.0.0_linux_amd64.tar.gz`. `goreleaser --snapshot` stamps -`dev-<12 chars of the commit sha>` rather than inventing the next patch -number. `vaultik version` calls a build a development build when its -version is `dev`, `unknown`, `dev-`, the short commit sha or -`--g`, with or without `-dirty`; only a plain tag, -such as `v1.0.0` or `1.0.0`, is a release. If `script/version` cannot -be run at all, `make` stops with an error instead of building an -unversioned binary, and a binary that somehow carries an empty version -string still reports itself as a development build. - -### cutting a release - -Releases are cut by CI from a tag, not from a workstation: +### S3 ``` -git tag -a v1.2.3 -m 'v1.2.3' -git push origin v1.2.3 +s3://bucket/prefix?endpoint=s3.example.com®ion=us-east-1 ``` -`.gitea/workflows/release.yml` triggers on `v*` tags, installs a Go -toolchain and the pinned `goreleaser`, and runs `script/release`, which -builds -`linux,darwin × amd64,arm64` archives plus `checksums.txt` and publishes -them to this repository's Gitea releases as a draft. `.goreleaser.yaml` -has a `gitea_urls:` block pointing at `https://git.eeqj.de/api/v1`; -without it `goreleaser` would talk to the GitHub API. +Any S3-compatible object store works. -The workflow needs one repository Actions secret: +- `endpoint` is the host of the store. The default is `s3.amazonaws.com`. +- `region` overrides `s3.region`. +- `ssl=false` selects plain HTTP for an endpoint written without a scheme. +- No other parameter is accepted. -| Secret | What it is | -| --------------- | ------------------------------------------------------------------------------------------------------- | -| `RELEASE_TOKEN` | A Gitea access token with `write:repository` scope, owned by an account that can publish releases here. | +The credentials come from `s3.access_key_id` and `s3.secret_access_key` in the +config. A URL that carries credentials is refused. -It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic -token is deliberately not used: it is not guaranteed to carry release -write access. - -The Go toolchain that compiles the released binaries is installed by -`script/install-go`, which downloads the version named by `go.mod` -(currently `1.26.1`, the same version the `Dockerfile` builder stage -pins by digest) and refuses the archive unless its sha256 matches the -value committed in the script. `goreleaser` shells out to `go` for every -cross-compile, so without that step the release would either fail -outright or ship binaries built by whatever unpinned toolchain the -runner happened to carry. - -To rehearse the whole build without publishing or tagging anything: +### Local directory ``` -make release-snapshot +file:///path/to/backup ``` -Artifacts land in `./dist`, which is gitignored. +- The path must be absolute. +- A percent-escape in the path is decoded, so `my%20stick` names `my stick`. +- The directory must exist before the first backup. `vaultik` never creates it. -Release artifacts are not signed, carry no SBOM, and are not built -reproducibly; the archives contain the binary, `LICENSE`, and -`README.md` only (no shell completions or man page). +A directory on a removable volume is gone while the volume is not mounted. A +backup then fails. It does not write to the disk under the mount point. + +This does not hold when the destination is the mount point itself. The empty +mount point stays behind when the volume is unplugged. There only +[the marker file](#the-marker-file) stops the backup. + +### rclone + +``` +rclone://remote/path +``` + +This uses any storage provider that rclone supports. Define the remote first +with `rclone config`. `vaultik` reads rclone's own config file. + +The URL takes no parameters and no credentials. + +### Interrupted uploads + +An upload that is cut off must not leave a partial object under the final name. +A later backup would take that object for complete. + +- S3 shows an object only once its upload has finished. +- The local directory destination writes a temporary file and then renames it. +- An rclone remote that can move objects is written the same way. The temporary + name ends in `.partial`. +- An rclone remote that cannot move objects is written in place. On such a + remote, a killed upload can leave a partial object under the final name. + +A leftover `.partial` file is ignored. You can delete it. + +### Destination permissions + +Each command needs different permissions at the destination. + +- `snapshot create` needs only permission to write objects. A machine you back + up can therefore hold credentials that can neither read nor delete. +- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list + and read. +- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete. +- `prune` and `snapshot create --prune` list, read and delete. + +A machine that runs `snapshot create --prune` can delete its own backups. To +keep it to write-only credentials, run `prune` from a separate, trusted machine. + +A backup tries to look up the marker file and each blob before it writes. It +goes on when the destination refuses those look-ups. + +### The marker file + +Every backup writes `vaultik-destination.txt` at the top of the destination. The +file holds one fixed line of text. + +The marker tells an emptied or missing destination apart from a new one. An +unplugged volume leaves an empty directory at its mount point. Without the +marker, that directory would look like a destination whose snapshots are all +gone. + +The check applies while the local index holds snapshots. When the destination +then answers that the marker does not exist, commands treat the destination as +not present: + +- `snapshot create` refuses to write to it. +- `prune`, `snapshot purge` and `remote nuke` fail. +- `snapshot list` shows the local index alone. +- `snapshot remove` deletes nothing at the destination. + +To fix this, mount the volume. If you emptied the destination on purpose, run +`vaultik database delete` and back up again. + +The check is skipped when the look-up fails for any other reason. S3 refuses the +look-up to credentials that can only write. A machine with such credentials +therefore does not notice an emptied destination. Its next backup would leave +out the data its local index records as stored. After you empty a destination, +run `vaultik database delete` on such a machine before it backs up again. + +### Changing the destination + +The local index belongs to one destination. It records which blobs are stored +there. + +The first command that changes data records the configured destination in the +local index. Later, these commands refuse to run when the config names a +different destination: + +- `snapshot create` +- `snapshot purge` +- `snapshot remove` +- `prune` +- `remote nuke` + +The destination is `storage_url`, or else `s3.endpoint`, `s3.bucket` and +`s3.prefix`. + +There are two ways out: + +- Change the config back to the earlier destination. +- Run `vaultik database delete`. The next backup then starts a new index for the + new destination. + +### Sharing a destination + +Give each machine and each user its own destination, such as its own prefix or +directory. The rest of this section applies only when two of them share one. + +A prune from one machine can meet a backup from another that is still running. A +backup uploads its blobs first, then its database, and its manifest last. Until +the manifest exists, no manifest lists the new blobs. + +`prune_grace_period` protects those blobs. A prune keeps a blob that no manifest +lists until the blob is older than the grace period. It keeps a directory under +`metadata/` that has no manifest for the same time. Set the grace period longer +than the longest backup of any machine or user that shares the destination. + +The grace period does not protect one case. A backup does not upload a blob that +is already at the destination. Such a blob keeps the time of its first upload. +It can be left over from an earlier backup that stopped before its manifest. A +prune from another machine can then delete that blob while the new backup runs. +The new snapshot cannot be restored afterwards. + +`remote nuke` ignores the grace period. + +## Locking + +Commands that change data take a lock, so that only one of them runs at a time. +These commands take it: + +- `snapshot create` +- `snapshot purge` +- `snapshot remove` +- `prune` +- `remote nuke` + +A second one does not wait. It exits at once with an "already running" error. + +The lock is the file `vaultik.pid` in the platform data directory. That is +`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/` +on macOS. + +The lock covers one user on one machine. It does not stop another user or +another machine from writing to the same destination. See +[Sharing a destination](#sharing-a-destination). + +No other command takes the lock. `snapshot list`, `snapshot verify`, +`snapshot restore`, `info` and `remote info` run while a backup is in progress. + +## Output + +### stdout and stderr + +stdout carries the output you asked for: + +- status and progress lines +- tables and reports +- JSON documents +- `config get` values +- completion scripts + +stderr carries the rest: + +- the startup banner +- log records, including those from `--verbose` and `--debug` +- the error that a failed command ends with + +So `vaultik snapshot list > out.txt` captures the list and leaves the +diagnostics on your terminal. + +Log records are colored lines when stderr is a terminal. They are JSON, one +object per line, when stderr is redirected. + +Status lines are colored only on a terminal, and never when `NO_COLOR` is set. + +### `--json` + +Under `--json`, stdout holds one JSON document and nothing else. The banner and +the status lines are left out, and warnings go to stderr. A command can +therefore be piped as it is: + +```sh +vaultik snapshot list --json | jq . +``` + +### `--quiet` and `--cron` + +`--quiet` leaves out the banner, the status lines, the tables and the reports. +`--cron` on `snapshot create` does the same. + +Neither flag hides these: + +- warnings and errors +- `config get` values +- JSON documents +- confirmation prompts + +### Exit status + +- `0`: the command succeeded. +- `1`: the command failed. +- `130`: the command was interrupted by Ctrl-C or `SIGTERM`. + +An interrupted command removes its temporary files before it exits. It gives up +after 30 seconds. + +## Temporary files + +Temporary files go to `$TMPDIR`, or to `/tmp` when `TMPDIR` is not set. + +A backup needs free temporary space for these: + +- One blob at a time. Each blob is written whole to a temporary file before it + is uploaded, so allow about `blob_size_limit`. +- About twice that for an rclone remote that cannot stream uploads. +- Copies of the local index, made when the snapshot's database is prepared. + +With `TMPDIR` not set, SQLite writes one of those index copies to `/var/tmp`. + +A restore keeps the downloaded blobs on disk, decrypted, until no remaining file +needs them. `snapshot restore` and `snapshot verify --deep` also write the +decrypted snapshot database there. + +These files are readable only by their owner, and are removed when the command +ends. A `kill -9` or a power loss leaves them behind. Delete any leftover +`vaultik-*` entry in the temporary directory yourself. + +## What the destination reveals + +File contents, names, paths and metadata are encrypted. Someone who can read the +destination still learns these: + +- when each backup finished +- how many blobs each snapshot uses, and their sizes +- which blobs two snapshots share +- whether a hostname and snapshot name they guess exist there + +The last point holds because a remote key is a plain hash of the snapshot ID, +made without a secret. + +[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the +destination, what it protects and the risks it accepts. + +## Limitations + +- **No extended attributes.** ACLs, macOS Finder metadata, quarantine flags and + SELinux labels are not backed up. +- **No hard links.** Two hard links to one file are backed up as two files. The + content is stored once, but a restore creates two separate files. +- **No sparse files.** A sparse file is backed up at its full size, zeros + included, and restored as a full file. +- **No bandwidth limit.** Uploads and downloads use all the bandwidth there is. +- **One download at a time.** A restore downloads blobs one after another. +- **No device nodes, named pipes or sockets.** They are skipped without a + message. +- **No consistent view of changing files.** A file that changes while it is read + can be stored in a mixed state. +- **Owners need root.** A restore sets owners and groups only when it runs as + root. +- **No resume.** An interrupted restore starts again from the beginning. +- **The local index can become unusable after an update.** This applies before + version 1.0. Run `vaultik database delete` and then a full backup. The + destination is not affected. See + [Schema migrations](docs/DATAMODEL.md#schema-migrations). + +## Development + +[ARCHITECTURE.md](ARCHITECTURE.md) describes how the code is organised. +[docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index. +[`AGENTS.md`](AGENTS.md) holds the rules for AI agents. +[`REPO_POLICIES.md`](REPO_POLICIES.md) holds the repository policies. + +### Requirements + +- Go 1.26.1 or later. +- Docker, with a daemon you can reach. The tests and the linter run only inside + Docker. + +### Workflow + +- Make every change on a feature branch. +- Write the tests with the change. +- Update the documentation that the change affects. +- Run `make fmt`, then `make check`, before each commit. + +`make check` runs the tests, the linter and the format check. All three must +pass. + +### Entrypoints + +The scripts in `script/` are the entrypoints. The `Makefile` targets only call +them. The layout follows +[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all). + +- `script/bootstrap`: installs what development needs. It installs Go from + `script/install-go` when the machine has none, and it fails without Docker. +- `script/setup`: prepares a fresh clone. It runs `script/bootstrap` and + `script/install-precommit`. +- `script/test`: runs the whole test suite, by building the `test` stage of the + `Dockerfile`. +- `script/lint`: runs the linter, by building the `lint` stage of the + `Dockerfile`. The linter never runs on your machine. +- `script/lint-fix`: applies the linter's automatic fixes to your files. It + needs a local Docker daemon. Run `script/lint` afterwards. +- `script/fmt`: formats the Go code. +- `script/fmt-check`: checks the formatting and changes nothing. +- `script/check`: runs `script/test`, `script/lint` and `script/fmt-check`. +- `script/precommit`: runs `go mod tidy`, `go fmt` and `script/check`. The git + pre-commit hook calls it. +- `script/install-precommit`: installs that hook. +- `script/docker`: builds the Docker image. The build runs the linter and the + tests too. +- `script/cibuild`: what CI runs on every push. It runs `script/bootstrap`, + `script/check` and the image build. +- `script/version`: prints the version to build into the binary. +- `script/projectname`: prints the project name, which tags the Docker image. +- `script/install-go`: installs the Go version that `go.mod` names into + `.tool/go`. It checks the download against a checksum in the script. +- `script/install-goreleaser`: installs the pinned `goreleaser` into + `.tool/bin`, checked the same way. +- `script/release`: builds and publishes the release files. +- `script/release-snapshot`: builds the release files into `./dist` and + publishes nothing. + +Every `docker build` in these scripts uses `--no-cache`. A cached layer would +skip the tests or the linter and still report success. + +The test suite has no separate integration target. Every test runs every time. +Each test binary is stopped after 90 seconds. + +To change the Go version, change these together: `go.mod`, the version and +checksums in `script/install-go`, and the two `golang` image digests in the +`Dockerfile`. + +### Messages to the user + +Status lines, warnings and errors go through `internal/ui`. The data a command +exists to print is written plain, with no marker and no color. That covers +tables, reports, JSON documents and `config get` values. + +`internal/ui` has these kinds of message: + +- **Banner**: the startup line. No marker. +- **Begin**: an operation is starting. Marker `》`. +- **Complete**: an operation has finished. Marker `》`, in green. +- **Info**: a neutral status line. Marker `》`. +- **Notice**: a note the user should see. Marker `》`, in cyan. +- **Warning**: a problem the command survives. Prefix `⚠️ Warning:`. +- **Error**: the command stops. Prefix `🛑 ERROR:`. +- **Progress** and **Detail**: an indented line under another. Marker ` 》`. + +Messages follow these conventions: + +- A message is a full sentence that ends with a period. +- Name things in full. Write "backup destination store", not "storage". +- An operation that prints a Complete also prints a Begin. +- Use the tense to show the state: "Uploading" for a Begin, "Uploaded" for a + Complete. +- Say what a time belongs to, as in "blob upload elapsed: 30s". +- "ETA" is a clock time, not a remaining time. Print it with `ui.Time`, and + print a remaining time with `ui.Duration`. + +Build a message from the value helpers in `internal/ui`, and never write color +codes by hand: + +- `Hex`: a blob or chunk hash, cut to 12 characters. +- `Snapshot`: a snapshot ID. +- `Path`: a filesystem path. +- `Size`: a byte count. +- `Speed`: a transfer rate, in bits per second. +- `Duration`: an elapsed or remaining time. +- `Time`: a clock time, in local time. +- `Count`: a whole number, with thousands separators. +- `Percent`: a percentage. + +## Releasing + +### Version numbers + +The version a binary reports comes from git. It is the output of +`git describe --tags --always --dirty`: + +- On a tagged commit, it is the tag, such as `v1.0.0`. +- On a commit after a tag, it is `--g`. +- With no tag, it is the short commit. +- With uncommitted changes to tracked files, `-dirty` is added. + +`script/version` gives this value to the `Makefile`. `script/docker` and +`script/cibuild` compute it themselves. + +A binary built without git reports `dev`. An image built by `script/docker` or +`script/cibuild` outside a git checkout reports `unknown`. + +`goreleaser` builds the release binaries. It uses the tag without its leading +`v`, so the tag `v1.0.0` gives `vaultik 1.0.0`. `script/release-snapshot` stamps +`dev-` followed by 12 characters of the commit. + +`vaultik version` calls a build a release only when its version is a plain tag, +such as `v1.0.0` or `1.0.0`. Every other version is a development build. + +`make` stops with an error when `script/version` gives no version. + +### The release workflow + +`.gitea/workflows/release.yml` runs when a tag that starts with `v` is pushed. +It does this: + +- It installs Go with `script/install-go` and `goreleaser` with + `script/install-goreleaser`. +- It runs `script/release`. +- `script/release` builds archives for Linux and macOS on `amd64` and `arm64`, + and a `checksums.txt`. +- It publishes them to this repository's releases, as a draft. + +The workflow needs one repository secret, `RELEASE_TOKEN`. It is a Gitea access +token with the `write:repository` scope, from an account that can publish +releases here. + +`script/release` refuses a `goreleaser` on `PATH` that is not the pinned +version. + +`make release-snapshot` builds the same files into `./dist` and publishes +nothing. + +Each archive holds the binary, `LICENSE` and `README.md`. The release files are +not signed, have no SBOM and are not built reproducibly. ## license