From 8058b775d6c257d451a7cf70af801c095cfee297 Mon Sep 17 00:00:00 2001 From: sneak Date: Fri, 6 Nov 2026 18:21:23 +0000 Subject: [PATCH] Rewrite README.md sentence by sentence Rewritten from scratch against the code on next: what the program is, how to use it, a command reference that says what each command needs, the configuration reference as nested bullets, then development and releasing. snapshot verify --deep now says it needs the private key, from VAULTIK_AGE_SECRET_KEY or age_secret_key, as snapshot restore does. The roadmap section is removed: it described features that are not built. Model: fable-5-1 --- README.md | 2204 +++++++++++++++++++++++++++++------------------------ 1 file changed, 1214 insertions(+), 990 deletions(-) diff --git a/README.md b/README.md index 06a531b..21d55a9 100644 --- a/README.md +++ b/README.md @@ -1,1120 +1,1345 @@ # vaultik (ваултик) -`vaultik` is an incremental backup tool written in Go. It encrypts data using an -`age` public key and uploads each encrypted blob directly to a remote -S3-compatible object store. It requires no private keys, secrets, or credentials -(other than those required to PUT to encrypted object storage, such as S3 API -keys) stored on the backed-up system. +`vaultik` is an incremental backup tool. It encrypts your files with a public +key and uploads them to an S3-compatible object store, a local directory or an +rclone remote. -## quickstart +The machine being backed up holds only the public key. It cannot decrypt its own +backups. You keep the private key somewhere else and need it only to restore. -```sh -# install -go install sneak.berlin/go/vaultik/cmd/vaultik@latest +## Why -# create a default config file (prints the path it wrote to) -vaultik config init +Other backup tools, such as `restic`, `borg` and `duplicity`, are designed for +machines that can hold a decryption key. `vaultik` is for machines that should +not hold one. -# generate an age keypair; keep the private key file somewhere safe and -# offline — you need it to restore, and the backed-up machine does not need it -age-keygen -o vaultik_backup_private_key.txt -grep 'public key' vaultik_backup_private_key.txt - -# configure the encryption key and backup destination -vaultik config set age_recipients.0 age1YOUR_PUBLIC_KEY_HERE -vaultik config set storage_url "file:///Volumes/usbstick/mybackup" - -# create the destination directory; vaultik never creates it, so a backup -# to a volume that is not plugged in fails instead of filling the disk -mkdir /Volumes/usbstick/mybackup - -# macOS only: grant your terminal app Full Disk Access first -# (System Settings → Privacy & Security → Full Disk Access), otherwise -# the backup will abort with a permission error on protected directories - -# run your first backup (the default config backs up ~, and /Applications -# on macOS only, with sensible excludes) -vaultik snapshot create - -# see what you have -vaultik snapshot list -``` - -Features: - -- modern encryption ([age](https://age-encryption.org/), X25519 + - ChaCha20-Poly1305) -- content-defined chunking with deduplication (FastCDC) -- incremental backups (a file is re-chunked only when it changed or a chunk it - lists is held by no uploaded blob) -- multithreaded zstd compression at configurable levels -- content-addressed immutable storage -- local state tracking in SQLite (enables write-only incremental backups) -- no mutable remote metadata -- no plaintext file paths or metadata in remote storage -- packs small files into large blobs (keeps S3 operation counts down) -- backs up regular files, symlinks, empty directories, and file permissions -- pluggable storage backends: S3, local filesystem, rclone (70+ providers) -- pure Go (no CGO), cross-compiles to linux/darwin × amd64/arm64 - -## why - -Other backup tools like `restic`, `borg`, and `duplicity` are designed for -environments where the source host can store secrets and has access to -decryption keys. `vaultik` is for environments where you don't want to store -backup decryption keys on your hosts — only public keys for encryption. - -Requirements that no existing tool meets: +It was written to meet these requirements together: - open source -- no passphrases or private keys on the source host +- no passphrase or private key on the backed-up machine - incremental - compressed - encrypted -- s3 compatible without an intermediate step or tool +- direct upload to S3-compatible storage, with no intermediate step or tool -## daily use +## Features + +- Encryption with [age](https://age-encryption.org/) (X25519 and + ChaCha20-Poly1305), to one or more public keys. +- Incremental backups. A file is read again only when it has changed. +- Deduplication. Identical content is stored once, across files and across + snapshots. +- zstd compression at a level you choose. +- Small files are packed together into large objects. This keeps the number of + S3 requests low. +- File contents, names, paths and metadata are encrypted at the destination. +- Three kinds of destination: S3, a local directory and any rclone remote. +- Regular files, symlinks and directories are backed up, with their permissions, + owners and modification times. +- Pure Go, with no C code. It builds for Linux and macOS on `amd64` and `arm64`. + +## Terms + +- **Snapshot**: one backup of the paths listed under one name in the config, + taken at one time. +- **Snapshot ID**: the name of a snapshot, in the form + `__`. An example is + `server1_home_2025-06-01T12:00:00Z`. +- **Destination**: the place the backups are stored. +- **Chunk**: a piece of a file's content. `vaultik` cuts each file into chunks + and stores each distinct chunk once. +- **Blob**: one object at the destination. It holds many chunks, compressed and + then encrypted. +- **Local index**: a SQLite database on the backed-up machine. It records what + is already stored, so the next backup uploads only what changed. +- **Manifest**: the list of blobs one snapshot uses. It is stored at the + destination without encryption. +- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory + at the destination, so the destination never shows the snapshot ID. + +## Install ```sh -# verify a snapshot (shallow: checks all blobs are present with the listed size) +go install sneak.berlin/go/vaultik/cmd/vaultik@latest +``` + +This needs Go 1.26.1 or later. Generating a key pair needs `age-keygen`, which +comes with [age](https://age-encryption.org/). + +## Quick start + +```sh +# Write a default config file. The command prints the path it wrote. +vaultik config init + +# Generate a key pair. Keep this file somewhere safe, away from this machine. +age-keygen -o vaultik_backup_private_key.txt +grep 'public key' vaultik_backup_private_key.txt + +# Set the public key and the destination. +vaultik config set age_recipients.0 age1YOUR_PUBLIC_KEY_HERE +vaultik config set storage_url "file:///Volumes/usbstick/mybackup" + +# Create the destination directory. vaultik never creates it. +mkdir /Volumes/usbstick/mybackup + +# Run the first backup. +vaultik snapshot create + +# See what is stored. +vaultik snapshot list +``` + +The default config is written for macOS. It backs up your home directory and +`/Applications`, with a list of excludes. A configured path that does not exist +stops the backup, so on another system run `vaultik config edit` first and +change the `snapshots` section. + +On macOS, grant your terminal app Full Disk Access before the first backup. The +setting is under System Settings, Privacy & Security, Full Disk Access. Without +it the backup stops with a permission error on a protected directory. + +## Everyday use + +```sh +# Check that a snapshot's blobs are all present, with the listed sizes. vaultik snapshot verify -# put the private key file in the environment (reading it from the file -# keeps the key out of your shell history); the whole age-keygen file, -# with one or more identities, is accepted +# Put the private key in the environment. Reading it from the file keeps it +# out of your shell history. export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" -# deep verify (downloads every blob, decrypts it, and re-hashes it to -# detect corruption — this checks integrity, not who wrote the blob) +# Download, decrypt and re-hash every blob of a snapshot. vaultik snapshot verify --deep -# restore (requires the private key). Restore into a new directory you own, -# writable only by you — not a shared location like /tmp +# Restore a snapshot into a new directory that only you can write. vaultik snapshot restore ~/vaultik-restore -# daily cron job: back up, keep a 4-week rolling window of snapshots +# A daily cron line: back up, then keep four weeks of snapshots. # 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w ``` -On a destination shared between hosts or users, the prune in that cron line -relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup -another one is still running, and its directory under `metadata/` that has no -manifest yet, so it must be longer than the longest backup run. It does not -protect a blob that backup reuses from an earlier run that stopped before its -manifest: the prune can delete such a blob, and the new snapshot then cannot be -restored. To not depend on the grace period, give each host its own prefix or -directory. See [locking](#locking). +`--prune` deletes data at the destination. Read +[Sharing a destination](#sharing-a-destination) before you use it on a +destination that more than one machine or user writes to. -## restoring on another machine +## Restoring on another machine -Restoring on a host that never ran the backup — a replacement machine after the -original is gone — is the case vaultik is built for. That host needs only three -things: the `vaultik` binary, the age **private** key, and the storage -credentials for the destination. It does **not** need the local index, the -original config file, or the original hostname. +You can restore on a machine that never ran the backup, such as a replacement +for a lost one. That machine needs three things: + +- the `vaultik` binary +- the age private key +- the destination, and credentials that can list and read it + +It does not need the original local index, config file or hostname. ```sh -# install -go install sneak.berlin/go/vaultik/cmd/vaultik@latest - -# create a config and point it at the ORIGINAL backup destination +# Create a config and point it at the original destination. vaultik config init vaultik config set storage_url "s3://bucket/prefix?endpoint=https://s3.example.com" vaultik config set s3.access_key_id "..." vaultik config set s3.secret_access_key "..." -# see what is on the destination store +# See what the destination holds. This needs no private key. vaultik snapshot list ``` -`snapshot list` reads the destination store without the private key. A snapshot -that is not in this host's (empty) local index is shown as remote-only: its row -is identified by `` rather than by a `hostname_name_timestamp` -name, because the name lives only in the local index and the encrypted database -and cannot be recovered from the store. Its timestamp and compressed size are -real. (See the `snapshot list` description under -[command details](#command-details) for the full explanation.) +On this machine the local index is empty, so the list cannot show snapshot IDs. +Each snapshot appears as ``, with the first 12 characters of +its remote key. The snapshot ID is stored only in the local index and in the +snapshot's encrypted database. -Use that remote key — the hex printed inside ``, or the full -`remote_key` from `snapshot list --json` — to restore and verify: +Use that remote key in place of the snapshot ID: ```sh -# put the private key file in the environment (reading it from the file -# keeps the key out of your shell history) export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" -# restore everything to a new directory you own (writable only by you, not a -# shared location like /tmp), then check every restored file's chunk hashes +# Restore everything, then check the chunk hashes of every restored file. vaultik snapshot restore --verify ~/vaultik-restore -# optionally, deep-verify the snapshot against the store (downloads every -# blob, decrypts it, and re-hashes it to detect corruption — this checks -# integrity, not who wrote the blob) +# Optionally, also check every blob of the snapshot at the destination. vaultik snapshot verify --deep ``` -`age_recipients` (the public key) is not needed to restore — only the private -key in `VAULTIK_AGE_SECRET_KEY`. Both the abbreviated key printed in the table -and the full 64-character key from `--json` are accepted; a leading part of the -key is enough as long as it is unambiguous. +Any leading part of the remote key works, as long as it matches one snapshot. +`snapshot list --json` gives the full 64-character key as `remote_key`. ---- +A machine that only restores does not need `age_recipients`. -## cli - -### commands +## Commands ```sh -vaultik [--config ] config init -vaultik [--config ] config edit -vaultik [--config ] config get -vaultik [--config ] config set -vaultik [--config ] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than ] -vaultik [--config ] snapshot list [--json] # alias: ls -vaultik [--config ] snapshot verify [--deep] [--json] -vaultik [--config ] snapshot purge [--keep-latest | --older-than ] [--snapshot ...] [--force] -vaultik [--config ] snapshot remove [--dry-run] [--force] [--local-only] [--json] # alias: rm -vaultik [--config ] snapshot restore [paths...] [--verify] -vaultik [--config ] prune [--force] [--json] -vaultik [--config ] info -vaultik [--config ] remote info [--json] -vaultik [--config ] remote nuke --force -vaultik [--config ] database delete [--force] +vaultik config init +vaultik config edit +vaultik config get +vaultik config set +vaultik snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than ] +vaultik snapshot list [--json] +vaultik snapshot verify [--deep] [--json] +vaultik snapshot purge (--keep-latest | --older-than ) [--snapshot ...] [--force] +vaultik snapshot remove [--dry-run] [--force] [--local-only] [--json] +vaultik snapshot restore [paths...] [--verify] +vaultik prune [--force] [--json] +vaultik info +vaultik remote info [--json] +vaultik remote nuke --force +vaultik database delete [--force] vaultik completion vaultik version ``` -### global flags +`snapshot list` has the alias `snapshot ls`. `snapshot remove` has the alias +`snapshot rm`. -- `--config `: Path to config file (default: `$VAULTIK_CONFIG`, then - platform config dir, then `/etc/vaultik/config.yml`) -- `--verbose`, `-v`: Enable verbose output (on stderr — see below) -- `--debug`: Enable debug output (on stderr — see below) -- `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner) -- `--skip-errors`: Skip files that cannot be read when creating a snapshot, or - that cannot be restored when restoring, instead of aborting. Packing and - storage errors while creating a snapshot (which would leave a chunk recorded - but not stored) still abort the run. A run that skipped anything still - finishes but exits non-zero: `snapshot create` keeps the snapshot and counts - each file or directory it skipped as failed. A file removed during the backup - is also counted as failed but does not change the exit status. +### What every command needs -### locking +`config init`, `completion` and `version` need nothing. -Commands that write persistent state — `snapshot create`, `snapshot purge`, -`snapshot remove`, `prune`, and `remote nuke` — take a process-wide lock at -`$XDG_DATA_HOME/vaultik/vaultik.pid` (`~/.local/share/vaultik/vaultik.pid` on -Linux) for the whole run. Only one of them runs at a time: a second one exits -immediately with an "already running" error rather than waiting, so two writers -can never corrupt the local index. +`config edit`, `config get` and `config set` need a config file that exists. It +does not have to be complete. -The lock covers one user on one machine. It does not stop another user, or -another host, from writing to the same destination store at the same time. A -backup uploads its blobs, and then its `db.zst.age`, before its manifest, so a -prune from elsewhere at that moment finds blobs no snapshot lists and a -directory under `metadata/` with no manifest. It deletes such a blob only once -it is older than `prune_grace_period` (default `24h`), and such a directory only -once every object in it is, so the grace period must be longer than the longest -backup run of any host or user sharing the destination. The grace period does -not protect a blob a backup reuses from an earlier run that stopped before its -manifest: a backup skips the upload of a blob already on the destination, so -that blob keeps the time of its first upload, and a prune from elsewhere during -the backup can delete it; the new snapshot then cannot be restored. -`remote nuke` deletes everything regardless. To not depend on the grace period, -give each host or user its own prefix (or directory) on the destination. +Every other command needs a config file that loads. See +[Configuration](#configuration) for the checks. -Read-only commands — `info`, `snapshot list`, `snapshot verify`, and -`remote info` — do not take the lock and are never blocked, so they run even -while a backup is in progress. `snapshot restore` does not take the lock either: -it writes only to the target directory you name, not the local index or the -destination store. `config`, `database delete`, `completion`, and `version` do -not take the lock. +Every other command except `database delete` also needs these: -### stdout and stderr +- Write access to the local index. The command creates the file, and its + directory, when they do not exist. +- For an `rclone://` destination, an rclone config file that defines the remote. -Log output — everything from `--verbose` and `--debug`, and every warning and -error the logger emits — goes to **stderr**, and so does the startup banner. -stdout carries the output you asked for: tables, the documents produced by -`--json`, `config get` values, and completion scripts. +Each command below lists what it needs beyond this. No command needs root, but a +backup can read only what its user can read, and a restore sets file owners only +as root. -This means `vaultik snapshot list --verbose > out.txt` captures the listing and -leaves the diagnostics on your terminal. To capture both, redirect stderr as -well (`> out.txt 2> log.txt`, or `> out.txt 2>&1` to interleave them). +### Global flags -The split is what makes `--json` usable from a script. Warnings and errors are -never suppressed — not by `--quiet`, not by `--cron` — so a logger on stdout -would eventually land a log line inside a JSON document and break the parse. A -config file with group- or world-readable permissions is enough to trigger it. +- `--config `: the config file to use. See + [Where the config file is](#where-the-config-file-is). +- `--verbose`, `-v`: write informational log records to stderr. +- `--debug`: write debug log records to stderr. +- `--quiet`, `-q`: print only warnings, errors and the output a script reads. + See [Output](#output). +- `--skip-errors`: skip a file that cannot be read during a backup, or cannot be + restored during a restore. Without it the command stops at the first such + file. -Format follows the stream: when stderr is a terminal the records are colorized -one-liners, and when it is redirected or piped they are JSON, one object per -line. +`--skip-errors` applies to `snapshot create` and `snapshot restore` only. A +command that skipped anything still finishes, and then exits non-zero. -Under `--json`, stdout holds the document and nothing else. The startup banner -is suppressed, as `--quiet` and `--cron` suppress it, and the progress narration -a command would otherwise print — such as the stale local records `prune` -reconciles away — is suppressed too, so it cannot land ahead of the document. -Every `--json` command therefore pipes on its own, with no additional flag: -`vaultik snapshot list --json | jq .` and `vaultik prune --json | jq .` both -work as written. +### Durations -### environment variables +`--keep-newer-than`, `--older-than` and `prune_grace_period` take a duration. -- `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for - `snapshot restore` and `snapshot verify --deep`). May hold the whole - `age-keygen` file — comments and every identity in it are accepted. Set it - from the file, e.g. - `export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"`, so - the key is not typed into your shell history. -- `VAULTIK_CONFIG`: Path to config file (overridden by `--config`) -- `VAULTIK_INDEX_PATH`: Override local SQLite index path -- `VAULTIK_CPUPROFILE`: Write a CPU profile to this path for the duration of the - run (development/debugging) -- `VAULTIK_MEMPROFILE`: Write a heap profile to this path when the run exits - (development/debugging) +- `d`: days. +- `w`: weeks. +- `mo`: months of 30 days. +- `y`: years of 365 days. +- `h`, `m` and `s`: hours, minutes and seconds, as in `36h` or `90m`. -### shell completion +`m` means minutes and `mo` means months. Days, weeks, months and years take +whole numbers and can be combined, as in `2w3d`. + +`prune_grace_period` accepts only hours, minutes and seconds. + +### `config init` + +Writes a default config file, with a comment that explains each setting it +lists. + +- It writes to the path from `--config`, or else from `VAULTIK_CONFIG`. +- Otherwise it writes to `config.yml` in the platform config directory. That is + `~/.config/vaultik/` on Linux and `~/Library/Application Support/vaultik/` on + macOS. +- As root it writes to `/etc/vaultik/config.yml` instead. +- It creates the file with mode `0600`, because the file can hold credentials. +- It refuses to overwrite a file that exists. + +### `config edit` + +Opens the config file in `$EDITOR`. It uses `vi` when `EDITOR` is not set. + +### `config get ` + +Prints one value from the config file. The key is a dotted path, as in +`vaultik config get storage_url` or `vaultik config get age_recipients.0`. A +list or a map prints as YAML. + +### `config set ` + +Sets one value in the config file. The key is a dotted path, as in +`vaultik config set compression_level 9`. + +- Comments and formatting in the file are kept. +- A key that does not exist yet is created. A missing parent becomes a list when + the next part of the path is a number, and a map otherwise. +- In a list, the index one past the end appends an entry. +- The command prints the key it set, and never the value. +- It sets the file's mode to `0600` when group or others could read it. + +The command refuses a change that would stop the config from loading. An example +is `vaultik config set compression_level 99`. The file is then left unchanged. + +A config that does not load yet is checked less strictly. An example is the file +`config init` writes, which has no `storage_url`. There the command refuses only +a value of the wrong type, such as a word where a number belongs. + +### `snapshot create` + +Backs up the configured snapshots. With no names it backs up all of them. Each +name gets its own snapshot. + +It needs: + +- At least one public key in `age_recipients`. +- Read access to the configured paths. +- Permission to write objects at the destination. See + [Destination permissions](#destination-permissions). +- Free temporary space. See [Temporary files](#temporary-files). +- The [lock](#locking). + +It does not need the private key. + +A file is read again when any of these has changed since the last backup: + +- its size +- its modification time +- its mode +- its owner or group +- its link target + +Every other file is recorded in the new snapshot without being read. + +Flags: + +- `--cron`: print nothing when the backup succeeds. Warnings and errors are + still printed. +- `--prune`: delete old snapshots and unused data after the backup. See below. +- `--keep-newer-than `: with `--prune`, keep every snapshot newer than + this. + +The backup stops before it writes anything in these cases: + +- A `file://` destination directory does not exist. +- The destination has no marker file, although the local index holds snapshots. + See [The marker file](#the-marker-file). +- The destination in the config is not the one the local index was used with. + See [Changing the destination](#changing-the-destination). + +The backup also stops when a configured path is a symlink whose target does not +exist. It stops when a configured path does not exist, unless `--skip-errors` is +given. + +With `--skip-errors`, a file or directory that cannot be read is skipped. The +snapshot is kept and counts each skipped entry as failed, and the command exits +non-zero. An error while packing or uploading still stops the backup. + +A file that is removed during the backup is counted as failed. It does not +change the exit status. + +#### `--prune` + +After a successful backup, `--prune` does two things: + +- It removes the older snapshots of each name it just backed up. It keeps only + the newest one, or with `--keep-newer-than` every snapshot newer than that. +- It then deletes at the destination what [`prune`](#prune) deletes. + +`--prune` also needs permission to list, read and delete at the destination. + +`--keep-newer-than` without `--prune` is an error. So is a value that is not a +duration. Both are reported before the backup starts. + +The prune does not run after a backup that may have missed data. An older +snapshot may then hold the only copy of what was missed. This applies when: + +- `--skip-errors` skipped a file or directory that could not be read. +- A configured path held nothing, as the mount point of an unmounted volume + does. +- An earlier snapshot of the same name found a mount point that this backup did + not find. + +In each case every older snapshot is kept. The command says why, names the path +and exits non-zero. It does so on every backup until the cause is gone. + +A mount point here is a directory at or below a configured path that is on a +different filesystem from its parent. It counts only when the backup finds an +entry in it that is not excluded. A backup does not find it when the volume is +not mounted, or when the directory is now excluded or no longer configured. + +If the cause is intended, such as a retired volume, remove the older snapshots +yourself with `snapshot purge`. Take a path that is empty on purpose out of the +config. + +### `snapshot list` + +Lists the snapshots in the local index and the snapshots at the destination, as +one table. Before it, the command prints the snapshot names and paths in the +config. + +It needs permission to list and read at the destination. It needs no private +key. + +The table has these columns: + +- `SNAPSHOT ID` +- `TIMESTAMP`: in UTC. +- `COMPRESSED SIZE`: the total size of the blobs the snapshot uses. +- `UNCOMPRESSED SIZE`: the total size of its distinct chunks before compression. +- `NEW CHUNK SIZE`: the size of the chunks this snapshot stored that were not + stored before. + +A snapshot at the destination that is not in the local index is shown as +``. The dots stand for the first 12 characters of its remote +key. Its last two columns read ``, because those sizes come from +the local index. Its timestamp is the time its backup finished. + +The command reports these below the table: + +- Snapshots in the local index that are missing at the destination. Run + `vaultik prune` to drop those records. +- Remote-only snapshots whose manifest cannot be read. They are left out of the + table. +- Remote-only snapshots beyond the first 1000. They are left out of the table. +- A local index that was used with a different destination. See + [Changing the destination](#changing-the-destination). + +When the destination cannot be listed, the command warns and shows the local +index alone. It still exits zero. It does the same when the destination has no +marker file although the local index holds snapshots. + +`--json` prints a JSON array instead. Each entry has these fields: + +- `id`: the snapshot ID. It is empty for a remote-only snapshot. +- `remote_key`: the full 64-character remote key. +- `timestamp` +- `compressed_size` +- `uncompressed_size` and `new_chunk_size`: left out for a remote-only snapshot. +- `locally_tracked`: whether the snapshot is in the local index. +- `remote_present`: whether the snapshot is at the destination. It is `null` + when the destination could not be listed. + +Under `--json`, every warning above goes to stderr. + +### `snapshot verify ` + +Checks a snapshot at the destination. The argument is a snapshot ID, a remote +key or the leading part of a remote key. + +Without `--deep`, it checks two things: + +- Every blob the manifest lists is present, with the size the manifest records. +- The snapshot's encrypted database is present. + +This reads no blob contents and needs no private key. It needs permission to +read at the destination, and to list it when you give a remote key. + +Flags: + +- `--deep`: download, decrypt and re-hash every blob. See below. +- `--json`: print the result as JSON. + +The command exits non-zero when a check fails or cannot be made. + +#### `--deep` + +`--deep` needs the age private key. Set it in the `VAULTIK_AGE_SECRET_KEY` +environment variable, or as `age_secret_key` in the config file. The environment +variable wins when both are set. See [The private key](#the-private-key). + +`--deep` does these checks, in order: + +- It downloads and decrypts the snapshot's database. It checks that this is the + database of the snapshot you named. +- It checks that the manifest and the database list the same blobs, with the + same sizes. +- It checks that every blob is present with the recorded size. +- It downloads and decrypts every blob. It checks the hash of each chunk, and + the hash that names the blob. + +It stops at the first blob that fails. + +`--deep` proves that the stored data is intact and decrypts with your key. It +does not prove who wrote the data. + +`--deep` writes the decrypted database to a temporary directory and removes it +afterwards. See [Temporary files](#temporary-files). + +### `snapshot purge` + +Removes old snapshots. You must give exactly one of `--keep-latest` and +`--older-than`. + +It needs the [lock](#locking), and permission to list and delete at the +destination. It needs no private key. + +Flags: + +- `--keep-latest`: keep only the newest snapshot of each name. +- `--older-than `: remove every snapshot older than this. +- `--snapshot `: act only on snapshots with this name. Repeat the flag for + more names. +- `--force`: do not ask for confirmation. + +The command acts only on snapshots in the local index. `--keep-latest` keeps the +newest snapshot of each name, not the newest overall. + +A removed snapshot leaves the local index, and its directory under `metadata/` +at the destination is deleted. Its blobs stay. Run `vaultik prune` afterwards to +delete the blobs that no snapshot uses any more. + +The command first drops local records of snapshots that are no longer at the +destination. It fails and removes nothing when the destination cannot be listed +or has no marker file. + +### `snapshot remove ` + +Removes one snapshot. The argument must be a snapshot ID. A remote key is not +accepted. + +It needs the [lock](#locking), and permission to list and delete at the +destination. It needs no private key. + +Flags: + +- `--force`, `-f`: do not ask for confirmation. +- `--dry-run`: say what would be removed, and remove nothing. +- `--local-only`: remove the snapshot from the local index only. The destination + is not changed. +- `--json`: print the result as JSON. This also skips the confirmation. + +The command removes the snapshot from the local index. It also deletes the +snapshot's directory under `metadata/` at the destination. Its blobs stay. Run +`vaultik prune` afterwards to delete the blobs that no snapshot uses any more. + +An ID that is in neither the local index nor the destination is an error. With +`--local-only`, the snapshot must be in the local index. + +The deletion at the destination can fail, for example when the destination is +unreachable or has no marker file. What happens then depends on where the +snapshot was: + +- A snapshot in the local index is still removed from it, with a warning. Run + the same command again later to delete its directory at the destination. + `vaultik prune` does not delete it. +- For a snapshot that is only at the destination, the failure is an error. + +### `snapshot restore [paths...]` + +Restores files from a snapshot into ``. The first argument is a +snapshot ID, a remote key or the leading part of a remote key. + +It needs: + +- The age private key, as for `snapshot verify --deep`. See + [The private key](#the-private-key). +- Permission to read at the destination, and to list it when you give a remote + key. +- Write access to ``. The command creates the directory when it does + not exist. +- Free temporary space for the decrypted blobs. See + [Temporary files](#temporary-files). + +It does not need `age_recipients`, and it does not take the lock. + +Each file is restored under its original absolute path inside ``. +For example, `/home/user/notes.txt` is restored to +`/home/user/notes.txt`. + +With no paths, the command restores the whole snapshot. Each path names a file +or a directory by its original absolute path: ```sh -# zsh: load for the current session +vaultik snapshot restore ~/vaultik-restore /home/user/documents +``` + +A path that matches nothing is named in a warning. The command restores the +other paths and then exits non-zero. + +The command restores these: + +- regular files, symlinks and directories, empty directories included +- permissions and modification times +- owners and groups, when it runs as root + +Without root, restored files belong to the user who runs the command. The +command then warns once if any owner or group differs from the snapshot. + +A file that already exists in `` is replaced. Running the same +restore again starts from the beginning. + +Restore into a new directory that only you can write, not a shared one such as +`/tmp`. No other user can then change the files while they are written. + +Flags: + +- `--verify`: after the restore, read every restored file and check its chunk + hashes. + +With `--skip-errors`, a file that cannot be restored is reported and skipped, +and the command exits non-zero. `--verify` does not check a skipped file. + +### `prune` + +Removes what is no longer needed, from the local index and from the destination. + +It needs the [lock](#locking), and permission to list, read and delete at the +destination. It needs no private key. + +It works in three steps: + +- It drops each snapshot from the local index whose directory is gone from the + destination. +- It removes unfinished snapshots from the local index. It also removes the file, + chunk and blob records that no snapshot uses. +- It reads every manifest at the destination. It then deletes each blob that no + manifest lists and that is older than `prune_grace_period`. + +The third step also deletes a directory under `metadata/` that has no manifest. +A backup leaves one when it stops before it uploads its manifest. Such a +directory is deleted only when all of these hold: + +- Every object in it is older than `prune_grace_period`. +- Its snapshot is not in the local index. +- Its name has the form of a remote key. + +`prune_grace_period` defaults to `24h`. A newer blob is kept, because a backup +that is still running may need it. See +[Sharing a destination](#sharing-a-destination) for what the grace period does +not protect. + +The command deletes nothing at the destination in these cases: + +- The destination cannot be listed. +- The destination has no marker file, although the local index holds snapshots. +- A manifest cannot be read. + +It exits non-zero when a blob or a directory cannot be deleted. + +The time to read the manifests grows with the number of snapshots. + +Flags: + +- `--force`: do not ask for confirmation before deleting at the destination. +- `--json`: print the result as JSON. This also skips the confirmation. + +The JSON document has these fields: + +- `blobs_found`: the unused blobs old enough to delete. +- `blobs_deleted` +- `blobs_failed`: blobs that could not be deleted. Left out when zero. +- `bytes_freed` +- `blobs_too_new`: unused blobs that were kept. They are newer than the grace + period, or the destination gave no time for them. +- `metadata_dirs_deleted` +- `metadata_dirs_failed`: left out when zero. + +### `info` + +Prints the system details, the settings in use and figures from the local index. +The figures are the size of the index and its counts of snapshots, blobs and +files. + +### `remote info` + +Reports what the destination holds: + +- the kind of destination and its location +- the size of each snapshot's database and manifest +- the number and size of all blobs +- the number and size of the blobs that manifests list +- the number and size of the blobs that no manifest lists + +It needs permission to list and read at the destination. It needs no private +key. + +When a manifest cannot be read, the last figures are reported as unknown. That +snapshot's own blob count and size are unknown too. + +The command skips a name under `metadata/` that is not a remote key, with a +warning. It does not count a file under `blobs/` whose name is not a blob hash. + +`--json` prints the report as JSON. Unknown figures are `null`. +`unreadable_manifests` lists the remote key of each manifest that could not be +read. `skipped_manifest_count` counts the manifests under skipped names. + +### `remote nuke --force` + +Deletes everything `vaultik` stored at the destination. This cannot be undone. + +It needs `--force`, the [lock](#locking), and permission to list and delete at +the destination. + +It deletes these, in order: + +- every object under `metadata/` +- every object under `blobs/` +- the marker file + +It also removes each snapshot from the local index, once that snapshot's +directory at the destination is deleted. + +The command fails before it deletes anything when the destination cannot be +listed, or has no marker file although the local index holds snapshots. + +If objects remain afterwards, the command names each one and exits non-zero. +Running it again deletes what is left. + +### `database delete` + +Deletes the local index file. The destination is not changed. + +It needs a config file that loads, because the config gives the path of the +index. + +The next backup then reads every file again. It skips the upload of a blob that +the destination already holds under the same name. + +Use this command after you point the config at a different destination. See +[Changing the destination](#changing-the-destination). + +`--force` skips the confirmation. + +### `completion ` + +Prints a completion script for `bash`, `zsh`, `fish` or `powershell`. + +```sh +# zsh, for the current session source <(vaultik completion zsh) -# zsh: install permanently +# zsh, permanently vaultik completion zsh > "${fpath[1]}/_vaultik" -# bash: load for the current session +# bash, for the current session source <(vaultik completion bash) -# bash: install permanently (Linux) +# bash, permanently, on Linux vaultik completion bash > /etc/bash_completion.d/vaultik # fish vaultik completion fish > ~/.config/fish/completions/vaultik.fish ``` -### command details +### `version` -**`config init`**: Write a default config file with commented explanations for -every setting. Writes to the path from `--config`, `$VAULTIK_CONFIG`, or the -platform config directory (`~/Library/Application Support/vaultik/` on macOS, -`~/.config/vaultik/` on Linux, `/etc/vaultik/` as root). Refuses to overwrite an -existing file. Created with mode `0600` since it will contain credentials. +Prints the version, the commit, the build date and the platform. It also says +when the binary is a development build, not a release. -**`config edit`**: Open the config file in `$EDITOR` (falls back to `vi`). +## Configuration -**`config get`**: Print a config value addressed by dotted YAML path (e.g. -`vaultik config get storage_url`). Non-scalar values print as YAML. +The config file is YAML. `vaultik config init` writes one to start from, and +[`config.example.yml`](config.example.yml) is a longer example. -**`config set`**: Set a scalar config value by dotted YAML path (e.g. -`vaultik config set compression_level 9`, -`vaultik config set storage_url "file:///mnt/backups"`). Comments and formatting -in the file are preserved. Missing keys along the path are created: as a list -when the next path element is a number (so -`vaultik config set age_recipients.0 age1...` works on a file without -`age_recipients`), otherwise as a map. If the config loads, a change after which -it would not, such as `vaultik config set compression_level 99`, is refused and -the file is left unchanged. A config that does not load is checked one setting -at a time. Examples are the file `config init` writes before `storage_url` is -set, and one with a `${ENV:...}` reference to a variable not set in this shell. -In such a config, an unknown key or a value of the wrong type is refused. +A command refuses to start when the config does not load. The config loads when +all of these hold: -**`snapshot create`**: Perform incremental backup of configured snapshots. +- Every entry in `age_recipients` is an age X25519 public key. +- `snapshots` has at least one name, and every name has at least one path. +- A destination is set, with credentials when it is S3. +- Every other setting below is within its limits. -- Optional snapshot names argument to create specific snapshots (default: all) -- On macOS, the terminal application running vaultik needs Full Disk Access - (System Settings → Privacy & Security → Full Disk Access) to read - TCC-protected directories; without it the backup aborts with a permission - error that explains how to fix it -- `--cron`: Silent on total success; warnings and errors are still printed (for - crontab) -- `--prune`: After backup, drop older snapshots of each backed-up name, remove - orphaned blobs older than `prune_grace_period` from remote storage, and delete - each directory under `metadata/` that has no `manifest.json.zst` once every - object in it is older than `prune_grace_period` and its snapshot is not in the - local index, as `prune` does. By default keeps only the latest snapshot per - name; use `--keep-newer-than` for a rolling window. When `--skip-errors` - skipped a file or directory it could not read or list, the prune does not run - and every older snapshot is kept, since one of them may be the only one that - holds what was skipped; the command says so and exits non-zero. When the - backup found nothing in a configured path, as in the mount point of a volume - that is not mounted, the prune does not run either: every older snapshot is - kept, and the command names the path and exits non-zero, on every run until - the path holds something again. Take a path that is empty on purpose out of - the config, or remove the older snapshots by hand with `snapshot purge`. When - an earlier snapshot of the same name in the local index found a volume mounted - at or below a configured path (a directory on a different filesystem from its - parent, with an entry in it that the backup does not exclude) and this backup - does not, the prune does not run either: every older snapshot is kept, and the - command names the directory and exits non-zero, on every run until the volume - is mounted again. A backup also does not find the volume when it finds no such - entry in its directory, as when an automounted volume cannot be mounted, or - when that directory is no longer backed up, because it is excluded or its - configured path was removed. If any of this is on purpose, as for a retired - volume, remove the older snapshots by hand with `snapshot purge`. A file - removed during the backup does not stop the prune. -- `--keep-newer-than `: With `--prune`, keep snapshots newer than this - duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without - `--prune`, or with a value that is not a valid duration, the command fails - before the backup starts. -- Fails before it writes anything when a `file://` destination directory does - not exist, or when the local index holds snapshots and the destination answers - that it has no `vaultik-destination.txt`. When that look-up fails for any - reason other than not found, for example when S3 refuses it to credentials - that allow only PUT, the check is skipped (see - [remote storage layout](#remote-storage-layout)) -- Fails, before anything is pruned, when a configured path is a symlink whose - target does not exist, as one into a volume that is not mounted +`vaultik` warns when group or others can read the config file. -**`snapshot list`**: Show every snapshot known to this host — the union of the -local index and the backup destination store — with timestamps and three sizes -per snapshot (compressed remote size; total uncompressed chunk size; size of the -chunks that snapshot stored that were not stored before). Both chunk sizes are -recorded when the snapshot's backup runs. +A value written as `${ENV:NAME}` is replaced by the environment variable `NAME`. +This keeps a credential out of the file. -Listing the destination store does **not** require the age secret key, so it -works in vaultik's intended configuration, where the backed-up host holds only -the public key. A host that has lost its local index can still see what it has -backed up. +A leading `~` is expanded to your home directory in snapshot paths, in +`index_path` and in a `file://~/...` destination. -What that host cannot see is a remote-only snapshot's name. The snapshot ID is -hashed at the storage boundary and the manifest records only the hash, so -hostname and snapshot name exist solely in the local index and in the encrypted -per-snapshot database. Snapshots found only on the destination store are -therefore listed as `>` and show -`` in the uncompressed and "new chunk" columns, which come only -from the local index. Their timestamp and compressed size are real, read from -the manifest. +### Where the config file is -Snapshots in the local index with no counterpart on the destination store are -reported below the table as drift, with the `vaultik prune` invocation that -reconciles them. If the local index is bound to a different destination than the -configured one (after a change to `storage_url`, or `s3.endpoint`, `s3.bucket` -or `s3.prefix`), the command says so instead and gives the same two ways out -that `prune` gives: revert the configuration, or run `vaultik database delete`. +`vaultik` uses the first of these that applies: -If the destination store cannot be listed (unmounted volume, permission denied, -network down), the command warns, falls back to the local index alone, and still -exits zero. It does the same when the destination answers that it has no -`vaultik-destination.txt` while the local index holds snapshots stored there, as -when a volume mounted at the destination is unplugged and leaves its empty mount -point behind. +- the path given with `--config` +- the path in the `VAULTIK_CONFIG` environment variable +- `config.yml` in the platform config directory, which is `~/.config/vaultik/` + on Linux and `~/Library/Application Support/vaultik/` on macOS +- `/etc/vaultik/config.yml` -- `--json`: Output in JSON format. Each entry carries `locally_tracked` (whether - the snapshot is in the local index), `remote_key` (the full 64-character - storage key), and `remote_present` (whether it was seen on the destination - store, or `null` if the destination could not be listed). Warnings about an - unlistable destination, a local index bound to a different destination (with - both destinations and the same two ways out), unreadable manifests, and a - truncated listing all go to stderr through the logger, so stdout stays a - single parseable document. +A path from `--config` or `VAULTIK_CONFIG` must exist. -**`snapshot verify`**: Verify snapshot integrity. +### Settings -- Default (shallow): checks that every blob the manifest lists is present in - storage with the size the manifest records, and that the encrypted database is - present. It does not read blob contents. -- `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the - encrypted metadata database -- Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a - remote-only snapshot's remote key (or an unambiguous leading part of it) -- `--json`: Output results as JSON +- `age_recipients` + - The age public keys that backups are encrypted to. + - Default: none. `snapshot create` needs at least one. No other command + needs one. + - Any one of the matching private keys can decrypt. + - Adding a key later does not re-encrypt what is already stored. See + [Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). +- `age_secret_key` + - The age private key. + - Default: not set. + - Only `snapshot restore` and `snapshot verify --deep` use it. + - The `VAULTIK_AGE_SECRET_KEY` environment variable overrides it. + - Do not set it on a machine you back up. See + [The private key](#the-private-key). +- `snapshots` + - The named sets of paths to back up. Required. + - `snapshots..paths`: the paths to back up under this name. + - `snapshots..exclude`: exclude patterns for this name only. +- `exclude` + - Exclude patterns that apply to every snapshot. See + [Exclude patterns](#exclude-patterns). + - Default: none. +- `storage_url` + - The destination. See [Destinations](#destinations). + - Required, unless the `s3` settings name the destination. +- `s3.access_key_id` and `s3.secret_access_key` + - The S3 credentials. Required for an S3 destination. +- `s3.region` + - The S3 region. Default: `us-east-1`. +- `s3.part_size` + - The size of each part of a multipart upload. Default: `5MiB`. + - It must be between `5MiB` and `5GiB`. + - A blob too large for 10,000 parts of this size is uploaded in larger + parts. +- `s3.endpoint`, `s3.bucket`, `s3.prefix` and `s3.use_ssl` + - An older way to name an S3 destination. They are used only when + `storage_url` is not set. + - `s3.endpoint` and `s3.bucket` are then required. + - `s3.use_ssl: false` selects plain HTTP for an endpoint written without a + scheme. +- `chunk_size` + - The average size of a chunk. Default: `10MB`. Minimum: `1MB`. + - A chunk is between a quarter of this size and four times this size. + - A smaller size finds more duplicate content and makes the index larger. +- `blob_size_limit` + - The largest size of a blob. Default: `10GB`. + - It must be at least four times `chunk_size`. +- `compression_level` + - The zstd compression level, from `1` to `19`. Default: `3`. + - Levels `1` to `2`, `3` to `5`, `6` to `9` and `10` to `19` form four + groups. Levels in one group compress the same way. +- `prune_grace_period` + - How old unused data must be before `prune` deletes it. Default: `24h`. + - It applies to a blob that no manifest lists. It also applies to every + object in a directory under `metadata/` that has no manifest. + - `0s` deletes such data at once. A negative value is refused. + - See [Sharing a destination](#sharing-a-destination). +- `hostname` + - The hostname used in snapshot IDs. Default: the system hostname. + - Only the part before the first `.` is used. +- `index_path` + - The path of the local index. + - Default: `index.sqlite` in the platform data directory. That is + `~/.local/share/vaultik/` on Linux and + `~/Library/Application Support/vaultik/` on macOS. -**`snapshot purge`**: Remove old snapshots based on criteria. Retention is -per-snapshot-name (`--keep-latest` keeps the latest of each name, not the latest -globally). +Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB` +and `GiB` are powers of 1024. -- `--keep-latest`: Keep only the most recent snapshot of each name -- `--older-than `: Remove snapshots older than duration (e.g. `30d`, - `4w`, `6mo`, `1y`; `m` is minutes, `mo` is months) -- `--snapshot `: Restrict to specific snapshot names (repeat for multiple) -- `--force`: Skip confirmation prompt +### Environment variables -**`snapshot remove`**: Remove one snapshot. By default this removes the snapshot -from the local index and strips the snapshot's metadata from the backup -destination store. Blobs are NOT touched — deleting blobs requires reading every -remaining remote manifest (the destination store may hold snapshots this host -doesn't know about), which is what `vaultik prune` does. The command prints the -`vaultik prune` invocation to run as a follow-up only when it removed the -snapshot's metadata from the destination store, and never under `--json`. An ID -that names no snapshot in the local index or on the destination store is an -error; with `--local-only`, the snapshot must be in the local index. Local row -cleanup (files, chunks, blobs the snapshot was the last referrer for) runs -automatically. If removing the snapshot's metadata from the destination store -fails (for example because the store is unreachable, or the local index holds -snapshots and the destination answers that it has no `vaultik-destination.txt`), -the local-DB removal still completes and a warning is emitted; run -`vaultik snapshot remove ` again once the store is reachable to -remove the snapshot's metadata from it (`vaultik prune` does not). For a -snapshot that is only on the destination store, that failure is an error. To -wipe everything on the destination in one go, use `vaultik remote nuke --force`. +- `VAULTIK_AGE_SECRET_KEY`: the age private key. See + [The private key](#the-private-key). +- `VAULTIK_CONFIG`: the path of the config file. `--config` overrides it. +- `VAULTIK_INDEX_PATH`: the path of the local index. It overrides `index_path`. +- `RCLONE_CONFIG`: the path of the rclone config file, for an `rclone://` + destination. +- `EDITOR`: the editor that `config edit` starts. +- `TMPDIR`: the directory for temporary files. +- `NO_COLOR`: when set, status lines have no color. +- `VAULTIK_CPUPROFILE`: write a CPU profile to this path. For debugging. +- `VAULTIK_MEMPROFILE`: write a heap profile to this path on exit. For + debugging. -- `--local-only`: Skip remote cleanup; only touch the local index -- `--dry-run`: Show what would be deleted without deleting -- `--force`: Skip confirmation prompt -- `--json`: Output result as JSON. Also skips the confirmation prompt, as - `--force` does. +### The private key -**`snapshot restore`**: Restore files from a backup snapshot. +`snapshot restore` and `snapshot verify --deep` decrypt data, so they need the +age private key. No other command needs it. -- Requires `VAULTIK_AGE_SECRET_KEY` environment variable -- Accepts a snapshot ID, or — for a snapshot only on the destination store — its - remote key (or an unambiguous leading part of it) as shown by `snapshot list`. - See [restoring on another machine](#restoring-on-another-machine). -- Optional path arguments to restore specific files/directories (default: all). - A path that matches nothing in the snapshot is named in a warning, and the - command exits non-zero after restoring the paths that match. -- Preserves file permissions, timestamps, ownership (ownership requires root), - symlinks, and empty directories -- `--verify`: After restoring, verify every restored file's chunk hashes match. - A file skipped under `--skip-errors` is reported as failed to restore and is - not verified. +Give the key in one of two ways: -**`prune`**: Tidy up everything that isn't needed. Runs three passes: (1) -reconcile the local index against the destination store — any local snapshot -whose remote metadata is missing is dropped from the local index; (2) delete -orphaned local rows (files, chunks, blobs no longer referenced by any completed -snapshot); (3) list every remote manifest on the destination store to compute -the still-referenced blob set and delete any blob not in that set that is older -than `prune_grace_period` (default `24h`). A newer one is kept and counted: it -may belong to a backup still running, on this host or another sharing the -destination store, that has not yet uploaded its manifest. The grace period does -not protect a blob such a backup reuses from an earlier run that stopped before -its manifest: that blob keeps the time of its first upload, so it can be -deleted. Step (3) also deletes each directory under `metadata/` that has no -`manifest.json.zst`, as a backup leaves when it stops after uploading -`db.zst.age` but before its manifest, once every object in it is older than -`prune_grace_period` and its snapshot is not in the local index. An object the -listing gives no time for keeps its directory, and a directory whose name is not -a remote key is left alone. Step (3) reads all remote manifests — network cost -scales with the number of snapshots. `snapshot create --prune` runs the same -cleanup automatically; this is the manual entry point for the same work. If the -destination store cannot be listed, or the local index holds snapshots and the -destination answers that it has no `vaultik-destination.txt` (an unplugged -volume's empty mount point), prune fails and deletes nothing. `snapshot purge` -does the same. If a blob or such a directory cannot be deleted, the command (and -`snapshot create --prune`) exits non-zero. +- the `VAULTIK_AGE_SECRET_KEY` environment variable +- `age_secret_key` in the config file -- `--force`: Skip confirmation prompt -- `--json`: Output stats as JSON. Also skips the confirmation prompt, as - `--force` does. A blob that cannot be deleted is counted in `blobs_failed`, - and the document is printed before the non-zero exit. A blob kept for being - newer than `prune_grace_period`, or because the listing of the destination - gives no time for it, is counted in `blobs_too_new` and not in `blobs_found`. - A directory under `metadata/` with no manifest is counted in - `metadata_dirs_deleted` once deleted, or in `metadata_dirs_failed` if it - cannot be. +The environment variable wins when both are set. -**`info`**: Display system configuration, storage settings, encryption -recipients, and local database statistics. +The value can be the whole file that `age-keygen` writes. Comments are ignored, +and every key in the file is used. -**`remote info`**: Show storage backend type and location plus detailed remote -storage inventory: per-snapshot metadata sizes, blob counts, and orphaned blob -detection. A name under `metadata/` that is not a remote key is skipped with a -warning and is not printed. If a listed `manifest.json.zst` cannot be read, or -sits under a skipped name, the orphaned blob figures are reported as unknown; -`--json` gives them as `null`, lists the remote key of each unreadable manifest -in `unreadable_manifests` and counts the manifests under skipped names in -`skipped_manifest_count`. An unreadable manifest also leaves its snapshot's blob -count and blob size unknown, `null` in `--json`. A file under `blobs/` whose -name is not a blob hash is not counted as a blob, since `prune` never deletes -it. +Set the variable from the file, so that the key is not typed into your shell +history: -- `--json`: Output as JSON - -**`remote nuke`**: Delete every object under `metadata/` and `blobs/` on the -backup destination store, whatever its name, and then `vaultik-destination.txt`, -leaving the bucket prefix empty. Destructive and irreversible. This is the -single supported way to wipe the entire destination store. A snapshot is removed -from the local index only once its metadata is deleted from the destination -store. If the destination store cannot be listed, or the local index holds -snapshots and the destination answers that it has no `vaultik-destination.txt` -(an unplugged volume's empty mount point), the command fails before it removes -anything. If a listing afterwards still finds anything under `metadata/` or -`blobs/`, the command names each object left and exits non-zero; running it -again deletes what is left. - -- `--force`: Required to confirm destruction. - -**`database delete`**: Delete the local SQLite state database file entirely. -Remote storage is unaffected; the next backup will do a full scan and -re-deduplicate against existing remote blobs, and the local index will re-bind -to the currently configured storage destination. Use this after changing -`storage_url`, or `s3.endpoint`, `s3.bucket` or `s3.prefix`, to a different -destination. - -- `--force`: Skip confirmation prompt - ---- - -## storage backends - -vaultik supports three storage backends, selected via the `storage_url` config -field: - -**S3** (`s3://bucket/prefix?endpoint=host®ion=us-east-1`): Any S3-compatible -object store. Credentials are read from `s3.access_key_id` and -`s3.secret_access_key` in the config file. - -**Local filesystem** (`file:///path/to/backup`): Stores blobs and metadata on a -local or mounted filesystem. Useful for testing or backing up to a NAS. The path -must be absolute; a relative one is refused when the config loads. -Percent-escapes in the path are decoded, so `my%20stick` and `my stick` name the -same directory. The directory must exist before the first backup: vaultik never -creates it. A directory on a removable volume is gone while the volume is not -mounted, so a backup then fails instead of writing to the disk under the mount -point. The mount point itself stays behind when the volume is unplugged: there -only `vaultik-destination.txt` stops such a backup, and only while the local -index holds snapshots. - -**Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud -providers. Requires rclone to be configured separately (`rclone config`). - -An upload cut off part-way leaves nothing under the object's name on S3, which -shows an object only once its upload has completed, and on the local filesystem -backend, which writes a temporary file and renames it into place. Rclone remotes -with a server-side move (local and sftp among them) are written under a -temporary name ending in `.partial` and moved into place. Rclone remotes without -one are written in place: where such a remote shows a file while it is still -being written, a killed upload can leave a truncated object under its name, -which a later backup takes for complete. A leftover `.partial` file is ignored -and can be deleted. - -A destination shared between hosts or users, each with its own local index, -relies on `prune_grace_period` to keep one's prune from deleting the blobs of -another's backup still running, or its directory under `metadata/` that has no -manifest yet, so it must be longer than the longest backup run. It does not -protect a blob that backup reuses from an earlier run that stopped before its -manifest, which the prune can delete. To not depend on the grace period, give -each host its own prefix (or directory). See [locking](#locking). - -Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is -still supported for backward compatibility. `storage_url` takes precedence if -both are set. - ---- - -## architecture - -For an implementation-level view of the internals — the data model, the `fx` -dependency-injection wiring, and the scanner — see -[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md). - -### remote storage layout - -``` -// -├── vaultik-destination.txt # Marker written by every backup -├── blobs/ -│ └── // -└── metadata/ - └── / - ├── db.zst.age # Encrypted binary SQLite database - └── manifest.json.zst # Unencrypted blob list (for pruning) +```sh +export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" ``` -- `vaultik-destination.txt` holds one fixed line of text, with no host name, - path or time. While the local index holds snapshots stored there, a - destination that answers a look-up of it with not found is treated as not - present: `snapshot create` refuses to write to it, `prune`, `snapshot purge` - and `remote nuke` fail, and `snapshot list` and `snapshot remove` behave as - for a destination that cannot be listed. This keeps an unplugged volume's - empty mount point from being taken for a destination whose snapshots are all - gone. Mount the volume if it is removable; if the destination was emptied on - purpose, run `vaultik database delete` and back up again. When the look-up - fails for any reason other than not found, the check is skipped and the - command goes on. S3 refuses it on every run to credentials that allow only - PUT, so a host with such credentials does not notice a destination emptied - since its last backup, and its next backup would leave out the data its local - index records as already stored. After the destination is emptied, run - `vaultik database delete` on such a host before it backs up again. -- Blobs are two-level directory sharded using the first 4 hex chars of the blob - hash -- `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted) - containing all file metadata, chunk mappings, and relationships for the - snapshot -- `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling - pruning without the private key +Do not put the private key on a machine you back up. Anyone who breaks into that +machine could then decrypt its backups. Supply the key only on the machine you +restore on. -Snapshot IDs follow the human-readable format -`__` (e.g. -`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the -destination store in plaintext. Each snapshot's metadata directory is named with -its ``, a one-way double SHA-256 hash of the ID, so a plain listing -of the store shows no hostname or snapshot name. The hash uses no secret, -though, so an observer who guesses a candidate hostname and snapshot name can -hash it and confirm the snapshot is present; the remote key keeps names out of a -listing but does not hide them from a guess. The backup time is not hidden -either: manifest.json.zst carries a plaintext timestamp, and object modification -times are visible at the storage layer regardless. For example, -`server1_home_2025-06-01T12:00:00Z` is stored under -`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`. -See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the -derivation, and -[Security Considerations](docs/REPOSTRUCTURE.md#security-considerations) -(including [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks)) for what the -format does and does not protect. +### Exclude patterns -### data flow +A pattern excludes a file or a whole directory from a backup. Patterns are +matched against the path below the configured path. -**backup:** +- A pattern that starts with `/` matches from the configured path. In a snapshot + of `~`, `/Library/Caches` matches only `~/Library/Caches`. +- Any other pattern matches at every depth. `.cache` matches every file or + directory named `.cache`. +- `*` and `?` match within one name. `**` matches across directories. +- A pattern that matches a directory excludes everything in it. -1. Open local SQLite index, load known files and chunks into memory -2. Walk source directories, compare mtime/size/mode against index -3. For changed/new files: chunk using content-defined chunking (FastCDC) -4. For symlinks and directories: record metadata (no chunking) -5. For each chunk: hash, check dedup, add to blob packer -6. When blob reaches size threshold: compress (zstd), encrypt (age), upload -7. Build snapshot metadata database, compress, encrypt, upload -8. Create unencrypted blob manifest for pruning support +`exclude` at the top of the config applies to every snapshot. `exclude` under a +snapshot name adds patterns for that snapshot. -**restore:** +## Destinations -1. Download and decrypt `metadata//db.zst.age` -2. Open the binary SQLite database -3. Query files (optionally filtered by paths) -4. Download and decrypt required blobs -5. Extract chunks, reconstruct files -6. Restore permissions, timestamps, ownership, symlinks +`storage_url` selects one of three kinds of destination. -**prune:** - -1. List all snapshot manifests -2. Build set of all referenced blob hashes -3. List all blobs in storage -4. Delete any blob not in the referenced set that is older than - `prune_grace_period` -5. Delete each directory under `metadata/` with no manifest whose objects are - all older than `prune_grace_period`, unless its snapshot is in the local - index - -### chunking and deduplication - -- Content-defined chunking using the FastCDC algorithm -- Average chunk size: configurable (default 10MB) -- Deduplication at file level (unchanged files skipped, unless a chunk the file - lists is held by no uploaded blob) and chunk level (identical chunks across - files stored once) -- Multiple chunks packed into blobs to reduce object count - -### encryption - -- Asymmetric encryption using age (X25519 + ChaCha20-Poly1305) -- Only the public key is needed on the source host -- Each blob and each metadata database is encrypted independently -- Multiple recipients supported (encrypt to multiple keys) - -### compression - -- zstd compression at configurable level (1-19, default 3). The level is - accepted as 1-19 but maps onto zstd's four internal speed presets: 1-2 - fastest, 3-5 default, 6-9 better, 10-19 best. Levels within the same band - compress identically. -- Applied before encryption at the blob level - ---- - -## configuration reference - -Run `vaultik config init` to generate a fully commented config file; a complete -annotated example also lives in -[`configs/config.example.yml`](configs/config.example.yml). Key fields: - -| Field | Default | Description | -| -------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty | -| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. | -| `snapshots` | (required) | Named snapshot definitions with paths and excludes | -| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) | -| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) | -| `exclude` | | Global exclude patterns (applied to all snapshots) | -| `chunk_size` | `10MB` | Average chunk size for content-defined chunking | -| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` | -| `compression_level` | `3` | zstd compression level (1-19) | -| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once | -| `hostname` | system hostname | Hostname used in snapshot IDs | -| `index_path` | platform data dir | Local SQLite index path | - ---- - -## limitations - -- **No extended attributes (xattrs).** ACLs, macOS Finder metadata, quarantine - flags, SELinux labels, and other extended attributes are not backed up or - restored. -- **No hard link detection.** Two hard links to the same inode are backed up as - independent files. Content deduplication means the data is stored once, but - the hard link relationship is lost on restore. -- **No sparse file support.** Sparse files are fully materialized during backup. - A 100 GB sparse VM disk that is mostly zeros will consume the full - (compressed) size in storage. -- **No bandwidth limiting.** Uploads and downloads use whatever bandwidth is - available. There is no `--bwlimit` flag yet. -- **No parallel blob downloads during restore.** Blobs are fetched sequentially. - Restore speed is bound by single-stream throughput. -- **Device nodes, named pipes, and sockets are silently skipped.** Only regular - files, directories, and symlinks are backed up. -- **Before 1.0, an update can make the local index unusable.** Vaultik supports - schema migrations, but none are added before 1.0 because there is no installed - base yet. If an update leaves your local index unusable, run - `vaultik database delete` and then a full backup; remote storage is - unaffected. After 1.0, the local index is migrated when vaultik is updated. - See [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). -- **Files that change during backup may be inconsistent.** There is no - filesystem snapshot or freeze. If a file is modified between the scan and - chunk phases, the backed-up copy may reflect a partial write. -- **Ownership restoration requires root.** File uid/gid are recorded and - restored, but `chown` requires elevated privileges. Without root, files are - restored with the current user's ownership. - ---- - -## roadmap - -Work planned after 1.0. Loosely ordered by priority. - -### correctness and operability - -- **Outside security audit.** Before 1.0 the encryption and blob-generation code - was reviewed: every bug the review found was fixed, and the risks it accepted - are listed in [Accepted Risks](docs/REPOSTRUCTURE.md#accepted-risks). No - outside audit has been done. age + zstd + content-defined chunking is mostly - off-the-shelf pieces, but the seams (key handling, recipient parsing, manifest - trust boundary, restore-time identity validation) need an outside read. -- **Verify restored content end-to-end in CI.** The current integration test - does this for a small synthetic snapshot but not at scale. A nightly job - against a multi-GB representative snapshot would catch silent regressions in - the chunker, packer, or restore planner. - -### performance - -- **Parallel blob downloads during restore.** Single-stream right now. With a - fast S3 endpoint and a multi-core machine restore is bound by per-blob fetch + - decrypt + decompress; running N of those in parallel against the disk cache - would close most of the remaining gap. Needs to interact correctly with the - locality planner and sweeper. -- **Bandwidth limiting (`--bwlimit`).** Both upload and download. Useful for - backing up over a shared link. Tricky to make work correctly with the - parallel-download story. -- **Restart of interrupted restore.** Today restore is restartable in the sense - that re-running it overwrites partial output; it doesn't resume from where it - stopped or skip already-present files. A `--resume` mode that checks targets - before fetching blobs would matter for very large restores. -- **Daemon mode.** A long-running mode that watches for file changes so frequent - backups, such as hourly, skip the full scan. It adds little for the usual runs - from cron every 12 to 36 hours. See - [issue #204](https://git.eeqj.de/sneak/vaultik/issues/204). - -### usability - -- **Man pages and richer `--help` examples.** Cobra generates basic help; man - pages would be a separate target. -- **`vaultik snapshot diff `** — show which files changed between two - snapshots without restoring either. -- **Status reporting hook for `--cron`.** When a backup fails silently in cron, - the user has no idea. A configurable webhook / email / `notify-send` hook on - completion (success and failure) would close the loop. - -### infrastructure - -- **Schema migrations after 1.0.** Migrations are supported, but none are added - before 1.0 because there is no installed base yet. After 1.0, each schema - change is a new migration, so an existing local index is migrated when vaultik - is updated (see [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations)). -- **Storage backend coverage tests.** S3, file://, and rclone:// all share the - Storer interface but the rclone path is the least exercised in CI. - ---- - -## output style - -Every command's user-facing output is governed by `internal/ui`, in one of two -ways. Color is enabled when the stream written to is a TTY and the `NO_COLOR` -environment variable is unset (https://no-color.org/). - -- **Status, progress, warnings, and errors** go through the `internal/ui` - message methods below: marker-prefixed, colored on a TTY, and — except - warnings and errors — silenced by `--quiet`. This is the operational narration - of the long-running commands (`snapshot create`, `prune`, `snapshot restore`, - and the like) and the confirmations of `config init`, `config set`, and - `database delete`. -- **The data a command exists to produce** is written plain, with no marker and - no color, because a marker would corrupt a table or a parsed document. This - covers the `version`, `info`, `remote info` and `snapshot verify` reports, the - `snapshot list` table, `config get` values, and every `--json` document. - `--quiet` silences the human reports and tables (`version`, `info`, - `remote info`, `snapshot verify`, `snapshot list`) but never the - machine-consumed `config get` value or the `--json` documents, which a script - depends on. The confirmation prompts of `database delete`, `prune`, - `snapshot purge` and `snapshot remove`, each with what it asks to delete, are - also written this way and always shown: they are an interactive exchange the - operator must see. - -`internal/ui` writes to stdout; it is the output the user asked for. The -exceptions are the startup banner and the error a failed command ends with, -which go to stderr. Structured log records are a different thing and go through -`internal/log`, which writes to stderr (see "stdout and stderr" above). - -Message classes: - -| Class | Marker | Alignment | Use for | -| -------- | ------------------------------ | --------- | ------------------------------------------------------------------------------ | -| Banner | none | column 0 | The startup line printed once per invocation, on stderr | -| Begin | `》` (white) | column 0 | An operation is about to start (present-continuous verb) | -| Complete | `》` (green) | column 0 | An operation just finished (past-tense verb) | -| Info | `》` (white) | column 0 | Neutral status update | -| Notice | `》` (cyan) | column 0 | Important note that is not a warning | -| Warning | `⚠️ Warning:` (orange/yellow) | column 0 | Recoverable problem | -| Error | `🛑 ERROR:` (red) | column 0 | Operation aborted | -| Progress | ` 》` (white) | column 2 | Heartbeat or per-item status during a long-running operation | -| Detail | ` 》` (white) | column 2 | Continuation/sub-line of a preceding Complete (visually identical to Progress) | - -Conventions: - -- Messages are complete English sentences ending with a period. -- Fully qualify terms — say "backup destination store" instead of "storage", - "snapshot source files enumeration" instead of "scan", "local index database" - instead of "database". -- Every operation that emits a Complete also emits a corresponding Begin. - Operations that print only a Begin (because completion is obvious from a later - Begin) should be rare and intentional. -- Use natural verb tense to signal state: "Uploading" for Begin, "Uploaded" for - Complete. Never write the words "begin" or "complete" in the body — the marker - color already conveys that. -- All elapsed and remaining-time fields are explicitly scoped to their subject: - write "blob upload elapsed: 30s, blob upload ETA: 03:15:00 (est remain 14s)", - never just "elapsed 30s, ETA 14s". -- "ETA" means an absolute clock time (when the operation will finish), not a - remaining-duration. Use `ui.Time()` for the former and `ui.Duration()` for the - latter, and label both. -- `ui.Time` formats same-day times as `HH:MM:SS` and other-day times as - `YYYY-MM-DD HH:MM:SS`. No timezone — local time is implied. - -Value colorizers in `internal/ui` colorize specific value types consistently. -Compose messages from these helpers rather than embedding ANSI escapes inline: - -| Helper | Color | Use for | -| ---------- | --------- | --------------------------------------------------------- | -| `Hex` | cyan | Blob hashes, chunk hashes (truncated to 12 chars + `...`) | -| `Snapshot` | bold cyan | Snapshot IDs (untruncated) | -| `Path` | blue | Filesystem paths | -| `Size` | magenta | Byte counts (human-readable) | -| `Speed` | magenta | Bytes-per-second rates | -| `Duration` | yellow | Elapsed or remaining time | -| `Time` | yellow | Absolute clock times | -| `Count` | magenta | Integer counts with thousands separators | -| `Percent` | magenta | Percentages | - -When `NO_COLOR` is set or output is not a TTY, all helpers return plain text and -the marker prefixes (`》`, `Warning:`, `ERROR:`) emit without ANSI escapes. The -emoji prefixes on Warning and Error are always emitted regardless of color -setting (emoji are not color). - -## requirements - -- Go 1.26 or later -- Docker, with a reachable daemon, to test, lint, check, or commit: - `script/test` and `script/lint` build the `test` and `lint` phases of the - `Dockerfile`, and `make check` and the pre-commit hook both run them. A - `golangci-lint` installed on `PATH` is not a substitute and is never used on a - host, whatever its version. -- S3-compatible object storage (or local filesystem, or rclone remote) - -## development workflow - -All changes follow this workflow. No exceptions. - -1. Create a feature branch off `main`. -2. Write tests. -3. Write the implementation. -4. Fix implementation errors until it compiles and tests pass. -5. Fix linting errors (`make lint`). -6. Update documentation and README as required by the change. -7. Format code (`make fmt`). -8. Run `make check` (lint + fmt-check + test). Fix any issues. Repeat until - clean. -9. Commit on the branch. -10. Merge to `main`. -11. Push. - -Do not commit directly to `main`. Do not skip steps. - -Repository policies for AI agents are in [`AGENTS.md`](AGENTS.md). - -## Entrypoints - -This repository adheres to the -[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) -standard: normalized scripts in `script/` are the entrypoints for the -development workflow, and the Makefile targets are thin shims that call them. We -provide: - -- `script/bootstrap` — install all development dependencies (Go, Go module - download, and node, yarn and `prettier` for Markdown). Node comes from the - host, or else at a pinned version through `nvm`. A host without Go gets the - `go.mod` version through `script/install-go`, in `.tool/go`, which - `script/bootstrap` itself, the `Makefile`, `script/fmt`, `script/fmt-check`, - `script/precommit` and `script/release` add to their `PATH`. It deliberately - does not install `golangci-lint`; see `script/lint` below. -- `script/setup` — make a fresh clone ready for development: runs - `script/bootstrap`, then `script/install-precommit` -- `script/projectname` — print the project name (used for the Docker image tag) -- `script/version` — print the version string to bake into the binary. The - `Makefile`'s `LDFLAGS` call this. See [releasing](#releasing) for the rules. -- `script/install-goreleaser` — install the pinned `goreleaser` into `.tool/bin` - from a sha256-verified release archive. Idempotent, and called by - `script/bootstrap`; the release workflow calls it directly because it needs - `goreleaser` but not the Docker daemon `script/bootstrap` insists on. -- `script/install-go` — install the Go toolchain named by `go.mod`'s `go` - directive into `.tool/go` from a sha256-verified `go.dev` archive, for Linux - or macOS on amd64 or arm64. Idempotent. On a CI runner it also puts - `.tool/go/bin` on `PATH` for the steps that follow. Called by the release - workflow, which needs a host Go for `goreleaser` to shell out to, and by - `script/bootstrap` on a host without Go. `actions/setup-go` is not used - because it verifies the downloaded toolchain against no value in this repo. - Bumping Go edits `go.mod`, the checksums in this script, and the two `golang` - digests in the `Dockerfile` together. -- `script/release` — cross-compile and publish the release artifacts with the - pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose version is not the - pinned one, because a different version would build a different release from - the same tag. -- `script/release-snapshot` — the same build with no publishing and no tagging, - into `./dist` -- `script/test` — run the test suite by building the `test` phase of the - `Dockerfile` (verbose rerun on failure). This runs _everything_: there is no - separate integration target and no build-tagged subset held back, so the full - round-trip tests in `internal/vaultik/integration_test.go` run on every - invocation. The 90s `-timeout` is a hang backstop rather than a performance - budget; it applies per test binary, to test execution only. -- `script/lint` — lint by building the `lint` phase of the `Dockerfile`, which - runs `golangci-lint config verify` and then - `golangci-lint run --config .golangci.yml ./...` as build steps in the - digest-pinned `golangci-lint` image. Nothing lints on the host. That `FROM` - line is the only pin of the linter version, and it changes in the same commit - as a re-vendored `.golangci.yml`. -- `script/lint-fix` — apply the linter's autofixes (rewrites files), using the - same pinned image, parsed out of the `lint` phase's `FROM` line. It cannot be - a build step, because fixes have to land in the worktree, so it bind-mounts - the tree into a `docker run` and therefore needs a _local_ daemon. It is a - developer convenience and never a gate: no gate reads its exit status. Run - `make lint` afterwards to find out whether the tree is clean. -- `script/fmt` — format all code (writes): Go with `go fmt`, Markdown with - `prettier` -- `script/fmt-check` — check formatting (read-only). It runs `gofmt` on the - host, over every Go file outside `.tool`, and `prettier --check` over every - Markdown file. -- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`. -- `script/docker` — build the Docker image tagged via `script/projectname`, - stamped with the version `git describe --tags --always --dirty` gives on the - host, or `unknown` outside a git checkout. The image's build stage depends on - the `lint` and `test` phases, so this lints and tests too. -- `script/cibuild` — CI entrypoint: runs `script/bootstrap`, `script/check`, and - then the same image build as `script/docker`. `.gitea/workflows/check.yml` - runs it on every push. - - Every `docker build` in these scripts passes `--no-cache`, because on an - unchanged tree a cached check layer is replayed without running and the - build still exits 0. A plain `docker build .` is therefore no evidence that - the checks ran. The cost is that `script/cibuild` runs the `lint` and `test` - phases twice: once in `script/check` and again in the image build. - -- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must not - change files), then `script/check` -- `script/install-precommit` — install the git pre-commit hook that runs - `script/precommit` - -## releasing - -### version numbers - -The version a binary reports comes from git, not from a constant in a file. It -is `git describe --tags --always --dirty`, which `script/version` runs for the -`Makefile`, and `script/docker` and `script/cibuild` run themselves: - -- `HEAD` is exactly on a tag → that tag, such as `v1.0.0`. -- a commit after a tag → `--g`. -- no tag reachable → the short commit sha. -- any of these, with uncommitted changes to tracked files → a `-dirty` suffix, - because a modified checkout of a tag is not that tag. - -A `docker build .` of a clone, with no build arguments, runs the same -`git describe` (without `--dirty`) on the `.git` in its build context, so it -stamps the same value for a clean commit; the build fails if the context carries -`.git` and no version, commit or commit date comes out. `make` outside a git -checkout and `docker build .` without `.git` both stamp `dev`. `script/docker` -and `script/cibuild` stamp `unknown` when they run outside a git checkout. - -A binary built by plain `go install` or `go build` carries no stamped version. -It reports the version Go recorded in it instead: - -- a `go install` of the tag `v1.0.0` → `v1.0.0`. -- a `go build` of an untagged commit → a Go pseudo-version, such as - `v1.0.1-0.20261110231226-0123456789ab`. -- a build where Go recorded no version → `dev`. - -`goreleaser` stamps a release binary with the tag minus its leading `v`, so the -tag `v1.0.0` produces `vaultik 1.0.0`, matching the archive name -`vaultik_1.0.0_linux_amd64.tar.gz`. `goreleaser --snapshot` stamps -`dev-<12 chars of the commit sha>` rather than inventing the next patch number. - -`vaultik version` calls a build a development build when its version is one of -these: - -- `dev` or `unknown`. -- `dev-`. -- the short commit sha. -- `--g`. -- a Go pseudo-version. -- any version with a `-dirty` or `+dirty` suffix. - -Only a plain tag, such as `v1.0.0` or `1.0.0`, is a release. If `script/version` -cannot be run at all, `make` stops with an error instead of building an -unversioned binary. A binary stamped with an empty version reports the version -Go recorded in it; if Go recorded none, it is a development build. - -### cutting a release - -Releases are cut by CI from a tag, not from a workstation: +### S3 ``` -git tag -a v1.2.3 -m 'v1.2.3' -git push origin v1.2.3 +s3://bucket/prefix?endpoint=s3.example.com®ion=us-east-1 ``` -`.gitea/workflows/release.yml` triggers on `v*` tags, installs a Go toolchain -and the pinned `goreleaser`, and runs `script/release`, which builds -`linux,darwin × amd64,arm64` archives plus `checksums.txt` and publishes them to -this repository's Gitea releases as a draft. `.goreleaser.yaml` has a -`gitea_urls:` block pointing at `https://git.eeqj.de/api/v1`; without it -`goreleaser` would talk to the GitHub API. +Any S3-compatible object store works. -The workflow needs one repository Actions secret: +- `endpoint` is the host of the store. The default is `s3.amazonaws.com`. +- `region` overrides `s3.region`. +- `ssl=false` selects plain HTTP for an endpoint written without a scheme. +- No other parameter is accepted. -| Secret | What it is | -| --------------- | ------------------------------------------------------------------------------------------------------- | -| `RELEASE_TOKEN` | A Gitea access token with `write:repository` scope, owned by an account that can publish releases here. | +The credentials come from `s3.access_key_id` and `s3.secret_access_key` in the +config. A URL that carries credentials is refused. -It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic token is -deliberately not used: it is not guaranteed to carry release write access. - -The Go toolchain that compiles the released binaries is installed by -`script/install-go`, which downloads the version named by `go.mod` (currently -`1.26.1`, the same version the `Dockerfile` builder stage pins by digest) and -refuses the archive unless its sha256 matches the value committed in the script. -`goreleaser` shells out to `go` for every cross-compile, so without that step -the release would either fail outright or ship binaries built by whatever -unpinned toolchain the runner happened to carry. - -To rehearse the whole build without publishing or tagging anything: +### Local directory ``` -make release-snapshot +file:///path/to/backup ``` -Artifacts land in `./dist`, which is gitignored. +- The path must be absolute. +- A percent-escape in the path is decoded, so `my%20stick` names `my stick`. +- The directory must exist before the first backup. `vaultik` never creates it. -Release artifacts are not signed, carry no SBOM, and are not built reproducibly; -the archives contain the binary, `LICENSE`, and `README.md` only (no shell -completions or man page). +A directory on a removable volume is gone while the volume is not mounted. A +backup then fails. It does not write to the disk under the mount point. + +This does not hold when the destination is the mount point itself. The empty +mount point stays behind when the volume is unplugged. There only +[the marker file](#the-marker-file) stops the backup. + +### rclone + +``` +rclone://remote/path +``` + +This uses any storage provider that rclone supports. Define the remote first +with `rclone config`. `vaultik` reads rclone's own config file. + +The URL takes no parameters and no credentials. + +### Interrupted uploads + +An upload that is cut off must not leave a partial object under the final name. +A later backup would take that object for complete. + +- S3 shows an object only once its upload has finished. +- The local directory destination writes a temporary file and then renames it. +- An rclone remote that can move objects is written the same way. The temporary + name ends in `.partial`. +- An rclone remote that cannot move objects is written in place. On such a + remote, a killed upload can leave a partial object under the final name. + +A leftover `.partial` file is ignored. You can delete it. + +### Destination permissions + +Each command needs different permissions at the destination. + +- `snapshot create` needs only permission to write objects. A machine you back + up can therefore hold credentials that can neither read nor delete. +- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list + and read. +- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete. +- `prune` and `snapshot create --prune` list, read and delete. + +A machine that runs `snapshot create --prune` can delete its own backups. To +keep it to write-only credentials, run `prune` from a separate, trusted machine. + +A backup tries to look up the marker file and each blob before it writes. It +goes on when the destination refuses those look-ups. + +### The marker file + +Every backup writes `vaultik-destination.txt` at the top of the destination. The +file holds one fixed line of text. + +The marker tells an emptied or missing destination apart from a new one. An +unplugged volume leaves an empty directory at its mount point. Without the +marker, that directory would look like a destination whose snapshots are all +gone. + +The check applies while the local index holds snapshots. When the destination +then answers that the marker does not exist, commands treat the destination as +not present: + +- `snapshot create` refuses to write to it. +- `prune`, `snapshot purge` and `remote nuke` fail. +- `snapshot list` shows the local index alone. +- `snapshot remove` deletes nothing at the destination. + +To fix this, mount the volume. If you emptied the destination on purpose, run +`vaultik database delete` and back up again. + +The check is skipped when the look-up fails for any other reason. S3 refuses the +look-up to credentials that can only write. A machine with such credentials +therefore does not notice an emptied destination. Its next backup would leave +out the data its local index records as stored. After you empty a destination, +run `vaultik database delete` on such a machine before it backs up again. + +### Changing the destination + +The local index belongs to one destination. It records which blobs are stored +there. + +The first command that changes data records the configured destination in the +local index. Later, these commands refuse to run when the config names a +different destination: + +- `snapshot create` +- `snapshot purge` +- `snapshot remove` +- `prune` +- `remote nuke` + +The destination is `storage_url`, or else `s3.endpoint`, `s3.bucket` and +`s3.prefix`. + +There are two ways out: + +- Change the config back to the earlier destination. +- Run `vaultik database delete`. The next backup then starts a new index for the + new destination. + +### Sharing a destination + +Give each machine and each user its own destination, such as its own prefix or +directory. The rest of this section applies only when two of them share one. + +A prune from one machine can meet a backup from another that is still running. A +backup uploads its blobs first, then its database, and its manifest last. Until +the manifest exists, no manifest lists the new blobs. + +`prune_grace_period` protects those blobs. A prune keeps a blob that no manifest +lists until the blob is older than the grace period. It keeps a directory under +`metadata/` that has no manifest for the same time. Set the grace period longer +than the longest backup of any machine or user that shares the destination. + +The grace period does not protect one case. A backup does not upload a blob that +is already at the destination. Such a blob keeps the time of its first upload. +It can be left over from an earlier backup that stopped before its manifest. A +prune from another machine can then delete that blob while the new backup runs. +The new snapshot cannot be restored afterwards. + +`remote nuke` ignores the grace period. + +## Locking + +Commands that change data take a lock, so that only one of them runs at a time. +These commands take it: + +- `snapshot create` +- `snapshot purge` +- `snapshot remove` +- `prune` +- `remote nuke` + +A second one does not wait. It exits at once with an "already running" error. + +The lock is the file `vaultik.pid` in the platform data directory. That is +`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/` +on macOS. + +The lock covers one user on one machine. It does not stop another user or +another machine from writing to the same destination. See +[Sharing a destination](#sharing-a-destination). + +No other command takes the lock. `snapshot list`, `snapshot verify`, +`snapshot restore`, `info` and `remote info` run while a backup is in progress. + +## Output + +### stdout and stderr + +stdout carries the output you asked for: + +- status and progress lines +- tables and reports +- JSON documents +- `config get` values +- completion scripts + +stderr carries the rest: + +- the startup banner +- log records, including those from `--verbose` and `--debug` +- the error that a failed command ends with + +So `vaultik snapshot list > out.txt` captures the list and leaves the +diagnostics on your terminal. + +Log records are colored lines when stderr is a terminal. They are JSON, one +object per line, when stderr is redirected. + +Status lines are colored only on a terminal, and never when `NO_COLOR` is set. + +### `--json` + +Under `--json`, stdout holds one JSON document and nothing else. The banner and +the status lines are left out, and warnings go to stderr. A command can +therefore be piped as it is: + +```sh +vaultik snapshot list --json | jq . +``` + +### `--quiet` and `--cron` + +`--quiet` leaves out the banner, the status lines, the tables and the reports. +`--cron` on `snapshot create` does the same. + +Neither flag hides these: + +- warnings and errors +- `config get` values +- JSON documents +- confirmation prompts + +### Exit status + +- `0`: the command succeeded. +- `1`: the command failed. +- `130`: the command was interrupted by Ctrl-C or `SIGTERM`. + +An interrupted command removes its temporary files before it exits. It gives up +after 30 seconds. + +## Temporary files + +Temporary files go to `$TMPDIR`, or to `/tmp` when `TMPDIR` is not set. + +A backup needs free temporary space for these: + +- One blob at a time. Each blob is written whole to a temporary file before it + is uploaded, so allow about `blob_size_limit`. +- About twice that for an rclone remote that cannot stream uploads. +- Copies of the local index, made when the snapshot's database is prepared. + +With `TMPDIR` not set, SQLite writes one of those index copies to `/var/tmp`. + +A restore keeps the downloaded blobs on disk, decrypted, until no remaining file +needs them. `snapshot restore` and `snapshot verify --deep` also write the +decrypted snapshot database there. + +These files are readable only by their owner, and are removed when the command +ends. A `kill -9` or a power loss leaves them behind. Delete any leftover +`vaultik-*` entry in the temporary directory yourself. + +## What the destination reveals + +File contents, names, paths and metadata are encrypted. Someone who can read the +destination still learns these: + +- when each backup finished +- how many blobs each snapshot uses, and their sizes +- which blobs two snapshots share +- whether a hostname and snapshot name they guess exist there + +The last point holds because a remote key is a plain hash of the snapshot ID, +made without a secret. + +[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the +destination, what it protects and the risks it accepts. + +## Limitations + +- **No extended attributes.** ACLs, macOS Finder metadata, quarantine flags and + SELinux labels are not backed up. +- **No hard links.** Two hard links to one file are backed up as two files. The + content is stored once, but a restore creates two separate files. +- **No sparse files.** A sparse file is backed up at its full size, zeros + included, and restored as a full file. +- **No bandwidth limit.** Uploads and downloads use all the bandwidth there is. +- **One download at a time.** A restore downloads blobs one after another. +- **No device nodes, named pipes or sockets.** They are skipped without a + message. +- **No consistent view of changing files.** A file that changes while it is read + can be stored in a mixed state. +- **Owners need root.** A restore sets owners and groups only when it runs as + root. +- **No resume.** An interrupted restore starts again from the beginning. +- **The local index can become unusable after an update.** This applies before + version 1.0. Run `vaultik database delete` and then a full backup. The + destination is not affected. See + [Schema migrations](docs/DATAMODEL.md#schema-migrations). + +## Development + +[ARCHITECTURE.md](ARCHITECTURE.md) describes how the code is organised. +[docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index. +[`AGENTS.md`](AGENTS.md) holds the rules for AI agents. +[`REPO_POLICIES.md`](REPO_POLICIES.md) holds the repository policies. + +### Requirements + +- Go 1.26.1 or later. +- Docker, with a daemon you can reach. The tests and the linter run only inside + Docker. + +### Workflow + +- Make every change on a feature branch. +- Write the tests with the change. +- Update the documentation that the change affects. +- Run `make fmt`, then `make check`, before each commit. + +`make check` runs the tests, the linter and the format check. All three must +pass. + +### Entrypoints + +The scripts in `script/` are the entrypoints. The `Makefile` targets only call +them. The layout follows +[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all). + +- `script/bootstrap`: installs what development needs. It installs Go from + `script/install-go` when the machine has none, and it fails without Docker. +- `script/setup`: prepares a fresh clone. It runs `script/bootstrap` and + `script/install-precommit`. +- `script/test`: runs the whole test suite, by building the `test` stage of the + `Dockerfile`. +- `script/lint`: runs the linter, by building the `lint` stage of the + `Dockerfile`. The linter never runs on your machine. +- `script/lint-fix`: applies the linter's automatic fixes to your files. It + needs a local Docker daemon. Run `script/lint` afterwards. +- `script/fmt`: formats the Go code. +- `script/fmt-check`: checks the formatting and changes nothing. +- `script/check`: runs `script/test`, `script/lint` and `script/fmt-check`. +- `script/precommit`: runs `go mod tidy`, `go fmt` and `script/check`. The git + pre-commit hook calls it. +- `script/install-precommit`: installs that hook. +- `script/docker`: builds the Docker image. The build runs the linter and the + tests too. +- `script/cibuild`: what CI runs on every push. It runs `script/bootstrap`, + `script/check` and the image build. +- `script/version`: prints the version to build into the binary. +- `script/projectname`: prints the project name, which tags the Docker image. +- `script/install-go`: installs the Go version that `go.mod` names into + `.tool/go`. It checks the download against a checksum in the script. +- `script/install-goreleaser`: installs the pinned `goreleaser` into + `.tool/bin`, checked the same way. +- `script/release`: builds and publishes the release files. +- `script/release-snapshot`: builds the release files into `./dist` and + publishes nothing. + +Every `docker build` in these scripts uses `--no-cache`. A cached layer would +skip the tests or the linter and still report success. + +The test suite has no separate integration target. Every test runs every time. +Each test binary is stopped after 90 seconds. + +To change the Go version, change these together: `go.mod`, the version and +checksums in `script/install-go`, and the two `golang` image digests in the +`Dockerfile`. + +### Messages to the user + +Status lines, warnings and errors go through `internal/ui`. The data a command +exists to print is written plain, with no marker and no color. That covers +tables, reports, JSON documents and `config get` values. + +`internal/ui` has these kinds of message: + +- **Banner**: the startup line. No marker. +- **Begin**: an operation is starting. Marker `》`. +- **Complete**: an operation has finished. Marker `》`, in green. +- **Info**: a neutral status line. Marker `》`. +- **Notice**: a note the user should see. Marker `》`, in cyan. +- **Warning**: a problem the command survives. Prefix `⚠️ Warning:`. +- **Error**: the command stops. Prefix `🛑 ERROR:`. +- **Progress** and **Detail**: an indented line under another. Marker ` 》`. + +Messages follow these conventions: + +- A message is a full sentence that ends with a period. +- Name things in full. Write "backup destination store", not "storage". +- An operation that prints a Complete also prints a Begin. +- Use the tense to show the state: "Uploading" for a Begin, "Uploaded" for a + Complete. +- Say what a time belongs to, as in "blob upload elapsed: 30s". +- "ETA" is a clock time, not a remaining time. Print it with `ui.Time`, and + print a remaining time with `ui.Duration`. + +Build a message from the value helpers in `internal/ui`, and never write color +codes by hand: + +- `Hex`: a blob or chunk hash, cut to 12 characters. +- `Snapshot`: a snapshot ID. +- `Path`: a filesystem path. +- `Size`: a byte count. +- `Speed`: a transfer rate, in bits per second. +- `Duration`: an elapsed or remaining time. +- `Time`: a clock time, in local time. +- `Count`: a whole number, with thousands separators. +- `Percent`: a percentage. + +## Releasing + +### Version numbers + +The version a binary reports comes from git. It is the output of +`git describe --tags --always --dirty`: + +- On a tagged commit, it is the tag, such as `v1.0.0`. +- On a commit after a tag, it is `--g`. +- With no tag, it is the short commit. +- With uncommitted changes to tracked files, `-dirty` is added. + +`script/version` gives this value to the `Makefile`. `script/docker` and +`script/cibuild` compute it themselves. + +A binary built without git reports `dev`. An image built by `script/docker` or +`script/cibuild` outside a git checkout reports `unknown`. + +`goreleaser` builds the release binaries. It uses the tag without its leading +`v`, so the tag `v1.0.0` gives `vaultik 1.0.0`. `script/release-snapshot` stamps +`dev-` followed by 12 characters of the commit. + +`vaultik version` calls a build a release only when its version is a plain tag, +such as `v1.0.0` or `1.0.0`. Every other version is a development build. + +`make` stops with an error when `script/version` gives no version. + +### The release workflow + +`.gitea/workflows/release.yml` runs when a tag that starts with `v` is pushed. +It does this: + +- It installs Go with `script/install-go` and `goreleaser` with + `script/install-goreleaser`. +- It runs `script/release`. +- `script/release` builds archives for Linux and macOS on `amd64` and `arm64`, + and a `checksums.txt`. +- It publishes them to this repository's releases, as a draft. + +The workflow needs one repository secret, `RELEASE_TOKEN`. It is a Gitea access +token with the `write:repository` scope, from an account that can publish +releases here. + +`script/release` refuses a `goreleaser` on `PATH` that is not the pinned +version. + +`make release-snapshot` builds the same files into `./dist` and publishes +nothing. + +Each archive holds the binary, `LICENSE` and `README.md`. The release files are +not signed, have no SBOM and are not built reproducibly. ## license @@ -1122,8 +1347,7 @@ completions or man page). ## author -Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in -Berlin in the summer of 2025. +Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in Berlin in the summer of 2025. Released as a free software gift to the world, no strings attached.