# vaultik (ваултик) `vaultik` is an incremental backup tool. It encrypts your files with a public key and uploads them to an S3-compatible object store, a local directory or an rclone remote. The machine being backed up holds only the public key. It cannot decrypt its own backups. You keep the private key somewhere else and need it only to restore. ## Why Other backup tools, such as `restic`, `borg` and `duplicity`, are designed for machines that can hold a decryption key. `vaultik` is for machines that should not hold one. It was written to meet these requirements together: - open source - no passphrase or private key on the backed-up machine - incremental - compressed - encrypted - direct upload to S3-compatible storage, with no intermediate step or tool ## Features - Encryption with [age](https://age-encryption.org/) (X25519 and ChaCha20-Poly1305), to one or more public keys. - Incremental backups. A file is read again only when it has changed. - Deduplication. Identical content is stored once, across files and across snapshots. - zstd compression at a level you choose. - Small files are packed together into large objects. This keeps the number of S3 requests low. - File contents, names, paths and metadata are encrypted at the destination. - Three kinds of destination: S3, a local directory and any rclone remote. - Regular files, symlinks and directories are backed up, with their permissions, owners and modification times. - Pure Go, with no C code. It builds for Linux and macOS on `amd64` and `arm64`. ## Terms - **Snapshot**: one backup of the paths listed under one name in the config, taken at one time. - **Snapshot ID**: the name of a snapshot, in the form `__`. An example is `server1_home_2025-06-01T12:00:00Z`. - **Destination**: the place the backups are stored. - **Chunk**: a piece of a file's content. `vaultik` cuts each file into chunks and stores each distinct chunk once. - **Blob**: one object at the destination. It holds many chunks, compressed and then encrypted. - **Local index**: a SQLite database on the backed-up machine. It records what is already stored, so the next backup uploads only what changed. - **Manifest**: the list of blobs one snapshot uses. It is stored at the destination without encryption. - **Remote key**: a hash of the snapshot ID. It names the snapshot's directory at the destination, so the destination never shows the snapshot ID. ## Install ```sh go install sneak.berlin/go/vaultik/cmd/vaultik@latest ``` This needs Go 1.26.1 or later. Generating a key pair needs `age-keygen`, which comes with [age](https://age-encryption.org/). ## Quick start ```sh # Write a default config file. The command prints the path it wrote. vaultik config init # Generate a key pair. Keep this file somewhere safe, away from this machine. age-keygen -o vaultik_backup_private_key.txt grep 'public key' vaultik_backup_private_key.txt # Set the public key and the destination. vaultik config set age_recipients.0 age1YOUR_PUBLIC_KEY_HERE vaultik config set storage_url "file:///Volumes/usbstick/mybackup" # Create the destination directory. vaultik never creates it. mkdir /Volumes/usbstick/mybackup # Run the first backup. vaultik snapshot create # See what is stored. vaultik snapshot list ``` The default config is written for macOS. It backs up your home directory and `/Applications`, with a list of excludes. A configured path that does not exist stops the backup, so on another system run `vaultik config edit` first and change the `snapshots` section. On macOS, grant your terminal app Full Disk Access before the first backup. The setting is under System Settings, Privacy & Security, Full Disk Access. Without it the backup stops with a permission error on a protected directory. ## Everyday use ```sh # Check that a snapshot's blobs are all present, with the listed sizes. vaultik snapshot verify # Put the private key in the environment. Reading it from the file keeps it # out of your shell history. export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" # Download, decrypt and re-hash every blob of a snapshot. vaultik snapshot verify --deep # Restore a snapshot into a new directory that only you can write. vaultik snapshot restore ~/vaultik-restore # A daily cron line: back up, then keep four weeks of snapshots. # 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w ``` `--prune` deletes data at the destination. Read [Sharing a destination](#sharing-a-destination) before you use it on a destination that more than one machine or user writes to. ## Restoring on another machine You can restore on a machine that never ran the backup, such as a replacement for a lost one. That machine needs three things: - the `vaultik` binary - the age private key - the destination, and credentials that can list and read it It does not need the original local index, config file or hostname. ```sh # Create a config and point it at the original destination. vaultik config init vaultik config set storage_url "s3://bucket/prefix?endpoint=https://s3.example.com" vaultik config set s3.access_key_id "..." vaultik config set s3.secret_access_key "..." # See what the destination holds. This needs no private key. vaultik snapshot list ``` On this machine the local index is empty, so the list cannot show snapshot IDs. Each snapshot appears as ``, with the first 12 characters of its remote key. The snapshot ID is stored only in the local index and in the snapshot's encrypted database. Use that remote key in place of the snapshot ID: ```sh export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" # Restore everything, then check the chunk hashes of every restored file. vaultik snapshot restore --verify ~/vaultik-restore # Optionally, also check every blob of the snapshot at the destination. vaultik snapshot verify --deep ``` Any leading part of the remote key works, as long as it matches one snapshot. `snapshot list --json` gives the full 64-character key as `remote_key`. A machine that only restores does not need `age_recipients`. ## Commands ```sh vaultik config init vaultik config edit vaultik config get vaultik config set vaultik snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than ] vaultik snapshot list [--json] vaultik snapshot verify [--deep] [--json] vaultik snapshot purge (--keep-latest | --older-than ) [--snapshot ...] [--force] vaultik snapshot remove [--dry-run] [--force] [--local-only] [--json] vaultik snapshot restore [paths...] [--verify] vaultik prune [--force] [--json] vaultik info vaultik remote info [--json] vaultik remote nuke --force vaultik database delete [--force] vaultik completion vaultik version ``` `snapshot list` has the alias `snapshot ls`. `snapshot remove` has the alias `snapshot rm`. ### What every command needs `config init`, `completion` and `version` need nothing. `config edit`, `config get` and `config set` need a config file that exists. It does not have to be complete. Every other command needs a config file that loads. See [Configuration](#configuration) for the checks. Every other command except `database delete` also needs these: - Write access to the local index. The command creates the file, and its directory, when they do not exist. - For an `rclone://` destination, an rclone config file that defines the remote. Each command below lists what it needs beyond this. No command needs root, but a backup can read only what its user can read, and a restore sets file owners only as root. ### Global flags - `--config `: the config file to use. See [Where the config file is](#where-the-config-file-is). - `--verbose`, `-v`: write informational log records to stderr. - `--debug`: write debug log records to stderr. - `--quiet`, `-q`: print only warnings, errors and the output a script reads. See [Output](#output). - `--skip-errors`: skip a file that cannot be read during a backup, or cannot be restored during a restore. Without it the command stops at the first such file. `--skip-errors` applies to `snapshot create` and `snapshot restore` only. A command that skipped anything still finishes, and then exits non-zero. ### Durations `--keep-newer-than`, `--older-than` and `prune_grace_period` take a duration. - `d`: days. - `w`: weeks. - `mo`: months of 30 days. - `y`: years of 365 days. - `h`, `m` and `s`: hours, minutes and seconds, as in `36h` or `90m`. `m` means minutes and `mo` means months. Days, weeks, months and years take whole numbers and can be combined, as in `2w3d`. `prune_grace_period` accepts only hours, minutes and seconds. ### `config init` Writes a default config file, with a comment that explains each setting it lists. - It writes to the path from `--config`, or else from `VAULTIK_CONFIG`. - Otherwise it writes to `config.yml` in the platform config directory. That is `~/.config/vaultik/` on Linux and `~/Library/Application Support/vaultik/` on macOS. - As root it writes to `/etc/vaultik/config.yml` instead. - It creates the file with mode `0600`, because the file can hold credentials. - It refuses to overwrite a file that exists. ### `config edit` Opens the config file in `$EDITOR`. It uses `vi` when `EDITOR` is not set. ### `config get ` Prints one value from the config file. The key is a dotted path, as in `vaultik config get storage_url` or `vaultik config get age_recipients.0`. A list or a map prints as YAML. ### `config set ` Sets one value in the config file. The key is a dotted path, as in `vaultik config set compression_level 9`. - Comments and formatting in the file are kept. - A key that does not exist yet is created. A missing parent becomes a list when the next part of the path is a number, and a map otherwise. - In a list, the index one past the end appends an entry. - The command prints the key it set, and never the value. - It sets the file's mode to `0600` when group or others could read it. The command refuses a change that would stop the config from loading. An example is `vaultik config set compression_level 99`. The file is then left unchanged. A config that does not load yet is checked less strictly. An example is the file `config init` writes, which has no `storage_url`. There the command refuses only a value of the wrong type, such as a word where a number belongs. ### `snapshot create` Backs up the configured snapshots. With no names it backs up all of them. Each name gets its own snapshot. It needs: - At least one public key in `age_recipients`. - Read access to the configured paths. - Permission to write objects at the destination. See [Destination permissions](#destination-permissions). - Free temporary space. See [Temporary files](#temporary-files). - The [lock](#locking). It does not need the private key. A file is read again when any of these has changed since the last backup: - its size - its modification time - its mode - its owner or group - its link target Every other file is recorded in the new snapshot without being read. Flags: - `--cron`: print nothing when the backup succeeds. Warnings and errors are still printed. - `--prune`: delete old snapshots and unused data after the backup. See below. - `--keep-newer-than `: with `--prune`, keep every snapshot newer than this. The backup stops before it writes anything in these cases: - A `file://` destination directory does not exist. - The destination has no marker file, although the local index holds snapshots. See [The marker file](#the-marker-file). - The destination in the config is not the one the local index was used with. See [Changing the destination](#changing-the-destination). The backup also stops when a configured path is a symlink whose target does not exist. It stops when a configured path does not exist, unless `--skip-errors` is given. With `--skip-errors`, a file or directory that cannot be read is skipped. The snapshot is kept and counts each skipped entry as failed, and the command exits non-zero. An error while packing or uploading still stops the backup. A file that is removed during the backup is counted as failed. It does not change the exit status. #### `--prune` After a successful backup, `--prune` does two things: - It removes the older snapshots of each name it just backed up. It keeps only the newest one, or with `--keep-newer-than` every snapshot newer than that. - It then deletes at the destination what [`prune`](#prune) deletes. `--prune` also needs permission to list, read and delete at the destination. `--keep-newer-than` without `--prune` is an error. So is a value that is not a duration. Both are reported before the backup starts. The prune does not run after a backup that may have missed data. An older snapshot may then hold the only copy of what was missed. This applies when: - `--skip-errors` skipped a file or directory that could not be read. - A configured path held nothing, as the mount point of an unmounted volume does. - An earlier snapshot of the same name found a mount point that this backup did not find. In each case every older snapshot is kept. The command says why, names the path and exits non-zero. It does so on every backup until the cause is gone. A mount point here is a directory at or below a configured path that is on a different filesystem from its parent. It counts only when the backup finds an entry in it that is not excluded. A backup does not find it when the volume is not mounted, or when the directory is now excluded or no longer configured. If the cause is intended, such as a retired volume, remove the older snapshots yourself with `snapshot purge`. Take a path that is empty on purpose out of the config. ### `snapshot list` Lists the snapshots in the local index and the snapshots at the destination, as one table. Before it, the command prints the snapshot names and paths in the config. It needs permission to list and read at the destination. It needs no private key. The table has these columns: - `SNAPSHOT ID` - `TIMESTAMP`: in UTC. - `COMPRESSED SIZE`: the total size of the blobs the snapshot uses. - `UNCOMPRESSED SIZE`: the total size of its distinct chunks before compression. - `NEW CHUNK SIZE`: the size of the chunks this snapshot stored that were not stored before. A snapshot at the destination that is not in the local index is shown as ``. The dots stand for the first 12 characters of its remote key. Its last two columns read ``, because those sizes come from the local index. Its timestamp is the time its backup finished. The command reports these below the table: - Snapshots in the local index that are missing at the destination. Run `vaultik prune` to drop those records. - Remote-only snapshots whose manifest cannot be read. They are left out of the table. - Remote-only snapshots beyond the first 1000. They are left out of the table. - A local index that was used with a different destination. See [Changing the destination](#changing-the-destination). When the destination cannot be listed, the command warns and shows the local index alone. It still exits zero. It does the same when the destination has no marker file although the local index holds snapshots. `--json` prints a JSON array instead. Each entry has these fields: - `id`: the snapshot ID. It is empty for a remote-only snapshot. - `remote_key`: the full 64-character remote key. - `timestamp` - `compressed_size` - `uncompressed_size` and `new_chunk_size`: left out for a remote-only snapshot. - `locally_tracked`: whether the snapshot is in the local index. - `remote_present`: whether the snapshot is at the destination. It is `null` when the destination could not be listed. Under `--json`, every warning above goes to stderr. ### `snapshot verify ` Checks a snapshot at the destination. The argument is a snapshot ID, a remote key or the leading part of a remote key. Without `--deep`, it checks two things: - Every blob the manifest lists is present, with the size the manifest records. - The snapshot's encrypted database is present. This reads no blob contents and needs no private key. It needs permission to read at the destination, and to list it when you give a remote key. Flags: - `--deep`: download, decrypt and re-hash every blob. See below. - `--json`: print the result as JSON. The command exits non-zero when a check fails or cannot be made. #### `--deep` `--deep` needs the age private key. Set it in the `VAULTIK_AGE_SECRET_KEY` environment variable, or as `age_secret_key` in the config file. The environment variable wins when both are set. See [The private key](#the-private-key). `--deep` does these checks, in order: - It downloads and decrypts the snapshot's database. It checks that this is the database of the snapshot you named. - It checks that the manifest and the database list the same blobs, with the same sizes. - It checks that every blob is present with the recorded size. - It downloads and decrypts every blob. It checks the hash of each chunk, and the hash that names the blob. It stops at the first blob that fails. `--deep` proves that the stored data is intact and decrypts with your key. It does not prove who wrote the data. `--deep` writes the decrypted database to a temporary directory and removes it afterwards. See [Temporary files](#temporary-files). ### `snapshot purge` Removes old snapshots. You must give exactly one of `--keep-latest` and `--older-than`. It needs the [lock](#locking), and permission to list and delete at the destination. It needs no private key. Flags: - `--keep-latest`: keep only the newest snapshot of each name. - `--older-than `: remove every snapshot older than this. - `--snapshot `: act only on snapshots with this name. Repeat the flag for more names. - `--force`: do not ask for confirmation. The command acts only on snapshots in the local index. `--keep-latest` keeps the newest snapshot of each name, not the newest overall. A removed snapshot leaves the local index, and its directory under `metadata/` at the destination is deleted. Its blobs stay. Run `vaultik prune` afterwards to delete the blobs that no snapshot uses any more. The command first drops local records of snapshots that are no longer at the destination. It fails and removes nothing when the destination cannot be listed or has no marker file. ### `snapshot remove ` Removes one snapshot. The argument must be a snapshot ID. A remote key is not accepted. It needs the [lock](#locking), and permission to list and delete at the destination. It needs no private key. Flags: - `--force`, `-f`: do not ask for confirmation. - `--dry-run`: say what would be removed, and remove nothing. - `--local-only`: remove the snapshot from the local index only. The destination is not changed. - `--json`: print the result as JSON. This also skips the confirmation. The command removes the snapshot from the local index. It also deletes the snapshot's directory under `metadata/` at the destination. Its blobs stay. Run `vaultik prune` afterwards to delete the blobs that no snapshot uses any more. An ID that is in neither the local index nor the destination is an error. With `--local-only`, the snapshot must be in the local index. The deletion at the destination can fail, for example when the destination is unreachable or has no marker file. What happens then depends on where the snapshot was: - A snapshot in the local index is still removed from it, with a warning. Run the same command again later to delete its directory at the destination. `vaultik prune` does not delete it. - For a snapshot that is only at the destination, the failure is an error. ### `snapshot restore [paths...]` Restores files from a snapshot into ``. The first argument is a snapshot ID, a remote key or the leading part of a remote key. It needs: - The age private key, as for `snapshot verify --deep`. See [The private key](#the-private-key). - Permission to read at the destination, and to list it when you give a remote key. - Write access to ``. The command creates the directory when it does not exist. - Free temporary space for the decrypted blobs. See [Temporary files](#temporary-files). It does not need `age_recipients`, and it does not take the lock. Each file is restored under its original absolute path inside ``. For example, `/home/user/notes.txt` is restored to `/home/user/notes.txt`. With no paths, the command restores the whole snapshot. Each path names a file or a directory by its original absolute path: ```sh vaultik snapshot restore ~/vaultik-restore /home/user/documents ``` A path that matches nothing is named in a warning. The command restores the other paths and then exits non-zero. The command restores these: - regular files, symlinks and directories, empty directories included - permissions and modification times - owners and groups, when it runs as root Without root, restored files belong to the user who runs the command. The command then warns once if any owner or group differs from the snapshot. A file that already exists in `` is replaced. Running the same restore again starts from the beginning. Restore into a new directory that only you can write, not a shared one such as `/tmp`. No other user can then change the files while they are written. Flags: - `--verify`: after the restore, read every restored file and check its chunk hashes. With `--skip-errors`, a file that cannot be restored is reported and skipped, and the command exits non-zero. `--verify` does not check a skipped file. ### `prune` Removes what is no longer needed, from the local index and from the destination. It needs the [lock](#locking), and permission to list, read and delete at the destination. It needs no private key. It works in three steps: - It drops each snapshot from the local index whose directory is gone from the destination. - It removes unfinished snapshots from the local index. It also removes the file, chunk and blob records that no snapshot uses. - It reads every manifest at the destination. It then deletes each blob that no manifest lists and that is older than `prune_grace_period`. The third step also deletes a directory under `metadata/` that has no manifest. A backup leaves one when it stops before it uploads its manifest. Such a directory is deleted only when all of these hold: - Every object in it is older than `prune_grace_period`. - Its snapshot is not in the local index. - Its name has the form of a remote key. `prune_grace_period` defaults to `24h`. A newer blob is kept, because a backup that is still running may need it. See [Sharing a destination](#sharing-a-destination) for what the grace period does not protect. The command deletes nothing at the destination in these cases: - The destination cannot be listed. - The destination has no marker file, although the local index holds snapshots. - A manifest cannot be read. It exits non-zero when a blob or a directory cannot be deleted. The time to read the manifests grows with the number of snapshots. Flags: - `--force`: do not ask for confirmation before deleting at the destination. - `--json`: print the result as JSON. This also skips the confirmation. The JSON document has these fields: - `blobs_found`: the unused blobs old enough to delete. - `blobs_deleted` - `blobs_failed`: blobs that could not be deleted. Left out when zero. - `bytes_freed` - `blobs_too_new`: unused blobs that were kept. They are newer than the grace period, or the destination gave no time for them. - `metadata_dirs_deleted` - `metadata_dirs_failed`: left out when zero. ### `info` Prints the system details, the settings in use and figures from the local index. The figures are the size of the index and its counts of snapshots, blobs and files. ### `remote info` Reports what the destination holds: - the kind of destination and its location - the size of each snapshot's database and manifest - the number and size of all blobs - the number and size of the blobs that manifests list - the number and size of the blobs that no manifest lists It needs permission to list and read at the destination. It needs no private key. When a manifest cannot be read, the last figures are reported as unknown. That snapshot's own blob count and size are unknown too. The command skips a name under `metadata/` that is not a remote key, with a warning. It does not count a file under `blobs/` whose name is not a blob hash. `--json` prints the report as JSON. Unknown figures are `null`. `unreadable_manifests` lists the remote key of each manifest that could not be read. `skipped_manifest_count` counts the manifests under skipped names. ### `remote nuke --force` Deletes everything `vaultik` stored at the destination. This cannot be undone. It needs `--force`, the [lock](#locking), and permission to list and delete at the destination. It deletes these, in order: - every object under `metadata/` - every object under `blobs/` - the marker file It also removes each snapshot from the local index, once that snapshot's directory at the destination is deleted. The command fails before it deletes anything when the destination cannot be listed, or has no marker file although the local index holds snapshots. If objects remain afterwards, the command names each one and exits non-zero. Running it again deletes what is left. ### `database delete` Deletes the local index file. The destination is not changed. It needs a config file that loads, because the config gives the path of the index. The next backup then reads every file again. It skips the upload of a blob that the destination already holds under the same name. Use this command after you point the config at a different destination. See [Changing the destination](#changing-the-destination). `--force` skips the confirmation. ### `completion ` Prints a completion script for `bash`, `zsh`, `fish` or `powershell`. ```sh # zsh, for the current session source <(vaultik completion zsh) # zsh, permanently vaultik completion zsh > "${fpath[1]}/_vaultik" # bash, for the current session source <(vaultik completion bash) # bash, permanently, on Linux vaultik completion bash > /etc/bash_completion.d/vaultik # fish vaultik completion fish > ~/.config/fish/completions/vaultik.fish ``` ### `version` Prints the version, the commit, the build date and the platform. It also says when the binary is a development build, not a release. ## Configuration The config file is YAML. `vaultik config init` writes one to start from, and [`config.example.yml`](config.example.yml) is a longer example. A command refuses to start when the config does not load. The config loads when all of these hold: - Every entry in `age_recipients` is an age X25519 public key. - `snapshots` has at least one name, and every name has at least one path. - A destination is set, with credentials when it is S3. - Every other setting below is within its limits. `vaultik` warns when group or others can read the config file. A value written as `${ENV:NAME}` is replaced by the environment variable `NAME`. This keeps a credential out of the file. A leading `~` is expanded to your home directory in snapshot paths, in `index_path` and in a `file://~/...` destination. ### Where the config file is `vaultik` uses the first of these that applies: - the path given with `--config` - the path in the `VAULTIK_CONFIG` environment variable - `config.yml` in the platform config directory, which is `~/.config/vaultik/` on Linux and `~/Library/Application Support/vaultik/` on macOS - `/etc/vaultik/config.yml` A path from `--config` or `VAULTIK_CONFIG` must exist. ### Settings - `age_recipients` - The age public keys that backups are encrypted to. - Default: none. `snapshot create` needs at least one. No other command needs one. - Any one of the matching private keys can decrypt. - Adding a key later does not re-encrypt what is already stored. See [Accepted risks](docs/REPOSTRUCTURE.md#accepted-risks). - `age_secret_key` - The age private key. - Default: not set. - Only `snapshot restore` and `snapshot verify --deep` use it. - The `VAULTIK_AGE_SECRET_KEY` environment variable overrides it. - Do not set it on a machine you back up. See [The private key](#the-private-key). - `snapshots` - The named sets of paths to back up. Required. - `snapshots..paths`: the paths to back up under this name. - `snapshots..exclude`: exclude patterns for this name only. - `exclude` - Exclude patterns that apply to every snapshot. See [Exclude patterns](#exclude-patterns). - Default: none. - `storage_url` - The destination. See [Destinations](#destinations). - Required, unless the `s3` settings name the destination. - `s3.access_key_id` and `s3.secret_access_key` - The S3 credentials. Required for an S3 destination. - `s3.region` - The S3 region. Default: `us-east-1`. - `s3.part_size` - The size of each part of a multipart upload. Default: `5MiB`. - It must be between `5MiB` and `5GiB`. - A blob too large for 10,000 parts of this size is uploaded in larger parts. - `s3.endpoint`, `s3.bucket`, `s3.prefix` and `s3.use_ssl` - An older way to name an S3 destination. They are used only when `storage_url` is not set. - `s3.endpoint` and `s3.bucket` are then required. - `s3.use_ssl: false` selects plain HTTP for an endpoint written without a scheme. - `chunk_size` - The average size of a chunk. Default: `10MB`. Minimum: `1MB`. - A chunk is between a quarter of this size and four times this size. - A smaller size finds more duplicate content and makes the index larger. - `blob_size_limit` - The largest size of a blob. Default: `10GB`. - It must be at least four times `chunk_size`. - `compression_level` - The zstd compression level, from `1` to `19`. Default: `3`. - Levels `1` to `2`, `3` to `5`, `6` to `9` and `10` to `19` form four groups. Levels in one group compress the same way. - `prune_grace_period` - How old unused data must be before `prune` deletes it. Default: `24h`. - It applies to a blob that no manifest lists. It also applies to every object in a directory under `metadata/` that has no manifest. - `0s` deletes such data at once. A negative value is refused. - See [Sharing a destination](#sharing-a-destination). - `hostname` - The hostname used in snapshot IDs. Default: the system hostname. - Only the part before the first `.` is used. - `index_path` - The path of the local index. - Default: `index.sqlite` in the platform data directory. That is `~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/` on macOS. Sizes take a number of bytes or a unit. `MB` and `GB` are powers of 1000. `MiB` and `GiB` are powers of 1024. ### Environment variables - `VAULTIK_AGE_SECRET_KEY`: the age private key. See [The private key](#the-private-key). - `VAULTIK_CONFIG`: the path of the config file. `--config` overrides it. - `VAULTIK_INDEX_PATH`: the path of the local index. It overrides `index_path`. - `RCLONE_CONFIG`: the path of the rclone config file, for an `rclone://` destination. - `EDITOR`: the editor that `config edit` starts. - `TMPDIR`: the directory for temporary files. - `NO_COLOR`: when set, status lines have no color. - `VAULTIK_CPUPROFILE`: write a CPU profile to this path. For debugging. - `VAULTIK_MEMPROFILE`: write a heap profile to this path on exit. For debugging. ### The private key `snapshot restore` and `snapshot verify --deep` decrypt data, so they need the age private key. No other command needs it. Give the key in one of two ways: - the `VAULTIK_AGE_SECRET_KEY` environment variable - `age_secret_key` in the config file The environment variable wins when both are set. The value can be the whole file that `age-keygen` writes. Comments are ignored, and every key in the file is used. Set the variable from the file, so that the key is not typed into your shell history: ```sh export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)" ``` Do not put the private key on a machine you back up. Anyone who breaks into that machine could then decrypt its backups. Supply the key only on the machine you restore on. ### Exclude patterns A pattern excludes a file or a whole directory from a backup. Patterns are matched against the path below the configured path. - A pattern that starts with `/` matches from the configured path. In a snapshot of `~`, `/Library/Caches` matches only `~/Library/Caches`. - Any other pattern matches at every depth. `.cache` matches every file or directory named `.cache`. - `*` and `?` match within one name. `**` matches across directories. - A pattern that matches a directory excludes everything in it. `exclude` at the top of the config applies to every snapshot. `exclude` under a snapshot name adds patterns for that snapshot. ## Destinations `storage_url` selects one of three kinds of destination. ### S3 ``` s3://bucket/prefix?endpoint=s3.example.com®ion=us-east-1 ``` Any S3-compatible object store works. - `endpoint` is the host of the store. The default is `s3.amazonaws.com`. - `region` overrides `s3.region`. - `ssl=false` selects plain HTTP for an endpoint written without a scheme. - No other parameter is accepted. The credentials come from `s3.access_key_id` and `s3.secret_access_key` in the config. A URL that carries credentials is refused. ### Local directory ``` file:///path/to/backup ``` - The path must be absolute. - A percent-escape in the path is decoded, so `my%20stick` names `my stick`. - The directory must exist before the first backup. `vaultik` never creates it. A directory on a removable volume is gone while the volume is not mounted. A backup then fails. It does not write to the disk under the mount point. This does not hold when the destination is the mount point itself. The empty mount point stays behind when the volume is unplugged. There only [the marker file](#the-marker-file) stops the backup. ### rclone ``` rclone://remote/path ``` This uses any storage provider that rclone supports. Define the remote first with `rclone config`. `vaultik` reads rclone's own config file. The URL takes no parameters and no credentials. ### Interrupted uploads An upload that is cut off must not leave a partial object under the final name. A later backup would take that object for complete. - S3 shows an object only once its upload has finished. - The local directory destination writes a temporary file and then renames it. - An rclone remote that can move objects is written the same way. The temporary name ends in `.partial`. - An rclone remote that cannot move objects is written in place. On such a remote, a killed upload can leave a partial object under the final name. A leftover `.partial` file is ignored. You can delete it. ### Destination permissions Each command needs different permissions at the destination. - `snapshot create` needs only permission to write objects. A machine you back up can therefore hold credentials that can neither read nor delete. - `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list and read. - `snapshot purge`, `snapshot remove` and `remote nuke` list and delete. - `prune` and `snapshot create --prune` list, read and delete. A machine that runs `snapshot create --prune` can delete its own backups. To keep it to write-only credentials, run `prune` from a separate, trusted machine. A backup tries to look up the marker file and each blob before it writes. It goes on when the destination refuses those look-ups. ### The marker file Every backup writes `vaultik-destination.txt` at the top of the destination. The file holds one fixed line of text. The marker tells an emptied or missing destination apart from a new one. An unplugged volume leaves an empty directory at its mount point. Without the marker, that directory would look like a destination whose snapshots are all gone. The check applies while the local index holds snapshots. When the destination then answers that the marker does not exist, commands treat the destination as not present: - `snapshot create` refuses to write to it. - `prune`, `snapshot purge` and `remote nuke` fail. - `snapshot list` shows the local index alone. - `snapshot remove` deletes nothing at the destination. To fix this, mount the volume. If you emptied the destination on purpose, run `vaultik database delete` and back up again. The check is skipped when the look-up fails for any other reason. S3 refuses the look-up to credentials that can only write. A machine with such credentials therefore does not notice an emptied destination. Its next backup would leave out the data its local index records as stored. After you empty a destination, run `vaultik database delete` on such a machine before it backs up again. ### Changing the destination The local index belongs to one destination. It records which blobs are stored there. The first command that changes data records the configured destination in the local index. Later, these commands refuse to run when the config names a different destination: - `snapshot create` - `snapshot purge` - `snapshot remove` - `prune` - `remote nuke` The destination is `storage_url`, or else `s3.endpoint`, `s3.bucket` and `s3.prefix`. There are two ways out: - Change the config back to the earlier destination. - Run `vaultik database delete`. The next backup then starts a new index for the new destination. ### Sharing a destination Give each machine and each user its own destination, such as its own prefix or directory. The rest of this section applies only when two of them share one. A prune from one machine can meet a backup from another that is still running. A backup uploads its blobs first, then its database, and its manifest last. Until the manifest exists, no manifest lists the new blobs. `prune_grace_period` protects those blobs. A prune keeps a blob that no manifest lists until the blob is older than the grace period. It keeps a directory under `metadata/` that has no manifest for the same time. Set the grace period longer than the longest backup of any machine or user that shares the destination. The grace period does not protect one case. A backup does not upload a blob that is already at the destination. Such a blob keeps the time of its first upload. It can be left over from an earlier backup that stopped before its manifest. A prune from another machine can then delete that blob while the new backup runs. The new snapshot cannot be restored afterwards. `remote nuke` ignores the grace period. ## Locking Commands that change data take a lock, so that only one of them runs at a time. These commands take it: - `snapshot create` - `snapshot purge` - `snapshot remove` - `prune` - `remote nuke` A second one does not wait. It exits at once with an "already running" error. The lock is the file `vaultik.pid` in the platform data directory. That is `~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/` on macOS. The lock covers one user on one machine. It does not stop another user or another machine from writing to the same destination. See [Sharing a destination](#sharing-a-destination). No other command takes the lock. `snapshot list`, `snapshot verify`, `snapshot restore`, `info` and `remote info` run while a backup is in progress. ## Output ### stdout and stderr stdout carries the output you asked for: - status and progress lines - tables and reports - JSON documents - `config get` values - completion scripts stderr carries the rest: - the startup banner - log records, including those from `--verbose` and `--debug` - the error that a failed command ends with So `vaultik snapshot list > out.txt` captures the list and leaves the diagnostics on your terminal. Log records are colored lines when stderr is a terminal. They are JSON, one object per line, when stderr is redirected. Status lines are colored only on a terminal, and never when `NO_COLOR` is set. ### `--json` Under `--json`, stdout holds one JSON document and nothing else. The banner and the status lines are left out, and warnings go to stderr. A command can therefore be piped as it is: ```sh vaultik snapshot list --json | jq . ``` ### `--quiet` and `--cron` `--quiet` leaves out the banner, the status lines, the tables and the reports. `--cron` on `snapshot create` does the same. Neither flag hides these: - warnings and errors - `config get` values - JSON documents - confirmation prompts ### Exit status - `0`: the command succeeded. - `1`: the command failed. - `130`: the command was interrupted by Ctrl-C or `SIGTERM`. An interrupted command removes its temporary files before it exits. It gives up after 30 seconds. ## Temporary files Temporary files go to `$TMPDIR`, or to `/tmp` when `TMPDIR` is not set. A backup needs free temporary space for these: - One blob at a time. Each blob is written whole to a temporary file before it is uploaded, so allow about `blob_size_limit`. - About twice that for an rclone remote that cannot stream uploads. - Copies of the local index, made when the snapshot's database is prepared. With `TMPDIR` not set, SQLite writes one of those index copies to `/var/tmp`. A restore keeps the downloaded blobs on disk, decrypted, until no remaining file needs them. `snapshot restore` and `snapshot verify --deep` also write the decrypted snapshot database there. These files are readable only by their owner, and are removed when the command ends. A `kill -9` or a power loss leaves them behind. Delete any leftover `vaultik-*` entry in the temporary directory yourself. ## What the destination reveals File contents, names, paths and metadata are encrypted. Someone who can read the destination still learns these: - when each backup finished - how many blobs each snapshot uses, and their sizes - which blobs two snapshots share - whether a hostname and snapshot name they guess exist there The last point holds because a remote key is a plain hash of the snapshot ID, made without a secret. [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the destination, what it protects and the risks it accepts. ## Limitations - **No extended attributes.** ACLs, macOS Finder metadata, quarantine flags and SELinux labels are not backed up. - **No hard links.** Two hard links to one file are backed up as two files. The content is stored once, but a restore creates two separate files. - **No sparse files.** A sparse file is backed up at its full size, zeros included, and restored as a full file. - **No bandwidth limit.** Uploads and downloads use all the bandwidth there is. - **One download at a time.** A restore downloads blobs one after another. - **No device nodes, named pipes or sockets.** They are skipped without a message. - **No consistent view of changing files.** A file that changes while it is read can be stored in a mixed state. - **Owners need root.** A restore sets owners and groups only when it runs as root. - **No resume.** An interrupted restore starts again from the beginning. - **The local index can become unusable after an update.** This applies before version 1.0. Run `vaultik database delete` and then a full backup. The destination is not affected. See [Schema migrations](docs/DATAMODEL.md#schema-migrations). ## Development [ARCHITECTURE.md](ARCHITECTURE.md) describes how the code is organised. [docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index. [`AGENTS.md`](AGENTS.md) holds the rules for AI agents. [`REPO_POLICIES.md`](REPO_POLICIES.md) holds the repository policies. ### Requirements - Go 1.26.1 or later. - Docker, with a daemon you can reach. The tests and the linter run only inside Docker. ### Workflow - Make every change on a feature branch. - Write the tests with the change. - Update the documentation that the change affects. - Run `make fmt`, then `make check`, before each commit. `make check` runs the tests, the linter and the format check. All three must pass. ### Entrypoints The scripts in `script/` are the entrypoints. The `Makefile` targets only call them. The layout follows [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all). - `script/bootstrap`: installs what development needs. It installs Go from `script/install-go` when the machine has none, and it fails without Docker. - `script/setup`: prepares a fresh clone. It runs `script/bootstrap` and `script/install-precommit`. - `script/test`: runs the whole test suite, by building the `test` stage of the `Dockerfile`. - `script/lint`: runs the linter, by building the `lint` stage of the `Dockerfile`. The linter never runs on your machine. - `script/lint-fix`: applies the linter's automatic fixes to your files. It needs a local Docker daemon. Run `script/lint` afterwards. - `script/fmt`: formats the Go code. - `script/fmt-check`: checks the formatting and changes nothing. - `script/check`: runs `script/test`, `script/lint` and `script/fmt-check`. - `script/precommit`: runs `go mod tidy`, `go fmt` and `script/check`. The git pre-commit hook calls it. - `script/install-precommit`: installs that hook. - `script/docker`: builds the Docker image. The build runs the linter and the tests too. - `script/cibuild`: what CI runs on every push. It runs `script/bootstrap`, `script/check` and the image build. - `script/version`: prints the version to build into the binary. - `script/projectname`: prints the project name, which tags the Docker image. - `script/install-go`: installs the Go version that `go.mod` names into `.tool/go`. It checks the download against a checksum in the script. - `script/install-goreleaser`: installs the pinned `goreleaser` into `.tool/bin`, checked the same way. - `script/release`: builds and publishes the release files. - `script/release-snapshot`: builds the release files into `./dist` and publishes nothing. Every `docker build` in these scripts uses `--no-cache`. A cached layer would skip the tests or the linter and still report success. The test suite has no separate integration target. Every test runs every time. Each test binary is stopped after 90 seconds. To change the Go version, change these together: `go.mod`, the version and checksums in `script/install-go`, and the two `golang` image digests in the `Dockerfile`. ### Messages to the user Status lines, warnings and errors go through `internal/ui`. The data a command exists to print is written plain, with no marker and no color. That covers tables, reports, JSON documents and `config get` values. `internal/ui` has these kinds of message: - **Banner**: the startup line. No marker. - **Begin**: an operation is starting. Marker `》`. - **Complete**: an operation has finished. Marker `》`, in green. - **Info**: a neutral status line. Marker `》`. - **Notice**: a note the user should see. Marker `》`, in cyan. - **Warning**: a problem the command survives. Prefix `⚠️ Warning:`. - **Error**: the command stops. Prefix `🛑 ERROR:`. - **Progress** and **Detail**: an indented line under another. Marker ` 》`. Messages follow these conventions: - A message is a full sentence that ends with a period. - Name things in full. Write "backup destination store", not "storage". - An operation that prints a Complete also prints a Begin. - Use the tense to show the state: "Uploading" for a Begin, "Uploaded" for a Complete. - Say what a time belongs to, as in "blob upload elapsed: 30s". - "ETA" is a clock time, not a remaining time. Print it with `ui.Time`, and print a remaining time with `ui.Duration`. Build a message from the value helpers in `internal/ui`, and never write color codes by hand: - `Hex`: a blob or chunk hash, cut to 12 characters. - `Snapshot`: a snapshot ID. - `Path`: a filesystem path. - `Size`: a byte count. - `Speed`: a transfer rate, in bits per second. - `Duration`: an elapsed or remaining time. - `Time`: a clock time, in local time. - `Count`: a whole number, with thousands separators. - `Percent`: a percentage. ## Releasing ### Version numbers The version a binary reports comes from git. It is the output of `git describe --tags --always --dirty`: - On a tagged commit, it is the tag, such as `v1.0.0`. - On a commit after a tag, it is `--g`. - With no tag, it is the short commit. - With uncommitted changes to tracked files, `-dirty` is added. `script/version` gives this value to the `Makefile`. `script/docker` and `script/cibuild` compute it themselves. A binary built without git reports `dev`. An image built by `script/docker` or `script/cibuild` outside a git checkout reports `unknown`. `goreleaser` builds the release binaries. It uses the tag without its leading `v`, so the tag `v1.0.0` gives `vaultik 1.0.0`. `script/release-snapshot` stamps `dev-` followed by 12 characters of the commit. `vaultik version` calls a build a release only when its version is a plain tag, such as `v1.0.0` or `1.0.0`. Every other version is a development build. `make` stops with an error when `script/version` gives no version. ### The release workflow `.gitea/workflows/release.yml` runs when a tag that starts with `v` is pushed. It does this: - It installs Go with `script/install-go` and `goreleaser` with `script/install-goreleaser`. - It runs `script/release`. - `script/release` builds archives for Linux and macOS on `amd64` and `arm64`, and a `checksums.txt`. - It publishes them to this repository's releases, as a draft. The workflow needs one repository secret, `RELEASE_TOKEN`. It is a Gitea access token with the `write:repository` scope, from an account that can publish releases here. `script/release` refuses a `goreleaser` on `PATH` that is not the pinned version. `make release-snapshot` builds the same files into `./dist` and publishes nothing. Each archive holds the binary, `LICENSE` and `README.md`. The release files are not signed, have no SBOM and are not built reproducibly. ## license [MIT](https://opensource.org/license/mit/) ## author Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in Berlin in the summer of 2025. Released as a free software gift to the world, no strings attached. Contact: [sneak@sneak.berlin](mailto:sneak@sneak.berlin) [https://keys.openpgp.org/vks/v1/by-fingerprint/5539AD00DE4C42F3AFE11575052443F4DF2A55C2](https://keys.openpgp.org/vks/v1/by-fingerprint/5539AD00DE4C42F3AFE11575052443F4DF2A55C2)