vaultik/README.md

# vaultik (ваултик)

`vaultik` is an incremental backup tool written in Go. It encrypts data
using an `age` public key and uploads each encrypted blob directly to a
remote S3-compatible object store. It requires no private keys, secrets, or
credentials (other than those required to PUT to encrypted object storage,
such as S3 API keys) stored on the backed-up system.

## quickstart

```sh
# install
go install sneak.berlin/go/vaultik/cmd/vaultik@latest

# create a default config file (prints the path it wrote to)
vaultik config init

# generate an age keypair; keep the private key file somewhere safe and
# offline — you need it to restore, and the backed-up machine does not need it
age-keygen -o vaultik_backup_private_key.txt
grep 'public key' vaultik_backup_private_key.txt

# configure the encryption key and backup destination
vaultik config set age_recipients.0 age1YOUR_PUBLIC_KEY_HERE
vaultik config set storage_url "file:///Volumes/usbstick/mybackup"

# macOS only: grant your terminal app Full Disk Access first
# (System Settings → Privacy & Security → Full Disk Access), otherwise
# the backup will abort with a permission error on protected directories

# run your first backup (the default config backs up ~ and /Applications
# with sensible excludes)
vaultik snapshot create

# see what you have
vaultik snapshot list
```

Features:

* modern encryption ([age](https://age-encryption.org/), X25519 + XChaCha20-Poly1305)
* content-defined chunking with deduplication (FastCDC)
* incremental backups (only changed files are re-chunked)
* multithreaded zstd compression at configurable levels
* content-addressed immutable storage
* local state tracking in SQLite (enables write-only incremental backups)
* no mutable remote metadata
* no plaintext file paths or metadata in remote storage
* packs small files into large blobs (keeps S3 operation counts down)
* backs up regular files, symlinks, empty directories, and file permissions
* pluggable storage backends: S3, local filesystem, rclone (70+ providers)
* pure Go (no CGO), cross-compiles to linux/darwin × amd64/arm64

## why

Other backup tools like `restic`, `borg`, and `duplicity` are designed for
environments where the source host can store secrets and has access to
decryption keys. `vaultik` is for environments where you don't want to
store backup decryption keys on your hosts — only public keys for
encryption.

Requirements that no existing tool meets:

* open source
* no passphrases or private keys on the source host
* incremental
* compressed
* encrypted
* s3 compatible without an intermediate step or tool

## daily use

```sh
# verify a snapshot (shallow: checks all blobs exist)
vaultik snapshot verify <snapshot-id>

# deep verify (downloads and cryptographically verifies every blob)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot verify --deep <snapshot-id>

# restore (requires the private key)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot restore <snapshot-id> /tmp/restored

# daily cron job: back up, keep a 4-week rolling window of snapshots
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
```

## restoring on another machine

Restoring on a host that never ran the backup — a replacement machine
after the original is gone — is the case vaultik is built for. That host
needs only three things: the `vaultik` binary, the age **private** key,
and the storage credentials for the destination. It does **not** need the
local index, the original config file, or the original hostname.

```sh
# install
go install sneak.berlin/go/vaultik/cmd/vaultik@latest

# create a config and point it at the ORIGINAL backup destination
vaultik config init
vaultik config set storage_url "s3://bucket/prefix?endpoint=https://s3.example.com"
vaultik config set s3.access_key_id "..."
vaultik config set s3.secret_access_key "..."

# see what is on the destination store
vaultik snapshot list
```

`snapshot list` reads the destination store without the private key. A
snapshot that is not in this host's (empty) local index is shown as
remote-only: its row is identified by `<remote only:...>` rather than by
a `hostname_name_timestamp` name, because the name lives only in the
local index and the encrypted database and cannot be recovered from the
store. Its timestamp and compressed size are real. (See the `snapshot
list` description under [command details](#command-details) for the full
explanation.)

Use that remote key — the hex printed inside `<remote only:...>`, or the
full `remote_key` from `snapshot list --json` — to restore and verify:

```sh
# restore everything to /tmp/restored, then check every restored file's
# chunk hashes
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
  vaultik snapshot restore --verify <remote-key> /tmp/restored

# optionally, deep-verify the snapshot against the store (downloads and
# cryptographically checks every blob)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
  vaultik snapshot verify --deep <remote-key>
```

`age_recipients` (the public key) is not needed to restore — only the
private key in `VAULTIK_AGE_SECRET_KEY`. Both the abbreviated key printed
in the table and the full 64-character key from `--json` are accepted; a
leading part of the key is enough as long as it is unambiguous.

---

## cli

### commands

```sh
vaultik [--config <path>] config init
vaultik [--config <path>] config edit
vaultik [--config <path>] config get <key>
vaultik [--config <path>] config set <key> <value>
vaultik [--config <path>] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>]
vaultik [--config <path>] snapshot list [--json]
vaultik [--config <path>] snapshot verify <snapshot-id> [--deep] [--json]
vaultik [--config <path>] snapshot purge [--keep-latest | --older-than <duration>] [--snapshot <name>...] [--force]
vaultik [--config <path>] snapshot remove <snapshot-id> [--dry-run] [--force] [--local-only] [--json]
vaultik [--config <path>] snapshot restore <snapshot-id> <target-dir> [paths...] [--verify]
vaultik [--config <path>] prune [--force] [--json]
vaultik [--config <path>] info
vaultik [--config <path>] remote info [--json]
vaultik [--config <path>] remote nuke --force
vaultik [--config <path>] database delete [--force]
vaultik completion <bash|zsh|fish|powershell>
vaultik version
```

### global flags

* `--config <path>`: Path to config file (default: `$VAULTIK_CONFIG`, then platform config dir, then `/etc/vaultik/config.yml`)
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
* `--debug`: Enable debug output (on stderr — see below)
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
* `--skip-errors`: Continue past per-file errors instead of aborting (applies to `snapshot create` and `restore`)

### stdout and stderr

Log output — everything from `--verbose` and `--debug`, and every
warning and error the logger emits — goes to **stderr**. stdout carries
the output you asked for: tables, and the documents produced by `--json`.

This means `vaultik snapshot list --verbose > out.txt` captures the
listing and leaves the diagnostics on your terminal. To capture both,
redirect stderr as well (`> out.txt 2> log.txt`, or `> out.txt 2>&1` to
interleave them).

The split is what makes `--json` usable from a script. Warnings and
errors are never suppressed — not by `--quiet`, not by `--cron` — so a
logger on stdout would eventually land a log line inside a JSON
document and break the parse. A config file with group- or
world-readable permissions is enough to trigger it.

Format follows the stream: when stderr is a terminal the records are
colorized one-liners, and when it is redirected or piped they are
JSON, one object per line.

Under `--json`, stdout holds the document and nothing else. The startup
banner is suppressed, as `--quiet` and `--cron` suppress it, and the
progress narration a command would otherwise print — such as the stale
local records `prune` reconciles away — is suppressed too, so it cannot
land ahead of the document. Every `--json` command therefore pipes on
its own, with no additional flag: `vaultik snapshot list --json | jq .`
and `vaultik prune --json | jq .` both work as written.

### environment variables

* `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for `snapshot restore` and `snapshot verify --deep`)
* `VAULTIK_CONFIG`: Path to config file (overridden by `--config`)
* `VAULTIK_INDEX_PATH`: Override local SQLite index path

### shell completion

```sh
# zsh: load for the current session
source <(vaultik completion zsh)

# zsh: install permanently
vaultik completion zsh > "${fpath[1]}/_vaultik"

# bash: load for the current session
source <(vaultik completion bash)

# bash: install permanently (Linux)
vaultik completion bash > /etc/bash_completion.d/vaultik

# fish
vaultik completion fish > ~/.config/fish/completions/vaultik.fish
```

### command details

**`config init`**: Write a default config file with commented explanations for
every setting. Writes to the path from `--config`, `$VAULTIK_CONFIG`, or the
platform config directory (`~/Library/Application Support/vaultik/` on macOS,
`~/.config/vaultik/` on Linux, `/etc/vaultik/` as root). Refuses to overwrite an
existing file. Created with mode `0600` since it will contain credentials.

**`config edit`**: Open the config file in `$EDITOR` (falls back to `vi`).

**`config get`**: Print a config value addressed by dotted YAML path
(e.g. `vaultik config get storage_url`). Non-scalar values print as YAML.

**`config set`**: Set a scalar config value by dotted YAML path
(e.g. `vaultik config set compression_level 9`,
`vaultik config set storage_url "file:///mnt/backups"`). Comments and
formatting in the file are preserved; intermediate maps are created as
needed.

**`snapshot create`**: Perform incremental backup of configured snapshots.
* Optional snapshot names argument to create specific snapshots (default: all)
* On macOS, the terminal application running vaultik needs Full Disk Access
  (System Settings → Privacy & Security → Full Disk Access) to read
  TCC-protected directories; without it the backup aborts with a permission
  error that explains how to fix it
* `--cron`: Silent on total success; warnings and errors are still printed
  (for crontab)
* `--prune`: After backup, drop older snapshots of each backed-up name and
  remove orphaned blobs from remote storage. By default keeps only the latest
  snapshot per name; use `--keep-newer-than` for a rolling window.
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
  this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`)

**`snapshot list`**: Show every snapshot known to this host — the union
of the local index and the backup destination store — with timestamps
and three sizes per snapshot (compressed remote size; total
uncompressed chunk size; size of chunks newly referenced by that
snapshot).

Listing the destination store does **not** require the age secret key,
so it works in vaultik's intended configuration, where the backed-up
host holds only the public key. A host that has lost its local index
can still see what it has backed up.

What that host cannot see is a remote-only snapshot's name. The
snapshot ID is hashed at the storage boundary and the manifest records
only the hash, so hostname and snapshot name exist solely in the local
index and in the encrypted per-snapshot database. Snapshots found only
on the destination store are therefore listed as
`<remote only:<abbreviated remote key>>` and show `<remote only>` in
the uncompressed and "new chunk" columns, which can only be computed
from the local index. Their timestamp and compressed size are real,
read from the manifest.

Snapshots in the local index with no counterpart on the destination
store are reported below the table as drift, with the `vaultik prune`
invocation that reconciles them.

If the destination store cannot be listed (unmounted volume,
permission denied, network down), the command warns, falls back to the
local index alone, and still exits zero.
* `--json`: Output in JSON format. Each entry carries `locally_tracked`
  (whether the snapshot is in the local index), `remote_key` (the full
  64-character storage key), and `remote_present` (whether it was seen
  on the destination store, or `null` if the destination could not be
  listed). Warnings about an unlistable destination, unreadable
  manifests, and a truncated listing all go to stderr through the
  logger, so stdout stays a single parseable document.

**`snapshot verify`**: Verify snapshot integrity.
* Default (shallow): checks that all blobs referenced in the manifest exist in storage
* `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the
  encrypted metadata database
* Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a
  remote-only snapshot's remote key (or an unambiguous leading part of it)
* `--json`: Output results as JSON

**`snapshot purge`**: Remove old snapshots based on criteria. Retention is
per-snapshot-name (`--keep-latest` keeps the latest of each name, not the
latest globally).
* `--keep-latest`: Keep only the most recent snapshot of each name
* `--older-than <duration>`: Remove snapshots older than duration (e.g. `30d`,
  `4w`, `6mo`, `1y`; `m` is minutes, `mo` is months)
* `--snapshot <name>`: Restrict to specific snapshot names (repeat for multiple)
* `--force`: Skip confirmation prompt

**`snapshot remove`**: Remove one snapshot. By default this removes the
snapshot from the local index and strips the snapshot's metadata from
the backup destination store. Blobs are NOT touched — deleting blobs
requires reading every remaining remote manifest (the destination store
may hold snapshots this host doesn't know about), which is what
`vaultik prune` does. On success the command prints the exact `vaultik
prune` invocation to run as a follow-up. Local row cleanup (files,
chunks, blobs the snapshot was the last referrer for) runs
automatically. If the destination store is unreachable, the local-DB
removal still completes and a warning is emitted; rerun `vaultik prune`
once the store is reachable to finish remote cleanup. To wipe everything
on the destination in one go, use `vaultik remote nuke --force`.
* `--local-only`: Skip remote cleanup; only touch the local index
* `--dry-run`: Show what would be deleted without deleting
* `--force`: Skip confirmation prompt
* `--json`: Output result as JSON

**`snapshot restore`**: Restore files from a backup snapshot.
* Requires `VAULTIK_AGE_SECRET_KEY` environment variable
* Accepts a snapshot ID, or — for a snapshot only on the destination
  store — its remote key (or an unambiguous leading part of it) as shown
  by `snapshot list`. See
  [restoring on another machine](#restoring-on-another-machine).
* Optional path arguments to restore specific files/directories (default: all)
* Preserves file permissions, timestamps, ownership (ownership requires root),
  symlinks, and empty directories
* `--verify`: After restoring, verify every file's chunk hashes match

**`prune`**: Tidy up everything that isn't needed. Runs three passes:
(1) reconcile the local index against the destination store — any
local snapshot whose remote metadata is missing is dropped from the
local index; (2) delete orphaned local rows (files, chunks, blobs no
longer referenced by any completed snapshot); (3) list every remote
manifest on the destination store to compute the still-referenced blob
set and delete any blob not in that set. Step (3) reads all remote
manifests — network cost scales with the number of snapshots. `snapshot
create --prune` runs the same cleanup automatically; this is the
manual entry point for the same work.
* `--force`: Skip confirmation prompt
* `--json`: Output stats as JSON

**`info`**: Display system configuration, storage settings, encryption
recipients, and local database statistics.

**`remote info`**: Show storage backend type and location plus detailed
remote storage inventory: per-snapshot metadata sizes, blob counts, and
orphaned blob detection.
* `--json`: Output as JSON

**`remote nuke`**: Delete every snapshot's metadata and every blob from the
backup destination store, leaving the bucket prefix empty. Destructive and
irreversible. This is the single supported way to wipe the entire
destination store.
* `--force`: Required to confirm destruction.

**`database delete`**: Delete the local SQLite state database file
entirely. Remote storage is unaffected; the next backup will do a full
scan and re-deduplicate against existing remote blobs, and the local
index will re-bind to the currently configured storage destination.
Use this after changing `storage_url` to a different destination.
* `--force`: Skip confirmation prompt

---

## storage backends

vaultik supports three storage backends, selected via the `storage_url` config field:

**S3** (`s3://bucket/prefix?endpoint=host&region=us-east-1`): Any S3-compatible
object store. Credentials are read from `s3.access_key_id` and
`s3.secret_access_key` in the config file.

**Local filesystem** (`file:///path/to/backup`): Stores blobs and metadata on
a local or mounted filesystem. Useful for testing or backing up to a NAS.

**Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud
providers. Requires rclone to be configured separately (`rclone config`).

Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is
still supported for backward compatibility. `storage_url` takes precedence if
both are set.

---

## architecture

### remote storage layout

```
<bucket>/<prefix>/
├── blobs/
│   └── <aa>/<bb>/<full_blob_hash>
└── metadata/
    └── <remote-key>/
        ├── db.zst.age          # Encrypted binary SQLite database
        └── manifest.json.zst   # Unencrypted blob list (for pruning)
```

* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
* `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted)
  containing all file metadata, chunk mappings, and relationships for the snapshot
* `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling
  pruning without the private key

Snapshot IDs follow the human-readable format
`<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g.
`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the
destination store in plaintext. Each snapshot's metadata directory is named
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a listing
of the store reveals no hostname or snapshot name. The backup time is not
hidden: manifest.json.zst carries a plaintext timestamp, and object
modification times are visible at the storage layer regardless. For example,
`server1_home_2025-06-01T12:00:00Z` is stored under
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`.
See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation.

### data flow

**backup:**

1. Open local SQLite index, load known files and chunks into memory
2. Walk source directories, compare mtime/size/mode against index
3. For changed/new files: chunk using content-defined chunking (FastCDC)
4. For symlinks and directories: record metadata (no chunking)
5. For each chunk: hash, check dedup, add to blob packer
6. When blob reaches size threshold: compress (zstd), encrypt (age), upload
7. Build snapshot metadata database, compress, encrypt, upload
8. Create unencrypted blob manifest for pruning support

**restore:**

1. Download and decrypt `metadata/<remote-key>/db.zst.age`
2. Open the binary SQLite database
3. Query files (optionally filtered by paths)
4. Download and decrypt required blobs
5. Extract chunks, reconstruct files
6. Restore permissions, timestamps, ownership, symlinks

**prune:**

1. List all snapshot manifests
2. Build set of all referenced blob hashes
3. List all blobs in storage
4. Delete any blob not in the referenced set

### chunking and deduplication

* Content-defined chunking using the FastCDC algorithm
* Average chunk size: configurable (default 10MB)
* Deduplication at file level (unchanged files skipped) and chunk level
  (identical chunks across files stored once)
* Multiple chunks packed into blobs to reduce object count

### encryption

* Asymmetric encryption using age (X25519 + XChaCha20-Poly1305)
* Only the public key is needed on the source host
* Each blob and each metadata database is encrypted independently
* Multiple recipients supported (encrypt to multiple keys)

### compression

* zstd compression at configurable level (1-19, default 3)
* Applied before encryption at the blob level

---

## configuration reference

Run `vaultik config init` to generate a fully commented config file.
Key fields:

| Field | Default | Description |
|-------|---------|-------------|
| `age_recipients` | (required) | Age public keys for encryption |
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
| `exclude` | | Global exclude patterns (applied to all snapshots) |
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting |
| `compression_level` | `3` | zstd compression level (1-19) |
| `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path |

---

## limitations

* **No extended attributes (xattrs).** ACLs, macOS Finder metadata,
  quarantine flags, SELinux labels, and other extended attributes are not
  backed up or restored.
* **No hard link detection.** Two hard links to the same inode are backed
  up as independent files. Content deduplication means the data is stored
  once, but the hard link relationship is lost on restore.
* **No sparse file support.** Sparse files are fully materialized during
  backup. A 100 GB sparse VM disk that is mostly zeros will consume the
  full (compressed) size in storage.
* **No bandwidth limiting.** Uploads and downloads use whatever bandwidth
  is available. There is no `--bwlimit` flag yet.
* **No parallel blob downloads during restore.** Blobs are fetched
  sequentially. Restore speed is bound by single-stream throughput.
* **Device nodes, named pipes, and sockets are silently skipped.** Only
  regular files, directories, and symlinks are backed up.
* **No database migrations.** If the local SQLite schema changes between
  versions, delete the local database (`vaultik database delete`) and run
  a full backup. Remote storage is unaffected.
* **Files that change during backup may be inconsistent.** There is no
  filesystem snapshot or freeze. If a file is modified between the scan
  and chunk phases, the backed-up copy may reflect a partial write.
* **Ownership restoration requires root.** File uid/gid are recorded
  and restored, but `chown` requires elevated privileges. Without root,
  files are restored with the current user's ownership.

---

## roadmap

Items still to do before / shortly after 1.0. Loosely ordered by
priority.

### correctness and operability

* **Security audit of the encryption implementation.** Pre-1.0
  blocker if we're advertising "secure" at the top of this README.
  age + zstd + content-defined chunking is mostly off-the-shelf
  pieces, but the seams (key handling, recipient parsing, manifest
  trust boundary, restore-time identity validation) need an outside
  read.
* **Error-condition tests.** Today's coverage is the happy path
  plus a few specific regressions. Need fault-injection coverage:
  network failures mid-blob, disk-full during restore, corrupted /
  truncated / missing blobs, partial uploads, kill -9 between
  manifest and db.zst.age writes.
* **Verify restored content end-to-end in CI.** The current
  integration test does this for a small synthetic snapshot but
  not at scale. A nightly job against a multi-GB representative
  snapshot would catch silent regressions in the chunker, packer,
  or restore planner.

### performance

* **Parallel blob downloads during restore.** Single-stream right
  now. With a fast S3 endpoint and a multi-core machine restore is
  bound by per-blob fetch + decrypt + decompress; running N of
  those in parallel against the disk cache would close most of the
  remaining gap. Needs to interact correctly with the locality
  planner and sweeper.
* **Bandwidth limiting (`--bwlimit`).** Both upload and download.
  Useful for backing up over a shared link. Tricky to make work
  correctly with the parallel-download story.
* **Restart of interrupted restore.** Today restore is restartable
  in the sense that re-running it overwrites partial output; it
  doesn't resume from where it stopped or skip already-present
  files. A `--resume` mode that checks targets before fetching
  blobs would matter for very large restores.

### usability

* **Man pages and richer `--help` examples.** Cobra generates
  basic help; man pages would be a separate target.
* **`--bwlimit` style human-readable size flags** across the
  command surface where they're currently raw integers.
* **`vaultik snapshot diff <a> <b>`** — show which files changed
  between two snapshots without restoring either.
* **Status reporting hook for `--cron`.** When a backup fails
  silently in cron, the user has no idea. A configurable
  webhook / email / `notify-send` hook on completion (success and
  failure) would close the loop.

### infrastructure

* **Schema migrations.** Currently nonexistent — pre-1.0 schema
  changes are handled by `vaultik database delete` plus a full
  re-scan. Post-1.0 we'll need a migration story to keep existing
  index databases usable across upgrades.
* **Storage backend coverage tests.** S3, file://, and rclone://
  all share the Storer interface but the rclone path is the least
  exercised in CI.

---

## output style

All user-facing output goes through helpers in `internal/ui` and conforms
to a uniform style. Color is enabled when stdout is a TTY and the
`NO_COLOR` environment variable is unset (https://no-color.org/).

`internal/ui` writes to stdout; it is the output the user asked for.
Structured log records are a different thing and go through
`internal/log`, which writes to stderr (see "stdout and stderr" above).

Message classes:

| Class | Marker | Alignment | Use for |
|-------|--------|-----------|---------|
| Banner | none | column 0 | The startup line printed once per invocation |
| Begin | `》` (white) | column 0 | An operation is about to start (present-continuous verb) |
| Complete | `》` (green) | column 0 | An operation just finished (past-tense verb) |
| Info | `》` (white) | column 0 | Neutral status update |
| Notice | `》` (cyan) | column 0 | Important note that is not a warning |
| Warning | `⚠️  Warning:` (orange/yellow) | column 0 | Recoverable problem |
| Error | `🛑 ERROR:` (red) | column 0 | Operation aborted |
| Progress | `  》` (white) | column 2 | Heartbeat or per-item status during a long-running operation |
| Detail | `  》` (white) | column 2 | Continuation/sub-line of a preceding Complete (visually identical to Progress) |

Conventions:

* Messages are complete English sentences ending with a period.
* Fully qualify terms — say "backup destination store" instead of
  "storage", "snapshot source files enumeration" instead of "scan",
  "local index database" instead of "database".
* Every operation that emits a Complete also emits a corresponding
  Begin. Operations that print only a Begin (because completion is
  obvious from a later Begin) should be rare and intentional.
* Use natural verb tense to signal state: "Uploading" for Begin,
  "Uploaded" for Complete. Never write the words "begin" or "complete"
  in the body — the marker color already conveys that.
* All elapsed and remaining-time fields are explicitly scoped to their
  subject: write "blob upload elapsed: 30s, blob upload ETA: 03:15:00
  (est remain 14s)", never just "elapsed 30s, ETA 14s".
* "ETA" means an absolute clock time (when the operation will finish),
  not a remaining-duration. Use `ui.Time()` for the former and
  `ui.Duration()` for the latter, and label both.
* `ui.Time` formats same-day times as `HH:MM:SS` and other-day times as
  `YYYY-MM-DD HH:MM:SS`. No timezone — local time is implied.

Value colorizers in `internal/ui` colorize specific value types
consistently. Compose messages from these helpers rather than embedding
ANSI escapes inline:

| Helper | Color | Use for |
|--------|-------|---------|
| `Hex` | cyan | Blob hashes, chunk hashes (truncated to 12 chars + `...`) |
| `Snapshot` | bold cyan | Snapshot IDs (untruncated) |
| `Path` | blue | Filesystem paths |
| `Size` | magenta | Byte counts (human-readable) |
| `Speed` | magenta | Bytes-per-second rates |
| `Duration` | yellow | Elapsed or remaining time |
| `Time` | yellow | Absolute clock times |
| `Count` | magenta | Integer counts with thousands separators |
| `Percent` | magenta | Percentages |

When `NO_COLOR` is set or output is not a TTY, all helpers return plain
text and the marker prefixes (`》`, `Warning:`, `ERROR:`) emit without
ANSI escapes. The emoji prefixes on Warning and Error are always emitted
regardless of color setting (emoji are not color).

## requirements

* Go 1.26 or later
* Docker, with a reachable daemon, to lint, check, or commit:
  `script/lint` lints by building `Dockerfile.lint`, which runs the
  digest-pinned `golangci-lint` image as a build step, and `make check`
  and the pre-commit hook both run it. A `golangci-lint` installed on
  `PATH` is not a substitute and is never used on a host, whatever its
  version.
* S3-compatible object storage (or local filesystem, or rclone remote)

## development workflow

All changes follow this workflow. No exceptions.

1. Create a feature branch off `main`.
2. Write tests.
3. Write the implementation.
4. Fix implementation errors until it compiles and tests pass.
5. Fix linting errors (`make lint`).
6. Update documentation and README as required by the change.
7. Format code (`make fmt`).
8. Run `make check` (lint + fmt-check + test). Fix any issues. Repeat until clean.
9. Commit on the branch.
10. Merge to `main`.
11. Push.

Do not commit directly to `main`. Do not skip steps.

Repository policies for AI agents are in [`AGENTS.md`](AGENTS.md).

## Entrypoints

This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call
them. We provide:

* `script/bootstrap` — install all development dependencies (go, Go
  module download). It deliberately does not install `golangci-lint`;
  see `script/lint` below.
* `script/setup` — make a fresh clone ready for development: runs
  `script/bootstrap`, then `script/install-precommit`
* `script/projectname` — print the project name (used for the Docker
  image tag)
* `script/version` — print the version string to bake into the binary.
  The `Makefile`'s `LDFLAGS` call this; it is the single source of truth
  for the version. See [releasing](#releasing) for the rules.
* `script/install-goreleaser` — install the pinned `goreleaser` into
  `.tool/bin` from a sha256-verified release archive. Idempotent, and
  called by `script/bootstrap`; the release workflow calls it directly
  because it needs `goreleaser` but not the Docker daemon
  `script/bootstrap` insists on.
* `script/install-go` — install the Go toolchain named by `go.mod`'s
  `go` directive into `.tool/go` from a sha256-verified `go.dev`
  archive, and put it on `PATH`. Idempotent. Called only by the release
  workflow, which needs a host Go for `goreleaser` to shell out to;
  nothing else on the release runner does. `actions/setup-go` is not
  used because it verifies the downloaded toolchain against no value in
  this repo. Bumping Go edits `go.mod`, the checksum in this script, and
  the `Dockerfile` `golang` digest together.
* `script/release` — cross-compile and publish the release artifacts
  with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose
  version is not the pinned one, on the same reasoning as `script/lint`.
* `script/release-snapshot` — the same build with no publishing and no
  tagging, into `./dist`
* `script/test` — run the test suite (verbose rerun on failure). This
  runs *everything*: there is no separate integration target and no
  build-tagged subset held back, so the full round-trip tests in
  `internal/vaultik/integration_test.go` run on every invocation. It
  passes `-count=1`, which disables Go's test result cache. That is
  deliberate and it is not free: on this repo's suite it costs about 11
  seconds on every repeat run (measured, back to back: 0.4s cached
  versus 11.6s with `-count=1`). That is the price of the run meaning
  anything, because without it an unchanged package prints
  `ok <pkg> (cached)`, which is indistinguishable from a package that
  really ran, so the whole suite can report a full set of `ok` lines in
  under half a second having executed nothing. The `-timeout` is a hang
  backstop rather than a performance budget — it applies per test binary
  to test execution only, not to compilation — and is set well above the
  slowest package's measured runtime. Its 120s value deliberately
  diverges from the 30s `REPO_POLICIES.md` mandates; the reasoning is in
  the comment in the script, and issue #101 proposes amending the policy
  text.
* `script/lint` — lint by building `Dockerfile.lint`, which runs
  `golangci-lint run --config .golangci.yml ./...` as a build step
  inside the digest-pinned `golangci-lint` image, so a successful build
  *is* a clean lint. Nothing lints on the host, at any version, ever;
  the script requires Docker and fails loudly rather than falling back
  to a `golangci-lint` on `PATH`. That `FROM` line is the single source
  of truth for the linter version — bump it there and nowhere else.

  It takes no arguments, because a build step has no command line to
  pass flags to, and it passes a fresh `--build-arg CHECK_EPOCH` on
  every invocation so the lint layer cannot be replayed from cache (see
  `script/cibuild` below for what that mechanism defends against). To
  watch the linter execute, run it as
  `BUILDKIT_PROGRESS=plain script/lint` and check that the lint layer
  says `RUN … golangci-lint` rather than `CACHED`.

  One container per run means one lint cache and one `golangci-lint`
  lock per run, both private to it and discarded with it, so concurrent
  runs on one host cannot contaminate or block each other.
* `script/lint-fix` — apply the linter's autofixes (rewrites files),
  using the same pinned image, parsed out of `Dockerfile.lint`. It
  cannot be a build step, because fixes have to land in the worktree, so
  it bind-mounts the tree into a `docker run` and therefore needs a
  *local* daemon. It is a developer convenience and never a gate: no
  gate reads its exit status. Run `make lint` afterwards to find out
  whether the tree is clean.
* `script/fmt` — format all code (writes)
* `script/fmt-check` — check formatting (read-only)
* `script/check` — run `script/test`, `script/lint`, and
  `script/fmt-check`. This is authoritative *because* `script/lint`
  builds `Dockerfile.lint`: a local `make check` and CI cannot disagree
  about lint findings.
* `script/docker` — build the Docker image tagged via
  `script/projectname`. Passes a fresh `--build-arg CHECK_EPOCH` for the
  same reason `script/cibuild` does, so a local image build cannot be
  green on checks it replayed from cache. It builds the *product* image
  only, and the product `Dockerfile` has no lint stage, so it does not
  lint: a green here means formatted, tested, and it compiles.
* `script/cibuild` — CI entrypoint, and the full gate. Two builds, in
  order: `Dockerfile.lint` (the linter, as a build step) and then
  `Dockerfile` (`make fmt-check` and `make test` in its builder stage,
  then the product image). Either failing fails the script. It runs the
  checks in the same containers CI does, from a clean copy of the tree,
  so it also catches anything that depends on host state.
  `.gitea/workflows/check.yml` runs it on every push to `main` and
  `next` and on every pull request against either.

  It passes a fresh `--build-arg CHECK_EPOCH` to each build, unique per
  invocation, which both files declare immediately above their check
  `RUN`s and expand into each check command. Those layers are keyed on
  that value, so a new value re-runs them even on a byte-identical tree,
  and a green from this script means the checks executed. Dependency and
  module layers sit above the `ARG` and still cache, so a build is not
  cold.

  A build that supplies no `CHECK_EPOCH` — a bare `docker build .` or
  `docker build -f Dockerfile.lint .` — fails rather than lying. An
  unset `ARG` is an empty string and an empty string is a stable cache
  key, so without a guard such a build would serve every check layer
  from cache, execute nothing, and still exit 0. Each file therefore
  asserts the value is non-empty before running anything, and because
  failed steps are never cached that assertion fires on every
  invocation rather than once. Use `script/lint`, `script/docker` or
  `script/cibuild`, which pass the arg; a bare `docker build` is a loud
  error.
* `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
  not change files), then `script/check`
* `script/install-precommit` — install the git pre-commit hook that
  runs `script/precommit`

## releasing

### version numbers

The version a binary reports comes from git, not from a constant in a
file. `script/version` decides it, and everything that stamps a binary
agrees with it:

* `HEAD` is exactly on a tag → that tag with a leading `v` stripped, so
  the tag `v1.0.0` produces `vaultik 1.0.0`, matching the archive name
  `vaultik_1.0.0_linux_amd64.tar.gz`. `goreleaser` strips the prefix the
  same way.
* anything else → `dev-<12 chars of the commit sha>`.
* either, with uncommitted changes to tracked files → a `-dirty`
  suffix, because a modified checkout of a tag is not that tag.

A build that is not a release never names itself like one. `vaultik
version` says so in as many words on a development build, and
`goreleaser --snapshot` stamps the same `dev-<sha>` string rather than
inventing the next patch number. If `script/version` cannot be run at
all, `make` stops with an error instead of building an unversioned
binary, and a binary that somehow carries an empty version string still
reports itself as a development build.

### cutting a release

Releases are cut by CI from a tag, not from a workstation:

```
git tag -a v1.2.3 -m 'v1.2.3'
git push origin v1.2.3
```

`.gitea/workflows/release.yml` triggers on `v*` tags, installs a Go
toolchain and the pinned `goreleaser`, and runs `script/release`, which
builds
`linux,darwin × amd64,arm64` archives plus `checksums.txt` and publishes
them to this repository's Gitea releases as a draft. `.goreleaser.yaml`
has a `gitea_urls:` block pointing at `https://git.eeqj.de/api/v1`;
without it `goreleaser` would talk to the GitHub API.

The workflow needs one repository Actions secret:

| Secret          | What it is                                                                                              |
| --------------- | ------------------------------------------------------------------------------------------------------- |
| `RELEASE_TOKEN` | A Gitea access token with `write:repository` scope, owned by an account that can publish releases here. |

It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic
token is deliberately not used: it is not guaranteed to carry release
write access.

The Go toolchain that compiles the released binaries comes from an
`actions/setup-go` step pinned by commit sha, reading its version from
`go.mod` (currently `1.26.1`, the same version the `Dockerfile` builder
stage pins by digest). `goreleaser` shells out to `go` for every
cross-compile, so without that step the release would either fail
outright or ship binaries built by whatever unpinned toolchain the
runner happened to carry — the one unpinned thing in an otherwise
hash-pinned release path.

To rehearse the whole build without publishing or tagging anything:

```
make release-snapshot
```

Artifacts land in `./dist`, which is gitignored.

Release artifacts are not signed, carry no SBOM, and are not built
reproducibly; the archives contain the binary, `LICENSE`, and
`README.md` only (no shell completions or man page).

## license

[MIT](https://opensource.org/license/mit/)

## author

Made with love and lots of expensive SOTA AI by [sneak](https://sneak.berlin) in Berlin in the summer of 2025.

Released as a free software gift to the world, no strings attached.

Contact: [sneak@sneak.berlin](mailto:sneak@sneak.berlin)

[https://keys.openpgp.org/vks/v1/by-fingerprint/5539AD00DE4C42F3AFE11575052443F4DF2A55C2](https://keys.openpgp.org/vks/v1/by-fingerprint/5539AD00DE4C42F3AFE11575052443F4DF2A55C2)