Files
vaultik/README.md
sneak d257f8f658
All checks were successful
check / check (pull_request) Successful in 2m58s
Lint in a container as a build step, via Dockerfile.lint (closes #113)
Every lint run now happens inside its own container, invoked through
script/lint, and linting is a build step rather than a container
command: a successful build of the new root Dockerfile.lint IS a clean
lint. That shape also works where the docker daemon is remote and bind
mounts are impossible.

Its FROM line -- golangci/golangci-lint:v2.12.2, pinned by digest -- is
now the only pin of the linter version in this repo.

A container per run has its own lint cache and its own golangci-lint
lock, both discarded with it, so neither cross-worktree contamination
nor lock contention exists any more. The machinery that defended
against them is therefore gone: the per-worktree cache directories, the
lock-retry loop, and script/lint-audit, which existed to catch findings
replayed from a cache that no longer exists. So is the host lint path
in its entirety -- the native escape hatch, its version detection, and
VAULTIK_LINT_IN_CONTAINER in both script/lint and the Dockerfile.
Nothing lints on the host, at any version.

A cached build lints nothing, so the CHECK_EPOCH mechanism the product
Dockerfile already used is what makes a green mean something:
ARG CHECK_EPOCH with no default, placed below the module layers so
dependency caching survives, a `RUN [ -n "$CHECK_EPOCH" ] || exit 1`
guard so a build that withholds the arg fails instead of replaying, and
the value expanded into the lint command itself. script/lint computes
`epoch="$(date +%s%N)$$"` as a bare assignment on its own line, because
inline in the argument a failing substitution does not abort under
`set -eu` and yields a constant empty epoch -- which is exactly the
false green being prevented.

The product Dockerfile loses its lint stage rather than gaining a
second linter pin. That stage ran `make lint`, which is now
`docker build`: docker-in-docker inside a BuildKit step with no daemon.
Calling golangci-lint directly there instead would have meant two
independently bumpable digests for one tool. `make fmt-check` moves
beside `make test` in the builder stage, and script/cibuild now builds
Dockerfile.lint and then Dockerfile, each with its own fresh epoch,
failing on either. Consequence, stated in comments rather than left to
be discovered: script/docker builds the product image only and no
longer lints; script/check and script/cibuild are the gates.

`golangci-lint config verify` runs as its own epoch-keyed layer, above
the lint. It is not belt-and-braces: `golangci-lint run` rejects a
config it cannot PARSE but silently IGNORES an unknown top-level KEY.
Renaming .golangci.yml's `linters:` to `linterz:` -- one character --
discards `default: all`, the disable list and every threshold, leaves
only the small default linter set running, and exits 0 reporting
`0 issues.` on a tree the real config fails with an lll finding, in a
run whose lint layer demonstrably executed. That is a set-but-
ineffective config falling back to defaults instead of failing loudly,
sitting in the gate's own configuration. `config verify` catches it and
does so with the network genuinely off at this pin: under
`docker run --network none` against the pinned digest it exits 0 on
this repo's config and exits 3 on the `linterz:` variant. It is keyed
on CHECK_EPOCH like the lint itself, because a cached validation
validates nothing.

script/lint-fix is kept, reimplemented as a bind-mounted
docker run against the image parsed out of Dockerfile.lint -- a build
step cannot write fixes back to the worktree -- and its header states
outright that it is a developer convenience, never a gate, and needs a
local daemon.

cmd/vaultik/lintdocker_test.go parses both Dockerfiles and both scripts
and fails if any part of the mechanism is dropped: the digest pin, the
defaultless ARG below `go mod download`, the emptiness guard, the
expansion of the epoch into each check command, the bare per-invocation
epoch assignment in both scripts, cibuild building both files, the
config verification running before the lint, and -- structurally, not
by searching for one retired variable name -- that no script invokes
golangci-lint except through docker. Every one of those losses is
silent: the build still exits 0 and nothing is checked, which is why
they are asserted rather than trusted. The scanner behind the last of
those has its own test, because a structural check that goes blind
passes on every tree, including a broken one.

script/lint takes no arguments now, and says so instead of dropping
them: a build step has no command line to pass linter flags to.
2026-08-10 13:38:45 +00:00

37 KiB
Raw Permalink Blame History

vaultik (ваултик)

vaultik is an incremental backup tool written in Go. It encrypts data using an age public key and uploads each encrypted blob directly to a remote S3-compatible object store. It requires no private keys, secrets, or credentials (other than those required to PUT to encrypted object storage, such as S3 API keys) stored on the backed-up system.

quickstart

# install
go install sneak.berlin/go/vaultik/cmd/vaultik@latest

# create a default config file (prints the path it wrote to)
vaultik config init

# generate an age keypair; keep the private key file somewhere safe and
# offline — you need it to restore, and the backed-up machine does not need it
age-keygen -o vaultik_backup_private_key.txt
grep 'public key' vaultik_backup_private_key.txt

# configure the encryption key and backup destination
vaultik config set age_recipients.0 age1YOUR_PUBLIC_KEY_HERE
vaultik config set storage_url "file:///Volumes/usbstick/mybackup"

# macOS only: grant your terminal app Full Disk Access first
# (System Settings → Privacy & Security → Full Disk Access), otherwise
# the backup will abort with a permission error on protected directories

# run your first backup (the default config backs up ~ and /Applications
# with sensible excludes)
vaultik snapshot create

# see what you have
vaultik snapshot list

Features:

  • modern encryption (age, X25519 + XChaCha20-Poly1305)
  • content-defined chunking with deduplication (FastCDC)
  • incremental backups (only changed files are re-chunked)
  • multithreaded zstd compression at configurable levels
  • content-addressed immutable storage
  • local state tracking in SQLite (enables write-only incremental backups)
  • no mutable remote metadata
  • no plaintext file paths or metadata in remote storage
  • packs small files into large blobs (keeps S3 operation counts down)
  • backs up regular files, symlinks, empty directories, and file permissions
  • pluggable storage backends: S3, local filesystem, rclone (70+ providers)
  • pure Go (no CGO), cross-compiles to linux/darwin × amd64/arm64

why

Other backup tools like restic, borg, and duplicity are designed for environments where the source host can store secrets and has access to decryption keys. vaultik is for environments where you don't want to store backup decryption keys on your hosts — only public keys for encryption.

Requirements that no existing tool meets:

  • open source
  • no passphrases or private keys on the source host
  • incremental
  • compressed
  • encrypted
  • s3 compatible without an intermediate step or tool

daily use

# verify a snapshot (shallow: checks all blobs exist)
vaultik snapshot verify <snapshot-id>

# deep verify (downloads and cryptographically verifies every blob)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot verify --deep <snapshot-id>

# restore (requires the private key)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot restore <snapshot-id> /tmp/restored

# daily cron job: back up, keep a 4-week rolling window of snapshots
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w

cli

commands

vaultik [--config <path>] config init
vaultik [--config <path>] config edit
vaultik [--config <path>] config get <key>
vaultik [--config <path>] config set <key> <value>
vaultik [--config <path>] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>]
vaultik [--config <path>] snapshot list [--json]
vaultik [--config <path>] snapshot verify <snapshot-id> [--deep] [--json]
vaultik [--config <path>] snapshot purge [--keep-latest | --older-than <duration>] [--snapshot <name>...] [--force]
vaultik [--config <path>] snapshot remove <snapshot-id> [--dry-run] [--force] [--local-only] [--json]
vaultik [--config <path>] snapshot restore <snapshot-id> <target-dir> [paths...] [--verify]
vaultik [--config <path>] prune [--force] [--json]
vaultik [--config <path>] info
vaultik [--config <path>] remote info [--json]
vaultik [--config <path>] remote nuke --force
vaultik [--config <path>] database delete [--force]
vaultik completion <bash|zsh|fish|powershell>
vaultik version

global flags

  • --config <path>: Path to config file (default: $VAULTIK_CONFIG, then platform config dir, then /etc/vaultik/config.yml)
  • --verbose, -v: Enable verbose output (on stderr — see below)
  • --debug: Enable debug output (on stderr — see below)
  • --quiet, -q: Suppress non-error output (also suppresses startup banner)
  • --skip-errors: Continue past per-file errors instead of aborting (applies to snapshot create and restore)

stdout and stderr

Log output — everything from --verbose and --debug, and every warning and error the logger emits — goes to stderr. stdout carries the output you asked for: tables, and the documents produced by --json.

This means vaultik snapshot list --verbose > out.txt captures the listing and leaves the diagnostics on your terminal. To capture both, redirect stderr as well (> out.txt 2> log.txt, or > out.txt 2>&1 to interleave them).

The split is what makes --json usable from a script. Warnings and errors are never suppressed — not by --quiet, not by --cron — so a logger on stdout would eventually land a log line inside a JSON document and break the parse. A config file with group- or world-readable permissions is enough to trigger it.

Format follows the stream: when stderr is a terminal the records are colorized one-liners, and when it is redirected or piped they are JSON, one object per line.

Under --json, stdout holds the document and nothing else. The startup banner is suppressed, as --quiet and --cron suppress it, and the progress narration a command would otherwise print — such as the stale local records prune reconciles away — is suppressed too, so it cannot land ahead of the document. Every --json command therefore pipes on its own, with no additional flag: vaultik snapshot list --json | jq . and vaultik prune --json | jq . both work as written.

environment variables

  • VAULTIK_AGE_SECRET_KEY: Age private key for decryption (required for snapshot restore and snapshot verify --deep)
  • VAULTIK_CONFIG: Path to config file (overridden by --config)
  • VAULTIK_INDEX_PATH: Override local SQLite index path

shell completion

# zsh: load for the current session
source <(vaultik completion zsh)

# zsh: install permanently
vaultik completion zsh > "${fpath[1]}/_vaultik"

# bash: load for the current session
source <(vaultik completion bash)

# bash: install permanently (Linux)
vaultik completion bash > /etc/bash_completion.d/vaultik

# fish
vaultik completion fish > ~/.config/fish/completions/vaultik.fish

command details

config init: Write a default config file with commented explanations for every setting. Writes to the path from --config, $VAULTIK_CONFIG, or the platform config directory (~/Library/Application Support/vaultik/ on macOS, ~/.config/vaultik/ on Linux, /etc/vaultik/ as root). Refuses to overwrite an existing file. Created with mode 0600 since it will contain credentials.

config edit: Open the config file in $EDITOR (falls back to vi).

config get: Print a config value addressed by dotted YAML path (e.g. vaultik config get storage_url). Non-scalar values print as YAML.

config set: Set a scalar config value by dotted YAML path (e.g. vaultik config set compression_level 9, vaultik config set storage_url "file:///mnt/backups"). Comments and formatting in the file are preserved; intermediate maps are created as needed.

snapshot create: Perform incremental backup of configured snapshots.

  • Optional snapshot names argument to create specific snapshots (default: all)
  • On macOS, the terminal application running vaultik needs Full Disk Access (System Settings → Privacy & Security → Full Disk Access) to read TCC-protected directories; without it the backup aborts with a permission error that explains how to fix it
  • --cron: Silent on total success; warnings and errors are still printed (for crontab)
  • --prune: After backup, drop older snapshots of each backed-up name and remove orphaned blobs from remote storage. By default keeps only the latest snapshot per name; use --keep-newer-than for a rolling window.
  • --keep-newer-than <duration>: With --prune, keep snapshots newer than this duration instead of only the latest (e.g. 4w, 30d, 6mo, 1y)

snapshot list: Show every snapshot known to this host — the union of the local index and the backup destination store — with timestamps and three sizes per snapshot (compressed remote size; total uncompressed chunk size; size of chunks newly referenced by that snapshot).

Listing the destination store does not require the age secret key, so it works in vaultik's intended configuration, where the backed-up host holds only the public key. A host that has lost its local index can still see what it has backed up.

What that host cannot see is a remote-only snapshot's name. The snapshot ID is hashed at the storage boundary and the manifest records only the hash, so hostname and snapshot name exist solely in the local index and in the encrypted per-snapshot database. Snapshots found only on the destination store are therefore listed as <remote only:<abbreviated remote key>> and show <remote only> in the uncompressed and "new chunk" columns, which can only be computed from the local index. Their timestamp and compressed size are real, read from the manifest.

Snapshots in the local index with no counterpart on the destination store are reported below the table as drift, with the vaultik prune invocation that reconciles them.

If the destination store cannot be listed (unmounted volume, permission denied, network down), the command warns, falls back to the local index alone, and still exits zero.

  • --json: Output in JSON format. Each entry carries locally_tracked (whether the snapshot is in the local index), remote_key (the full 64-character storage key), and remote_present (whether it was seen on the destination store, or null if the destination could not be listed). Warnings about an unlistable destination, unreadable manifests, and a truncated listing all go to stderr through the logger, so stdout stays a single parseable document.

snapshot verify: Verify snapshot integrity.

  • Default (shallow): checks that all blobs referenced in the manifest exist in storage
  • --deep: Downloads and decrypts each blob, verifies chunk hashes against the encrypted metadata database
  • --json: Output results as JSON

snapshot purge: Remove old snapshots based on criteria. Retention is per-snapshot-name (--keep-latest keeps the latest of each name, not the latest globally).

  • --keep-latest: Keep only the most recent snapshot of each name
  • --older-than <duration>: Remove snapshots older than duration (e.g. 30d, 6m, 1y)
  • --snapshot <name>: Restrict to specific snapshot names (repeat for multiple)
  • --force: Skip confirmation prompt

snapshot remove: Remove one snapshot. By default this removes the snapshot from the local index and strips the snapshot's metadata from the backup destination store. Blobs are NOT touched — deleting blobs requires reading every remaining remote manifest (the destination store may hold snapshots this host doesn't know about), which is what vaultik prune does. On success the command prints the exact vaultik prune invocation to run as a follow-up. Local row cleanup (files, chunks, blobs the snapshot was the last referrer for) runs automatically. If the destination store is unreachable, the local-DB removal still completes and a warning is emitted; rerun vaultik prune once the store is reachable to finish remote cleanup. To wipe everything on the destination in one go, use vaultik remote nuke --force.

  • --local-only: Skip remote cleanup; only touch the local index
  • --dry-run: Show what would be deleted without deleting
  • --force: Skip confirmation prompt
  • --json: Output result as JSON

snapshot restore: Restore files from a backup snapshot.

  • Requires VAULTIK_AGE_SECRET_KEY environment variable
  • Optional path arguments to restore specific files/directories (default: all)
  • Preserves file permissions, timestamps, ownership (ownership requires root), symlinks, and empty directories
  • --verify: After restoring, verify every file's chunk hashes match

prune: Tidy up everything that isn't needed. Runs three passes: (1) reconcile the local index against the destination store — any local snapshot whose remote metadata is missing is dropped from the local index; (2) delete orphaned local rows (files, chunks, blobs no longer referenced by any completed snapshot); (3) list every remote manifest on the destination store to compute the still-referenced blob set and delete any blob not in that set. Step (3) reads all remote manifests — network cost scales with the number of snapshots. snapshot create --prune runs the same cleanup automatically; this is the manual entry point for the same work.

  • --force: Skip confirmation prompt
  • --json: Output stats as JSON

info: Display system configuration, storage settings, encryption recipients, and local database statistics.

remote info: Show storage backend type and location plus detailed remote storage inventory: per-snapshot metadata sizes, blob counts, and orphaned blob detection.

  • --json: Output as JSON

remote nuke: Delete every snapshot's metadata and every blob from the backup destination store, leaving the bucket prefix empty. Destructive and irreversible. This is the single supported way to wipe the entire destination store.

  • --force: Required to confirm destruction.

database delete: Delete the local SQLite state database file entirely. Remote storage is unaffected; the next backup will do a full scan and re-deduplicate against existing remote blobs, and the local index will re-bind to the currently configured storage destination. Use this after changing storage_url to a different destination.

  • --force: Skip confirmation prompt

storage backends

vaultik supports three storage backends, selected via the storage_url config field:

S3 (s3://bucket/prefix?endpoint=host&region=us-east-1): Any S3-compatible object store. Credentials are read from s3.access_key_id and s3.secret_access_key in the config file.

Local filesystem (file:///path/to/backup): Stores blobs and metadata on a local or mounted filesystem. Useful for testing or backing up to a NAS.

Rclone (rclone://remote/path): Uses rclone's 70+ supported cloud providers. Requires rclone to be configured separately (rclone config).

Legacy S3 configuration via s3.* fields (endpoint, bucket, prefix, etc.) is still supported for backward compatibility. storage_url takes precedence if both are set.


architecture

remote storage layout

<bucket>/<prefix>/
├── blobs/
│   └── <aa>/<bb>/<full_blob_hash>
└── metadata/
    └── <snapshot_id>/
        ├── db.zst.age          # Encrypted binary SQLite database
        └── manifest.json.zst   # Unencrypted blob list (for pruning)
  • Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
  • db.zst.age is a binary SQLite database (zstd compressed, age encrypted) containing all file metadata, chunk mappings, and relationships for the snapshot
  • manifest.json.zst is an unencrypted compressed JSON blob list, enabling pruning without the private key

Snapshot IDs follow the format <hostname>_<snapshot-name>_<RFC3339-timestamp> (e.g. server1_home_2025-06-01T12:00:00Z).

data flow

backup:

  1. Open local SQLite index, load known files and chunks into memory
  2. Walk source directories, compare mtime/size/mode against index
  3. For changed/new files: chunk using content-defined chunking (FastCDC)
  4. For symlinks and directories: record metadata (no chunking)
  5. For each chunk: hash, check dedup, add to blob packer
  6. When blob reaches size threshold: compress (zstd), encrypt (age), upload
  7. Build snapshot metadata database, compress, encrypt, upload
  8. Create unencrypted blob manifest for pruning support

restore:

  1. Download and decrypt metadata/<snapshot_id>/db.zst.age
  2. Open the binary SQLite database
  3. Query files (optionally filtered by paths)
  4. Download and decrypt required blobs
  5. Extract chunks, reconstruct files
  6. Restore permissions, timestamps, ownership, symlinks

prune:

  1. List all snapshot manifests
  2. Build set of all referenced blob hashes
  3. List all blobs in storage
  4. Delete any blob not in the referenced set

chunking and deduplication

  • Content-defined chunking using the FastCDC algorithm
  • Average chunk size: configurable (default 10MB)
  • Deduplication at file level (unchanged files skipped) and chunk level (identical chunks across files stored once)
  • Multiple chunks packed into blobs to reduce object count

encryption

  • Asymmetric encryption using age (X25519 + XChaCha20-Poly1305)
  • Only the public key is needed on the source host
  • Each blob and each metadata database is encrypted independently
  • Multiple recipients supported (encrypt to multiple keys)

compression

  • zstd compression at configurable level (1-19, default 3)
  • Applied before encryption at the blob level

configuration reference

Run vaultik config init to generate a fully commented config file. Key fields:

Field Default Description
age_recipients (required) Age public keys for encryption
snapshots (required) Named snapshot definitions with paths and excludes
storage_url Storage backend URL (s3://, file://, rclone://)
s3.* Legacy S3 configuration (endpoint, bucket, credentials)
exclude Global exclude patterns (applied to all snapshots)
chunk_size 10MB Average chunk size for content-defined chunking
blob_size_limit 10GB Maximum blob size before splitting
compression_level 3 zstd compression level (1-19)
hostname system hostname Hostname used in snapshot IDs
index_path platform data dir Local SQLite index path

limitations

  • No extended attributes (xattrs). ACLs, macOS Finder metadata, quarantine flags, SELinux labels, and other extended attributes are not backed up or restored.
  • No hard link detection. Two hard links to the same inode are backed up as independent files. Content deduplication means the data is stored once, but the hard link relationship is lost on restore.
  • No sparse file support. Sparse files are fully materialized during backup. A 100 GB sparse VM disk that is mostly zeros will consume the full (compressed) size in storage.
  • No bandwidth limiting. Uploads and downloads use whatever bandwidth is available. There is no --bwlimit flag yet.
  • No parallel blob downloads during restore. Blobs are fetched sequentially. Restore speed is bound by single-stream throughput.
  • Device nodes, named pipes, and sockets are silently skipped. Only regular files, directories, and symlinks are backed up.
  • No database migrations. If the local SQLite schema changes between versions, delete the local database (vaultik database delete) and run a full backup. Remote storage is unaffected.
  • Files that change during backup may be inconsistent. There is no filesystem snapshot or freeze. If a file is modified between the scan and chunk phases, the backed-up copy may reflect a partial write.
  • Ownership restoration requires root. File uid/gid are recorded and restored, but chown requires elevated privileges. Without root, files are restored with the current user's ownership.

roadmap

Items still to do before / shortly after 1.0. Loosely ordered by priority.

correctness and operability

  • Security audit of the encryption implementation. Pre-1.0 blocker if we're advertising "secure" at the top of this README. age + zstd + content-defined chunking is mostly off-the-shelf pieces, but the seams (key handling, recipient parsing, manifest trust boundary, restore-time identity validation) need an outside read.
  • Error-condition tests. Today's coverage is the happy path plus a few specific regressions. Need fault-injection coverage: network failures mid-blob, disk-full during restore, corrupted / truncated / missing blobs, partial uploads, kill -9 between manifest and db.zst.age writes.
  • Verify restored content end-to-end in CI. The current integration test does this for a small synthetic snapshot but not at scale. A nightly job against a multi-GB representative snapshot would catch silent regressions in the chunker, packer, or restore planner.

performance

  • Parallel blob downloads during restore. Single-stream right now. With a fast S3 endpoint and a multi-core machine restore is bound by per-blob fetch + decrypt + decompress; running N of those in parallel against the disk cache would close most of the remaining gap. Needs to interact correctly with the locality planner and sweeper.
  • Bandwidth limiting (--bwlimit). Both upload and download. Useful for backing up over a shared link. Tricky to make work correctly with the parallel-download story.
  • Restart of interrupted restore. Today restore is restartable in the sense that re-running it overwrites partial output; it doesn't resume from where it stopped or skip already-present files. A --resume mode that checks targets before fetching blobs would matter for very large restores.

usability

  • Man pages and richer --help examples. Cobra generates basic help; man pages would be a separate target.
  • --bwlimit style human-readable size flags across the command surface where they're currently raw integers.
  • vaultik snapshot diff <a> <b> — show which files changed between two snapshots without restoring either.
  • Status reporting hook for --cron. When a backup fails silently in cron, the user has no idea. A configurable webhook / email / notify-send hook on completion (success and failure) would close the loop.

infrastructure

  • Cross-machine restore documentation. The "restore from another host" workflow works but isn't documented as a first-class operation in this README. Worth a dedicated section once it's settled.
  • Schema migrations. Currently nonexistent — pre-1.0 schema changes are handled by vaultik database delete plus a full re-scan. Post-1.0 we'll need a migration story to keep existing index databases usable across upgrades.
  • Storage backend coverage tests. S3, file://, and rclone:// all share the Storer interface but the rclone path is the least exercised in CI.

output style

All user-facing output goes through helpers in internal/ui and conforms to a uniform style. Color is enabled when stdout is a TTY and the NO_COLOR environment variable is unset (https://no-color.org/).

internal/ui writes to stdout; it is the output the user asked for. Structured log records are a different thing and go through internal/log, which writes to stderr (see "stdout and stderr" above).

Message classes:

Class Marker Alignment Use for
Banner none column 0 The startup line printed once per invocation
Begin (white) column 0 An operation is about to start (present-continuous verb)
Complete (green) column 0 An operation just finished (past-tense verb)
Info (white) column 0 Neutral status update
Notice (cyan) column 0 Important note that is not a warning
Warning ⚠️ Warning: (orange/yellow) column 0 Recoverable problem
Error 🛑 ERROR: (red) column 0 Operation aborted
Progress (white) column 2 Heartbeat or per-item status during a long-running operation
Detail (white) column 2 Continuation/sub-line of a preceding Complete (visually identical to Progress)

Conventions:

  • Messages are complete English sentences ending with a period.
  • Fully qualify terms — say "backup destination store" instead of "storage", "snapshot source files enumeration" instead of "scan", "local index database" instead of "database".
  • Every operation that emits a Complete also emits a corresponding Begin. Operations that print only a Begin (because completion is obvious from a later Begin) should be rare and intentional.
  • Use natural verb tense to signal state: "Uploading" for Begin, "Uploaded" for Complete. Never write the words "begin" or "complete" in the body — the marker color already conveys that.
  • All elapsed and remaining-time fields are explicitly scoped to their subject: write "blob upload elapsed: 30s, blob upload ETA: 03:15:00 (est remain 14s)", never just "elapsed 30s, ETA 14s".
  • "ETA" means an absolute clock time (when the operation will finish), not a remaining-duration. Use ui.Time() for the former and ui.Duration() for the latter, and label both.
  • ui.Time formats same-day times as HH:MM:SS and other-day times as YYYY-MM-DD HH:MM:SS. No timezone — local time is implied.

Value colorizers in internal/ui colorize specific value types consistently. Compose messages from these helpers rather than embedding ANSI escapes inline:

Helper Color Use for
Hex cyan Blob hashes, chunk hashes (truncated to 12 chars + ...)
Snapshot bold cyan Snapshot IDs (untruncated)
Path blue Filesystem paths
Size magenta Byte counts (human-readable)
Speed magenta Bytes-per-second rates
Duration yellow Elapsed or remaining time
Time yellow Absolute clock times
Count magenta Integer counts with thousands separators
Percent magenta Percentages

When NO_COLOR is set or output is not a TTY, all helpers return plain text and the marker prefixes (, Warning:, ERROR:) emit without ANSI escapes. The emoji prefixes on Warning and Error are always emitted regardless of color setting (emoji are not color).

requirements

  • Go 1.26 or later
  • Docker, with a reachable daemon, to lint, check, or commit: script/lint lints by building Dockerfile.lint, which runs the digest-pinned golangci-lint image as a build step, and make check and the pre-commit hook both run it. A golangci-lint installed on PATH is not a substitute and is never used on a host, whatever its version.
  • sqlite3 CLI, which the test suite shells out to
  • S3-compatible object storage (or local filesystem, or rclone remote)

development workflow

All changes follow this workflow. No exceptions.

  1. Create a feature branch off main.
  2. Write tests.
  3. Write the implementation.
  4. Fix implementation errors until it compiles and tests pass.
  5. Fix linting errors (make lint).
  6. Update documentation and README as required by the change.
  7. Format code (make fmt).
  8. Run make check (lint + fmt-check + test). Fix any issues. Repeat until clean.
  9. Commit on the branch.
  10. Merge to main.
  11. Push.

Do not commit directly to main. Do not skip steps.

Repository policies for AI agents are in AGENTS.md.

Entrypoints

This repository adheres to the Scripts to Rule Them All standard: normalized scripts in script/ are the entrypoints for the development workflow, and the Makefile targets are thin shims that call them. We provide:

  • script/bootstrap — install all development dependencies (go, sqlite3, Go module download). It deliberately does not install golangci-lint; see script/lint below.

  • script/setup — make a fresh clone ready for development: runs script/bootstrap, then script/install-precommit

  • script/projectname — print the project name (used for the Docker image tag)

  • script/version — print the version string to bake into the binary. The Makefile's LDFLAGS call this; it is the single source of truth for the version. See releasing for the rules.

  • script/install-goreleaser — install the pinned goreleaser into .tool/bin from a sha256-verified release archive. Idempotent, and called by script/bootstrap; the release workflow calls it directly because it needs goreleaser but not the Docker daemon script/bootstrap insists on.

  • script/release — cross-compile and publish the release artifacts with the pinned goreleaser. Refuses a goreleaser on PATH whose version is not the pinned one, on the same reasoning as script/lint.

  • script/release-snapshot — the same build with no publishing and no tagging, into ./dist

  • script/test — run the test suite (verbose rerun on failure). This runs everything: there is no separate integration target and no build-tagged subset held back, so the full round-trip tests in internal/vaultik/integration_test.go run on every invocation. It passes -count=1, which disables Go's test result cache. That is deliberate and it is not free: on this repo's suite it costs about 11 seconds on every repeat run (measured, back to back: 0.4s cached versus 11.6s with -count=1). That is the price of the run meaning anything, because without it an unchanged package prints ok <pkg> (cached), which is indistinguishable from a package that really ran, so the whole suite can report a full set of ok lines in under half a second having executed nothing. The -timeout is a hang backstop rather than a performance budget — it applies per test binary to test execution only, not to compilation — and is set well above the slowest package's measured runtime. Its 120s value deliberately diverges from the 30s REPO_POLICIES.md mandates; the reasoning is in the comment in the script, and issue #101 proposes amending the policy text.

  • script/lint — lint by building Dockerfile.lint, which runs golangci-lint run --config .golangci.yml ./... as a build step inside the digest-pinned golangci-lint image, so a successful build is a clean lint. Nothing lints on the host, at any version, ever; the script requires Docker and fails loudly rather than falling back to a golangci-lint on PATH. That FROM line is the single source of truth for the linter version — bump it there and nowhere else.

    It takes no arguments, because a build step has no command line to pass flags to, and it passes a fresh --build-arg CHECK_EPOCH on every invocation so the lint layer cannot be replayed from cache (see script/cibuild below for what that mechanism defends against). To watch the linter execute, run it as BUILDKIT_PROGRESS=plain script/lint and check that the lint layer says RUN … golangci-lint rather than CACHED.

    One container per run means one lint cache and one golangci-lint lock per run, both private to it and discarded with it, so concurrent runs on one host cannot contaminate or block each other.

  • script/lint-fix — apply the linter's autofixes (rewrites files), using the same pinned image, parsed out of Dockerfile.lint. It cannot be a build step, because fixes have to land in the worktree, so it bind-mounts the tree into a docker run and therefore needs a local daemon. It is a developer convenience and never a gate: no gate reads its exit status. Run make lint afterwards to find out whether the tree is clean.

  • script/fmt — format all code (writes)

  • script/fmt-check — check formatting (read-only)

  • script/check — run script/test, script/lint, and script/fmt-check. This is authoritative because script/lint builds Dockerfile.lint: a local make check and CI cannot disagree about lint findings.

  • script/docker — build the Docker image tagged via script/projectname. Passes a fresh --build-arg CHECK_EPOCH for the same reason script/cibuild does, so a local image build cannot be green on checks it replayed from cache. It builds the product image only, and the product Dockerfile has no lint stage, so it does not lint: a green here means formatted, tested, and it compiles.

  • script/cibuild — CI entrypoint, and the full gate. Two builds, in order: Dockerfile.lint (the linter, as a build step) and then Dockerfile (make fmt-check and make test in its builder stage, then the product image). Either failing fails the script. It runs the checks in the same containers CI does, from a clean copy of the tree, so it also catches anything that depends on host state.

    It passes a fresh --build-arg CHECK_EPOCH to each build, unique per invocation, which both files declare immediately above their check RUNs and expand into each check command. Those layers are keyed on that value, so a new value re-runs them even on a byte-identical tree, and a green from this script means the checks executed. Dependency and module layers sit above the ARG and still cache, so a build is not cold.

    A build that supplies no CHECK_EPOCH — a bare docker build . or docker build -f Dockerfile.lint . — fails rather than lying. An unset ARG is an empty string and an empty string is a stable cache key, so without a guard such a build would serve every check layer from cache, execute nothing, and still exit 0. Each file therefore asserts the value is non-empty before running anything, and because failed steps are never cached that assertion fires on every invocation rather than once. Use script/lint, script/docker or script/cibuild, which pass the arg; a bare docker build is a loud error.

  • script/precommit — pre-commit gate: go mod tidy + go fmt (must not change files), then script/check

  • script/install-precommit — install the git pre-commit hook that runs script/precommit

releasing

version numbers

The version a binary reports comes from git, not from a constant in a file. script/version decides it, and everything that stamps a binary agrees with it:

  • HEAD is exactly on a tag → that tag with a leading v stripped, so the tag v1.0.0 produces vaultik 1.0.0, matching the archive name vaultik_1.0.0_linux_amd64.tar.gz. goreleaser strips the prefix the same way.
  • anything else → dev-<12 chars of the commit sha>.
  • either, with uncommitted changes to tracked files → a -dirty suffix, because a modified checkout of a tag is not that tag.

A build that is not a release never names itself like one. vaultik version says so in as many words on a development build, and goreleaser --snapshot stamps the same dev-<sha> string rather than inventing the next patch number. If script/version cannot be run at all, make stops with an error instead of building an unversioned binary, and a binary that somehow carries an empty version string still reports itself as a development build.

cutting a release

Releases are cut by CI from a tag, not from a workstation:

git tag -a v1.2.3 -m 'v1.2.3'
git push origin v1.2.3

.gitea/workflows/release.yml triggers on v* tags, installs a Go toolchain and the pinned goreleaser, and runs script/release, which builds linux,darwin × amd64,arm64 archives plus checksums.txt and publishes them to this repository's Gitea releases as a draft. .goreleaser.yaml has a gitea_urls: block pointing at https://git.eeqj.de/api/v1; without it goreleaser would talk to the GitHub API.

The workflow needs one repository Actions secret:

Secret What it is
RELEASE_TOKEN A Gitea access token with write:repository scope, owned by an account that can publish releases here.

It is passed to goreleaser as GITEA_TOKEN. The runner's automatic token is deliberately not used: it is not guaranteed to carry release write access.

The Go toolchain that compiles the released binaries comes from an actions/setup-go step pinned by commit sha, reading its version from go.mod (currently 1.26.1, the same version the Dockerfile builder stage pins by digest). goreleaser shells out to go for every cross-compile, so without that step the release would either fail outright or ship binaries built by whatever unpinned toolchain the runner happened to carry — the one unpinned thing in an otherwise hash-pinned release path.

To rehearse the whole build without publishing or tagging anything:

make release-snapshot

Artifacts land in ./dist, which is gitignored.

Release artifacts are not signed, carry no SBOM, and are not built reproducibly; the archives contain the binary, LICENSE, and README.md only (no shell completions or man page).

license

MIT

author

Made with love and lots of expensive SOTA AI by sneak in Berlin in the summer of 2025.

Released as a free software gift to the world, no strings attached.

Contact: sneak@sneak.berlin

https://keys.openpgp.org/vks/v1/by-fingerprint/5539AD00DE4C42F3AFE11575052443F4DF2A55C2