Files
vaultik/README.md
clawbot 696ed9ab4d
All checks were successful
check / check (push) Successful in 2m16s
Gate prune's local-cleanup output on --json, and make make build build (closes #108)
Closes #110.

CleanupLocalSnapshots wrote three prose lines to stdout with no --json
awareness, covering every branch, so `vaultik prune --json | jq` failed
on any input. -q never helped either: printlnStdout and stdoutf write
straight to v.Stdout and never consult v.UI, which is what SetQuiet
affects. It now takes *PruneOptions, symmetric with its sibling phase
PruneBlobs, and gates all three writes.

Threading opts.JSON was chosen over moving the lines to log.Info,
because internal/log/log.go defaults the level to Warn: log.Info would
not have relocated them to stderr, it would have deleted them from a
plain `vaultik prune`, and "Removing stale local record" narrates the
deletion of local index rows. The stale-record count is deliberately not
added to PruneBlobsResult - every field there is blob-scoped and produced
by the phase that runs after this reconciliation, so adding it would
change a published --json schema as a side effect of a stream fix.

Note for anyone reading the --json contract: under --json the
stale-record removal now produces no signal in either stream. stdout is
correctly gated, stderr is level-pinned to Warn because --json sets
Quiet, and the count is not in the document. That is inherited behaviour
- PruneBlobs' own log.Info calls are equally invisible under --json - not
something this change introduced, and it is tracked separately.

make build exited 0 and produced nothing: .PHONY listed build with no
build: rule, and a phony target with no prerequisites and no recipe is
considered already satisfied, which turns what would be a hard error into
a silent success. In a repo where `make build` is the documented way to
build, a caller checking the exit code concluded the build worked. Now
`build: vaultik`, verified in both directions - a clean build produces
the binary, a deliberately broken one exits non-zero and produces none.

All 19 .PHONY names were audited; build was the only one lacking a rule.
TestPhonyTargetsAllHaveRules keeps that true for names added later, so
the class is closed rather than the instance.
2026-08-09 19:53:46 +02:00

36 KiB
Raw Blame History

vaultik (ваултик)

vaultik is an incremental backup tool written in Go. It encrypts data using an age public key and uploads each encrypted blob directly to a remote S3-compatible object store. It requires no private keys, secrets, or credentials (other than those required to PUT to encrypted object storage, such as S3 API keys) stored on the backed-up system.

quickstart

# install
go install sneak.berlin/go/vaultik/cmd/vaultik@latest

# create a default config file (prints the path it wrote to)
vaultik config init

# generate an age keypair; keep the private key file somewhere safe and
# offline — you need it to restore, and the backed-up machine does not need it
age-keygen -o vaultik_backup_private_key.txt
grep 'public key' vaultik_backup_private_key.txt

# configure the encryption key and backup destination
vaultik config set age_recipients.0 age1YOUR_PUBLIC_KEY_HERE
vaultik config set storage_url "file:///Volumes/usbstick/mybackup"

# macOS only: grant your terminal app Full Disk Access first
# (System Settings → Privacy & Security → Full Disk Access), otherwise
# the backup will abort with a permission error on protected directories

# run your first backup (the default config backs up ~ and /Applications
# with sensible excludes)
vaultik snapshot create

# see what you have
vaultik snapshot list

Features:

  • modern encryption (age, X25519 + XChaCha20-Poly1305)
  • content-defined chunking with deduplication (FastCDC)
  • incremental backups (only changed files are re-chunked)
  • multithreaded zstd compression at configurable levels
  • content-addressed immutable storage
  • local state tracking in SQLite (enables write-only incremental backups)
  • no mutable remote metadata
  • no plaintext file paths or metadata in remote storage
  • packs small files into large blobs (keeps S3 operation counts down)
  • backs up regular files, symlinks, empty directories, and file permissions
  • pluggable storage backends: S3, local filesystem, rclone (70+ providers)
  • pure Go (no CGO), cross-compiles to linux/darwin × amd64/arm64

why

Other backup tools like restic, borg, and duplicity are designed for environments where the source host can store secrets and has access to decryption keys. vaultik is for environments where you don't want to store backup decryption keys on your hosts — only public keys for encryption.

Requirements that no existing tool meets:

  • open source
  • no passphrases or private keys on the source host
  • incremental
  • compressed
  • encrypted
  • s3 compatible without an intermediate step or tool

daily use

# verify a snapshot (shallow: checks all blobs exist)
vaultik snapshot verify <snapshot-id>

# deep verify (downloads and cryptographically verifies every blob)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot verify --deep <snapshot-id>

# restore (requires the private key)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot restore <snapshot-id> /tmp/restored

# daily cron job: back up, keep a 4-week rolling window of snapshots
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w

cli

commands

vaultik [--config <path>] config init
vaultik [--config <path>] config edit
vaultik [--config <path>] config get <key>
vaultik [--config <path>] config set <key> <value>
vaultik [--config <path>] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>]
vaultik [--config <path>] snapshot list [--json]
vaultik [--config <path>] snapshot verify <snapshot-id> [--deep] [--json]
vaultik [--config <path>] snapshot purge [--keep-latest | --older-than <duration>] [--snapshot <name>...] [--force]
vaultik [--config <path>] snapshot remove <snapshot-id> [--dry-run] [--force] [--local-only] [--json]
vaultik [--config <path>] snapshot restore <snapshot-id> <target-dir> [paths...] [--verify]
vaultik [--config <path>] prune [--force] [--json]
vaultik [--config <path>] info
vaultik [--config <path>] remote info [--json]
vaultik [--config <path>] remote nuke --force
vaultik [--config <path>] database delete [--force]
vaultik completion <bash|zsh|fish|powershell>
vaultik version

global flags

  • --config <path>: Path to config file (default: $VAULTIK_CONFIG, then platform config dir, then /etc/vaultik/config.yml)
  • --verbose, -v: Enable verbose output (on stderr — see below)
  • --debug: Enable debug output (on stderr — see below)
  • --quiet, -q: Suppress non-error output (also suppresses startup banner)
  • --skip-errors: Continue past per-file errors instead of aborting (applies to snapshot create and restore)

stdout and stderr

Log output — everything from --verbose and --debug, and every warning and error the logger emits — goes to stderr. stdout carries the output you asked for: tables, and the documents produced by --json.

This means vaultik snapshot list --verbose > out.txt captures the listing and leaves the diagnostics on your terminal. To capture both, redirect stderr as well (> out.txt 2> log.txt, or > out.txt 2>&1 to interleave them).

The split is what makes --json usable from a script. Warnings and errors are never suppressed — not by --quiet, not by --cron — so a logger on stdout would eventually land a log line inside a JSON document and break the parse. A config file with group- or world-readable permissions is enough to trigger it.

Format follows the stream: when stderr is a terminal the records are colorized one-liners, and when it is redirected or piped they are JSON, one object per line.

Under --json, stdout holds the document and nothing else. The startup banner is suppressed, as --quiet and --cron suppress it, and the progress narration a command would otherwise print — such as the stale local records prune reconciles away — is suppressed too, so it cannot land ahead of the document. Every --json command therefore pipes on its own, with no additional flag: vaultik snapshot list --json | jq . and vaultik prune --json | jq . both work as written.

environment variables

  • VAULTIK_AGE_SECRET_KEY: Age private key for decryption (required for snapshot restore and snapshot verify --deep)
  • VAULTIK_CONFIG: Path to config file (overridden by --config)
  • VAULTIK_INDEX_PATH: Override local SQLite index path

shell completion

# zsh: load for the current session
source <(vaultik completion zsh)

# zsh: install permanently
vaultik completion zsh > "${fpath[1]}/_vaultik"

# bash: load for the current session
source <(vaultik completion bash)

# bash: install permanently (Linux)
vaultik completion bash > /etc/bash_completion.d/vaultik

# fish
vaultik completion fish > ~/.config/fish/completions/vaultik.fish

command details

config init: Write a default config file with commented explanations for every setting. Writes to the path from --config, $VAULTIK_CONFIG, or the platform config directory (~/Library/Application Support/vaultik/ on macOS, ~/.config/vaultik/ on Linux, /etc/vaultik/ as root). Refuses to overwrite an existing file. Created with mode 0600 since it will contain credentials.

config edit: Open the config file in $EDITOR (falls back to vi).

config get: Print a config value addressed by dotted YAML path (e.g. vaultik config get storage_url). Non-scalar values print as YAML.

config set: Set a scalar config value by dotted YAML path (e.g. vaultik config set compression_level 9, vaultik config set storage_url "file:///mnt/backups"). Comments and formatting in the file are preserved; intermediate maps are created as needed.

snapshot create: Perform incremental backup of configured snapshots.

  • Optional snapshot names argument to create specific snapshots (default: all)
  • On macOS, the terminal application running vaultik needs Full Disk Access (System Settings → Privacy & Security → Full Disk Access) to read TCC-protected directories; without it the backup aborts with a permission error that explains how to fix it
  • --cron: Silent on total success; warnings and errors are still printed (for crontab)
  • --prune: After backup, drop older snapshots of each backed-up name and remove orphaned blobs from remote storage. By default keeps only the latest snapshot per name; use --keep-newer-than for a rolling window.
  • --keep-newer-than <duration>: With --prune, keep snapshots newer than this duration instead of only the latest (e.g. 4w, 30d, 6mo, 1y)

snapshot list: Show every snapshot known to this host — the union of the local index and the backup destination store — with timestamps and three sizes per snapshot (compressed remote size; total uncompressed chunk size; size of chunks newly referenced by that snapshot).

Listing the destination store does not require the age secret key, so it works in vaultik's intended configuration, where the backed-up host holds only the public key. A host that has lost its local index can still see what it has backed up.

What that host cannot see is a remote-only snapshot's name. The snapshot ID is hashed at the storage boundary and the manifest records only the hash, so hostname and snapshot name exist solely in the local index and in the encrypted per-snapshot database. Snapshots found only on the destination store are therefore listed as <remote only:<abbreviated remote key>> and show <remote only> in the uncompressed and "new chunk" columns, which can only be computed from the local index. Their timestamp and compressed size are real, read from the manifest.

Snapshots in the local index with no counterpart on the destination store are reported below the table as drift, with the vaultik prune invocation that reconciles them.

If the destination store cannot be listed (unmounted volume, permission denied, network down), the command warns, falls back to the local index alone, and still exits zero.

  • --json: Output in JSON format. Each entry carries locally_tracked (whether the snapshot is in the local index), remote_key (the full 64-character storage key), and remote_present (whether it was seen on the destination store, or null if the destination could not be listed). Warnings about an unlistable destination, unreadable manifests, and a truncated listing all go to stderr through the logger, so stdout stays a single parseable document.

snapshot verify: Verify snapshot integrity.

  • Default (shallow): checks that all blobs referenced in the manifest exist in storage
  • --deep: Downloads and decrypts each blob, verifies chunk hashes against the encrypted metadata database
  • --json: Output results as JSON

snapshot purge: Remove old snapshots based on criteria. Retention is per-snapshot-name (--keep-latest keeps the latest of each name, not the latest globally).

  • --keep-latest: Keep only the most recent snapshot of each name
  • --older-than <duration>: Remove snapshots older than duration (e.g. 30d, 6m, 1y)
  • --snapshot <name>: Restrict to specific snapshot names (repeat for multiple)
  • --force: Skip confirmation prompt

snapshot remove: Remove one snapshot. By default this removes the snapshot from the local index and strips the snapshot's metadata from the backup destination store. Blobs are NOT touched — deleting blobs requires reading every remaining remote manifest (the destination store may hold snapshots this host doesn't know about), which is what vaultik prune does. On success the command prints the exact vaultik prune invocation to run as a follow-up. Local row cleanup (files, chunks, blobs the snapshot was the last referrer for) runs automatically. If the destination store is unreachable, the local-DB removal still completes and a warning is emitted; rerun vaultik prune once the store is reachable to finish remote cleanup. To wipe everything on the destination in one go, use vaultik remote nuke --force.

  • --local-only: Skip remote cleanup; only touch the local index
  • --dry-run: Show what would be deleted without deleting
  • --force: Skip confirmation prompt
  • --json: Output result as JSON

snapshot restore: Restore files from a backup snapshot.

  • Requires VAULTIK_AGE_SECRET_KEY environment variable
  • Optional path arguments to restore specific files/directories (default: all)
  • Preserves file permissions, timestamps, ownership (ownership requires root), symlinks, and empty directories
  • --verify: After restoring, verify every file's chunk hashes match

prune: Tidy up everything that isn't needed. Runs three passes: (1) reconcile the local index against the destination store — any local snapshot whose remote metadata is missing is dropped from the local index; (2) delete orphaned local rows (files, chunks, blobs no longer referenced by any completed snapshot); (3) list every remote manifest on the destination store to compute the still-referenced blob set and delete any blob not in that set. Step (3) reads all remote manifests — network cost scales with the number of snapshots. snapshot create --prune runs the same cleanup automatically; this is the manual entry point for the same work.

  • --force: Skip confirmation prompt
  • --json: Output stats as JSON

info: Display system configuration, storage settings, encryption recipients, and local database statistics.

remote info: Show storage backend type and location plus detailed remote storage inventory: per-snapshot metadata sizes, blob counts, and orphaned blob detection.

  • --json: Output as JSON

remote nuke: Delete every snapshot's metadata and every blob from the backup destination store, leaving the bucket prefix empty. Destructive and irreversible. This is the single supported way to wipe the entire destination store.

  • --force: Required to confirm destruction.

database delete: Delete the local SQLite state database file entirely. Remote storage is unaffected; the next backup will do a full scan and re-deduplicate against existing remote blobs, and the local index will re-bind to the currently configured storage destination. Use this after changing storage_url to a different destination.

  • --force: Skip confirmation prompt

storage backends

vaultik supports three storage backends, selected via the storage_url config field:

S3 (s3://bucket/prefix?endpoint=host&region=us-east-1): Any S3-compatible object store. Credentials are read from s3.access_key_id and s3.secret_access_key in the config file.

Local filesystem (file:///path/to/backup): Stores blobs and metadata on a local or mounted filesystem. Useful for testing or backing up to a NAS.

Rclone (rclone://remote/path): Uses rclone's 70+ supported cloud providers. Requires rclone to be configured separately (rclone config).

Legacy S3 configuration via s3.* fields (endpoint, bucket, prefix, etc.) is still supported for backward compatibility. storage_url takes precedence if both are set.


architecture

remote storage layout

<bucket>/<prefix>/
├── blobs/
│   └── <aa>/<bb>/<full_blob_hash>
└── metadata/
    └── <snapshot_id>/
        ├── db.zst.age          # Encrypted binary SQLite database
        └── manifest.json.zst   # Unencrypted blob list (for pruning)
  • Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
  • db.zst.age is a binary SQLite database (zstd compressed, age encrypted) containing all file metadata, chunk mappings, and relationships for the snapshot
  • manifest.json.zst is an unencrypted compressed JSON blob list, enabling pruning without the private key

Snapshot IDs follow the format <hostname>_<snapshot-name>_<RFC3339-timestamp> (e.g. server1_home_2025-06-01T12:00:00Z).

data flow

backup:

  1. Open local SQLite index, load known files and chunks into memory
  2. Walk source directories, compare mtime/size/mode against index
  3. For changed/new files: chunk using content-defined chunking (FastCDC)
  4. For symlinks and directories: record metadata (no chunking)
  5. For each chunk: hash, check dedup, add to blob packer
  6. When blob reaches size threshold: compress (zstd), encrypt (age), upload
  7. Build snapshot metadata database, compress, encrypt, upload
  8. Create unencrypted blob manifest for pruning support

restore:

  1. Download and decrypt metadata/<snapshot_id>/db.zst.age
  2. Open the binary SQLite database
  3. Query files (optionally filtered by paths)
  4. Download and decrypt required blobs
  5. Extract chunks, reconstruct files
  6. Restore permissions, timestamps, ownership, symlinks

prune:

  1. List all snapshot manifests
  2. Build set of all referenced blob hashes
  3. List all blobs in storage
  4. Delete any blob not in the referenced set

chunking and deduplication

  • Content-defined chunking using the FastCDC algorithm
  • Average chunk size: configurable (default 10MB)
  • Deduplication at file level (unchanged files skipped) and chunk level (identical chunks across files stored once)
  • Multiple chunks packed into blobs to reduce object count

encryption

  • Asymmetric encryption using age (X25519 + XChaCha20-Poly1305)
  • Only the public key is needed on the source host
  • Each blob and each metadata database is encrypted independently
  • Multiple recipients supported (encrypt to multiple keys)

compression

  • zstd compression at configurable level (1-19, default 3)
  • Applied before encryption at the blob level

configuration reference

Run vaultik config init to generate a fully commented config file. Key fields:

Field Default Description
age_recipients (required) Age public keys for encryption
snapshots (required) Named snapshot definitions with paths and excludes
storage_url Storage backend URL (s3://, file://, rclone://)
s3.* Legacy S3 configuration (endpoint, bucket, credentials)
exclude Global exclude patterns (applied to all snapshots)
chunk_size 10MB Average chunk size for content-defined chunking
blob_size_limit 10GB Maximum blob size before splitting
compression_level 3 zstd compression level (1-19)
hostname system hostname Hostname used in snapshot IDs
index_path platform data dir Local SQLite index path

limitations

  • No extended attributes (xattrs). ACLs, macOS Finder metadata, quarantine flags, SELinux labels, and other extended attributes are not backed up or restored.
  • No hard link detection. Two hard links to the same inode are backed up as independent files. Content deduplication means the data is stored once, but the hard link relationship is lost on restore.
  • No sparse file support. Sparse files are fully materialized during backup. A 100 GB sparse VM disk that is mostly zeros will consume the full (compressed) size in storage.
  • No bandwidth limiting. Uploads and downloads use whatever bandwidth is available. There is no --bwlimit flag yet.
  • No parallel blob downloads during restore. Blobs are fetched sequentially. Restore speed is bound by single-stream throughput.
  • Device nodes, named pipes, and sockets are silently skipped. Only regular files, directories, and symlinks are backed up.
  • No database migrations. If the local SQLite schema changes between versions, delete the local database (vaultik database delete) and run a full backup. Remote storage is unaffected.
  • Files that change during backup may be inconsistent. There is no filesystem snapshot or freeze. If a file is modified between the scan and chunk phases, the backed-up copy may reflect a partial write.
  • Ownership restoration requires root. File uid/gid are recorded and restored, but chown requires elevated privileges. Without root, files are restored with the current user's ownership.

roadmap

Items still to do before / shortly after 1.0. Loosely ordered by priority.

correctness and operability

  • Security audit of the encryption implementation. Pre-1.0 blocker if we're advertising "secure" at the top of this README. age + zstd + content-defined chunking is mostly off-the-shelf pieces, but the seams (key handling, recipient parsing, manifest trust boundary, restore-time identity validation) need an outside read.
  • Error-condition tests. Today's coverage is the happy path plus a few specific regressions. Need fault-injection coverage: network failures mid-blob, disk-full during restore, corrupted / truncated / missing blobs, partial uploads, kill -9 between manifest and db.zst.age writes.
  • Verify restored content end-to-end in CI. The current integration test does this for a small synthetic snapshot but not at scale. A nightly job against a multi-GB representative snapshot would catch silent regressions in the chunker, packer, or restore planner.

performance

  • Parallel blob downloads during restore. Single-stream right now. With a fast S3 endpoint and a multi-core machine restore is bound by per-blob fetch + decrypt + decompress; running N of those in parallel against the disk cache would close most of the remaining gap. Needs to interact correctly with the locality planner and sweeper.
  • Bandwidth limiting (--bwlimit). Both upload and download. Useful for backing up over a shared link. Tricky to make work correctly with the parallel-download story.
  • Restart of interrupted restore. Today restore is restartable in the sense that re-running it overwrites partial output; it doesn't resume from where it stopped or skip already-present files. A --resume mode that checks targets before fetching blobs would matter for very large restores.

usability

  • Man pages and richer --help examples. Cobra generates basic help; man pages would be a separate target.
  • --bwlimit style human-readable size flags across the command surface where they're currently raw integers.
  • vaultik snapshot diff <a> <b> — show which files changed between two snapshots without restoring either.
  • Status reporting hook for --cron. When a backup fails silently in cron, the user has no idea. A configurable webhook / email / notify-send hook on completion (success and failure) would close the loop.

infrastructure

  • Cross-machine restore documentation. The "restore from another host" workflow works but isn't documented as a first-class operation in this README. Worth a dedicated section once it's settled.
  • Schema migrations. Currently nonexistent — pre-1.0 schema changes are handled by vaultik database delete plus a full re-scan. Post-1.0 we'll need a migration story to keep existing index databases usable across upgrades.
  • Storage backend coverage tests. S3, file://, and rclone:// all share the Storer interface but the rclone path is the least exercised in CI.

output style

All user-facing output goes through helpers in internal/ui and conforms to a uniform style. Color is enabled when stdout is a TTY and the NO_COLOR environment variable is unset (https://no-color.org/).

internal/ui writes to stdout; it is the output the user asked for. Structured log records are a different thing and go through internal/log, which writes to stderr (see "stdout and stderr" above).

Message classes:

Class Marker Alignment Use for
Banner none column 0 The startup line printed once per invocation
Begin (white) column 0 An operation is about to start (present-continuous verb)
Complete (green) column 0 An operation just finished (past-tense verb)
Info (white) column 0 Neutral status update
Notice (cyan) column 0 Important note that is not a warning
Warning ⚠️ Warning: (orange/yellow) column 0 Recoverable problem
Error 🛑 ERROR: (red) column 0 Operation aborted
Progress (white) column 2 Heartbeat or per-item status during a long-running operation
Detail (white) column 2 Continuation/sub-line of a preceding Complete (visually identical to Progress)

Conventions:

  • Messages are complete English sentences ending with a period.
  • Fully qualify terms — say "backup destination store" instead of "storage", "snapshot source files enumeration" instead of "scan", "local index database" instead of "database".
  • Every operation that emits a Complete also emits a corresponding Begin. Operations that print only a Begin (because completion is obvious from a later Begin) should be rare and intentional.
  • Use natural verb tense to signal state: "Uploading" for Begin, "Uploaded" for Complete. Never write the words "begin" or "complete" in the body — the marker color already conveys that.
  • All elapsed and remaining-time fields are explicitly scoped to their subject: write "blob upload elapsed: 30s, blob upload ETA: 03:15:00 (est remain 14s)", never just "elapsed 30s, ETA 14s".
  • "ETA" means an absolute clock time (when the operation will finish), not a remaining-duration. Use ui.Time() for the former and ui.Duration() for the latter, and label both.
  • ui.Time formats same-day times as HH:MM:SS and other-day times as YYYY-MM-DD HH:MM:SS. No timezone — local time is implied.

Value colorizers in internal/ui colorize specific value types consistently. Compose messages from these helpers rather than embedding ANSI escapes inline:

Helper Color Use for
Hex cyan Blob hashes, chunk hashes (truncated to 12 chars + ...)
Snapshot bold cyan Snapshot IDs (untruncated)
Path blue Filesystem paths
Size magenta Byte counts (human-readable)
Speed magenta Bytes-per-second rates
Duration yellow Elapsed or remaining time
Time yellow Absolute clock times
Count magenta Integer counts with thousands separators
Percent magenta Percentages

When NO_COLOR is set or output is not a TTY, all helpers return plain text and the marker prefixes (, Warning:, ERROR:) emit without ANSI escapes. The emoji prefixes on Warning and Error are always emitted regardless of color setting (emoji are not color).

requirements

  • Go 1.26 or later
  • Docker, with a reachable daemon, to lint, check, or commit: script/lint runs the digest-pinned golangci-lint image declared by the Dockerfile lint stage, and make check and the pre-commit hook both run it. A golangci-lint installed on PATH is not a substitute and is never used on a host, whatever its version.
  • sqlite3 CLI, which the test suite shells out to
  • S3-compatible object storage (or local filesystem, or rclone remote)

development workflow

All changes follow this workflow. No exceptions.

  1. Create a feature branch off main.
  2. Write tests.
  3. Write the implementation.
  4. Fix implementation errors until it compiles and tests pass.
  5. Fix linting errors (make lint).
  6. Update documentation and README as required by the change.
  7. Format code (make fmt).
  8. Run make check (lint + fmt-check + test). Fix any issues. Repeat until clean.
  9. Commit on the branch.
  10. Merge to main.
  11. Push.

Do not commit directly to main. Do not skip steps.

Repository policies for AI agents are in AGENTS.md.

Entrypoints

This repository adheres to the Scripts to Rule Them All standard: normalized scripts in script/ are the entrypoints for the development workflow, and the Makefile targets are thin shims that call them. We provide:

  • script/bootstrap — install all development dependencies (go, sqlite3, Go module download). It deliberately does not install golangci-lint; see script/lint below.

  • script/setup — make a fresh clone ready for development: runs script/bootstrap, then script/install-precommit

  • script/projectname — print the project name (used for the Docker image tag)

  • script/version — print the version string to bake into the binary. The Makefile's LDFLAGS call this; it is the single source of truth for the version. See releasing for the rules.

  • script/install-goreleaser — install the pinned goreleaser into .tool/bin from a sha256-verified release archive. Idempotent, and called by script/bootstrap; the release workflow calls it directly because it needs goreleaser but not the Docker daemon script/bootstrap insists on.

  • script/release — cross-compile and publish the release artifacts with the pinned goreleaser. Refuses a goreleaser on PATH whose version is not the pinned one, on the same reasoning as script/lint.

  • script/release-snapshot — the same build with no publishing and no tagging, into ./dist

  • script/test — run the test suite (verbose rerun on failure). This runs everything: there is no separate integration target and no build-tagged subset held back, so the full round-trip tests in internal/vaultik/integration_test.go run on every invocation. It passes -count=1, which disables Go's test result cache. That is deliberate and it is not free: on this repo's suite it costs about 11 seconds on every repeat run (measured, back to back: 0.4s cached versus 11.6s with -count=1). That is the price of the run meaning anything, because without it an unchanged package prints ok <pkg> (cached), which is indistinguishable from a package that really ran, so the whole suite can report a full set of ok lines in under half a second having executed nothing. The -timeout is a hang backstop rather than a performance budget — it applies per test binary to test execution only, not to compilation — and is set well above the slowest package's measured runtime. Its 120s value deliberately diverges from the 30s REPO_POLICIES.md mandates; the reasoning is in the comment in the script, and issue #101 proposes amending the policy text.

  • script/lint — run golangci-lint run ./... at the exact version CI uses, by running the digest-pinned golangci-lint image declared by the Dockerfile lint stage (requires Docker; it fails loudly rather than falling back to a differently versioned golangci-lint on PATH). That FROM line is the single source of truth for the linter version — bump it there and nowhere else.

  • script/lint-fix — apply the linter's autofixes (rewrites files), using the same pinned linter

  • script/fmt — format all code (writes)

  • script/fmt-check — check formatting (read-only)

  • script/check — run script/test, script/lint, and script/fmt-check. This is authoritative because script/lint uses the pinned linter: a local make check and CI cannot disagree about lint findings.

  • script/docker — build the Docker image tagged via script/projectname. Passes a fresh --build-arg CHECK_EPOCH for the same reason script/cibuild does, so a local image build cannot be green on checks it replayed from cache.

  • script/cibuild — CI entrypoint: docker build (the Dockerfile runs make fmt-check and make lint in its lint stage and make test in its builder stage). This is the full CI-equivalent gate — it runs the checks in the same containers CI does, from a clean copy of the tree, so it also catches anything that depends on host state. It passes a fresh --build-arg CHECK_EPOCH, unique per invocation, which the Dockerfile declares immediately above the check RUNs in both stages and expands into each check command. Those layers are keyed on that value, so a new value re-runs them even on a byte-identical tree, and a green from this script means the checks executed. Dependency and module layers sit above the ARG and still cache, so a build is not cold.

    A build that supplies no CHECK_EPOCH — a bare docker build . — fails rather than lying. An unset ARG is an empty string and an empty string is a stable cache key, so without a guard such a build would serve all three check layers from cache, execute nothing, and still exit 0. Each check stage therefore asserts the value is non-empty before running anything, and because failed steps are never cached that assertion fires on every invocation rather than once. Use script/cibuild (or script/docker, which passes the same arg); a bare docker build . is now a loud error.

  • script/precommit — pre-commit gate: go mod tidy + go fmt (must not change files), then script/check

  • script/install-precommit — install the git pre-commit hook that runs script/precommit

releasing

version numbers

The version a binary reports comes from git, not from a constant in a file. script/version decides it, and everything that stamps a binary agrees with it:

  • HEAD is exactly on a tag → that tag with a leading v stripped, so the tag v1.0.0 produces vaultik 1.0.0, matching the archive name vaultik_1.0.0_linux_amd64.tar.gz. goreleaser strips the prefix the same way.
  • anything else → dev-<12 chars of the commit sha>.
  • either, with uncommitted changes to tracked files → a -dirty suffix, because a modified checkout of a tag is not that tag.

A build that is not a release never names itself like one. vaultik version says so in as many words on a development build, and goreleaser --snapshot stamps the same dev-<sha> string rather than inventing the next patch number. If script/version cannot be run at all, make stops with an error instead of building an unversioned binary, and a binary that somehow carries an empty version string still reports itself as a development build.

cutting a release

Releases are cut by CI from a tag, not from a workstation:

git tag -a v1.2.3 -m 'v1.2.3'
git push origin v1.2.3

.gitea/workflows/release.yml triggers on v* tags, installs a Go toolchain and the pinned goreleaser, and runs script/release, which builds linux,darwin × amd64,arm64 archives plus checksums.txt and publishes them to this repository's Gitea releases as a draft. .goreleaser.yaml has a gitea_urls: block pointing at https://git.eeqj.de/api/v1; without it goreleaser would talk to the GitHub API.

The workflow needs one repository Actions secret:

Secret What it is
RELEASE_TOKEN A Gitea access token with write:repository scope, owned by an account that can publish releases here.

It is passed to goreleaser as GITEA_TOKEN. The runner's automatic token is deliberately not used: it is not guaranteed to carry release write access.

The Go toolchain that compiles the released binaries comes from an actions/setup-go step pinned by commit sha, reading its version from go.mod (currently 1.26.1, the same version the Dockerfile builder stage pins by digest). goreleaser shells out to go for every cross-compile, so without that step the release would either fail outright or ship binaries built by whatever unpinned toolchain the runner happened to carry — the one unpinned thing in an otherwise hash-pinned release path.

To rehearse the whole build without publishing or tagging anything:

make release-snapshot

Artifacts land in ./dist, which is gitignored.

Release artifacts are not signed, carry no SBOM, and are not built reproducibly; the archives contain the binary, LICENSE, and README.md only (no shell completions or man page).

license

MIT

author

Made with love and lots of expensive SOTA AI by sneak in Berlin in the summer of 2025.

Released as a free software gift to the world, no strings attached.

Contact: sneak@sneak.berlin

https://keys.openpgp.org/vks/v1/by-fingerprint/5539AD00DE4C42F3AFE11575052443F4DF2A55C2