vaultik/TODO.md

Workflow

• branch (from main)
• do the work in Next Step
• move Next Step to the top of Completed Steps
• move the top item of Future Steps into Next Step
• commit (TODO.md changes in the same commit as the work)
• merge to main if the branch is not protected, otherwise open a PR
• push

Status

pre-1.0

Next Step

Define the remaining scope for the first tagged release under the 1.0.0 milestone, then cut that tag. The mechanism to cut it now exists and is exercised; what is left is the scope decision, which is the owner's. This step deliberately names one version number: it previously said "cut v0.1.0" while the Makefile baked in 1.0.0-rc.1 and the issue milestone said 1.0.0, and three different answers to "what is the next release" is exactly the contradiction issue #65 was filed over.

Completed Steps

• 2026-08-09: Stopped the startup banner from contaminating --json documents (issue #106). Entry writes the banner to stdout before cobra parses anything, and the flag scan that suppresses it knew --quiet, -q and --cron but not --json, so every --json document arrived behind two lines of prose and a blank line, and vaultik snapshot list --json | jq failed. With the logger already on stderr from issue #82, this was the last writer that could put something on stdout that the caller did not ask for. The design question the issue raised — extend the raw-argv scan, or move the banner after parsing — is answered in favour of the scan: the banner is printed first deliberately, so that it still appears when cobra rejects the arguments and on --help, and after parsing there is no single place that covers those paths. The stated cost of the scan, that --json is a subcommand flag matched anywhere in the vector, is a cost --cron already carries — it exists only on snapshot create — so this adds an instance of an accepted imprecision rather than a new kind, and the two error directions are not symmetric: a false positive loses a decorative banner, a false negative corrupts a document. Regression tests at the CLI layer, where internal/vaultik's existing guard cannot reach: one runs Entry itself over the process's real stdout descriptor, through cobra and fx to the document, made hermetic by file:// storage; a second covers the argument vectors of all five --json commands; a third asserts the banner is still printed without a suppressing flag, so the first cannot be satisfied by deleting the banner. Also corrected AGENTS.md policy 9, which still keyed the structured-log format on stdout's TTY-ness after #82 moved that decision to stderr — a rules file that misdescribes the code misleads exactly the readers who trust it most. Two smaller findings from the same review: bytesAttrKey's human-readable byte formatting silently stopped applying under an open group, because the key reaching the comparison is group-qualified (transfer.bytes), now matched on its final segment and tested both ways; and listEnv.stderr in snapshot_list_test.go, assigned but never read since those tests began capturing the process's stderr, is removed. Vaultik.Stderr is kept — nothing writes to it today, which its comment now says outright.

• 2026-08-09: Moved the logger to stderr and fixed TTYHandler's discarded attributes (issue #82, issue #97). Two defects in internal/log, fixed together because both live in the handler construction path. The first: both handlers were built over os.Stdout, and WARN/ERROR are never suppressed, so a config file with group- or world-readable permissions was enough to put a log record inside a --json document and break jq. Diagnostics now go to stderr, and the TTY/JSON format choice follows stderr rather than stdout — testing the wrong stream would colorize records on a redirected stderr whenever stdout happened to be a terminal. This is user-visible: --verbose and --debug output moves to stderr too, which is documented in README.md under "stdout and stderr". It also let the local workaround in internal/vaultik/snapshot_list.go go: warnWhileListing had been hand-rolling structured-log formatting to reach a non-stdout writer, and the jsonOutput parameter threaded through the remote-listing helpers existed only to choose between the two writers. The collect-then-emit machinery around listingWarning stays, but on its remaining merit — warnings emitted in key order after group.Wait() are deterministic run to run, where emitting from the fetch workers would order them by network timing. The second defect: TTYHandler.WithAttrs and WithGroup discarded their arguments and returned the receiver while their doc comments claimed otherwise, so log.With attributes vanished on a terminal and appeared correctly in CI — failing precisely when someone is debugging interactively. Both now return a new handler (the receiver is never written to, since slog permits concurrent derivation), attributes persist across records, and grouping is implemented as dotted key prefixes, which is the only honest rendering for a format with nowhere to nest. New tests cover both, including one that feeds the same derivation chain to the TTY and JSON handlers and compares the attribute sets, so the two paths cannot drift apart again. Found and filed while verifying: the startup banner is written to stdout and --json does not suppress it (issue #106), which is a separate writer on a separate path and the remaining source of stdout contamination.

• 2026-08-09: Made the tagged-release path actually work on Gitea (issue #65). Three independent blockers, one of which was the whole release: .goreleaser.yaml had no gitea_urls: block, so goreleaser defaulted to the GitHub API and a goreleaser release from this repo would have failed or published where nobody is looking. It now points at https://git.eeqj.de/api/v1. The version is the second: it was a hardcoded VERSION := 1.0.0-rc.1 in the Makefile, so every local build claimed to be a release candidate that had never been tagged and did not exist, while git tag -l was empty and internal/globals defaulted to dev. Version now comes from git via the new script/version — the exact tag with a leading v stripped (so a make build and a goreleaser build of one commit report the same string, and it matches the archive names), otherwise dev-<12-char sha>, with -dirty appended in either case when tracked files are modified. Untracked files are deliberately not counted, matching git describe --dirty. The same honesty was owed by the snapshot path: snapshot.version_template was {{ incpatch .Version }}-next, which manufactures a release number from the last tag and, with no tags at all, from goreleaser's fabricated v0.0.0; it now emits the same dev-<sha>. The one non-obvious consequence is that internal/cli/version.go gated its "this is a development build" notice on the version being exactly dev, so the moment untagged builds began carrying a commit sha that notice would have gone silent and an unreleased binary would have read as a release — the gate is now globals.IsDevVersion, which is a predicate over a string rather than a comparison against a global precisely so it can be tested, and it is tested at the boundary (1.0.0-dev is a release, dev-<sha> is not). Release automation is the third blocker: a tag-triggered .gitea/workflows/release.yml runs the build in CI rather than from a laptop, with fetch-depth: 0 because a shallow checkout has no tags and would silently mislabel the release, and with the RELEASE_TOKEN repository secret passed as GITEA_TOKEN (documented in README.md; the runner's automatic token is not used because it is not guaranteed to carry release write scope). script/release unsets any GITHUB_TOKEN/GITLAB_TOKEN it finds, since goreleaser chooses its forge from whichever token variable is set and refuses to run when it sees more than one — a runner-provided token must not get to decide where these artifacts are published. make release and make release-snapshot, the last two Makefile targets that were not shims, now call script/release and script/release-snapshot, which resolve goreleaser exactly the way script/lint resolves the linter: a PATH binary is used only at the pinned version, never as a silent fallback. script/bootstrap installs it, from a sha256-verified GitHub release archive per REPO_POLICIES.md, via a separate script/install-goreleaser — separate because script/bootstrap hard-fails without a usable Docker daemon by design, and the release runner needs goreleaser without needing Docker. Verified by running the thing rather than reading it: make release-snapshot produced four archives and checksums.txt, and the linux/amd64 binary from dist/ reports dev-<sha> with the development-build notice. Tag handling was exercised in a throwaway repository rather than by tagging this one; no tag was created here, since that is the owner's call. Signing, SBOM, reproducible builds, completions and a man page are out of scope by the issue.

• 2026-08-09: Isolated the lint cache per worktree and context-gated the native lint path (issues #99, #80). One defect seen twice: script/lint decided whether it could skip the pinned image by asking what version was on PATH rather than where it was running, and cache isolation is part of that same question. The cache was one directory per repo, shared by every worktree on the host, so two checkouts with identical Go file contents collided and golangci-lint replayed the stored analysis — paths and all. The loud direction of that failure (a clean tree failed by a dirty sibling) is the harmless one; the silent direction, a dirty tree passed by a clean sibling, is a sixth way for a gate here to report a green it did not earn. The cache is now keyed on a digest of the worktree path, and every run is audited by the new script/lint-audit, which rejects output citing any file that is not in the tree being linted — a backstop that runs on clean output too, because that is the case nobody investigates. Caches record the worktree they belong to and are collected when it disappears, so throwaway worktrees do not accumulate them; the whole tree lives under XDG_CACHE_HOME and is disposable. The parallel golangci-lint is running refusal is now a bounded retry rather than a verdict: it is not a lint result, and exiting non-zero on it is indistinguishable to a caller from real findings (#88 showed a private cache does not remove that contention). The native path now requires VAULTIK_LINT_IN_CONTAINER=1, set only by the Dockerfile lint stage, in addition to matching the pin, so a developer's locally installed 2.12.2 no longer bypasses the digest pin; /.dockerenv was rejected as the signal because dockerd creates it for docker run and it is not reliably present during a BuildKit docker build, which is the case the exception exists for. Version detection uses golangci-lint version --short with the old banner scrape kept only as a fallback. script/bootstrap no longer prints bootstrap complete on a machine that cannot run the gate: a missing docker, or one whose daemon is unreachable, is a hard failure naming exactly what breaks. Verification was by reproduction rather than inspection — two concurrent lints from two worktrees of differing cleanliness, a real run made to report an outside path, a matching linter shimmed onto PATH, and a PATH with docker removed — and is recorded on the pull request.

• 2026-08-09: Closed the fifth false-green mechanism (issues #93, #69). script/test omitted -count=1, so Go's test result cache could satisfy the gate outright: a second back-to-back make test printed the full set of 14 ok lines, every one marked (cached), having executed no test at all. Since ok <pkg> (cached) is an ok line, the "14 ok lines means the suite ran" signal this repo leans on was forgeable, one level below the Docker layer cache that #85 addressed. Fixed with -count=1 unconditionally rather than only in the container, because the pre-commit hook runs the same script and a gate honest only in CI is dishonest where people rely on it most; test-coverage got the same flag, and script/check inherits it by calling script/test. In the same area, make test-integration was deleted rather than made real: no file in the repo carried a build tag, so -tags=integration selected nothing and the target was an exact duplicate of make test. Tagging a subset was rejected because the entire suite runs in well under a minute, and a scheme whose failure mode is "some tests silently stopped running" is a poor trade for those seconds in a repo with this particular history. The -timeout was raised from 30s after measuring rather than after assuming: the standing claim that cold-cache compilation is charged against -timeout is false, disproved by a containerised run that spent 46s compiling and still reported per-package durations within noise of a warm host run. -timeout reaches the test binary as -test.timeout and its clock starts inside testing.M.Run, after the build. The real exposure was margin, not compilation. The 120s landed on is a deliberate, documented divergence from REPO_POLICIES.md:192, which mandates 30s, and from that file's canonical recipe at :212-214; the divergence is recorded in script/test's comment because REPO_POLICIES.md is org-canonical and not editable here, and issue #101 proposes amending the policy text upstream. Numbers and the full verification are recorded once, on the pull request, and are deliberately not restated here.

• 2026-08-09: Triaged all fifteen stale remote branches (issue #71) and deleted fourteen of them; the full per-branch disposition with evidence is recorded on that issue. Method mattered more than the outcome here: a three-dot git diff main...branch diffs from the merge base, so it replays everything that landed on main after the branch diverged and makes any old branch look like it holds unlanded work. That artifact is what made golangci-v2.12.2 appear to carry 126 files of unpushed changes when its tree was byte-identical to main's. Every containment claim here therefore rests on two-dot tip diffs, tree-hash equality, git cherry, and git branch -r --merged. Nine branches were plain ancestors of main with zero git cherry + commits. golangci-v2.12.2 had landed squashed as cc58583, whose tree hash equals the branch tip's exactly; note the hash recorded in the issue had gone stale because main advanced, so the check had to be redone rather than repeated. fix/sync-snapshot-cleanup was redundant, its one line already on main in syncWithRemote. feature/restore-progress-bar was superseded by printRestoreProgress and the disk-backed blob cache, and had become actively regressive — it would have deleted internal/blobgen/compress_test.go, the #28 regression test that landed separately. The two branches this issue was filed for both turned out to be closed questions that main had already moved past by a recorded decision, so neither was landed and no regression test was owed: ctime no longer exists anywhere in the codebase after 1c72a37 removed the column, the File.CTime field and every use (#54/#55), and change detection compares size, mtime, mode, uid and gid only, exactly as ARCHITECTURE.md documents — so the silently-skipped-file data-loss risk that made this a 1.0 item does not exist. The SQL allow-list branch would have reverted bfd7334, which replaced that very allow-list with regex sanitisation on review feedback, and would have broken getTableCount("snapshots") because its allow-list omits that table. feature/daemon-mode is untouched and deferred to #94 pending an owner decision, so it is the one branch besides main still on the remote. The stale TODO.md entry named in the issue needed no fix: e496aa3 had already removed it. No product code changed.

• 2026-08-09: Adopted the remaining upstream CHECK_EPOCH hardening (issue #91), closing the gap #85 knowingly left open. Four changes, all four decided as adopt upstream in sneak/prompts #26. (1) Each check stage now asserts [ -n "$CHECK_EPOCH" ] || exit 1 before running anything, so a build that supplies no --build-arg fails instead of lying. This is the item that mattered: an unset ARG is an empty string and an empty string is a stable cache key, so the second and every later bare docker build . on an unchanged tree replayed all three check layers and still exited 0 — and docker build . is the command REPO_POLICIES.md names verbatim as a thing that must be green, so the documented command was precisely the one that lied. Failed steps are never cached, which is what makes the guard fire on every invocation rather than once. (2) The epoch is now expanded into each check command rather than left as a bare declaration, so the cache miss no longer depends on BuildKit's unreferenced-ARG handling staying as it is, and the value appears in the build log. (3) script/cibuild uses epoch="$(date +%s%N)$$", unique per invocation rather than per second; %N alone is insufficient because busybox drops it silently and exits 0, and $$ is what makes the guarantee hold regardless. The bare-assignment form is kept deliberately — inlined in an argument, a failing substitution does not abort under set -eu and would yield an empty constant epoch, restoring the exact false green being fixed. (4) script/docker passes the same fresh arg, so the two entrypoints cannot disagree about whether the tree is green; local builds are almost always warm, which made it the likelier fooling in practice. The ARG placement from #85 is unchanged, below apk add, COPY go.mod go.sum and go mod download, so dependency layers still cache and the build is not cold. Verified by negative control rather than inspection — a bare docker build . run twice back to back, plus back-to-back pairs of both scripts and a host-side make check; the measurements are recorded once, in the PR verification comment, rather than restated here. .golangci.yml, the lint-stage FROM line and its digest, script/lint, REPO_POLICIES.md and .gitea/workflows/check.yml are all untouched.

• 2026-08-09: Stopped script/cibuild from reporting a green it did not earn (issue #85). A bare docker build . let Docker serve the check layers from the layer cache whenever the tree had not changed: the checks never executed and the build still exited 0. The fix is an ARG CHECK_EPOCH declared immediately above the check RUNs in both the lint stage and the builder stage (ARG scope is per-stage, so each declares its own), with script/cibuild assigning epoch="$(date +%s)" and passing --build-arg CHECK_EPOCH="$epoch". The assignment is separate on purpose: under set -eu a command substitution that fails inside an argument does not abort the script, which would leave an empty constant CHECK_EPOCH and restore the very false green being fixed. Placement is the rest of the point — the ARG sits below the apk add, COPY go.mod go.sum, and go mod download layers, so only the checks are invalidated and the dependency layers still cache. The guarantee is conditional on a fresh value rather than absolute: a bare docker build . gets an empty CHECK_EPOCH and can still serve the check layers from cache, which README.md and the Dockerfile now say plainly, with issue #91 tracking the upstream hardening (expanded ARG form, unset guard, per-invocation epoch, script/docker) that would close it. Verified by re-running the reproduction plus the withheld---build-arg counterfactual; the measurements are recorded once, in the PR #89 verification comment, rather than restated here. .golangci.yml, the lint-stage FROM line and its digest, script/lint, and .gitea/workflows/check.yml are all untouched.

• 2026-08-09: Corrected the Vaultik.UI doc comment (issue #84). It claimed the cli layer replaces the writer with a discarding one in --cron mode; the actual mechanism is UI.SetQuiet(true) in setupGlobals, which drops Begin/Complete/Info/Notice/Detail/ Progress/Banner but still emits Warning and Error. The --cron line in README.md said "Silent unless error", which understated what survives, and now names warnings too. The other --cron comments (internal/log/log.go, internal/cli/snapshot.go, internal/vaultik/snapshot.go) were audited and already accurate. Comments and docs only, no behavior change.

• 2026-08-09: Made snapshot list list the destination store without the private key (issue #64). The listing is now the union of the local index and a single streamed listing of the metadata/ prefix, with no age_secret_key gate — the manifest is unencrypted, so a host holding only the public key can enumerate its own backups and a host that lost its local index can still see them. A remote-only snapshot's hostname and name are deliberately not recovered (they are not recoverable without the private key, and making them so would undo the privacy property tracked in issue #81); such rows are labelled by an abbreviation of their remote key and carry the real timestamp and compressed size from the manifest, with <remote only> in the two columns that require the local index. Local-only snapshots are reported as drift, and the hint now names vaultik prune, which exists, instead of vaultik snapshot cleanup, which does not. reportRemoteDrift collapsed into the merged view. Every remote manifest read in the codebase now goes through downloadManifestByKey, so issue #81 has one call site to change. Review rework: snapshot timestamps now normalize to UTC in scanSnapshotRows, the one place they enter the domain, so the merged TIMESTAMP column cannot show local time for a locally tracked row and UTC for a remote-only row on a non-UTC host; GetIncompleteByHostname was folded onto that same scanner. --json now reports the unreadable-manifest count and the 1000-row truncation on stderr instead of returning a silently short document (the document's shape is unchanged). The two per-snapshot log.Warn calls on the listing path now route through the same JSON-aware writer as the existing workaround, so one corrupt manifest can no longer put a log line on stdout ahead of the document and break | jq — still a local workaround pending issue #82. Verified with script/cibuild and with an uncached make check (0 issues., no cached test packages), plus end to end against a file:// destination with no secret key present.

• 2026-08-09: Closed the gap between make lint and CI (issue #78). script/lint now runs the digest-pinned golangci-lint image taken from the Dockerfile lint stage, which is the single source of truth for the linter version; the duplicate pin in the Makefile deps target and the unpinned golangci-lint install in script/bootstrap are gone. A golangci-lint on PATH is used only when its version is exactly the pinned one (which is how the lint stage runs it inside the container); anything else goes through Docker, and a missing or unreachable Docker daemon is a hard error rather than a silent fallback. Only the lint leg of make check became equivalent to script/cibuild; its tests and gofmt still run on the host against the host toolchain, as README.md states. An earlier version of this entry claimed make check was "as trustworthy as script/cibuild" outright, which overstated it; corrected under issue #80.

• 2026-08-09: Finished the lint remediation under the canonical .golangci.yml (issue #61, which also unblocks issue #59). The remaining findings were fixed behavior-preservingly: wsl_v5 whitespace, sqlclosecheck, and prealloc. The sqlclosecheck sites now close sql.Rows in a deferred closure instead of via the CloseRows helper, which the linter could not see through. Only the revive package-name findings remain suppressed, with per-site //nolint directives; the package-rename question behind them is tracked in issue #76. Verified with script/cibuild, which exits 0 — that is the only trustworthy gate, because script/lint runs whatever golangci-lint happens to be on PATH rather than the pinned v2.12.2 that CI and the Dockerfile use, so make check can report green on findings CI still fails. That tooling gap is tracked in issue #78.

• 2026-08-09: The earlier next step "reconcile the uncommitted ARCHITECTURE.md edits on main" needed no work: the working tree is clean and ARCHITECTURE.md is committed on main.

• 2026-08-07: Updated golangci-lint to v2.12.2 everywhere it is pinned (Dockerfile lint stage, Makefile deps target), replaced .golangci.yml with the canonical config (v2 schema, default: all), and remediated the bulk of the lint findings it surfaced (issue #61): behavior-preserving fixes across every package, 2,990 findings down to 80. make test and make fmt-check were green at that point but make lint was still red; the commit message claiming make check was green was wrong.

• 2026-08-07: Added the standard .golangci.yml and .editorconfig (issue #59); lint findings under the new config are tracked in issue #61. script/bootstrap now installs sqlite3 (needed by tests).

• 2026-07-07 Adopted scripts-to-rule-them-all: script/ entrypoints, Makefile shims, README Entrypoints section

• 2026-07-02: Consolidated CLI verbs, retired overlapping commands; bound the local index to its backup destination URL.

• 2026-06-28: snapshot rm now removes metadata only and prints the prune command; restore skips chown when running as non-root.

• 2026-06-26: Snapshot IDs hashed at the storage boundary; snapshot list made resilient to bad remote entries.

• 2026-06-24: Collapsed snapshot prune into vaultik prune; restore streams blobs to disk and restores files in blob-locality order; cron output fixes.

• 2026-06-17: Restore overhaul: ReadAt chunk reads from cached blobs, reference-counted blob sweeper, integration tests; new internal/ui output layer, banner, and progress lines.

• 2025-12-18: Added ARCHITECTURE.md and godoc coverage for exported API.

• 2025-07-26: End-to-end integration tests; manifest format refactor; renamed backup to snapshot; afero filesystem abstraction.

• 2025-07-20: Initial design and implementation: cobra + fx CLI skeleton, SQLite index database, UUID blob storage with streaming chunking.

Future Steps

None queued; the release-scoping item is now the Next Step.