version, info, remote info, config, and database delete wrote plain text straight to stdout, so they were unstyled and ignored --quiet. Output is now governed by internal/ui in two buckets. Status lines and confirmations (config init, config set, the database delete prompt) go through the ui message methods and are silenced by --quiet. The data a command exists to produce is written plain -- the version/info/remote-info reports, the snapshot list table, config get values, and the --json documents -- and is never suppressed, since a script depends on it and a marker would corrupt a table or document. The database delete confirmation prompt is always shown. Pure-cli commands reach ui through a small commandUI helper.
Model: opus-4-8
On SIGINT/SIGTERM the process could exit before the interrupted command cleanup defers ran, leaving decrypted data in the temp directory (the blob cache and the decrypted snapshot database).
RunApp now mirrors fx run sequence: start, block on app.Wait(), then app.Stop(), returning only after Stop completes. fx delivers both an OS interrupt and the finished operation Shutdowner.Shutdown() on one channel. Stop runs the OnStop hooks; the operation hook cancels the command and waits for its goroutine to return (bounded by shutdownTimeout) before exit. The old code returned as soon as app.Done fired, without Stop, so a real interrupt unwound to os.Exit while cleanup still ran. Restore loops check the context between chunks and blobs so the wait ends promptly. A cli test drives RunApp through the OnStop hook.
Model: opus-4-8
Production encryption and decryption already run through blobgen; the crypto package (Encryptor, Decryptor, UpdateRecipients, the fx Module) and Vaultik.GetEncryptor/GetDecryptor had no production caller. Delete crypto and route verify --deep through the same blobgen reader restore uses, parsing the age key once.
The second blob-ID hash step is now one exported blobgen.DoubleSHA256; Writer.Sum256 (the double hash) becomes Writer.ContentID so it no longer collides with Reader.Sum256 (the single plaintext hash). Also delete the never-adopted internal/types newtypes and the uncalled CleanupIncompleteSnapshots and its now-dead deleteSnapshot caller, and correct ARCHITECTURE.md. No production behavior changes.
Model: opus-4-8
The banner is printed to stdout before cobra parses, and
bannerSuppressedInArgs recognised only --quiet, -q and --cron. So every
--json document was preceded by two banner lines and a blank one, and
`vaultik snapshot list --json | jq` failed. Passing opts.JSON as
extraQuiet did not help: that calls UI.SetQuiet in an fx OnStart hook,
long after Entry has printed.
The raw-argv scan is extended rather than the banner moved after
parsing. root.go documents that the banner must survive cobra rejecting
its arguments and --help, and no single post-parse location covers those
paths. The subcommand-versus-persistent distinction does not decide it:
--cron is already in the suppression list and is itself subcommand-only,
existing on snapshot create alone, so this adds another instance of an
accepted imprecision rather than a new kind. The error directions are
asymmetric - a false positive loses a decorative banner, a false negative
corrupts a document - so the scan errs toward suppression, which is also
why --json=false suppresses, exactly as --quiet=false already does.
Four of the five --json commands now pipe into jq cleanly with no other
flags: snapshot list, snapshot verify, snapshot remove, remote info.
prune does not, because pruneLocalSnapshots writes three prose lines to
stdout with no --json awareness. That reproduces identically before this
change and -q never suppressed it either, since printlnStdout and
stdoutf bypass v.UI entirely. Tracked as #108.
Also fixed: TTYHandler's human-readable byte formatting did not survive
grouping, because the key check compared against the bare attribute name
and a grouped record presents it qualified. AGENTS.md policy 9 keyed the
log format on stdout's TTY-ness, which #82 made false by moving the
logger to stderr; it now names the log stream. Vaultik.Stderr keeps its
field with the comment amended to say outright that nothing writes to
it, and the dead listEnv.stderr is removed.
Closes#97.
internal/log attached both handlers to os.Stdout, so any record that was
not suppressed landed in the middle of a --json document. WARN and ERROR
are never suppressed, so this was not hypothetical: a config file with
permissions looser than 0600 was enough to break
`vaultik snapshot list --json | jq`.
Both handlers now write to os.Stderr, and the TTY-vs-JSON format choice
tests os.Stderr rather than os.Stdout - the format has to follow the
stream the records land on, or a redirected stderr gets colorized
whenever stdout happens to be a terminal.
User-visible: --verbose and --debug output moves to stderr too, so
`vaultik snapshot list -v > out.txt` no longer captures diagnostics.
--quiet and --cron semantics are unchanged.
TTYHandler.WithAttrs and WithGroup discarded their arguments and returned
the receiver, while their doc comments claimed otherwise, so attributes
passed through the exported log.With vanished. The effect was
environment-dependent in the worst direction: handler choice is by
TTY-ness, so attributes disappeared on a terminal - where a developer is
debugging - and appeared correctly in CI. Both now return a new handler
with copied state rather than mutating the receiver, since slog permits a
handler to be shared and derived from concurrently. A test asserts the
TTY and JSON handlers emit the same attribute set, which is the test that
would have caught the original defect.
The local workaround in snapshot_list.go is removed now that the logger
no longer writes to stdout. The collect-then-emit machinery is kept, but
for a different reason than it was added: emitting from the fetch workers
would order warnings by network timing, whereas key-order emission after
group.Wait() is deterministic run to run.
Not yet complete: --json stdout still carries the startup banner, which
internal/cli/entry.go writes before cobra parses and which
bannerSuppressedInArgs does not recognise --json for. That is the
remaining stdout contamination path and is tracked in #106.
The Vaultik.UI doc comment claimed the cli layer replaces the writer with
a discarding writer in --cron mode. It does not. UI is built once as
ui.New(os.Stdout) and never reassigned; internal/cli/app.go calls
UI.SetQuiet(true) when --cron or --quiet is set, which drops Begin,
Complete, Info, Notice, Detail, Progress and Banner - but Warningf and
Errorf have no quiet check and are still emitted.
That distinction matters: the end-of-run summary is deliberately routed
through UI.Warningf so cron delivers something, so a reader who believed
the comment would have concluded the opposite of how the code is meant to
work.
The README's --cron description carried the same imprecision ("Silent
unless error") and is corrected alongside it.
Comment and documentation only - the Go diff contains no non-comment
lines, so there is no behavior change.
Updates golangci-lint to v2.12.2 everywhere it is pinned and installs the canonical `.golangci.yml`, then remediates every finding the new linter/config surfaces so `make check` is green.
## Version bump
- `Dockerfile` lint stage: `golangci/golangci-lint:v2.11.3-alpine` -> `v2.12.2-alpine` (digest-pinned, date comment updated)
- `Makefile` `deps` target: `go install` moved from the old v1 module path at `@latest` to the pinned `github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2`
- `.golangci.yml` replaced with the canonical config (v2 schema; settings under `linters.settings` so the thresholds actually apply; `default: all` with the standard six disables)
- `script/bootstrap` installs golangci-lint via the system package manager and carries no version pin, so it is unchanged
- CI (`.gitea/workflows/check.yml`) only runs `script/cibuild`, so it needed no change
## Lint remediation
The canonical config surfaced ~3,300 findings across 56k lines. All are fixed, behavior-preserving; incorporates and supersedes the per-package mechanical passes already merged to `main` (refs #61). Highlights:
- `err113`: dynamic errors replaced with package sentinels + `%w` wrapping; comparisons via `errors.Is`
- `goprintffuncname`: printf-style helpers renamed with an `f` suffix (`ui.Writer` message methods, `cli.ReportErrorf`, `database.Fatalf`) and all call sites updated
- `revive` stutter renames: `blob.Handler`, `blob.WithReader`, `blob.ChunkPosition`, `storage.URL`, `storage.Info`; missing doc comments added
- `contextcheck`/`noctx`: `context.Context` threaded through `blob.Packer` and the scanner call sites; context-aware `exec`/`sql` variants
- `funlen`/`cyclop`/`gocognit`/`dupl`: oversized and duplicated functions split into focused helpers (production and test code)
- tests: `t.Parallel()` added where safe (global logger init kept in the serial phase for `-race`), `t.TempDir()`/`t.Helper()` adopted, several suites converted to external test packages
- `gosec`: bounded integer conversions, `ReadHeaderTimeout` on the test HTTP server; remaining warnings suppressed per-site with justifications
- remaining `nolint` directives are rare, targeted, and each carries a reason (e.g. `nilnil` not-found contract in the repository layer, fx module globals, on-disk snake_case struct tags)
- removed the deprecated `log.LogOptions` alias (callers migrated to `log.Options`)
`make check` (tests with `-race`, lint, fmt-check) passes.
Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #62
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Captures three behaviors the restore hot path must exhibit but
currently doesn't, all under one test:
* Peak blob disk cache occupancy ≤ 1. Smart restore ordering should
drain every file referencing the currently-cached blob before
downloading the next one, so the sweeper can free each blob the
moment its file set is exhausted.
* Every remote blob fetched exactly once (counter on a wrapping
Storer). Already true today; the test pins it so neither future
cache-eviction nor reorder regressions can introduce
re-downloads.
* blobDiskCache.Get is never called during restore — chunk
extraction must go through ReadAt so we never read the whole
blob from disk to slice out a few KB. The 10 GB
photo-snapshot --debug output showed ~900 ms per cache-hit chunk
extract; ReadAt should bring that to sub-millisecond.
Adds Get/ReadAt call counters and a peak-Len tracker to
blobDiskCache, plus an internal restoreCacheObserver hook on Vaultik
so the test can capture the production cache instance without
exporting unexported types.
Currently fails with peak_len=3, get_calls=24, readat_calls=0. The
fix follows in subsequent commits.
Restore and verify no longer use schollz/progressbar. Instead they emit
a periodic status line every 15 s via ui.Writer.Progress, matching the
cadence and shape of the snapshot create scanner output. The lines
include files done, byte counts, throughput in bits/sec, elapsed,
absolute ETA, and remaining duration — same conventions as snapshot
create. The progressbar dependency, the newProgressBar/isTerminal
helpers, and the unused printfStderr helper are removed; go.mod loses
schollz/progressbar plus its colorstring and uniseg transitive deps.
Adds --debug timing instrumentation throughout the restore hot path so
the next slow-restore report can pinpoint which stage is the
bottleneck. Per-file: file-chunks query, output Create, per-chunk blob
DB lookups, cache get/put, blob download, chunk write, sweeper call.
Per-blob-download: fetch-setup (Get + Stat) vs read+decrypt+decompress
vs close-and-verify. FetchBlob splits the Storage.Get and Storage.Stat
round-trips so an expensive size-stat is visible separately.
All user-facing output now goes through a single ui.Writer with a
uniform style:
》 (white) for begin / info / notice
》 (green) for complete / success
Warning: for warnings (orange)
ERROR: for errors (red)
》 (indented) for progress heartbeats
Color is enabled when stdout is a TTY and NO_COLOR is unset.
Standards:
- Complete-sentence messages with fully qualified terms ("backup
destination store", "local index database", "snapshot source
files enumeration").
- Every Complete has a matching Begin.
- Natural verb tense conveys state ("Uploading" -> "Uploaded"). The
words "begin"/"complete" never appear in message bodies; the marker
color carries that information.
- ETA means clock time, not duration. Progress lines say "estimated
remaining time (<dur>), finish at <time>" with both labeled.
Adds globals.CommitDate (populated by Makefile/Dockerfile/goreleaser
via ldflags from `git show -s --format=%cI HEAD`) and a startup banner
printed once per invocation.
Strips fx call-chain noise from startup errors so users see the actual
underlying error (e.g. "creating base path: mkdir /Volumes/BACKUPS:
permission denied" instead of three layers of "could not build
arguments for function ...").
README documents the output style and the ui package conventions.
Module path changed from git.eeqj.de/sneak/vaultik to
sneak.berlin/go/vaultik (vanity redirect). All imports, ldflags,
Dockerfile, goreleaser config, and docs updated. App data/config
directories now use plain "vaultik" instead of the reverse-DNS name.
README:
- New copy-pasteable quickstart at top: go install, config init,
age keypair, config set for key + file:// destination, home backup
- All command names in command details are code-quoted
- config set/get gained sequence index support (age_recipients.0)
so lists are settable from the CLI
- Dockerfile build is CGO_ENABLED=0 to match the pure-Go build
- Replace bare fmt.Scanln with v.scanStdin() helper in snapshot.go
- Remove duplicate FetchBlob from vaultik.go (canonical version in blob_fetch_stub.go)
- Remove duplicate FetchAndDecryptBlob from restore.go (canonical version in blob_fetch_stub.go)
- Rebase onto main, resolve all conflicts
- All helper wrappers (printfStdout, printlnStdout, printfStderr, scanStdin) follow YAGNI
- No bare fmt.Print*/fmt.Scan* calls remain outside helpers
- make test passes: lint clean, all tests pass
Address all four review concerns on PR #31:
1. Fix missed bare fmt.Println() in VerifySnapshotWithOptions (line 620)
2. Replace all direct fmt.Fprintf(v.Stdout,...) / fmt.Fprintln(v.Stdout,...) /
fmt.Fscanln(v.Stdin,...) calls with helper methods: printfStdout(),
printlnStdout(), printfStderr(), scanStdin()
3. Route progress bar and stderr output through v.Stderr instead of os.Stderr
in restore.go (concern #4: v.Stderr now actually used)
4. Rename exported Outputf to unexported printfStdout (YAGNI: only helpers
actually used are created)
These methods were referenced in main but never defined, causing compilation
failures. They were introduced by merges that assumed dependent PRs were
already merged.
- Implement deterministic blob hashing using double SHA256 of uncompressed
plaintext data, enabling deduplication even after local DB is cleared
- Add Stat() check before blob upload to skip existing blobs in storage
- Add rclone storage backend for additional remote storage options
- Add 'vaultik database purge' command to erase local state DB
- Add 'vaultik remote check' command to verify remote connectivity
- Show configured snapshots in 'vaultik snapshot list' output
- Skip macOS resource fork files (._*) when listing remote snapshots
- Use multi-threaded zstd compression (CPUs - 2 threads)
- Add writer tests for double hashing behavior
- Implement exclude patterns with anchored pattern support:
- Patterns starting with / only match from root of source dir
- Unanchored patterns match anywhere in path
- Support for glob patterns (*.log, .*, **/*.pack)
- Directory patterns skip entire subtrees
- Add gobwas/glob dependency for pattern matching
- Add 16 comprehensive tests for exclude functionality
- Add snapshot prune command to clean orphaned data:
- Removes incomplete snapshots from database
- Cleans orphaned files, chunks, and blobs
- Runs automatically at backup start for consistency
- Add snapshot remove command for deleting snapshots
- Add VAULTIK_AGE_SECRET_KEY environment variable support
- Fix duplicate fx module provider in restore command
- Change snapshot ID format to hostname_YYYY-MM-DDTHH:MM:SSZ
Storage backend:
- Add internal/storage package with Storer interface
- Implement FileStorer for local filesystem storage (file:// URLs)
- Implement S3Storer wrapping existing s3.Client
- Support storage_url config field (s3:// or file://)
- Migrate all consumers to use storage.Storer interface
PID locking:
- Add internal/pidlock package to prevent concurrent instances
- Acquire lock before app start, release on exit
- Detect stale locks from crashed processes
Scan progress improvements:
- Add fast file enumeration pass before stat() phase
- Use enumerated set for deletion detection (no extra filesystem access)
- Show progress with percentage, files/sec, elapsed time, and ETA
- Change "changed" to "changed/new" for clarity
Config improvements:
- Add tilde expansion for paths (~/)
- Use xdg library for platform-specific default index path
- Create comprehensive integration tests with mock S3 client
- Add in-memory filesystem and SQLite database support for testing
- Test full backup workflow including chunking, packing, and uploading
- Add test to verify encrypted blob content
- Fix scanner to use afero filesystem for temp file cleanup
- Demonstrate successful backup and verification with mock dependencies
clawbot
clawbot