internal/storage had no tests. This adds table-driven ParseStorageURL
coverage and a shared Storer conformance suite, defined once in
conformance_test.go and run against every backend that can run
in-process: file:// over a temp dir and s3:// over the same in-memory
S3 harness (gofakes3 + s3mem) that internal/s3 and the not-found test
use, so a new backend inherits the contract by passing its constructor.
The rclone backend gets construction and argument-shaping tests via
rclone's in-process ":local" backend, plus the missing-remote error
path; a comment records that its data-plane operations need a configured
remote and so cannot run in a unit test.
Tests only; no production code changed and no defect surfaced.
Model: opus-4-8
Prune read table row counts before and after to report how many orphaned files, chunks and blobs it removed, and discarded the error from every read. A failed query therefore reported as a count of 0, and the summary showed plausible wrong numbers.
A count that cannot be read is now logged as a warning (on stderr, also under --json) and shown as "unknown"; a difference computed from an unknown count is itself unknown. 0 still means the table was empty. No --json document carries these counts, so none can show a false 0.
model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
The file:// backend streamed each object straight to its final key, so an upload cut off mid-stream left a truncated object there. The next backup saw that Stat succeeded, recorded the blob as complete, and produced a snapshot that reported success but could not be restored.
Writes now go to a temporary file with a .partial suffix in the destination directory, are synced, then renamed onto the key. List and ListStream skip .partial files, so a leftover is never trusted as a blob and is overwritten when the key is written again. S3 PutObject is already atomic.
Disclosure: the containing directory is not synced after the rename, so a host crash right after it could still lose the object on some filesystems.
model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
The Storer interface documents that Get and Stat return storage.ErrNotFound for a missing object. The file and rclone backends did; the s3 backend returned the raw SDK error, so callers testing for ErrNotFound behaved differently on s3.
S3Storer.Get and Stat now wrap ErrNotFound when the SDK reports a missing object and leave every other error untouched. The SDK reports a missing key two ways (NoSuchKey from Get, NotFound from Head); both are recognised in one helper, s3.IsNotFound, which HeadObject now also uses. The mapping lives in the storage package because internal/s3 cannot import it.
model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
Two parseDuration functions existed with different grammars; only the one in internal/vaultik/helpers.go was reachable from a flag. The unused copy in internal/cli/duration.go is deleted, so no flag accepts anything it did not before.
The README gave 6m as the six-months example for snapshot purge --older-than, but m is minutes: that command removed every snapshot older than six minutes. The example is now 6mo, and the help for --older-than and --keep-newer-than states that m is minutes and mo is months.
The parser now rejects negative durations, which it used to accept or silently make positive.
model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
Co-authored-by: clawbot <clawbot@noreply.example.org>
The last step of verify --deep hashed the encrypted bytes it downloaded once with SHA256 and compared the result to the blob name. The name is the double SHA256 of the blob plaintext, so the two could never match and deep verification failed on every healthy blob with "blob hash mismatch".
It now hashes the decompressed plaintext as chunk verification streams it and compares the double SHA256 of that to the blob name, the same derivation the writer uses. A new test backs up a real snapshot, runs deep verify on it, then flips one byte in a stored blob and expects failure.
model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
snapshot create compacted the metadata database by running a sqlite3 command-line binary, after every blob had already been uploaded. On a host without that binary, which includes anyone who installed with go install, the backup failed at the last step, and two tests failed the same way.
VACUUM now runs through the Go sqlite driver the program already uses, and its error is returned to the caller. The runtime Docker image no longer installs the sqlite package, since nothing in the binary calls it.
model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
Removes code and fixtures nothing uses: the internal/models package and its test, a second SnapshotInfo type in package cli that had no references (the live one is in internal/vaultik), and two config fixtures, test-config.yml and test/integration-config.yml, whose keys the config loader no longer accepts. test/config.yaml stays; a test uses it.
Also fixes config set, which rewrote the whole file with 4-space indentation on the first set despite the documented promise to preserve formatting. It now encodes with 2-space indent like the default template, and a test asserts comments and indentation survive a set.
Deviation: one commit, not one per deletion as the issue asked.
model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
Three documents showed the remote layout with a plaintext snapshot ID as the metadata directory name, and docs/REPOSTRUCTURE.md blamed those IDs for the observable backup time. The store actually names each metadata directory with a one-way hash of the ID, so hostname and snapshot name are not visible; the backup time is, through the plaintext timestamp in the manifest, which is accepted behaviour.
README, ARCHITECTURE.md, docs/DATAMODEL.md and docs/REPOSTRUCTURE.md now show the hashed layout, the derivation is documented once, and the privacy section lists what the unencrypted manifest exposes. Two code comments that claimed the timestamp was hidden are corrected. No behaviour change.
Judgement call: docs/DATAMODEL.md was not named in the issue but had the same error.
model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
`--cron` sets the UI quiet, but `Warningf` and `Errorf` are unconditional, and the snapshot summary is routed through `Warningf` on purpose so cron delivers something on a successful run. The help string said `silent unless error`, so a user could read normal cron output as a failure. It now says `silent unless warning or error`, matching the README. String only; no behavior change.
Model: opus-4-8 (implementation and review)
model: claude-fable-5
Closes#110.
CleanupLocalSnapshots wrote three prose lines to stdout with no --json
awareness, covering every branch, so `vaultik prune --json | jq` failed
on any input. -q never helped either: printlnStdout and stdoutf write
straight to v.Stdout and never consult v.UI, which is what SetQuiet
affects. It now takes *PruneOptions, symmetric with its sibling phase
PruneBlobs, and gates all three writes.
Threading opts.JSON was chosen over moving the lines to log.Info,
because internal/log/log.go defaults the level to Warn: log.Info would
not have relocated them to stderr, it would have deleted them from a
plain `vaultik prune`, and "Removing stale local record" narrates the
deletion of local index rows. The stale-record count is deliberately not
added to PruneBlobsResult - every field there is blob-scoped and produced
by the phase that runs after this reconciliation, so adding it would
change a published --json schema as a side effect of a stream fix.
Note for anyone reading the --json contract: under --json the
stale-record removal now produces no signal in either stream. stdout is
correctly gated, stderr is level-pinned to Warn because --json sets
Quiet, and the count is not in the document. That is inherited behaviour
- PruneBlobs' own log.Info calls are equally invisible under --json - not
something this change introduced, and it is tracked separately.
make build exited 0 and produced nothing: .PHONY listed build with no
build: rule, and a phony target with no prerequisites and no recipe is
considered already satisfied, which turns what would be a hard error into
a silent success. In a repo where `make build` is the documented way to
build, a caller checking the exit code concluded the build worked. Now
`build: vaultik`, verified in both directions - a clean build produces
the binary, a deliberately broken one exits non-zero and produces none.
All 19 .PHONY names were audited; build was the only one lacking a rule.
TestPhonyTargetsAllHaveRules keeps that true for names added later, so
the class is closed rather than the instance.
The banner is printed to stdout before cobra parses, and
bannerSuppressedInArgs recognised only --quiet, -q and --cron. So every
--json document was preceded by two banner lines and a blank one, and
`vaultik snapshot list --json | jq` failed. Passing opts.JSON as
extraQuiet did not help: that calls UI.SetQuiet in an fx OnStart hook,
long after Entry has printed.
The raw-argv scan is extended rather than the banner moved after
parsing. root.go documents that the banner must survive cobra rejecting
its arguments and --help, and no single post-parse location covers those
paths. The subcommand-versus-persistent distinction does not decide it:
--cron is already in the suppression list and is itself subcommand-only,
existing on snapshot create alone, so this adds another instance of an
accepted imprecision rather than a new kind. The error directions are
asymmetric - a false positive loses a decorative banner, a false negative
corrupts a document - so the scan errs toward suppression, which is also
why --json=false suppresses, exactly as --quiet=false already does.
Four of the five --json commands now pipe into jq cleanly with no other
flags: snapshot list, snapshot verify, snapshot remove, remote info.
prune does not, because pruneLocalSnapshots writes three prose lines to
stdout with no --json awareness. That reproduces identically before this
change and -q never suppressed it either, since printlnStdout and
stdoutf bypass v.UI entirely. Tracked as #108.
Also fixed: TTYHandler's human-readable byte formatting did not survive
grouping, because the key check compared against the bare attribute name
and a grouped record presents it qualified. AGENTS.md policy 9 keyed the
log format on stdout's TTY-ness, which #82 made false by moving the
logger to stderr; it now names the log stream. Vaultik.Stderr keeps its
field with the comment amended to say outright that nothing writes to
it, and the dead listEnv.stderr is removed.
Closes#97.
internal/log attached both handlers to os.Stdout, so any record that was
not suppressed landed in the middle of a --json document. WARN and ERROR
are never suppressed, so this was not hypothetical: a config file with
permissions looser than 0600 was enough to break
`vaultik snapshot list --json | jq`.
Both handlers now write to os.Stderr, and the TTY-vs-JSON format choice
tests os.Stderr rather than os.Stdout - the format has to follow the
stream the records land on, or a redirected stderr gets colorized
whenever stdout happens to be a terminal.
User-visible: --verbose and --debug output moves to stderr too, so
`vaultik snapshot list -v > out.txt` no longer captures diagnostics.
--quiet and --cron semantics are unchanged.
TTYHandler.WithAttrs and WithGroup discarded their arguments and returned
the receiver, while their doc comments claimed otherwise, so attributes
passed through the exported log.With vanished. The effect was
environment-dependent in the worst direction: handler choice is by
TTY-ness, so attributes disappeared on a terminal - where a developer is
debugging - and appeared correctly in CI. Both now return a new handler
with copied state rather than mutating the receiver, since slog permits a
handler to be shared and derived from concurrently. A test asserts the
TTY and JSON handlers emit the same attribute set, which is the test that
would have caught the original defect.
The local workaround in snapshot_list.go is removed now that the logger
no longer writes to stdout. The collect-then-emit machinery is kept, but
for a different reason than it was added: emitting from the fetch workers
would order warnings by network timing, whereas key-order emission after
group.Wait() is deterministic run to run.
Not yet complete: --json stdout still carries the startup banner, which
internal/cli/entry.go writes before cobra parses and which
bannerSuppressedInArgs does not recognise --json for. That is the
remaining stdout contamination path and is tracked in #106.
No tag could be cut at all: .goreleaser.yaml had no gitea_urls block, so
goreleaser defaulted to the GitHub API, and the repo has zero tags.
.goreleaser.yaml now points at git.eeqj.de. Version derives from git via
a new script/version - exact tag with any leading v stripped, else
dev-<12-char sha>, with a -dirty suffix when tracked files are modified -
replacing the hardcoded 1.0.0-rc.1 that every local build was stamping
regardless of git state. A tag-triggered .gitea/workflows/release.yml
runs goreleaser with a scoped token (RELEASE_TOKEN); script/bootstrap
installs a sha256-verified goreleaser, and make release / release-snapshot
become script shims like every other target.
Two fabrications were removed rather than merely replaced. goreleaser's
snapshot.version_template was `{{ incpatch .Version }}-next`, which
invents a release number from the last tag - and with no tags, from
goreleaser's own fabricated v0.0.0. And internal/cli/version.go gated its
development-build notice on Version == "dev" exactly, so the moment
untagged builds carried a sha that notice would have gone silent and an
unreleased binary would have read as a release. Replaced with a tested
IsDevVersion predicate, and closed at both layers: the Makefile now
refuses to build when script/version yields nothing, and an empty version
counts as a development build - reachable today via
`docker build --build-arg VERSION=`.
The release workflow installs Go from a sha-pinned actions/setup-go
(v5.6.0) using go-version-file, so the compiler that produces released
binaries is pinned like every other external reference. Without it the
first tag push would either fail at goreleaser's before-hook or compile
the published artifacts with whatever unpinned Go the runner happened to
carry - the one unpinned thing in a release path that already refuses an
unpinned goreleaser.
Known gap: the Go tarball setup-go fetches is version-pinned but not
checksum-verified against a value in this repo, unlike the goreleaser
install and the Dockerfile digest.
The Vaultik.UI doc comment claimed the cli layer replaces the writer with
a discarding writer in --cron mode. It does not. UI is built once as
ui.New(os.Stdout) and never reassigned; internal/cli/app.go calls
UI.SetQuiet(true) when --cron or --quiet is set, which drops Begin,
Complete, Info, Notice, Detail, Progress and Banner - but Warningf and
Errorf have no quiet check and are still emitted.
That distinction matters: the end-of-run summary is deliberately routed
through UI.Warningf so cron delivers something, so a reader who believed
the comment would have concluded the opposite of how the code is meant to
work.
The README's --cron description carried the same imprecision ("Silent
unless error") and is corrected alongside it.
Comment and documentation only - the Go diff contains no non-comment
lines, so there is no behavior change.
ListSnapshots built its table entirely from the local SQLite index. The
only remote access, reportRemoteDrift, was gated on AgeSecretKey != "",
so on a correctly configured host - which by design holds no private key
- snapshot list never contacted the destination store at all. A user who
lost their local index could not see their own backups, and the
"<remote only>" cell the README documents was unreachable dead code.
The listing is now the union of the local index and the destination
store, with no age_secret_key gate. Remote-only snapshots cannot have
their hostname or name recovered - RemoteSnapshotKey is one-way and the
manifest stores the hash - so they are listed by abbreviated remote key
with the real timestamp and compressed size from the manifest, and
"<remote only>" in the two columns that require the local index. Nothing
new is written to remote storage and the human ID is never fabricated.
An unreachable destination degrades to local-only with a warning and a
zero exit code. remote_present is null rather than false in that case,
so "absent" and "unknown" stay distinguishable and no drift is claimed
from a listing that never happened.
Also:
- Snapshot timestamps are normalised to UTC in scanSnapshotRows, the
single point where they enter the domain. Previously one of three
scanners omitted .UTC(), so on a non-UTC host the same snapshot
rendered a different time depending on whether it was locally tracked.
- The 1000-row cap and the unreadable-manifest count are reported in
--json mode as well as table mode, so machine consumers cannot be
silently truncated. The JSON shape is unchanged.
- Warnings raised while listing are routed to stderr rather than the
logger, which writes to stdout and would corrupt the JSON document.
This is a local workaround for the logger bug tracked in #82 and
should be removed when that lands.
- downloadManifestByKey is now the only remote manifest reader, so the
manifest privacy question in #81 has a single call site to change.
- The orphaned "vaultik snapshot cleanup" hint now names vaultik prune;
that command was folded into prune by the 2026-07-02 consolidation.
Clears the final 80 golangci-lint findings under the canonical
.golangci.yml (sha256 021cc83f4e6fc7c31b95b34b846723dfcf20b66b7baeea1dc40406e643346bcb),
taking the repo from red to green: script/cibuild exits 0.
- wsl_v5 (60): blank line above defer/go statements sharing no variable
with the line above; blank-line-only diff.
- sqlclosecheck (10): the package-local CloseRows helper hid the close
from the analyzer. Helper removed; all 18 call sites now defer an
inline rows.Close(), preserving the fatal-on-close-error path. No
resource leak existed - the rows were always being closed.
- prealloc (3): append targets given a starting capacity.
- revive (3): package-name findings suppressed with per-site directives
pending the naming decision tracked in #76.
No gosec suppressions are needed under the pinned linter. .golangci.yml,
Dockerfile, Makefile, .gitea/ and script/ are byte-identical to main.
Verified with script/cibuild (digest-pinned golangci-lint v2.12.2), not
make check - the latter resolves the linter from PATH and is not a
trustworthy gate here; see #78.
Closes#59.
Updates golangci-lint to v2.12.2 everywhere it is pinned and installs the canonical `.golangci.yml`, then remediates every finding the new linter/config surfaces so `make check` is green.
## Version bump
- `Dockerfile` lint stage: `golangci/golangci-lint:v2.11.3-alpine` -> `v2.12.2-alpine` (digest-pinned, date comment updated)
- `Makefile` `deps` target: `go install` moved from the old v1 module path at `@latest` to the pinned `github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2`
- `.golangci.yml` replaced with the canonical config (v2 schema; settings under `linters.settings` so the thresholds actually apply; `default: all` with the standard six disables)
- `script/bootstrap` installs golangci-lint via the system package manager and carries no version pin, so it is unchanged
- CI (`.gitea/workflows/check.yml`) only runs `script/cibuild`, so it needed no change
## Lint remediation
The canonical config surfaced ~3,300 findings across 56k lines. All are fixed, behavior-preserving; incorporates and supersedes the per-package mechanical passes already merged to `main` (refs #61). Highlights:
- `err113`: dynamic errors replaced with package sentinels + `%w` wrapping; comparisons via `errors.Is`
- `goprintffuncname`: printf-style helpers renamed with an `f` suffix (`ui.Writer` message methods, `cli.ReportErrorf`, `database.Fatalf`) and all call sites updated
- `revive` stutter renames: `blob.Handler`, `blob.WithReader`, `blob.ChunkPosition`, `storage.URL`, `storage.Info`; missing doc comments added
- `contextcheck`/`noctx`: `context.Context` threaded through `blob.Packer` and the scanner call sites; context-aware `exec`/`sql` variants
- `funlen`/`cyclop`/`gocognit`/`dupl`: oversized and duplicated functions split into focused helpers (production and test code)
- tests: `t.Parallel()` added where safe (global logger init kept in the serial phase for `-race`), `t.TempDir()`/`t.Helper()` adopted, several suites converted to external test packages
- `gosec`: bounded integer conversions, `ReadHeaderTimeout` on the test HTTP server; remaining warnings suppressed per-site with justifications
- remaining `nolint` directives are rare, targeted, and each carries a reason (e.g. `nilnil` not-found contract in the repository layer, fx module globals, on-disk snake_case struct tags)
- removed the deprecated `log.LogOptions` alias (callers migrated to `log.Options`)
`make check` (tests with `-race`, lint, fmt-check) passes.
Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #62
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
The verb surface accumulated overlapping cleanup commands. Consolidate
so each cleanup verb has one meaning:
- Rename 'database purge' -> 'database delete'. The command removes the
SQLite file entirely; "purge" wrongly suggested purging contents.
- Fold 'snapshot cleanup' into 'prune'. Prune now runs three passes:
reconcile local snapshots against the remote (previously the
standalone cleanup command), drop orphaned local rows, then delete
unreferenced remote blobs. One command, one mental model.
- Delete 'store info'. Its output was a strict subset of 'remote info',
which already prints storage type + location. Any user reaching for
either should reach for 'remote info'.
- Drop 'snapshot remove --all'. It duplicated 'remote nuke --force'.
'remote nuke' is the single supported entry point for wiping the
destination store.
Also update the storage-binding error message to reference the new
'vaultik database delete' name.
The local index tracks which chunks and blobs already exist at the
backup destination. Nothing was recording *which* destination, so
changing storage_url and running a backup left the scanner treating
every already-seen chunk as still-present at the new (empty) location.
Uploads were skipped silently and the resulting snapshots pointed at
blobs that don't exist at the new destination.
Fix: record storage_url in a new local_meta key-value table on first
mutating command, and refuse to proceed when the configured URL later
differs from the stored one. The error explains the two recovery
paths (revert the config, or run 'vaultik database purge' to discard
the index and rebuild from a fresh full backup).
Wired into snapshot create / prune / snapshot remove / snapshot purge
/ snapshot cleanup. Read-only inspection commands (snapshot list,
remote info, store info) are exempt.
chown(2) requires root on every Unix-ish kernel. Restoring 39k files
as a non-root user produces 39k EPERM syscalls plus 39k matching debug
log lines, all for an operation that can't possibly succeed. Skip the
syscall entirely when euid != 0, and emit one warning at the end of the
restore so the user knows the on-disk UID/GID will reflect the running
user rather than the original owner.
The previous change had snapshot rm auto-prune unreferenced blobs. The
correct division of labor is: rm removes a snapshot (local DB + remote
metadata), prune cleans up blobs. Reverting the auto-prune means rm
stays a cheap, deterministic operation: it touches one snapshot's worth
of state and emits the exact 'vaultik prune' command the user should
run next to delete blobs no longer referenced by any remote manifest.
This is correct because prune must consult every remote manifest
(including snapshots this host doesn't know about) to determine which
blobs are still referenced, and folding that work into rm would
silently turn rm into an expensive O(remote snapshots) operation that
also assumes the remote is fully reachable.
snapshot rm <id> now does the full cleanup: removes the local index
entry, strips the snapshot's metadata from the destination store, and
prunes any blobs that were only referenced by the just-removed manifest.
The --remote flag is retired; --local-only opts out for the rare case
where the user wants to forget a snapshot locally without touching the
remote.
If the destination store is unreachable, the local-DB removal still
completes and a warning is emitted; the user can rerun 'vaultik prune'
to retry the remote half later.
RemoveAllSnapshots gets the same treatment: after deleting every
snapshot's metadata (local + remote + orphan keys), an automatic blob
prune sweep removes the now-unreferenced blob set.
Two related changes, both addressing leakage and brittleness around
the public bytes the destination store sees.
First, every remote storage path that previously embedded a human
snapshot ID (e.g. metadata/heraklion_berlin.sneak.fs.photos.2026.
catalog_2026-06-24T07:00:15Z/...) now uses the hashed remote key:
RemoteSnapshotKey(id) = hex(SHA256(SHA256("vaultik|" + id)))
Applied at:
* uploadSnapshotArtifacts (snapshot create write path)
* the manifest.json.zst snapshot_id field — manifest is
unencrypted, so the human ID would otherwise be readable to
anyone with bucket-list permission
* cleanupIncompleteSnapshots metadata-existence probe
* snapshot restore / verify (downloadSnapshotDB,
loadVerificationData)
* downloadManifestByKey, deleteRemoteSnapshotByKey
* CleanupLocalSnapshots reconciliation
* the locally-driven removal paths (RemoveSnapshot,
RemoveAllSnapshots, confirmAndExecutePurge)
The local index database keeps human IDs everywhere — the hash is a
boundary translation, not a rename. A directory listing of the
backup destination now looks like
"metadata/<64-hex>/{db.zst.age,manifest.json.zst}" with no host,
snapshot-name, or timestamp information visible.
Second, snapshot list no longer fails just because remote storage is
unreachable, and only consults the remote when the local machine can
plausibly decrypt:
* Listing is always driven by the local index database — that's
what holds the human IDs, timestamps, and per-snapshot stats
that the table actually shows.
* If no age secret key is configured, we skip remote listing
entirely (the box is treated as a write-only backup machine —
there's no value showing it remote-only keys it could never
restore).
* If a key IS configured, we try the remote listing; failures
(volume unmounted, permission denied, network error) downgrade
to a warning instead of aborting the command.
* When the remote listing succeeds, we cross-reference by hashing
each local human ID and diffing against the returned key set.
Local-only snapshots get the existing "stale local record"
cleanup hint; remote-only keys are surfaced as a single
"NOTE: N remote snapshot(s) found in backup destination store
but not in local database" line.
FileStorer construction also no longer does an eager mkdir — the
basePath is recorded and the directory is created lazily on first
write. A missing or unmounted destination during `snapshot list`
should NOT block the command, and now it doesn't.
RemoveAllSnapshots is rewritten to drive deletion from the local
index instead of from a remote listing, hashing each local ID to
find the corresponding remote key. Orphan remote keys (no matching
local snapshot) are handled separately and only deleted when
--remote is set. Existing tests are updated to hash storage paths
through the new RemoteSnapshotKey helper.
The hash format is a hard pre-1.0 break: existing remote snapshots
written under the human-ID path scheme are no longer readable; they
need to be either re-uploaded under the new scheme or manually
renamed. There is no fallback path; matching the project policy of
"no migrations pre-1.0."
--cron used to behave like full quiet mode for everything that
wasn't an error: warnings were swallowed both in the structured
log channel (LevelError gate) and at the snapshot terminus (the
"Finished (with N warnings)" line went through ui.Complete, which
is silenced under SetQuiet). A backup that, say, hit Full Disk
Access permission errors on a handful of files and skipped them
via --skip-errors would exit 0 and emit nothing — the cron job
would never page anyone.
--cron now obeys "silent only on total success":
* log.Initialize raises the cron/quiet log level from Error to
Warn so log.Warn output still reaches stdout (and therefore
cron's mail).
* The post-backup terminus message switches to ui.Warning when
WarningCount > 0. Warning is not silenced by SetQuiet, so cron
delivers the summary line whenever the count is non-zero. The
no-warnings path keeps ui.Complete, which IS silenced under
cron — that's the success path.
Separately, blob upload UI now names the actual destination
instead of the generic "backup destination store" string. The
Begin/Info lines emit the storer's reported Location (s3://bucket,
file:///mnt/usb/backup, rclone://remote/path, etc.), so anyone
watching a backup can see exactly where each blob is landing.
The CLI had two commands named "prune" doing different jobs (local
DB orphan cleanup vs. remote blob garbage collection), which was
confusing and forced a manual two-step workflow after deleting any
snapshot.
Single user-facing prune surface is now `vaultik prune`, which calls
PruneDatabase (local orphan cleanup) then PruneBlobs (remote unref
blob GC). Snapshot deletion paths (snapshot remove, snapshot remove
--all, snapshot purge) auto-run CleanupOrphanedData inline so the
local index database doesn't accumulate ghost rows after every
removal — the user observed ~39k orphaned files and 2 orphaned blobs
after a remove --all because that cleanup was previously a separate
opt-in command. `snapshot prune` is removed.
Also addresses the doc/help-string drift the user audit caught:
* cli/prune.go help text used to reference a non-existent
`vaultik purge` command.
* cli/config.go get/set short/long examples were S3-specific
(s3.bucket) when the primary storage configuration is
storage_url.
* vaultik/info.go printed S3 Bucket/Endpoint/Region labels
unconditionally; for file:// or rclone:// users those rows
were empty. The Storage Configuration block now prints the
storer's Type+Location first, the storage_url string when set,
and only emits S3 rows that are actually populated.
* vaultik/info.go's "Run 'vaultik prune --remote'" hint
referenced a flag that doesn't exist.
* vaultik/blobcache.go's doc comment claimed LRU eviction, which
is no longer the restore-time policy (the sweeper drives
eviction; LRU is the safety-net fallback when maxBytes is
finite).
* README.md listed `vaultik restore`, `vaultik snapshot prune`,
and `s3.bucket` example, all out of date.
README's roadmap section is rewritten with concrete pre-1.0 items
(security audit, error-condition tests, parallel blob downloads,
restart of interrupted restore, …) so the next-steps surface
matches what the project actually still needs.
The cleanup calls are guarded against a nil SnapshotManager so
tests that construct a bare Vaultik struct continue to work.
Three coordinated changes drop restore wall-clock by orders of
magnitude on real-world snapshots and bring memory use back under
control:
* Streaming download into the disk cache. New
blobDiskCache.PutFromReader takes an io.Reader and io.Copy's it
straight into the cache file. The old downloadBlob path did
io.ReadAll on the decrypted plaintext stream — for a 50 GB blob
that meant 50 GB in RAM before the cache write. The whole chain
(Storage.Get → age.Decrypt → zstd.NewReader → io.Copy) is now
fully streaming; peak RAM per blob is bounded by ~64 KB of
internal age/zstd buffers plus the io.Copy buffer.
* Chunk extraction via ReadAt. After a blob is on disk, restore
reads chunks via blobDiskCache.ReadAt(hash, offset, length) so
only the chunk's bytes ever touch RAM. The previous code path
called blobCache.Get for every cache-hit chunk, which read the
entire blob (e.g. 10 GB) from disk into a []byte just to slice
out a few KB — single-handedly ~900 ms per cache hit on the
user's photo snapshot.
* Locality-aware restore ordering. New restorePlan indexes
file→blob_set and blob→file_set at restore start, then drives
the loop so that every file whose full blob set is on disk is
drained before any new blob downloads. After the drain queue
empties, the planner picks the pending file with the smallest
uncached-blob count, downloads those blobs, and drains again.
A sweep is forced right before each download so the just-
completed blob is evicted before the next one is Put, keeping
peak disk-cache occupancy at 1 for path-order-adversarial
snapshots.
The restore hot path also moves onto a restoreSession struct so
restoreFile/restoreRegularFile/etc. take only the per-call file
argument instead of threading 9+ parameters through every signature.
The new BlobRepository.GetAll method lets the session build a single
blob-id → blob-hash map at start instead of doing one DB query per
chunk.
TestRestoreLocalityAndReadAt passes: peak_len ≤ 1, get_calls = 0,
readat_calls > 0, every blob fetched exactly once.
Captures three behaviors the restore hot path must exhibit but
currently doesn't, all under one test:
* Peak blob disk cache occupancy ≤ 1. Smart restore ordering should
drain every file referencing the currently-cached blob before
downloading the next one, so the sweeper can free each blob the
moment its file set is exhausted.
* Every remote blob fetched exactly once (counter on a wrapping
Storer). Already true today; the test pins it so neither future
cache-eviction nor reorder regressions can introduce
re-downloads.
* blobDiskCache.Get is never called during restore — chunk
extraction must go through ReadAt so we never read the whole
blob from disk to slice out a few KB. The 10 GB
photo-snapshot --debug output showed ~900 ms per cache-hit chunk
extract; ReadAt should bring that to sub-millisecond.
Adds Get/ReadAt call counters and a peak-Len tracker to
blobDiskCache, plus an internal restoreCacheObserver hook on Vaultik
so the test can capture the production cache instance without
exporting unexported types.
Currently fails with peak_len=3, get_calls=24, readat_calls=0. The
fix follows in subsequent commits.
sneak
clawbot