Insert the blank line wsl_v5 requires above `defer` and `go`
statements that share no variables with the statement above them.
Applied mechanically via `make lint-fix`; the diff is 60 added blank
lines and nothing else.
Updates golangci-lint to v2.12.2 everywhere it is pinned and installs the canonical `.golangci.yml`, then remediates every finding the new linter/config surfaces so `make check` is green.
## Version bump
- `Dockerfile` lint stage: `golangci/golangci-lint:v2.11.3-alpine` -> `v2.12.2-alpine` (digest-pinned, date comment updated)
- `Makefile` `deps` target: `go install` moved from the old v1 module path at `@latest` to the pinned `github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2`
- `.golangci.yml` replaced with the canonical config (v2 schema; settings under `linters.settings` so the thresholds actually apply; `default: all` with the standard six disables)
- `script/bootstrap` installs golangci-lint via the system package manager and carries no version pin, so it is unchanged
- CI (`.gitea/workflows/check.yml`) only runs `script/cibuild`, so it needed no change
## Lint remediation
The canonical config surfaced ~3,300 findings across 56k lines. All are fixed, behavior-preserving; incorporates and supersedes the per-package mechanical passes already merged to `main` (refs #61). Highlights:
- `err113`: dynamic errors replaced with package sentinels + `%w` wrapping; comparisons via `errors.Is`
- `goprintffuncname`: printf-style helpers renamed with an `f` suffix (`ui.Writer` message methods, `cli.ReportErrorf`, `database.Fatalf`) and all call sites updated
- `revive` stutter renames: `blob.Handler`, `blob.WithReader`, `blob.ChunkPosition`, `storage.URL`, `storage.Info`; missing doc comments added
- `contextcheck`/`noctx`: `context.Context` threaded through `blob.Packer` and the scanner call sites; context-aware `exec`/`sql` variants
- `funlen`/`cyclop`/`gocognit`/`dupl`: oversized and duplicated functions split into focused helpers (production and test code)
- tests: `t.Parallel()` added where safe (global logger init kept in the serial phase for `-race`), `t.TempDir()`/`t.Helper()` adopted, several suites converted to external test packages
- `gosec`: bounded integer conversions, `ReadHeaderTimeout` on the test HTTP server; remaining warnings suppressed per-site with justifications
- remaining `nolint` directives are rare, targeted, and each carries a reason (e.g. `nilnil` not-found contract in the repository layer, fx module globals, on-disk snake_case struct tags)
- removed the deprecated `log.LogOptions` alias (callers migrated to `log.Options`)
`make check` (tests with `-race`, lint, fmt-check) passes.
Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #62
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
The verb surface accumulated overlapping cleanup commands. Consolidate
so each cleanup verb has one meaning:
- Rename 'database purge' -> 'database delete'. The command removes the
SQLite file entirely; "purge" wrongly suggested purging contents.
- Fold 'snapshot cleanup' into 'prune'. Prune now runs three passes:
reconcile local snapshots against the remote (previously the
standalone cleanup command), drop orphaned local rows, then delete
unreferenced remote blobs. One command, one mental model.
- Delete 'store info'. Its output was a strict subset of 'remote info',
which already prints storage type + location. Any user reaching for
either should reach for 'remote info'.
- Drop 'snapshot remove --all'. It duplicated 'remote nuke --force'.
'remote nuke' is the single supported entry point for wiping the
destination store.
Also update the storage-binding error message to reference the new
'vaultik database delete' name.
The local index tracks which chunks and blobs already exist at the
backup destination. Nothing was recording *which* destination, so
changing storage_url and running a backup left the scanner treating
every already-seen chunk as still-present at the new (empty) location.
Uploads were skipped silently and the resulting snapshots pointed at
blobs that don't exist at the new destination.
Fix: record storage_url in a new local_meta key-value table on first
mutating command, and refuse to proceed when the configured URL later
differs from the stored one. The error explains the two recovery
paths (revert the config, or run 'vaultik database purge' to discard
the index and rebuild from a fresh full backup).
Wired into snapshot create / prune / snapshot remove / snapshot purge
/ snapshot cleanup. Read-only inspection commands (snapshot list,
remote info, store info) are exempt.
chown(2) requires root on every Unix-ish kernel. Restoring 39k files
as a non-root user produces 39k EPERM syscalls plus 39k matching debug
log lines, all for an operation that can't possibly succeed. Skip the
syscall entirely when euid != 0, and emit one warning at the end of the
restore so the user knows the on-disk UID/GID will reflect the running
user rather than the original owner.
The previous change had snapshot rm auto-prune unreferenced blobs. The
correct division of labor is: rm removes a snapshot (local DB + remote
metadata), prune cleans up blobs. Reverting the auto-prune means rm
stays a cheap, deterministic operation: it touches one snapshot's worth
of state and emits the exact 'vaultik prune' command the user should
run next to delete blobs no longer referenced by any remote manifest.
This is correct because prune must consult every remote manifest
(including snapshots this host doesn't know about) to determine which
blobs are still referenced, and folding that work into rm would
silently turn rm into an expensive O(remote snapshots) operation that
also assumes the remote is fully reachable.
snapshot rm <id> now does the full cleanup: removes the local index
entry, strips the snapshot's metadata from the destination store, and
prunes any blobs that were only referenced by the just-removed manifest.
The --remote flag is retired; --local-only opts out for the rare case
where the user wants to forget a snapshot locally without touching the
remote.
If the destination store is unreachable, the local-DB removal still
completes and a warning is emitted; the user can rerun 'vaultik prune'
to retry the remote half later.
RemoveAllSnapshots gets the same treatment: after deleting every
snapshot's metadata (local + remote + orphan keys), an automatic blob
prune sweep removes the now-unreferenced blob set.
Two related changes, both addressing leakage and brittleness around
the public bytes the destination store sees.
First, every remote storage path that previously embedded a human
snapshot ID (e.g. metadata/heraklion_berlin.sneak.fs.photos.2026.
catalog_2026-06-24T07:00:15Z/...) now uses the hashed remote key:
RemoteSnapshotKey(id) = hex(SHA256(SHA256("vaultik|" + id)))
Applied at:
* uploadSnapshotArtifacts (snapshot create write path)
* the manifest.json.zst snapshot_id field — manifest is
unencrypted, so the human ID would otherwise be readable to
anyone with bucket-list permission
* cleanupIncompleteSnapshots metadata-existence probe
* snapshot restore / verify (downloadSnapshotDB,
loadVerificationData)
* downloadManifestByKey, deleteRemoteSnapshotByKey
* CleanupLocalSnapshots reconciliation
* the locally-driven removal paths (RemoveSnapshot,
RemoveAllSnapshots, confirmAndExecutePurge)
The local index database keeps human IDs everywhere — the hash is a
boundary translation, not a rename. A directory listing of the
backup destination now looks like
"metadata/<64-hex>/{db.zst.age,manifest.json.zst}" with no host,
snapshot-name, or timestamp information visible.
Second, snapshot list no longer fails just because remote storage is
unreachable, and only consults the remote when the local machine can
plausibly decrypt:
* Listing is always driven by the local index database — that's
what holds the human IDs, timestamps, and per-snapshot stats
that the table actually shows.
* If no age secret key is configured, we skip remote listing
entirely (the box is treated as a write-only backup machine —
there's no value showing it remote-only keys it could never
restore).
* If a key IS configured, we try the remote listing; failures
(volume unmounted, permission denied, network error) downgrade
to a warning instead of aborting the command.
* When the remote listing succeeds, we cross-reference by hashing
each local human ID and diffing against the returned key set.
Local-only snapshots get the existing "stale local record"
cleanup hint; remote-only keys are surfaced as a single
"NOTE: N remote snapshot(s) found in backup destination store
but not in local database" line.
FileStorer construction also no longer does an eager mkdir — the
basePath is recorded and the directory is created lazily on first
write. A missing or unmounted destination during `snapshot list`
should NOT block the command, and now it doesn't.
RemoveAllSnapshots is rewritten to drive deletion from the local
index instead of from a remote listing, hashing each local ID to
find the corresponding remote key. Orphan remote keys (no matching
local snapshot) are handled separately and only deleted when
--remote is set. Existing tests are updated to hash storage paths
through the new RemoteSnapshotKey helper.
The hash format is a hard pre-1.0 break: existing remote snapshots
written under the human-ID path scheme are no longer readable; they
need to be either re-uploaded under the new scheme or manually
renamed. There is no fallback path; matching the project policy of
"no migrations pre-1.0."
--cron used to behave like full quiet mode for everything that
wasn't an error: warnings were swallowed both in the structured
log channel (LevelError gate) and at the snapshot terminus (the
"Finished (with N warnings)" line went through ui.Complete, which
is silenced under SetQuiet). A backup that, say, hit Full Disk
Access permission errors on a handful of files and skipped them
via --skip-errors would exit 0 and emit nothing — the cron job
would never page anyone.
--cron now obeys "silent only on total success":
* log.Initialize raises the cron/quiet log level from Error to
Warn so log.Warn output still reaches stdout (and therefore
cron's mail).
* The post-backup terminus message switches to ui.Warning when
WarningCount > 0. Warning is not silenced by SetQuiet, so cron
delivers the summary line whenever the count is non-zero. The
no-warnings path keeps ui.Complete, which IS silenced under
cron — that's the success path.
Separately, blob upload UI now names the actual destination
instead of the generic "backup destination store" string. The
Begin/Info lines emit the storer's reported Location (s3://bucket,
file:///mnt/usb/backup, rclone://remote/path, etc.), so anyone
watching a backup can see exactly where each blob is landing.
The CLI had two commands named "prune" doing different jobs (local
DB orphan cleanup vs. remote blob garbage collection), which was
confusing and forced a manual two-step workflow after deleting any
snapshot.
Single user-facing prune surface is now `vaultik prune`, which calls
PruneDatabase (local orphan cleanup) then PruneBlobs (remote unref
blob GC). Snapshot deletion paths (snapshot remove, snapshot remove
--all, snapshot purge) auto-run CleanupOrphanedData inline so the
local index database doesn't accumulate ghost rows after every
removal — the user observed ~39k orphaned files and 2 orphaned blobs
after a remove --all because that cleanup was previously a separate
opt-in command. `snapshot prune` is removed.
Also addresses the doc/help-string drift the user audit caught:
* cli/prune.go help text used to reference a non-existent
`vaultik purge` command.
* cli/config.go get/set short/long examples were S3-specific
(s3.bucket) when the primary storage configuration is
storage_url.
* vaultik/info.go printed S3 Bucket/Endpoint/Region labels
unconditionally; for file:// or rclone:// users those rows
were empty. The Storage Configuration block now prints the
storer's Type+Location first, the storage_url string when set,
and only emits S3 rows that are actually populated.
* vaultik/info.go's "Run 'vaultik prune --remote'" hint
referenced a flag that doesn't exist.
* vaultik/blobcache.go's doc comment claimed LRU eviction, which
is no longer the restore-time policy (the sweeper drives
eviction; LRU is the safety-net fallback when maxBytes is
finite).
* README.md listed `vaultik restore`, `vaultik snapshot prune`,
and `s3.bucket` example, all out of date.
README's roadmap section is rewritten with concrete pre-1.0 items
(security audit, error-condition tests, parallel blob downloads,
restart of interrupted restore, …) so the next-steps surface
matches what the project actually still needs.
The cleanup calls are guarded against a nil SnapshotManager so
tests that construct a bare Vaultik struct continue to work.
Three coordinated changes drop restore wall-clock by orders of
magnitude on real-world snapshots and bring memory use back under
control:
* Streaming download into the disk cache. New
blobDiskCache.PutFromReader takes an io.Reader and io.Copy's it
straight into the cache file. The old downloadBlob path did
io.ReadAll on the decrypted plaintext stream — for a 50 GB blob
that meant 50 GB in RAM before the cache write. The whole chain
(Storage.Get → age.Decrypt → zstd.NewReader → io.Copy) is now
fully streaming; peak RAM per blob is bounded by ~64 KB of
internal age/zstd buffers plus the io.Copy buffer.
* Chunk extraction via ReadAt. After a blob is on disk, restore
reads chunks via blobDiskCache.ReadAt(hash, offset, length) so
only the chunk's bytes ever touch RAM. The previous code path
called blobCache.Get for every cache-hit chunk, which read the
entire blob (e.g. 10 GB) from disk into a []byte just to slice
out a few KB — single-handedly ~900 ms per cache hit on the
user's photo snapshot.
* Locality-aware restore ordering. New restorePlan indexes
file→blob_set and blob→file_set at restore start, then drives
the loop so that every file whose full blob set is on disk is
drained before any new blob downloads. After the drain queue
empties, the planner picks the pending file with the smallest
uncached-blob count, downloads those blobs, and drains again.
A sweep is forced right before each download so the just-
completed blob is evicted before the next one is Put, keeping
peak disk-cache occupancy at 1 for path-order-adversarial
snapshots.
The restore hot path also moves onto a restoreSession struct so
restoreFile/restoreRegularFile/etc. take only the per-call file
argument instead of threading 9+ parameters through every signature.
The new BlobRepository.GetAll method lets the session build a single
blob-id → blob-hash map at start instead of doing one DB query per
chunk.
TestRestoreLocalityAndReadAt passes: peak_len ≤ 1, get_calls = 0,
readat_calls > 0, every blob fetched exactly once.
Captures three behaviors the restore hot path must exhibit but
currently doesn't, all under one test:
* Peak blob disk cache occupancy ≤ 1. Smart restore ordering should
drain every file referencing the currently-cached blob before
downloading the next one, so the sweeper can free each blob the
moment its file set is exhausted.
* Every remote blob fetched exactly once (counter on a wrapping
Storer). Already true today; the test pins it so neither future
cache-eviction nor reorder regressions can introduce
re-downloads.
* blobDiskCache.Get is never called during restore — chunk
extraction must go through ReadAt so we never read the whole
blob from disk to slice out a few KB. The 10 GB
photo-snapshot --debug output showed ~900 ms per cache-hit chunk
extract; ReadAt should bring that to sub-millisecond.
Adds Get/ReadAt call counters and a peak-Len tracker to
blobDiskCache, plus an internal restoreCacheObserver hook on Vaultik
so the test can capture the production cache instance without
exporting unexported types.
Currently fails with peak_len=3, get_calls=24, readat_calls=0. The
fix follows in subsequent commits.
Restore's per-chunk loop called blobCache.Get(blobHash) and sliced the
returned []byte to extract the chunk it actually wanted. Get reads the
entire blob from disk into memory — so for a 10 GB blob, every chunk
extraction was a 10 GB ReadFile to get back a few KB. With ~40k files
needing ~600ms per cache hit, that alone was burning ~6 hours of
wall-clock on a real restore.
Hot loop now:
- If the blob isn't cached: download (full plaintext into memory),
Put to disk cache, satisfy this chunk from the in-memory buffer.
- If it's cached: blobCache.ReadAt(hash, offset, length) — reads
only the chunk's bytes from the on-disk blob file.
ReadAt was already implemented on blobDiskCache; restore just wasn't
using it.
Debug timings from the user's photo-catalog restore showed
ms_cache_gets dominating every cache-hit file at 500-1000ms. With
ReadAt those should drop to sub-millisecond and the visible throughput
should be bound by single-stream blob download + decrypt/decompress
rather than disk-read amplification.
Restore and verify no longer use schollz/progressbar. Instead they emit
a periodic status line every 15 s via ui.Writer.Progress, matching the
cadence and shape of the snapshot create scanner output. The lines
include files done, byte counts, throughput in bits/sec, elapsed,
absolute ETA, and remaining duration — same conventions as snapshot
create. The progressbar dependency, the newProgressBar/isTerminal
helpers, and the unused printfStderr helper are removed; go.mod loses
schollz/progressbar plus its colorstring and uniseg transitive deps.
Adds --debug timing instrumentation throughout the restore hot path so
the next slow-restore report can pinpoint which stage is the
bottleneck. Per-file: file-chunks query, output Create, per-chunk blob
DB lookups, cache get/put, blob download, chunk write, sweeper call.
Per-blob-download: fetch-setup (Get + Stat) vs read+decrypt+decompress
vs close-and-verify. FetchBlob splits the Storage.Get and Storage.Stat
round-trips so an expensive size-stat is visible separately.
Writes 30 random 1 MB files plus 10 duplicates (40 files, 30 MB of
unique content), backs them up with a 10 MB blob_size_limit, then
restores through a counting storer that records every Get per key.
Each blob on disk must be downloaded exactly once during restore — a
re-download would mean the sweeper evicted a blob whose chunks were
still referenced by an unrestored file, and zero downloads would mean
the cache silently stopped being consulted.
The duplicates exercise the dedup path: the sweeper has to keep each
blob alive until every file (original AND duplicate) that references
any of its chunks has been restored.
Restore previously capped the blob disk cache at 4× the configured
blob_size_limit (so 40 GB by default). With large or heavily-deduped
snapshots a chunk-by-chunk file walk could blow past that cap and
trigger LRU eviction of blobs that were still needed by later files,
forcing repeated re-downloads — observed during a real restore as
single-stream throughput collapsing to under 1 MB/s.
Restore now allocates the cache with no practical size cap and drives
eviction explicitly:
* An in-memory set of restored file IDs accumulates as files finish.
* Every blob_size_limit/100 bytes of restored data (≈100 sweeps per
blob's worth of writes) the sweeper iterates the cache. For each
cached blob it queries the snapshot's local SQLite DB for every
file that references any chunk in the blob and deletes the cache
entry only when every such file is already in the restored set.
* blobStillNeeded returns true on any error so an unreadable DB
never causes premature eviction.
The cache itself gains Delete(key) and Keys() so the sweeper can drive
removal without touching internal LRU state.
Two output-style fixes plus a quiet-mode correction.
Banner: a manual scan of os.Args in CLIEntry decides whether to suppress
the banner (--quiet/-q/--cron), then prints it before cobra parses any
arguments. This makes the banner appear even when cobra rejects bad args
("requires at least 2 arg(s)") and on --help — paths that previously
skipped PersistentPreRun entirely. The cobra-side hook plumbing (sync.Once,
PersistentPreRun, custom HelpFunc) is removed.
Errors: rootCmd.SilenceErrors = true so cobra no longer prints its own
"Error: <msg>" line. Any error returned from Execute() goes through
ui.New(os.Stderr).Error(...), giving the documented "🛑 ERROR: <msg>"
format. A new helper cli.ReportError() formats errors from goroutine
paths that can't return through cobra's normal return chain; every
CLI command's fx-goroutine error path now calls it alongside the
existing structured log.Error so both channels record the failure.
Quiet mode: previously --quiet/--cron swapped Vaultik.UI to io.Discard,
which silenced Warning and Error messages too — contradicting the
documented "suppresses non-error output" semantics. ui.Writer now has
a SetQuiet flag that drops Begin/Complete/Info/Notice/Detail/Progress/
Banner only; Warning and Error always emit.
Also folds in restore.go cleanups the audit flagged: the hardcoded
"WARNING:" prefix on the failed-files block now uses ui.Warning +
ui.Detail, the post-restore "Restored N files" line uses ui.Complete,
and the "No files found to restore" branch emits both log.Warn and
ui.Warning so structured logs continue to capture it under --verbose.
The remote snapshot table now shows the total plaintext size of all
chunks referenced by each snapshot, plus the plaintext size of chunks
newly referenced by that snapshot (chunks not in any earlier completed
snapshot known to the local DB). The latter is the marginal data
introduced by each backup — useful for spotting which snapshots
actually added bytes vs. dedup'd against prior state.
Both new columns are computed from the local database only. Snapshots
that exist in remote storage but not in the local DB show
"<remote only>" in those cells; their COMPRESSED SIZE column still
reflects the value fetched from the remote manifest.
remote nuke: new subcommand that deletes every snapshot's metadata and
every blob from remote storage, leaving the bucket prefix empty.
Requires --force.
User-facing 'Processing' is now 'Backing up' everywhere it referred to
the chunking/upload phase. Files summary line says 'backed up' instead
of 'processed'.
ui.Speed now formats bytes/sec input as bits/sec output (bit/s, Kbit/s,
Mbit/s, Gbit/s). Network transfer rates are conventionally expressed
in bits — the per-blob heartbeat now matches the per-snapshot summary
line which has always been bits/sec.
The bug: fully-deduplicated snapshots (every chunk already in storage
from a prior run) had an empty snapshot_blobs table. The metadata-
export pipeline then dropped all blob/blob_chunks rows from the
exported database, leaving file_chunks references to chunks whose
blobs were no longer recorded. Restore fails on every file with
"chunk X not found in any blob".
Fix: at CompleteSnapshot time, run an INSERT OR IGNORE that links
every blob holding a chunk referenced by this snapshot's files into
snapshot_blobs. New blobs uploaded during the snapshot are already
recorded (no-op for them); dedup-referenced blobs are added.
The cleanup query in deleteOrphanedBlobs already restricts to
snapshot_blobs entries for the current snapshot — so once
snapshot_blobs is correctly populated, the exported database
contains the full set of blob/blob_chunks rows needed for restore.
Regression test: TestDedupOnlySnapshotRestores creates two
identical snapshots (the second uploads zero new blobs) and
restores the second. Without the fix, restore fails on every file.
restore aborts on the first per-file failure by default, surfacing
the file path and the underlying error and suggesting --skip-errors
to continue past failures.
--skip-errors moved from a 'snapshot create' subcommand flag to a
top-level persistent flag on the root command. It applies to both
snapshot create and restore. Old 'vaultik snapshot create --skip-
errors' still works because persistent flags are inherited.
- New ui.Detail method for indented continuation lines under a
preceding Complete (visually same as Progress: " 》" in white).
- Snapshot summary lines (Files/Data/Storage/Upload/Duration) are
now Detail lines indented under "Created snapshot X.".
- Local index database prune complete result lines (incomplete
snapshots, orphaned files/chunks/blobs) are also Detail lines
under a clean Complete header.
- "Files: ... to process" → "Files: ... processed" (they have been
processed by the time we emit the summary).
- "Data: ... (... to process)" → "Data: ... (... processed)".
- ui.Writer now tracks warning and error counts emitted; Vaultik
prints "Finished successfully." or "Finished (with N warnings)."
as the final line of CreateSnapshot.
Progress lines now use the form:
..., <subject> elapsed: <dur>, <subject> ETA: <time> (est remain <dur>).
ui.Time formats same-day times as HH:MM:SS and other-day times as
YYYY-MM-DD HH:MM:SS, with no timezone suffix (local time is implied).
The local-index-database prune complete line now shows remaining
counts for each category:
... 1 incomplete snapshots removed (3 remain), 3783 orphaned files
removed (42 remain), ...
All user-facing output now goes through a single ui.Writer with a
uniform style:
》 (white) for begin / info / notice
》 (green) for complete / success
Warning: for warnings (orange)
ERROR: for errors (red)
》 (indented) for progress heartbeats
Color is enabled when stdout is a TTY and NO_COLOR is unset.
Standards:
- Complete-sentence messages with fully qualified terms ("backup
destination store", "local index database", "snapshot source
files enumeration").
- Every Complete has a matching Begin.
- Natural verb tense conveys state ("Uploading" -> "Uploaded"). The
words "begin"/"complete" never appear in message bodies; the marker
color carries that information.
- ETA means clock time, not duration. Progress lines say "estimated
remaining time (<dur>), finish at <time>" with both labeled.
Adds globals.CommitDate (populated by Makefile/Dockerfile/goreleaser
via ldflags from `git show -s --format=%cI HEAD`) and a startup banner
printed once per invocation.
Strips fx call-chain noise from startup errors so users see the actual
underlying error (e.g. "creating base path: mkdir /Volumes/BACKUPS:
permission denied" instead of three layers of "could not build
arguments for function ...").
README documents the output style and the ui package conventions.
ResolveConfigPath now stats explicit paths from --config and
$VAULTIK_CONFIG and produces an actionable error naming the bad
path and suggesting 'vaultik config init' (with the right path
in the --config case). The default-search failure message lists
the paths it tried.
The scanner no longer hard-codes os.Stdout vs io.Discard based on
EnableProgress. ScannerConfig and ScannerParams take an explicit
Output io.Writer, and the Vaultik caller passes v.Stdout — which
itself is set to io.Discard in --cron mode. One knob controls
both scanner-level and Vaultik-level user-facing output.
The version command prints a hint when Version == "dev" telling
the user this is a development build without embedded version
metadata.
Module path changed from git.eeqj.de/sneak/vaultik to
sneak.berlin/go/vaultik (vanity redirect). All imports, ldflags,
Dockerfile, goreleaser config, and docs updated. App data/config
directories now use plain "vaultik" instead of the reverse-DNS name.
README:
- New copy-pasteable quickstart at top: go install, config init,
age keypair, config set for key + file:// destination, home backup
- All command names in command details are code-quoted
- config set/get gained sequence index support (age_recipients.0)
so lists are settable from the CLI
- Dockerfile build is CGO_ENABLED=0 to match the pure-Go build
These test files existed locally and ran in the suite but were never
committed due to the old .gitignore 'vaultik' pattern matching the
internal/vaultik/ directory.
--cron now sets Vaultik.Stdout to io.Discard so all user-facing output
is suppressed, not just the scanner progress. Errors still go to stderr
via the structured logger.
snapshot list now warns when local snapshot records have no matching
remote metadata, and suggests 'vaultik snapshot cleanup' instead of
silently deleting them.
snapshot cleanup is a new subcommand that explicitly removes stale
local snapshot records. syncWithRemote (used by purge) still does
this automatically since purge is already destructive.
.gitignore changed from 'vaultik' to '/vaultik' so it only matches
the binary at the repo root, not the internal/vaultik/ directory.
snapshot create --prune now accepts --keep-newer-than <duration> (e.g.
4w, 30d, 6mo) to keep a rolling window of snapshots instead of only
the latest. Supports d/w/mo/y units and combinations (2w3d).
Without --keep-newer-than, --prune still defaults to keep-latest-only.
Scanner now records symlinks (with their target) and directories
during the walk phase instead of skipping them. processFileStreaming
detects non-regular entries and writes the DB record without chunking.
The e2e test (TestEndToEndFileStorage) now verifies:
- Symlink target preserved through backup→restore
- Empty directory survives round-trip
- File permissions (0600) restored correctly
Scanner now writes all user-facing output to an io.Writer (os.Stdout
when progress is enabled, io.Discard in --cron mode). This fixes the
long-standing issue where --cron still printed progress lines.
S3 HeadObject now properly distinguishes not-found from other errors
instead of swallowing all errors as not-found.
Config/CLI error messages include actionable hints (where to find the
config, how to generate keys, what storage options exist).
- Adopt origin's SnapshotPurgeOptions naming and PurgeSnapshotsWithOptions
method, but extend with Names []string (repeatable --snapshot flag) and
Quiet bool for use by --prune.
- Adopt origin's parseSnapshotName helper.
- Fold the duplicate post-backup prune block into one runPostBackupPrune
call that filters retention to the snapshot names just backed up.
- Keep the shallow-verify timestamp parsing fix and the dead deep-verify
branch removal; use origin's printVerifyHeader/verifyManifestBlobsExist
helper extraction.
- Drop top-level vaultik purge and verify (duplicates of snapshot purge
and snapshot verify).
- Drop the resurrected daemon block from info.go (config fields no
longer exist).
- Combine Makefile targets: gofmt -l for fmt-check, -race for tests,
release/release-snapshot/docker/hooks/deps/test-coverage all included.
The --daemon flag, BackupInterval, FullScanInterval, MinTimeBetweenRun
config fields, and DirtyPath model were placeholders for a never-shipped
daemon mode and have been removed. Daemon mode is out of scope for 1.0.
## Summary
`PurgeSnapshots` now applies `--keep-latest` retention per snapshot name instead of globally across all names.
### Problem
Previously, `--keep-latest` would keep only the single most recent snapshot across ALL snapshot names. For example, with snapshots:
- `system_2024-01-15`
- `home_2024-01-14`
- `system_2024-01-13`
`--keep-latest` would keep only `system_2024-01-15` and delete the latest `home` snapshot too.
### Solution
1. **Per-name retention**: `--keep-latest` now groups snapshots by name and keeps the latest of each group. In the example above, both `system_2024-01-15` and `home_2024-01-14` would be kept.
2. **`--name` flag**: New flag to filter purge operations to a specific snapshot name. `--name home --keep-latest` only purges `home` snapshots, leaving all `system` snapshots untouched.
### Changes
- `internal/vaultik/helpers.go`: Add `parseSnapshotName()` to extract the snapshot name from a snapshot ID (`hostname_name_timestamp` format)
- `internal/vaultik/snapshot.go`: Add `SnapshotPurgeOptions` struct with `Name` field, add `PurgeSnapshotsWithOptions()` method, modify `--keep-latest` logic to group by name
- `internal/cli/purge.go` and `internal/cli/snapshot.go`: Add `--name` flag to both purge CLI surfaces
- `README.md`: Update CLI documentation
### Tests
- `helpers_test.go`: Unit tests for `parseSnapshotName()` and `parseSnapshotTimestamp()`
- `purge_per_name_test.go`: Integration tests covering:
- Per-name retention with multiple names
- Single-name retention
- `--name` filter with `--keep-latest`
- `--name` filter with `--older-than`
- No-match name filter (all snapshots retained)
- Legacy snapshots without name component
- Mixed named and legacy snapshots
- Three different snapshot names
### Backward Compatibility
The existing `PurgeSnapshots(keepLatest, olderThan, force)` signature is preserved as a wrapper around the new `PurgeSnapshotsWithOptions()`. The `--prune` flag in `snapshot create` continues to work unchanged.
`docker build .` passes (lint, fmt-check, all tests).
closes [#9](#9)
Co-authored-by: user <user@Mac.lan guest wan>
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #51
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
## Summary
Replace serial `getManifestSize()` calls in `ListSnapshots` with bounded concurrent downloads using `errgroup`. For each remote snapshot not in the local DB, manifest downloads now run in parallel (up to 10 concurrent goroutines) instead of one at a time.
## Changes
- Use `errgroup` with `SetLimit(10)` for bounded concurrency
- Collect remote-only snapshot IDs first, pre-add entries with zero size
- Download manifests concurrently, patch sizes from results
- Remove now-unused `getManifestSize` helper (logic inlined into goroutines)
- Promote `golang.org/x/sync` from indirect to direct dependency
## Testing
- `make check` passes (fmt-check, lint, tests)
- `docker build .` passes
closes #8
Co-authored-by: user <user@Mac.lan guest wan>
Reviewed-on: #50
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Replace linear scan deduplication of snapshot IDs in `RemoveAllSnapshots()` and `PruneBlobs()` with `map[string]bool` for O(1) lookups.
Previously, each new snapshot ID was checked against the entire collected slice via a linear scan, resulting in O(n²) overall complexity. Now a `seen` map provides constant-time membership checks while preserving insertion order in the slice.
**Changes:**
- `internal/vaultik/snapshot.go` (`RemoveAllSnapshots`): replaced linear `for` loop dedup with `seen` map
- `internal/vaultik/prune.go` (`PruneBlobs`): replaced linear `for` loop dedup with `seen` map
closes #12
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #45
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Renames `internal/vaultik/blob_fetch_stub.go` to `internal/vaultik/blob_fetch.go`.
The file contains production code (`hashVerifyReader`, `FetchAndDecryptBlob`), not stubs. The `_stub` suffix was a misnomer from the original implementation in [PR #39](#39).
Pure rename — no code changes. All tests, linting, and formatting pass.
closes#52
Co-authored-by: user <user@Mac.lan guest wan>
Reviewed-on: #53
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
## Summary
`ListSnapshots()` silently deleted local snapshot records not found in remote storage. A list/read operation should not have destructive side effects.
## Changes
1. **Removed destructive sync from `ListSnapshots()`** — the inline loop that deleted local snapshots not present in remote storage has been removed entirely. `ListSnapshots()` now only reads and displays data.
2. **Improved `syncWithRemote()` cascade cleanup** — updated `syncWithRemote()` to use `deleteSnapshotFromLocalDB()` instead of directly calling `Repositories.Snapshots.Delete()`. This ensures proper cascade deletion of related records (`snapshot_files`, `snapshot_blobs`, `snapshot_uploads`) before deleting the snapshot record itself, matching the thorough cleanup that the removed `ListSnapshots` code was doing.
The explicit sync behavior remains available via `syncWithRemote()`, which is called by `PurgeSnapshots()`.
## Testing
- `docker build .` passes (lint, fmt-check, all tests, compilation)
closes #15
Co-authored-by: clawbot <clawbot@eeqj.de>
Reviewed-on: #49
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
## Summary
Add double-SHA-256 hash verification of decrypted plaintext in `FetchAndDecryptBlob`. This ensures blob integrity during restore operations by comparing the computed hash against the expected blob hash before returning data to the caller.
The blob hash is `SHA256(SHA256(plaintext))` as produced by `blobgen.Writer.Sum256()`. Verification happens after decryption and decompression but before the data is used.
## Test
Added `blob_fetch_hash_test.go` with tests for:
- Correct hash passes verification
- Mismatched hash returns descriptive error
## make test output
```
golangci-lint run
0 issues.
ok git.eeqj.de/sneak/vaultik/internal/blob 4.563s
ok git.eeqj.de/sneak/vaultik/internal/blobgen 3.981s
ok git.eeqj.de/sneak/vaultik/internal/chunker 4.127s
ok git.eeqj.de/sneak/vaultik/internal/cli 1.499s
ok git.eeqj.de/sneak/vaultik/internal/config 1.905s
ok git.eeqj.de/sneak/vaultik/internal/crypto 0.519s
ok git.eeqj.de/sneak/vaultik/internal/database 4.590s
ok git.eeqj.de/sneak/vaultik/internal/globals 0.650s
ok git.eeqj.de/sneak/vaultik/internal/models 0.779s
ok git.eeqj.de/sneak/vaultik/internal/pidlock 2.945s
ok git.eeqj.de/sneak/vaultik/internal/s3 3.286s
ok git.eeqj.de/sneak/vaultik/internal/snapshot 3.979s
ok git.eeqj.de/sneak/vaultik/internal/vaultik 4.418s
```
All tests pass, 0 lint issues.
Co-authored-by: user <user@Mac.lan guest wan>
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #39
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
Co-committed-by: clawbot <sneak+clawbot@sneak.cloud>
Add an interactive progress bar (using schollz/progressbar) to the file restore loop, matching the existing pattern in verify. Shows bytes restored with ETA when output is a terminal.
Fixes#20
Co-authored-by: clawbot <clawbot@eeqj.de>
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #23
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
The disk blob cache now uses 4 * BlobSizeLimit from config instead of a
hardcoded 10 GiB default. This ensures the cache scales with the
configured blob size.
