A backup uploads its blobs before the manifest that lists them, and the
process lock covers one user on one machine, so a prune from another host
or user sharing the destination deleted a running backup's blobs.
Prune now deletes an unreferenced blob only once its listed time is older
than the new config key prune_grace_period (default 24h; a negative value
is refused). A newer blob, or one listed with no time, is kept and counted
(blobs_too_new). The rclone backend takes each object's time from the
listing (use_server_modtime).
Decision: a blob a backup reuses from an earlier run that stopped before
its manifest keeps its old time and is not protected; the docs and config
comments say so.
Model: opus-5-5
`remote nuke` took a destination store it could not list, or one
without `vaultik-destination.txt` while the local index held
snapshots, as a warning: it then cleared the local snapshot records,
and for an unplugged volume's empty mount point reported the store
empty and exited zero. Nuke now fails on either before it removes
anything, without its advice to run it again, which would fail the
same way. The bulk remove it runs fails on either too; only a
local-only remove still goes on with a warning. Nuke also deletes each
snapshot's remote metadata before its local record, so a failed delete
keeps the snapshot in the local index rather than leaving it on the
store as a remote-only snapshot.
Model: opus-5-5
Every backup writes vaultik-destination.txt at the destination root.
While the local index holds snapshots stored there, a destination
without it is not present: snapshot create refuses before writing,
prune and snapshot purge fail, and snapshot list and snapshot remove
act as for a destination that cannot be listed. snapshot create and
snapshot remove check it first. remote nuke deletes the marker last.
The file:// backend never creates its destination directory, which
protects a directory on a removable volume, not a mount point.
Tests back up into directories that exist; fixtures for a written
destination carry the marker.
Judgement call: the marker is rewritten on every backup.
Deviation: for the complexity limit, two blocks moved, unchanged, into
snapshotNamesToCreate and confirmRemoveSnapshot.
Model: opus-5-5
PruneBlobs, CleanupLocalSnapshots, RemoveSnapshot, RemoveAllSnapshots
and snapshot purge wrote their status lines straight to stdout, so
`snapshot create --cron --prune` printed on every successful run and
--quiet did not silence `prune`, `snapshot purge`, `snapshot remove` or
`remote nuke`. Those lines now go through internal/ui, which drops them
under --cron and --quiet; warnings, errors and --json documents still
print.
The confirmation prompts still write straight to stdout, as the
`database delete` prompt does, together with what they ask to delete:
the snapshot list before the purge prompt, and the blob count and size
before the prune prompt. With --force those lines are status lines.
The two purge test fixtures gained the UI writer production always sets.
Model: opus-5-5
remote nuke deleted only the snapshot metadata and blobs the snapshot
and blob listings recognise, so a snapshot's database whose manifest
was never uploaded, and any name under blobs/ that is not a blob hash,
survived while nuke reported the store empty.
After removing every snapshot, nuke now deletes every object listed
under metadata/ and blobs/, whatever its name, in place of the blob
prune step. A delete that fails is logged and the others still run. A
listing afterwards decides the result: nuke reports the store empty
only when it finds nothing, and otherwise names each object left and
fails.
Judgement call: nuke no longer prints the blob prune summary.
Model: opus-5-5
prune, and the prune step of snapshot create --prune, counted a blob
it could not delete and still exited 0. It now prints its summary or
--json document and then returns an error, which replaces the
"Failed to delete N blob(s)" line.
remote nuke only warned when a snapshot's metadata could not be
deleted, advised vaultik prune, which never removes metadata, and then
reported the store empty. It now stops with an error, does not report
the store empty, and says to run vaultik remote nuke --force again. It
checks that the local index belongs to the configured destination
before deleting anything, and that error carries no such advice. A
successful nuke no longer advises prune.
Model: opus-5-5
remote info counted every key under blobs/ with four or more path parts
as a blob, while prune skips a name that is not a blob hash. A stray
file such as .DS_Store was reported as an orphaned blob, with advice to
run prune, which then deleted nothing. remote info now takes its blob
list from prune's listAllRemoteBlobs, so its blob and orphan figures,
in the table and in --json, cover exactly what prune would consider.
blobKeyParts moves to prune.go, its only remaining user.
Model: opus-5-5
The file and rclone listings skip an object whose name ends in
`.partial`, the temporary name a `file://` or rclone upload writes
before moving the object into place. `remote nuke` deletes only what
the listings return, so it left the `.partial` objects killed uploads
leave behind and still reported the destination store empty.
Storer gains DeletePartialUploads. The file and rclone backends remove
every `.partial` object under the prefix; S3 has none to remove, since
it shows an object only once its upload completes. `remote nuke` calls
it for `metadata/` and `blobs/` as its last step.
Empty directories under a `file://` destination are still left behind.
Model: opus-5-5
A blob hash read back from the downloaded snapshot database or the store listing was trusted unchecked. A hostile remote could set a hash such as "aa/../../etc" and have a decrypted blob written outside the cache directory, or feed a short or negative value that panicked a command.
blobDiskCache.path now refuses any key with a path separator, and ReadAt rejects a negative offset or length, bounding so a sum cannot overflow past the check. A new isBlobHash helper gates FetchBlob, shallow and deep verify, and restore: buildBlobIndexes rejects every hash from the snapshot database before any fetch. The blobs/ and metadata/ listings skip a non-conforming name, and short-hash prefixes in log and error text go through a panic-safe shortHash helper.
Model: opus-4-8
Prune learned which blobs are in use by reading every snapshot's manifest, but merely logged and skipped one it could not download or decode. Blobs referenced only by that snapshot then looked unreferenced and were deleted, with a zero exit -- and snapshot create --prune runs this unattended. collectReferencedBlobs now errors, naming the remote key, so prune deletes nothing and exits non-zero.
Manifest generation likewise skipped a blob whose lookup failed or was missing, yielding a manifest short of what the snapshot needs; it now fails. Deep verify only warned when the manifest omitted a database blob; it now fails on any divergence. Docs corrected.
Model: opus-4-8
Closes#110.
CleanupLocalSnapshots wrote three prose lines to stdout with no --json
awareness, covering every branch, so `vaultik prune --json | jq` failed
on any input. -q never helped either: printlnStdout and stdoutf write
straight to v.Stdout and never consult v.UI, which is what SetQuiet
affects. It now takes *PruneOptions, symmetric with its sibling phase
PruneBlobs, and gates all three writes.
Threading opts.JSON was chosen over moving the lines to log.Info,
because internal/log/log.go defaults the level to Warn: log.Info would
not have relocated them to stderr, it would have deleted them from a
plain `vaultik prune`, and "Removing stale local record" narrates the
deletion of local index rows. The stale-record count is deliberately not
added to PruneBlobsResult - every field there is blob-scoped and produced
by the phase that runs after this reconciliation, so adding it would
change a published --json schema as a side effect of a stream fix.
Note for anyone reading the --json contract: under --json the
stale-record removal now produces no signal in either stream. stdout is
correctly gated, stderr is level-pinned to Warn because --json sets
Quiet, and the count is not in the document. That is inherited behaviour
- PruneBlobs' own log.Info calls are equally invisible under --json - not
something this change introduced, and it is tracked separately.
make build exited 0 and produced nothing: .PHONY listed build with no
build: rule, and a phony target with no prerequisites and no recipe is
considered already satisfied, which turns what would be a hard error into
a silent success. In a repo where `make build` is the documented way to
build, a caller checking the exit code concluded the build worked. Now
`build: vaultik`, verified in both directions - a clean build produces
the binary, a deliberately broken one exits non-zero and produces none.
All 19 .PHONY names were audited; build was the only one lacking a rule.
TestPhonyTargetsAllHaveRules keeps that true for names added later, so
the class is closed rather than the instance.
Updates golangci-lint to v2.12.2 everywhere it is pinned and installs the canonical `.golangci.yml`, then remediates every finding the new linter/config surfaces so `make check` is green.
## Version bump
- `Dockerfile` lint stage: `golangci/golangci-lint:v2.11.3-alpine` -> `v2.12.2-alpine` (digest-pinned, date comment updated)
- `Makefile` `deps` target: `go install` moved from the old v1 module path at `@latest` to the pinned `github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2`
- `.golangci.yml` replaced with the canonical config (v2 schema; settings under `linters.settings` so the thresholds actually apply; `default: all` with the standard six disables)
- `script/bootstrap` installs golangci-lint via the system package manager and carries no version pin, so it is unchanged
- CI (`.gitea/workflows/check.yml`) only runs `script/cibuild`, so it needed no change
## Lint remediation
The canonical config surfaced ~3,300 findings across 56k lines. All are fixed, behavior-preserving; incorporates and supersedes the per-package mechanical passes already merged to `main` (refs #61). Highlights:
- `err113`: dynamic errors replaced with package sentinels + `%w` wrapping; comparisons via `errors.Is`
- `goprintffuncname`: printf-style helpers renamed with an `f` suffix (`ui.Writer` message methods, `cli.ReportErrorf`, `database.Fatalf`) and all call sites updated
- `revive` stutter renames: `blob.Handler`, `blob.WithReader`, `blob.ChunkPosition`, `storage.URL`, `storage.Info`; missing doc comments added
- `contextcheck`/`noctx`: `context.Context` threaded through `blob.Packer` and the scanner call sites; context-aware `exec`/`sql` variants
- `funlen`/`cyclop`/`gocognit`/`dupl`: oversized and duplicated functions split into focused helpers (production and test code)
- tests: `t.Parallel()` added where safe (global logger init kept in the serial phase for `-race`), `t.TempDir()`/`t.Helper()` adopted, several suites converted to external test packages
- `gosec`: bounded integer conversions, `ReadHeaderTimeout` on the test HTTP server; remaining warnings suppressed per-site with justifications
- remaining `nolint` directives are rare, targeted, and each carries a reason (e.g. `nilnil` not-found contract in the repository layer, fx module globals, on-disk snake_case struct tags)
- removed the deprecated `log.LogOptions` alias (callers migrated to `log.Options`)
`make check` (tests with `-race`, lint, fmt-check) passes.
Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #62
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
The verb surface accumulated overlapping cleanup commands. Consolidate
so each cleanup verb has one meaning:
- Rename 'database purge' -> 'database delete'. The command removes the
SQLite file entirely; "purge" wrongly suggested purging contents.
- Fold 'snapshot cleanup' into 'prune'. Prune now runs three passes:
reconcile local snapshots against the remote (previously the
standalone cleanup command), drop orphaned local rows, then delete
unreferenced remote blobs. One command, one mental model.
- Delete 'store info'. Its output was a strict subset of 'remote info',
which already prints storage type + location. Any user reaching for
either should reach for 'remote info'.
- Drop 'snapshot remove --all'. It duplicated 'remote nuke --force'.
'remote nuke' is the single supported entry point for wiping the
destination store.
Also update the storage-binding error message to reference the new
'vaultik database delete' name.
The local index tracks which chunks and blobs already exist at the
backup destination. Nothing was recording *which* destination, so
changing storage_url and running a backup left the scanner treating
every already-seen chunk as still-present at the new (empty) location.
Uploads were skipped silently and the resulting snapshots pointed at
blobs that don't exist at the new destination.
Fix: record storage_url in a new local_meta key-value table on first
mutating command, and refuse to proceed when the configured URL later
differs from the stored one. The error explains the two recovery
paths (revert the config, or run 'vaultik database purge' to discard
the index and rebuild from a fresh full backup).
Wired into snapshot create / prune / snapshot remove / snapshot purge
/ snapshot cleanup. Read-only inspection commands (snapshot list,
remote info, store info) are exempt.
snapshot rm <id> now does the full cleanup: removes the local index
entry, strips the snapshot's metadata from the destination store, and
prunes any blobs that were only referenced by the just-removed manifest.
The --remote flag is retired; --local-only opts out for the rare case
where the user wants to forget a snapshot locally without touching the
remote.
If the destination store is unreachable, the local-DB removal still
completes and a warning is emitted; the user can rerun 'vaultik prune'
to retry the remote half later.
RemoveAllSnapshots gets the same treatment: after deleting every
snapshot's metadata (local + remote + orphan keys), an automatic blob
prune sweep removes the now-unreferenced blob set.
Two related changes, both addressing leakage and brittleness around
the public bytes the destination store sees.
First, every remote storage path that previously embedded a human
snapshot ID (e.g. metadata/heraklion_berlin.sneak.fs.photos.2026.
catalog_2026-06-24T07:00:15Z/...) now uses the hashed remote key:
RemoteSnapshotKey(id) = hex(SHA256(SHA256("vaultik|" + id)))
Applied at:
* uploadSnapshotArtifacts (snapshot create write path)
* the manifest.json.zst snapshot_id field — manifest is
unencrypted, so the human ID would otherwise be readable to
anyone with bucket-list permission
* cleanupIncompleteSnapshots metadata-existence probe
* snapshot restore / verify (downloadSnapshotDB,
loadVerificationData)
* downloadManifestByKey, deleteRemoteSnapshotByKey
* CleanupLocalSnapshots reconciliation
* the locally-driven removal paths (RemoveSnapshot,
RemoveAllSnapshots, confirmAndExecutePurge)
The local index database keeps human IDs everywhere — the hash is a
boundary translation, not a rename. A directory listing of the
backup destination now looks like
"metadata/<64-hex>/{db.zst.age,manifest.json.zst}" with no host,
snapshot-name, or timestamp information visible.
Second, snapshot list no longer fails just because remote storage is
unreachable, and only consults the remote when the local machine can
plausibly decrypt:
* Listing is always driven by the local index database — that's
what holds the human IDs, timestamps, and per-snapshot stats
that the table actually shows.
* If no age secret key is configured, we skip remote listing
entirely (the box is treated as a write-only backup machine —
there's no value showing it remote-only keys it could never
restore).
* If a key IS configured, we try the remote listing; failures
(volume unmounted, permission denied, network error) downgrade
to a warning instead of aborting the command.
* When the remote listing succeeds, we cross-reference by hashing
each local human ID and diffing against the returned key set.
Local-only snapshots get the existing "stale local record"
cleanup hint; remote-only keys are surfaced as a single
"NOTE: N remote snapshot(s) found in backup destination store
but not in local database" line.
FileStorer construction also no longer does an eager mkdir — the
basePath is recorded and the directory is created lazily on first
write. A missing or unmounted destination during `snapshot list`
should NOT block the command, and now it doesn't.
RemoveAllSnapshots is rewritten to drive deletion from the local
index instead of from a remote listing, hashing each local ID to
find the corresponding remote key. Orphan remote keys (no matching
local snapshot) are handled separately and only deleted when
--remote is set. Existing tests are updated to hash storage paths
through the new RemoteSnapshotKey helper.
The hash format is a hard pre-1.0 break: existing remote snapshots
written under the human-ID path scheme are no longer readable; they
need to be either re-uploaded under the new scheme or manually
renamed. There is no fallback path; matching the project policy of
"no migrations pre-1.0."
The CLI had two commands named "prune" doing different jobs (local
DB orphan cleanup vs. remote blob garbage collection), which was
confusing and forced a manual two-step workflow after deleting any
snapshot.
Single user-facing prune surface is now `vaultik prune`, which calls
PruneDatabase (local orphan cleanup) then PruneBlobs (remote unref
blob GC). Snapshot deletion paths (snapshot remove, snapshot remove
--all, snapshot purge) auto-run CleanupOrphanedData inline so the
local index database doesn't accumulate ghost rows after every
removal — the user observed ~39k orphaned files and 2 orphaned blobs
after a remove --all because that cleanup was previously a separate
opt-in command. `snapshot prune` is removed.
Also addresses the doc/help-string drift the user audit caught:
* cli/prune.go help text used to reference a non-existent
`vaultik purge` command.
* cli/config.go get/set short/long examples were S3-specific
(s3.bucket) when the primary storage configuration is
storage_url.
* vaultik/info.go printed S3 Bucket/Endpoint/Region labels
unconditionally; for file:// or rclone:// users those rows
were empty. The Storage Configuration block now prints the
storer's Type+Location first, the storage_url string when set,
and only emits S3 rows that are actually populated.
* vaultik/info.go's "Run 'vaultik prune --remote'" hint
referenced a flag that doesn't exist.
* vaultik/blobcache.go's doc comment claimed LRU eviction, which
is no longer the restore-time policy (the sweeper drives
eviction; LRU is the safety-net fallback when maxBytes is
finite).
* README.md listed `vaultik restore`, `vaultik snapshot prune`,
and `s3.bucket` example, all out of date.
README's roadmap section is rewritten with concrete pre-1.0 items
(security audit, error-condition tests, parallel blob downloads,
restart of interrupted restore, …) so the next-steps surface
matches what the project actually still needs.
The cleanup calls are guarded against a nil SnapshotManager so
tests that construct a bare Vaultik struct continue to work.
remote nuke: new subcommand that deletes every snapshot's metadata and
every blob from remote storage, leaving the bucket prefix empty.
Requires --force.
User-facing 'Processing' is now 'Backing up' everywhere it referred to
the chunking/upload phase. Files summary line says 'backed up' instead
of 'processed'.
ui.Speed now formats bytes/sec input as bits/sec output (bit/s, Kbit/s,
Mbit/s, Gbit/s). Network transfer rates are conventionally expressed
in bits — the per-blob heartbeat now matches the per-snapshot summary
line which has always been bits/sec.
Module path changed from git.eeqj.de/sneak/vaultik to
sneak.berlin/go/vaultik (vanity redirect). All imports, ldflags,
Dockerfile, goreleaser config, and docs updated. App data/config
directories now use plain "vaultik" instead of the reverse-DNS name.
README:
- New copy-pasteable quickstart at top: go install, config init,
age keypair, config set for key + file:// destination, home backup
- All command names in command details are code-quoted
- config set/get gained sequence index support (age_recipients.0)
so lists are settable from the CLI
- Dockerfile build is CGO_ENABLED=0 to match the pure-Go build
Address all four review concerns on PR #31:
1. Fix missed bare fmt.Println() in VerifySnapshotWithOptions (line 620)
2. Replace all direct fmt.Fprintf(v.Stdout,...) / fmt.Fprintln(v.Stdout,...) /
fmt.Fscanln(v.Stdin,...) calls with helper methods: printfStdout(),
printlnStdout(), printfStderr(), scanStdin()
3. Route progress bar and stderr output through v.Stderr instead of os.Stderr
in restore.go (concern #4: v.Stderr now actually used)
4. Rename exported Outputf to unexported printfStdout (YAGNI: only helpers
actually used are created)
Multiple methods wrote directly to os.Stdout instead of using the injectable
v.Stdout writer, breaking the TestVaultik testing infrastructure and making
output impossible to capture or redirect.
Fixed in: ListSnapshots, PurgeSnapshots, VerifySnapshotWithOptions,
PruneBlobs, outputPruneBlobsJSON, outputRemoveJSON, ShowInfo, RemoteInfo.
- Add global --quiet/-q flag to suppress non-error output
- Add --json flag to verify, snapshot rm, and prune commands
- Add config file permission check (warns if world/group readable)
- Update TODO.md to remove completed items
Storage backend:
- Add internal/storage package with Storer interface
- Implement FileStorer for local filesystem storage (file:// URLs)
- Implement S3Storer wrapping existing s3.Client
- Support storage_url config field (s3:// or file://)
- Migrate all consumers to use storage.Storer interface
PID locking:
- Add internal/pidlock package to prevent concurrent instances
- Acquire lock before app start, release on exit
- Detect stale locks from crashed processes
Scan progress improvements:
- Add fast file enumeration pass before stat() phase
- Use enumerated set for deletion detection (no extra filesystem access)
- Show progress with percentage, files/sec, elapsed time, and ETA
- Change "changed" to "changed/new" for clarity
Config improvements:
- Add tilde expansion for paths (~/)
- Use xdg library for platform-specific default index path
- Create comprehensive integration tests with mock S3 client
- Add in-memory filesystem and SQLite database support for testing
- Test full backup workflow including chunking, packing, and uploading
- Add test to verify encrypted blob content
- Fix scanner to use afero filesystem for temp file cleanup
- Demonstrate successful backup and verification with mock dependencies
clawbot
