Commit Graph

145 Commits

Author SHA1 Message Date
3bcdbcfd83 Correct what --cron actually suppresses (closes #84)
All checks were successful
check / check (push) Successful in 6s
The Vaultik.UI doc comment claimed the cli layer replaces the writer with
a discarding writer in --cron mode. It does not. UI is built once as
ui.New(os.Stdout) and never reassigned; internal/cli/app.go calls
UI.SetQuiet(true) when --cron or --quiet is set, which drops Begin,
Complete, Info, Notice, Detail, Progress and Banner - but Warningf and
Errorf have no quiet check and are still emitted.

That distinction matters: the end-of-run summary is deliberately routed
through UI.Warningf so cron delivers something, so a reader who believed
the comment would have concluded the opposite of how the code is meant to
work.

The README's --cron description carried the same imprecision ("Silent
unless error") and is corrected alongside it.

Comment and documentation only - the Go diff contains no non-comment
lines, so there is no behavior change.
2026-08-09 07:43:45 +02:00
50e20b460e List remote snapshots without requiring the private key (closes #64)
All checks were successful
check / check (push) Successful in 6s
ListSnapshots built its table entirely from the local SQLite index. The
only remote access, reportRemoteDrift, was gated on AgeSecretKey != "",
so on a correctly configured host - which by design holds no private key
- snapshot list never contacted the destination store at all. A user who
lost their local index could not see their own backups, and the
"<remote only>" cell the README documents was unreachable dead code.

The listing is now the union of the local index and the destination
store, with no age_secret_key gate. Remote-only snapshots cannot have
their hostname or name recovered - RemoteSnapshotKey is one-way and the
manifest stores the hash - so they are listed by abbreviated remote key
with the real timestamp and compressed size from the manifest, and
"<remote only>" in the two columns that require the local index. Nothing
new is written to remote storage and the human ID is never fabricated.

An unreachable destination degrades to local-only with a warning and a
zero exit code. remote_present is null rather than false in that case,
so "absent" and "unknown" stay distinguishable and no drift is claimed
from a listing that never happened.

Also:

- Snapshot timestamps are normalised to UTC in scanSnapshotRows, the
  single point where they enter the domain. Previously one of three
  scanners omitted .UTC(), so on a non-UTC host the same snapshot
  rendered a different time depending on whether it was locally tracked.
- The 1000-row cap and the unreadable-manifest count are reported in
  --json mode as well as table mode, so machine consumers cannot be
  silently truncated. The JSON shape is unchanged.
- Warnings raised while listing are routed to stderr rather than the
  logger, which writes to stdout and would corrupt the JSON document.
  This is a local workaround for the logger bug tracked in #82 and
  should be removed when that lands.
- downloadManifestByKey is now the only remote manifest reader, so the
  manifest privacy question in #81 has a single call site to change.
- The orphaned "vaultik snapshot cleanup" hint now names vaultik prune;
  that command was folded into prune by the 2026-07-02 consolidation.
2026-08-09 07:34:15 +02:00
e496aa334b Finish the lint remediation: script/cibuild exits 0 (closes #61)
All checks were successful
check / check (push) Successful in 5s
Clears the final 80 golangci-lint findings under the canonical
.golangci.yml (sha256 021cc83f4e6fc7c31b95b34b846723dfcf20b66b7baeea1dc40406e643346bcb),
taking the repo from red to green: script/cibuild exits 0.

- wsl_v5 (60): blank line above defer/go statements sharing no variable
  with the line above; blank-line-only diff.
- sqlclosecheck (10): the package-local CloseRows helper hid the close
  from the analyzer. Helper removed; all 18 call sites now defer an
  inline rows.Close(), preserving the fatal-on-close-error path. No
  resource leak existed - the rows were always being closed.
- prealloc (3): append targets given a starting capacity.
- revive (3): package-name findings suppressed with per-site directives
  pending the naming decision tracked in #76.

No gosec suppressions are needed under the pinned linter. .golangci.yml,
Dockerfile, Makefile, .gitea/ and script/ are byte-identical to main.

Verified with script/cibuild (digest-pinned golangci-lint v2.12.2), not
make check - the latter resolves the linter from PATH and is not a
trustworthy gate here; see #78.

Closes #59.
2026-08-09 04:25:11 +02:00
cc58583130 Update golangci-lint to v2.12.2 with canonical config (#62)
All checks were successful
check / check (push) Successful in 5s
Updates golangci-lint to v2.12.2 everywhere it is pinned and installs the canonical `.golangci.yml`, then remediates every finding the new linter/config surfaces so `make check` is green.

## Version bump

- `Dockerfile` lint stage: `golangci/golangci-lint:v2.11.3-alpine` -> `v2.12.2-alpine` (digest-pinned, date comment updated)
- `Makefile` `deps` target: `go install` moved from the old v1 module path at `@latest` to the pinned `github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2`
- `.golangci.yml` replaced with the canonical config (v2 schema; settings under `linters.settings` so the thresholds actually apply; `default: all` with the standard six disables)
- `script/bootstrap` installs golangci-lint via the system package manager and carries no version pin, so it is unchanged
- CI (`.gitea/workflows/check.yml`) only runs `script/cibuild`, so it needed no change

## Lint remediation

The canonical config surfaced ~3,300 findings across 56k lines. All are fixed, behavior-preserving; incorporates and supersedes the per-package mechanical passes already merged to `main` (refs #61). Highlights:

- `err113`: dynamic errors replaced with package sentinels + `%w` wrapping; comparisons via `errors.Is`
- `goprintffuncname`: printf-style helpers renamed with an `f` suffix (`ui.Writer` message methods, `cli.ReportErrorf`, `database.Fatalf`) and all call sites updated
- `revive` stutter renames: `blob.Handler`, `blob.WithReader`, `blob.ChunkPosition`, `storage.URL`, `storage.Info`; missing doc comments added
- `contextcheck`/`noctx`: `context.Context` threaded through `blob.Packer` and the scanner call sites; context-aware `exec`/`sql` variants
- `funlen`/`cyclop`/`gocognit`/`dupl`: oversized and duplicated functions split into focused helpers (production and test code)
- tests: `t.Parallel()` added where safe (global logger init kept in the serial phase for `-race`), `t.TempDir()`/`t.Helper()` adopted, several suites converted to external test packages
- `gosec`: bounded integer conversions, `ReadHeaderTimeout` on the test HTTP server; remaining warnings suppressed per-site with justifications
- remaining `nolint` directives are rare, targeted, and each carries a reason (e.g. `nilnil` not-found contract in the repository layer, fx module globals, on-disk snake_case struct tags)
- removed the deprecated `log.LogOptions` alias (callers migrated to `log.Options`)

`make check` (tests with `-race`, lint, fmt-check) passes.

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #62
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-08-07 23:22:48 +02:00
e7b49d58ab Fix noinlineerr findings: internal/vaultik (refs #61) 2026-08-07 16:59:56 +00:00
919229f224 Fix noinlineerr findings: internal/storage (refs #61) 2026-08-07 16:59:56 +00:00
68cffba35d Fix noinlineerr findings: internal/snapshot (refs #61) 2026-08-07 16:59:56 +00:00
26cbb63749 Fix noinlineerr findings: internal/pidlock (refs #61) 2026-08-07 16:59:56 +00:00
bf1d3c6bad Fix noinlineerr findings: internal/database (refs #61) 2026-08-07 16:59:56 +00:00
dca3c50cd2 Fix noinlineerr findings: internal/crypto (refs #61) 2026-08-07 16:59:56 +00:00
1ee0d291ca Fix noinlineerr findings: internal/config (refs #61) 2026-08-07 16:59:56 +00:00
2f7e37153c Fix noinlineerr findings: internal/cli (refs #61) 2026-08-07 16:59:56 +00:00
26909b058a Fix noinlineerr findings: internal/chunker (refs #61) 2026-08-07 16:59:56 +00:00
2dfdc5f095 Fix noinlineerr findings: internal/blobgen (refs #61) 2026-08-07 16:59:56 +00:00
217d60eeaa Fix noinlineerr findings: internal/blob (refs #61) 2026-08-07 16:59:56 +00:00
82eb352eb5 Apply linter autofixes: internal/vaultik (refs #61) 2026-08-07 16:53:23 +00:00
0296e26210 Apply linter autofixes: internal/storage, types, ui (refs #61) 2026-08-07 16:53:23 +00:00
1e05fa0dd7 Apply linter autofixes: internal/snapshot (refs #61) 2026-08-07 16:53:23 +00:00
bec964fc20 Apply linter autofixes: internal/globals, log, models, pidlock, s3 (refs #61) 2026-08-07 16:53:23 +00:00
b1451bb17e Apply linter autofixes: internal/database (refs #61) 2026-08-07 16:53:19 +00:00
ee83f50281 Apply linter autofixes: internal/crypto (refs #61) 2026-08-07 16:53:19 +00:00
76f79af733 Apply linter autofixes: internal/config (refs #61) 2026-08-07 16:53:19 +00:00
070a8a5447 Apply linter autofixes: internal/cli (refs #61) 2026-08-07 16:53:19 +00:00
40516d1263 Apply linter autofixes: internal/chunker (refs #61) 2026-08-07 16:53:19 +00:00
a66e1f9844 Apply linter autofixes: internal/blobgen (refs #61) 2026-08-07 16:53:19 +00:00
5e4df7d04f Apply linter autofixes: internal/blob (refs #61) 2026-08-07 16:53:19 +00:00
34a4d163f2 Consolidate CLI verbs; retire overlapping commands
The verb surface accumulated overlapping cleanup commands. Consolidate
so each cleanup verb has one meaning:

- Rename 'database purge' -> 'database delete'. The command removes the
  SQLite file entirely; "purge" wrongly suggested purging contents.
- Fold 'snapshot cleanup' into 'prune'. Prune now runs three passes:
  reconcile local snapshots against the remote (previously the
  standalone cleanup command), drop orphaned local rows, then delete
  unreferenced remote blobs. One command, one mental model.
- Delete 'store info'. Its output was a strict subset of 'remote info',
  which already prints storage type + location. Any user reaching for
  either should reach for 'remote info'.
- Drop 'snapshot remove --all'. It duplicated 'remote nuke --force'.
  'remote nuke' is the single supported entry point for wiping the
  destination store.

Also update the storage-binding error message to reference the new
'vaultik database delete' name.
2026-07-02 16:42:20 +02:00
d330f9f031 Bind the local index to its backup destination
The local index tracks which chunks and blobs already exist at the
backup destination. Nothing was recording *which* destination, so
changing storage_url and running a backup left the scanner treating
every already-seen chunk as still-present at the new (empty) location.
Uploads were skipped silently and the resulting snapshots pointed at
blobs that don't exist at the new destination.

Fix: record storage_url in a new local_meta key-value table on first
mutating command, and refuse to proceed when the configured URL later
differs from the stored one. The error explains the two recovery
paths (revert the config, or run 'vaultik database purge' to discard
the index and rebuild from a fresh full backup).

Wired into snapshot create / prune / snapshot remove / snapshot purge
/ snapshot cleanup. Read-only inspection commands (snapshot list,
remote info, store info) are exempt.
2026-07-02 16:35:41 +02:00
1a97a80a81 Skip chown when restore runs as non-root; warn at end
chown(2) requires root on every Unix-ish kernel. Restoring 39k files
as a non-root user produces 39k EPERM syscalls plus 39k matching debug
log lines, all for an operation that can't possibly succeed. Skip the
syscall entirely when euid != 0, and emit one warning at the end of the
restore so the user knows the on-disk UID/GID will reflect the running
user rather than the original owner.
2026-06-28 07:48:28 +02:00
c17426b556 snapshot rm: remove metadata only, print prune command for blobs
The previous change had snapshot rm auto-prune unreferenced blobs. The
correct division of labor is: rm removes a snapshot (local DB + remote
metadata), prune cleans up blobs. Reverting the auto-prune means rm
stays a cheap, deterministic operation: it touches one snapshot's worth
of state and emits the exact 'vaultik prune' command the user should
run next to delete blobs no longer referenced by any remote manifest.

This is correct because prune must consult every remote manifest
(including snapshots this host doesn't know about) to determine which
blobs are still referenced, and folding that work into rm would
silently turn rm into an expensive O(remote snapshots) operation that
also assumes the remote is fully reachable.
2026-06-28 06:20:09 +02:00
b39d765374 Make snapshot rm clean up the remote by default
snapshot rm <id> now does the full cleanup: removes the local index
entry, strips the snapshot's metadata from the destination store, and
prunes any blobs that were only referenced by the just-removed manifest.
The --remote flag is retired; --local-only opts out for the rare case
where the user wants to forget a snapshot locally without touching the
remote.

If the destination store is unreachable, the local-DB removal still
completes and a warning is emitted; the user can rerun 'vaultik prune'
to retry the remote half later.

RemoveAllSnapshots gets the same treatment: after deleting every
snapshot's metadata (local + remote + orphan keys), an automatic blob
prune sweep removes the now-unreferenced blob set.
2026-06-28 06:10:26 +02:00
fd759a921a Hash snapshot IDs at the storage boundary; make snapshot list resilient
Two related changes, both addressing leakage and brittleness around
the public bytes the destination store sees.

First, every remote storage path that previously embedded a human
snapshot ID (e.g. metadata/heraklion_berlin.sneak.fs.photos.2026.
catalog_2026-06-24T07:00:15Z/...) now uses the hashed remote key:

  RemoteSnapshotKey(id) = hex(SHA256(SHA256("vaultik|" + id)))

Applied at:

  * uploadSnapshotArtifacts (snapshot create write path)
  * the manifest.json.zst snapshot_id field — manifest is
    unencrypted, so the human ID would otherwise be readable to
    anyone with bucket-list permission
  * cleanupIncompleteSnapshots metadata-existence probe
  * snapshot restore / verify (downloadSnapshotDB,
    loadVerificationData)
  * downloadManifestByKey, deleteRemoteSnapshotByKey
  * CleanupLocalSnapshots reconciliation
  * the locally-driven removal paths (RemoveSnapshot,
    RemoveAllSnapshots, confirmAndExecutePurge)

The local index database keeps human IDs everywhere — the hash is a
boundary translation, not a rename. A directory listing of the
backup destination now looks like
"metadata/<64-hex>/{db.zst.age,manifest.json.zst}" with no host,
snapshot-name, or timestamp information visible.

Second, snapshot list no longer fails just because remote storage is
unreachable, and only consults the remote when the local machine can
plausibly decrypt:

  * Listing is always driven by the local index database — that's
    what holds the human IDs, timestamps, and per-snapshot stats
    that the table actually shows.
  * If no age secret key is configured, we skip remote listing
    entirely (the box is treated as a write-only backup machine —
    there's no value showing it remote-only keys it could never
    restore).
  * If a key IS configured, we try the remote listing; failures
    (volume unmounted, permission denied, network error) downgrade
    to a warning instead of aborting the command.
  * When the remote listing succeeds, we cross-reference by hashing
    each local human ID and diffing against the returned key set.
    Local-only snapshots get the existing "stale local record"
    cleanup hint; remote-only keys are surfaced as a single
    "NOTE: N remote snapshot(s) found in backup destination store
    but not in local database" line.

FileStorer construction also no longer does an eager mkdir — the
basePath is recorded and the directory is created lazily on first
write. A missing or unmounted destination during `snapshot list`
should NOT block the command, and now it doesn't.

RemoveAllSnapshots is rewritten to drive deletion from the local
index instead of from a remote listing, hashing each local ID to
find the corresponding remote key. Orphan remote keys (no matching
local snapshot) are handled separately and only deleted when
--remote is set. Existing tests are updated to hash storage paths
through the new RemoteSnapshotKey helper.

The hash format is a hard pre-1.0 break: existing remote snapshots
written under the human-ID path scheme are no longer readable; they
need to be either re-uploaded under the new scheme or manually
renamed. There is no fallback path; matching the project policy of
"no migrations pre-1.0."
2026-06-26 01:54:35 +02:00
5ce1dfa39e Restore --cron warning visibility; show destination on blob upload
--cron used to behave like full quiet mode for everything that
wasn't an error: warnings were swallowed both in the structured
log channel (LevelError gate) and at the snapshot terminus (the
"Finished (with N warnings)" line went through ui.Complete, which
is silenced under SetQuiet). A backup that, say, hit Full Disk
Access permission errors on a handful of files and skipped them
via --skip-errors would exit 0 and emit nothing — the cron job
would never page anyone.

--cron now obeys "silent only on total success":

  * log.Initialize raises the cron/quiet log level from Error to
    Warn so log.Warn output still reaches stdout (and therefore
    cron's mail).
  * The post-backup terminus message switches to ui.Warning when
    WarningCount > 0. Warning is not silenced by SetQuiet, so cron
    delivers the summary line whenever the count is non-zero. The
    no-warnings path keeps ui.Complete, which IS silenced under
    cron — that's the success path.

Separately, blob upload UI now names the actual destination
instead of the generic "backup destination store" string. The
Begin/Info lines emit the storer's reported Location (s3://bucket,
file:///mnt/usb/backup, rclone://remote/path, etc.), so anyone
watching a backup can see exactly where each blob is landing.
2026-06-24 08:58:31 +02:00
1f22b9c603 Collapse snapshot prune into vaultik prune; auto-clean on removal
The CLI had two commands named "prune" doing different jobs (local
DB orphan cleanup vs. remote blob garbage collection), which was
confusing and forced a manual two-step workflow after deleting any
snapshot.

Single user-facing prune surface is now `vaultik prune`, which calls
PruneDatabase (local orphan cleanup) then PruneBlobs (remote unref
blob GC). Snapshot deletion paths (snapshot remove, snapshot remove
--all, snapshot purge) auto-run CleanupOrphanedData inline so the
local index database doesn't accumulate ghost rows after every
removal — the user observed ~39k orphaned files and 2 orphaned blobs
after a remove --all because that cleanup was previously a separate
opt-in command. `snapshot prune` is removed.

Also addresses the doc/help-string drift the user audit caught:

  * cli/prune.go help text used to reference a non-existent
    `vaultik purge` command.
  * cli/config.go get/set short/long examples were S3-specific
    (s3.bucket) when the primary storage configuration is
    storage_url.
  * vaultik/info.go printed S3 Bucket/Endpoint/Region labels
    unconditionally; for file:// or rclone:// users those rows
    were empty. The Storage Configuration block now prints the
    storer's Type+Location first, the storage_url string when set,
    and only emits S3 rows that are actually populated.
  * vaultik/info.go's "Run 'vaultik prune --remote'" hint
    referenced a flag that doesn't exist.
  * vaultik/blobcache.go's doc comment claimed LRU eviction, which
    is no longer the restore-time policy (the sweeper drives
    eviction; LRU is the safety-net fallback when maxBytes is
    finite).
  * README.md listed `vaultik restore`, `vaultik snapshot prune`,
    and `s3.bucket` example, all out of date.

README's roadmap section is rewritten with concrete pre-1.0 items
(security audit, error-condition tests, parallel blob downloads,
restart of interrupted restore, …) so the next-steps surface
matches what the project actually still needs.

The cleanup calls are guarded against a nil SnapshotManager so
tests that construct a bare Vaultik struct continue to work.
2026-06-24 08:55:00 +02:00
7ae49a1b2c Stream blobs to disk and restore files in blob-locality order
Three coordinated changes drop restore wall-clock by orders of
magnitude on real-world snapshots and bring memory use back under
control:

  * Streaming download into the disk cache. New
    blobDiskCache.PutFromReader takes an io.Reader and io.Copy's it
    straight into the cache file. The old downloadBlob path did
    io.ReadAll on the decrypted plaintext stream — for a 50 GB blob
    that meant 50 GB in RAM before the cache write. The whole chain
    (Storage.Get → age.Decrypt → zstd.NewReader → io.Copy) is now
    fully streaming; peak RAM per blob is bounded by ~64 KB of
    internal age/zstd buffers plus the io.Copy buffer.

  * Chunk extraction via ReadAt. After a blob is on disk, restore
    reads chunks via blobDiskCache.ReadAt(hash, offset, length) so
    only the chunk's bytes ever touch RAM. The previous code path
    called blobCache.Get for every cache-hit chunk, which read the
    entire blob (e.g. 10 GB) from disk into a []byte just to slice
    out a few KB — single-handedly ~900 ms per cache hit on the
    user's photo snapshot.

  * Locality-aware restore ordering. New restorePlan indexes
    file→blob_set and blob→file_set at restore start, then drives
    the loop so that every file whose full blob set is on disk is
    drained before any new blob downloads. After the drain queue
    empties, the planner picks the pending file with the smallest
    uncached-blob count, downloads those blobs, and drains again.
    A sweep is forced right before each download so the just-
    completed blob is evicted before the next one is Put, keeping
    peak disk-cache occupancy at 1 for path-order-adversarial
    snapshots.

The restore hot path also moves onto a restoreSession struct so
restoreFile/restoreRegularFile/etc. take only the per-call file
argument instead of threading 9+ parameters through every signature.
The new BlobRepository.GetAll method lets the session build a single
blob-id → blob-hash map at start instead of doing one DB query per
chunk.

TestRestoreLocalityAndReadAt passes: peak_len ≤ 1, get_calls = 0,
readat_calls > 0, every blob fetched exactly once.
2026-06-24 08:33:22 +02:00
a92b1a82ad Add failing test for restore blob-cache locality and ReadAt usage
Captures three behaviors the restore hot path must exhibit but
currently doesn't, all under one test:

  * Peak blob disk cache occupancy ≤ 1. Smart restore ordering should
    drain every file referencing the currently-cached blob before
    downloading the next one, so the sweeper can free each blob the
    moment its file set is exhausted.
  * Every remote blob fetched exactly once (counter on a wrapping
    Storer). Already true today; the test pins it so neither future
    cache-eviction nor reorder regressions can introduce
    re-downloads.
  * blobDiskCache.Get is never called during restore — chunk
    extraction must go through ReadAt so we never read the whole
    blob from disk to slice out a few KB. The 10 GB
    photo-snapshot --debug output showed ~900 ms per cache-hit chunk
    extract; ReadAt should bring that to sub-millisecond.

Adds Get/ReadAt call counters and a peak-Len tracker to
blobDiskCache, plus an internal restoreCacheObserver hook on Vaultik
so the test can capture the production cache instance without
exporting unexported types.

Currently fails with peak_len=3, get_calls=24, readat_calls=0. The
fix follows in subsequent commits.
2026-06-17 08:14:55 +02:00
39d5d21d48 Revert "Merge fix/restore-cache-readat"
All checks were successful
check / check (push) Successful in 4s
This reverts commit 44c9008e7e, reversing
changes made to b55d5763ad.
2026-06-17 08:01:56 +02:00
8036d93914 Read chunks from cached blobs via ReadAt instead of full-blob Get
Restore's per-chunk loop called blobCache.Get(blobHash) and sliced the
returned []byte to extract the chunk it actually wanted. Get reads the
entire blob from disk into memory — so for a 10 GB blob, every chunk
extraction was a 10 GB ReadFile to get back a few KB. With ~40k files
needing ~600ms per cache hit, that alone was burning ~6 hours of
wall-clock on a real restore.

Hot loop now:
  - If the blob isn't cached: download (full plaintext into memory),
    Put to disk cache, satisfy this chunk from the in-memory buffer.
  - If it's cached: blobCache.ReadAt(hash, offset, length) — reads
    only the chunk's bytes from the on-disk blob file.

ReadAt was already implemented on blobDiskCache; restore just wasn't
using it.

Debug timings from the user's photo-catalog restore showed
ms_cache_gets dominating every cache-hit file at 500-1000ms. With
ReadAt those should drop to sub-millisecond and the visible throughput
should be bound by single-stream blob download + decrypt/decompress
rather than disk-read amplification.
2026-06-17 07:58:01 +02:00
53febb48d2 Replace restore progress bars with periodic ui.Progress lines
Restore and verify no longer use schollz/progressbar. Instead they emit
a periodic status line every 15 s via ui.Writer.Progress, matching the
cadence and shape of the snapshot create scanner output. The lines
include files done, byte counts, throughput in bits/sec, elapsed,
absolute ETA, and remaining duration — same conventions as snapshot
create. The progressbar dependency, the newProgressBar/isTerminal
helpers, and the unused printfStderr helper are removed; go.mod loses
schollz/progressbar plus its colorstring and uniseg transitive deps.

Adds --debug timing instrumentation throughout the restore hot path so
the next slow-restore report can pinpoint which stage is the
bottleneck. Per-file: file-chunks query, output Create, per-chunk blob
DB lookups, cache get/put, blob download, chunk write, sweeper call.
Per-blob-download: fetch-setup (Get + Stat) vs read+decrypt+decompress
vs close-and-verify. FetchBlob splits the Storage.Get and Storage.Stat
round-trips so an expensive size-stat is visible separately.
2026-06-17 07:54:14 +02:00
d9319dc0fb Add integration test for restore sweeper
Writes 30 random 1 MB files plus 10 duplicates (40 files, 30 MB of
unique content), backs them up with a 10 MB blob_size_limit, then
restores through a counting storer that records every Get per key.
Each blob on disk must be downloaded exactly once during restore — a
re-download would mean the sweeper evicted a blob whose chunks were
still referenced by an unrestored file, and zero downloads would mean
the cache silently stopped being consulted.

The duplicates exercise the dedup path: the sweeper has to keep each
blob alive until every file (original AND duplicate) that references
any of its chunks has been restored.
2026-06-17 07:20:07 +02:00
683fb0b103 Replace LRU eviction in restore with reference-counted sweeper
Restore previously capped the blob disk cache at 4× the configured
blob_size_limit (so 40 GB by default). With large or heavily-deduped
snapshots a chunk-by-chunk file walk could blow past that cap and
trigger LRU eviction of blobs that were still needed by later files,
forcing repeated re-downloads — observed during a real restore as
single-stream throughput collapsing to under 1 MB/s.

Restore now allocates the cache with no practical size cap and drives
eviction explicitly:

  * An in-memory set of restored file IDs accumulates as files finish.
  * Every blob_size_limit/100 bytes of restored data (≈100 sweeps per
    blob's worth of writes) the sweeper iterates the cache. For each
    cached blob it queries the snapshot's local SQLite DB for every
    file that references any chunk in the blob and deletes the cache
    entry only when every such file is already in the restored set.
  * blobStillNeeded returns true on any error so an unreadable DB
    never causes premature eviction.

The cache itself gains Delete(key) and Keys() so the sweeper can drive
removal without touching internal LRU state.
2026-06-17 07:15:22 +02:00
a63c729fbc Print banner before cobra parsing; route arg errors through ui.Error
Two output-style fixes plus a quiet-mode correction.

Banner: a manual scan of os.Args in CLIEntry decides whether to suppress
the banner (--quiet/-q/--cron), then prints it before cobra parses any
arguments. This makes the banner appear even when cobra rejects bad args
("requires at least 2 arg(s)") and on --help — paths that previously
skipped PersistentPreRun entirely. The cobra-side hook plumbing (sync.Once,
PersistentPreRun, custom HelpFunc) is removed.

Errors: rootCmd.SilenceErrors = true so cobra no longer prints its own
"Error: <msg>" line. Any error returned from Execute() goes through
ui.New(os.Stderr).Error(...), giving the documented "🛑 ERROR: <msg>"
format. A new helper cli.ReportError() formats errors from goroutine
paths that can't return through cobra's normal return chain; every
CLI command's fx-goroutine error path now calls it alongside the
existing structured log.Error so both channels record the failure.

Quiet mode: previously --quiet/--cron swapped Vaultik.UI to io.Discard,
which silenced Warning and Error messages too — contradicting the
documented "suppresses non-error output" semantics. ui.Writer now has
a SetQuiet flag that drops Begin/Complete/Info/Notice/Detail/Progress/
Banner only; Warning and Error always emit.

Also folds in restore.go cleanups the audit flagged: the hardcoded
"WARNING:" prefix on the failed-files block now uses ui.Warning +
ui.Detail, the post-restore "Restored N files" line uses ui.Complete,
and the "No files found to restore" branch emits both log.Warn and
ui.Warning so structured logs continue to capture it under --verbose.
2026-06-17 06:56:34 +02:00
0e9c96c8b5 Add uncompressed-size and new-chunk-size columns to snapshot list
The remote snapshot table now shows the total plaintext size of all
chunks referenced by each snapshot, plus the plaintext size of chunks
newly referenced by that snapshot (chunks not in any earlier completed
snapshot known to the local DB). The latter is the marginal data
introduced by each backup — useful for spotting which snapshots
actually added bytes vs. dedup'd against prior state.

Both new columns are computed from the local database only. Snapshots
that exist in remote storage but not in the local DB show
"<remote only>" in those cells; their COMPRESSED SIZE column still
reflects the value fetched from the remote manifest.
2026-06-17 06:33:59 +02:00
7a0d5bfd73 Move restore to snapshot restore subcommand
Renames the top-level `restore` command to `vaultik snapshot restore`
for consistency with `vaultik snapshot create`. The factory follows the
sibling pattern (newSnapshotRestoreCommand) and its file is renamed to
snapshot_restore.go to match.
2026-06-17 06:27:44 +02:00
e75367c594 Add 'vaultik remote nuke', rename Processing→Backing up, bits/sec rates
remote nuke: new subcommand that deletes every snapshot's metadata and
every blob from remote storage, leaving the bucket prefix empty.
Requires --force.

User-facing 'Processing' is now 'Backing up' everywhere it referred to
the chunking/upload phase. Files summary line says 'backed up' instead
of 'processed'.

ui.Speed now formats bytes/sec input as bits/sec output (bit/s, Kbit/s,
Mbit/s, Gbit/s). Network transfer rates are conventionally expressed
in bits — the per-blob heartbeat now matches the per-snapshot summary
line which has always been bits/sec.
2026-06-17 06:21:21 +02:00
132f7149ca Populate snapshot_blobs for dedup-referenced blobs at completion
The bug: fully-deduplicated snapshots (every chunk already in storage
from a prior run) had an empty snapshot_blobs table. The metadata-
export pipeline then dropped all blob/blob_chunks rows from the
exported database, leaving file_chunks references to chunks whose
blobs were no longer recorded. Restore fails on every file with
"chunk X not found in any blob".

Fix: at CompleteSnapshot time, run an INSERT OR IGNORE that links
every blob holding a chunk referenced by this snapshot's files into
snapshot_blobs. New blobs uploaded during the snapshot are already
recorded (no-op for them); dedup-referenced blobs are added.

The cleanup query in deleteOrphanedBlobs already restricts to
snapshot_blobs entries for the current snapshot — so once
snapshot_blobs is correctly populated, the exported database
contains the full set of blob/blob_chunks rows needed for restore.

Regression test: TestDedupOnlySnapshotRestores creates two
identical snapshots (the second uploads zero new blobs) and
restores the second. Without the fix, restore fails on every file.
2026-06-17 06:05:52 +02:00
d8edf90fac Restore fails fast on first error; --skip-errors is now global
restore aborts on the first per-file failure by default, surfacing
the file path and the underlying error and suggesting --skip-errors
to continue past failures.

--skip-errors moved from a 'snapshot create' subcommand flag to a
top-level persistent flag on the root command. It applies to both
snapshot create and restore. Old 'vaultik snapshot create --skip-
errors' still works because persistent flags are inherited.
2026-06-17 06:02:15 +02:00
9f537b9c4c Print startup banner on every invocation (except -q / --cron)
Adds maybePrintBanner() called from three cobra hooks:
  - PersistentPreRun on root: covers every subcommand invocation
  - Custom HelpFunc on root: covers --help and group-level help
  - Run on root: covers bare 'vaultik' with no subcommand

bannerOnce sync.Once ensures the banner prints exactly once per
process regardless of which hook(s) fire.

Removed the duplicate banner-print from fx setupGlobals; that hook
still handles the --cron/--quiet UI swap for the rest of the output.
2026-06-17 05:57:21 +02:00
3113014b58 Print banner when vaultik is invoked with no subcommand
Cobra's default 'no subcommand → print help' path bypasses fx, so
the startup banner never ran for bare 'vaultik'. Add a Run handler
on the root command that prints the banner and then calls Help.

Extracted the banner-printing logic into writeStartupBanner() so
both this path and the fx setupGlobals hook share one implementation.
2026-06-17 05:54:48 +02:00
75564a504e Bold the startup banner on TTY; blank line after banner 2026-06-17 05:52:03 +02:00