❌ is a thin black-and-white cross that gets lost against terminal
backgrounds and the ANSI red text. 🛑 is a solid red octagon that
reads unmistakably as 'stop/error' at a glance, even when the user
isn't reading the line carefully.
All user-facing output now goes through a single ui.Writer with a
uniform style:
》 (white) for begin / info / notice
》 (green) for complete / success
Warning: for warnings (orange)
ERROR: for errors (red)
》 (indented) for progress heartbeats
Color is enabled when stdout is a TTY and NO_COLOR is unset.
Standards:
- Complete-sentence messages with fully qualified terms ("backup
destination store", "local index database", "snapshot source
files enumeration").
- Every Complete has a matching Begin.
- Natural verb tense conveys state ("Uploading" -> "Uploaded"). The
words "begin"/"complete" never appear in message bodies; the marker
color carries that information.
- ETA means clock time, not duration. Progress lines say "estimated
remaining time (<dur>), finish at <time>" with both labeled.
Adds globals.CommitDate (populated by Makefile/Dockerfile/goreleaser
via ldflags from `git show -s --format=%cI HEAD`) and a startup banner
printed once per invocation.
Strips fx call-chain noise from startup errors so users see the actual
underlying error (e.g. "creating base path: mkdir /Volumes/BACKUPS:
permission denied" instead of three layers of "could not build
arguments for function ...").
README documents the output style and the ui package conventions.
Long-running uploads (multi-GB blobs over slow links) previously
produced silence between the start of the upload and the "Blob
stored" line at the end. Now we print:
Uploading blob: <hash> (<size>)
before the upload starts, and a heartbeat line at most every 15s:
uploading <hash>: <done>/<total> (NN%), <speed>/sec, <elapsed> elapsed, ETA <eta>
This gives the user visible progress on large uploads, especially
over SMB or remote storage where 10+ second stalls are normal.
ResolveConfigPath now stats explicit paths from --config and
$VAULTIK_CONFIG and produces an actionable error naming the bad
path and suggesting 'vaultik config init' (with the right path
in the --config case). The default-search failure message lists
the paths it tried.
The scanner no longer hard-codes os.Stdout vs io.Discard based on
EnableProgress. ScannerConfig and ScannerParams take an explicit
Output io.Writer, and the Vaultik caller passes v.Stdout — which
itself is set to io.Discard in --cron mode. One knob controls
both scanner-level and Vaultik-level user-facing output.
The version command prints a hint when Version == "dev" telling
the user this is a development build without embedded version
metadata.
When the scanner hits a permission-denied error (TCC-protected
directories on macOS without Full Disk Access, or any other EPERM),
the error now names the offending path and includes platform-specific
remediation instructions. On macOS it points the user at System
Settings -> Privacy & Security -> Full Disk Access. On other
platforms it suggests --skip-errors.
The error wraps os.ErrPermission so errors.Is still works for callers
that care about the underlying error.
README quickstart and snapshot create docs now mention the macOS FDA
requirement.
Module path changed from git.eeqj.de/sneak/vaultik to
sneak.berlin/go/vaultik (vanity redirect). All imports, ldflags,
Dockerfile, goreleaser config, and docs updated. App data/config
directories now use plain "vaultik" instead of the reverse-DNS name.
README:
- New copy-pasteable quickstart at top: go install, config init,
age keypair, config set for key + file:// destination, home backup
- All command names in command details are code-quoted
- config set/get gained sequence index support (age_recipients.0)
so lists are settable from the CLI
- Dockerfile build is CGO_ENABLED=0 to match the pure-Go build
These test files existed locally and ran in the suite but were never
committed due to the old .gitignore 'vaultik' pattern matching the
internal/vaultik/ directory.
The config command group manages the config file:
config init - write default config (moved from top-level init)
config edit - open the config in $EDITOR (falls back to vi)
config get - print a value by dotted YAML path (s3.bucket)
config set - set a scalar value by dotted YAML path
get/set operate on the yaml.Node tree so comments and formatting in
the config file are preserved across edits. set creates intermediate
maps as needed.
Removed /.docker (small, contains registry auth config worth keeping)
and /Library/Parallels (small app support; the actual VM disks live in
~/Parallels) from the default excludes.
Added recommended excludes for data that should never be backed up:
- Language/toolchain caches (npm, cargo, rustup, go modules, maven,
vagrant boxes, node_modules, __pycache__, .venv)
- VM disk images (Parallels, VMware Fusion, VirtualBox, OrbStack, UTM)
- Downloaded LLM models (ollama, LM Studio)
- Cloud-synced storage (~/Library/CloudStorage, iCloud Drive) — synced
elsewhere, and dataless placeholder files would be force-downloaded
- Android SDK and emulator images
The init-generated config now ships with a comprehensive home snapshot
exclude list (caches, trash, cloud-synced data, rebuildable app state,
device backups) derived from a battle-tested rsync backup script, plus
an apps snapshot for /Applications excluding Apple-redownloadable apps
(Safari, GarageBand, iWork, iMovie) and large third-party installs.
Obsolete pre-Catalina app entries (Dashboard, iTunes, DVD Player, etc.)
were dropped — OS apps live in /System/Applications on modern macOS and
never appear in /Applications.
Adds a test asserting the template parses as valid YAML with the
expected snapshot structure.
New init command writes a default config file with commented
explanations for every setting. Uses XDG config directory via
github.com/adrg/xdg for platform-appropriate paths:
macOS: ~/Library/Application Support/vaultik/config.yml
Linux: ~/.config/vaultik/config.yml
root: /etc/vaultik/config.yml
Config resolution now searches the XDG path before /etc/vaultik/.
Refuses to overwrite an existing file. Created with 0600 permissions.
README quickstart rewritten as a single copy-pasteable shell block
walking through install, keygen, init, edit, first backup, verify,
and cron setup.
--cron now sets Vaultik.Stdout to io.Discard so all user-facing output
is suppressed, not just the scanner progress. Errors still go to stderr
via the structured logger.
snapshot list now warns when local snapshot records have no matching
remote metadata, and suggests 'vaultik snapshot cleanup' instead of
silently deleting them.
snapshot cleanup is a new subcommand that explicitly removes stale
local snapshot records. syncWithRemote (used by purge) still does
this automatically since purge is already destructive.
.gitignore changed from 'vaultik' to '/vaultik' so it only matches
the binary at the repo root, not the internal/vaultik/ directory.
snapshot create --prune now accepts --keep-newer-than <duration> (e.g.
4w, 30d, 6mo) to keep a rolling window of snapshots instead of only
the latest. Supports d/w/mo/y units and combinations (2w3d).
Without --keep-newer-than, --prune still defaults to keep-latest-only.
Scanner now records symlinks (with their target) and directories
during the walk phase instead of skipping them. processFileStreaming
detects non-regular entries and writes the DB record without chunking.
The e2e test (TestEndToEndFileStorage) now verifies:
- Symlink target preserved through backup→restore
- Empty directory survives round-trip
- File permissions (0600) restored correctly
Scanner now writes all user-facing output to an io.Writer (os.Stdout
when progress is enabled, io.Discard in --cron mode). This fixes the
long-standing issue where --cron still printed progress lines.
S3 HeadObject now properly distinguishes not-found from other errors
instead of swallowing all errors as not-found.
Config/CLI error messages include actionable hints (where to find the
config, how to generate keys, what storage options exist).
- Adopt origin's SnapshotPurgeOptions naming and PurgeSnapshotsWithOptions
method, but extend with Names []string (repeatable --snapshot flag) and
Quiet bool for use by --prune.
- Adopt origin's parseSnapshotName helper.
- Fold the duplicate post-backup prune block into one runPostBackupPrune
call that filters retention to the snapshot names just backed up.
- Keep the shallow-verify timestamp parsing fix and the dead deep-verify
branch removal; use origin's printVerifyHeader/verifyManifestBlobsExist
helper extraction.
- Drop top-level vaultik purge and verify (duplicates of snapshot purge
and snapshot verify).
- Drop the resurrected daemon block from info.go (config fields no
longer exist).
- Combine Makefile targets: gofmt -l for fmt-check, -race for tests,
release/release-snapshot/docker/hooks/deps/test-coverage all included.
The --daemon flag, BackupInterval, FullScanInterval, MinTimeBetweenRun
config fields, and DirtyPath model were placeholders for a never-shipped
daemon mode and have been removed. Daemon mode is out of scope for 1.0.
Closes #57
Adopts the [pixa migration pattern](sneak/pixa#36) for schema management. Replaces the monolithic `schema.sql` embed with a numbered migration system.
## Changes
### New: `schema/000.sql` — Bootstrap migration
- Creates `schema_migrations` table with `INTEGER PRIMARY KEY` version column
- Self-contained: includes both `CREATE TABLE IF NOT EXISTS` and `INSERT OR IGNORE` for version 0
- Go code does zero INSERTs for bootstrap — just reads and executes 000.sql
### Renamed: `schema.sql` → `schema/001.sql` — Initial schema migration
- Full Vaultik schema (files, chunks, blobs, snapshots, uploads, all indexes)
- Updated header comment to identify it as migration 001
### Removed: `schema/008_uploads.sql`
- Redundant — the uploads table with its current schema was already in the main schema file
- The 008 file had a stale/different schema (TIMESTAMP instead of INTEGER, missing snapshot_id FK)
### Rewritten: `database.go` — Migration engine
- `//go:embed schema/*.sql` replaces `//go:embed schema.sql`
- `bootstrapMigrationsTable()`: checks if `schema_migrations` table exists, applies 000.sql if missing
- `applyMigrations()`: iterates through numbered .sql files, checks `schema_migrations` for each version, applies and records pending ones
- `collectMigrations()`: reads embedded schema dir, returns sorted filenames
- `ParseMigrationVersion()`: extracts numeric version from filenames like `001.sql` or `001_description.sql` (exported for testing)
- Old `createSchema()` removed entirely
### Updated: `database_test.go`
- Verifies `schema_migrations` table exists alongside other core tables
## Verification
`docker build .` passes — formatting, linting, all tests green.
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #58
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
## Summary
`PurgeSnapshots` now applies `--keep-latest` retention per snapshot name instead of globally across all names.
### Problem
Previously, `--keep-latest` would keep only the single most recent snapshot across ALL snapshot names. For example, with snapshots:
- `system_2024-01-15`
- `home_2024-01-14`
- `system_2024-01-13`
`--keep-latest` would keep only `system_2024-01-15` and delete the latest `home` snapshot too.
### Solution
1. **Per-name retention**: `--keep-latest` now groups snapshots by name and keeps the latest of each group. In the example above, both `system_2024-01-15` and `home_2024-01-14` would be kept.
2. **`--name` flag**: New flag to filter purge operations to a specific snapshot name. `--name home --keep-latest` only purges `home` snapshots, leaving all `system` snapshots untouched.
### Changes
- `internal/vaultik/helpers.go`: Add `parseSnapshotName()` to extract the snapshot name from a snapshot ID (`hostname_name_timestamp` format)
- `internal/vaultik/snapshot.go`: Add `SnapshotPurgeOptions` struct with `Name` field, add `PurgeSnapshotsWithOptions()` method, modify `--keep-latest` logic to group by name
- `internal/cli/purge.go` and `internal/cli/snapshot.go`: Add `--name` flag to both purge CLI surfaces
- `README.md`: Update CLI documentation
### Tests
- `helpers_test.go`: Unit tests for `parseSnapshotName()` and `parseSnapshotTimestamp()`
- `purge_per_name_test.go`: Integration tests covering:
- Per-name retention with multiple names
- Single-name retention
- `--name` filter with `--keep-latest`
- `--name` filter with `--older-than`
- No-match name filter (all snapshots retained)
- Legacy snapshots without name component
- Mixed named and legacy snapshots
- Three different snapshot names
### Backward Compatibility
The existing `PurgeSnapshots(keepLatest, olderThan, force)` signature is preserved as a wrapper around the new `PurgeSnapshotsWithOptions()`. The `--prune` flag in `snapshot create` continues to work unchanged.
`docker build .` passes (lint, fmt-check, all tests).
closes [#9](#9)
Co-authored-by: user <user@Mac.lan guest wan>
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #51
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
## Summary
Replace serial `getManifestSize()` calls in `ListSnapshots` with bounded concurrent downloads using `errgroup`. For each remote snapshot not in the local DB, manifest downloads now run in parallel (up to 10 concurrent goroutines) instead of one at a time.
## Changes
- Use `errgroup` with `SetLimit(10)` for bounded concurrency
- Collect remote-only snapshot IDs first, pre-add entries with zero size
- Download manifests concurrently, patch sizes from results
- Remove now-unused `getManifestSize` helper (logic inlined into goroutines)
- Promote `golang.org/x/sync` from indirect to direct dependency
## Testing
- `make check` passes (fmt-check, lint, tests)
- `docker build .` passes
closes #8
Co-authored-by: user <user@Mac.lan guest wan>
Reviewed-on: #50
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Replace linear scan deduplication of snapshot IDs in `RemoveAllSnapshots()` and `PruneBlobs()` with `map[string]bool` for O(1) lookups.
Previously, each new snapshot ID was checked against the entire collected slice via a linear scan, resulting in O(n²) overall complexity. Now a `seen` map provides constant-time membership checks while preserving insertion order in the slice.
**Changes:**
- `internal/vaultik/snapshot.go` (`RemoveAllSnapshots`): replaced linear `for` loop dedup with `seen` map
- `internal/vaultik/prune.go` (`PruneBlobs`): replaced linear `for` loop dedup with `seen` map
closes #12
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #45
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Remove all ctime from the codebase per sneak's decision on [PR #48](#48).
## Rationale
- ctime means different things on macOS (birth time) vs Linux (inode change time) — ambiguous cross-platform
- Vaultik never uses ctime operationally (scanning triggers on mtime change)
- Cannot be restored on either platform
- Write-only forensic data with no consumer
## Changes
- **Schema** (`internal/database/schema.sql`): Removed `ctime` column from `files` table
- **Model** (`internal/database/models.go`): Removed `CTime` field from `File` struct
- **Database layer** (`internal/database/files.go`): Removed ctime from all INSERT/SELECT queries, ON CONFLICT updates, and scan targets in both `scanFile` and `scanFileRows` helpers; updated `CreateBatch` accordingly
- **Scanner** (`internal/snapshot/scanner.go`): Removed `CTime: info.ModTime()` assignment in `checkFileInMemory()`
- **Tests**: Removed all `CTime` field assignments from 8 test files
- **Documentation**: Removed ctime references from `ARCHITECTURE.md` and `docs/DATAMODEL.md`
`docker build .` passes clean (lint, fmt-check, all tests).
closes#54
Co-authored-by: user <user@Mac.lan guest wan>
Reviewed-on: #55
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Add `ON DELETE CASCADE` to the two foreign keys that were missing it:
- `snapshot_files.file_id` → `files(id)`
- `snapshot_blobs.blob_id` → `blobs(id)`
This ensures that when a file or blob row is deleted, the corresponding snapshot junction rows are automatically cleaned up, consistent with the other CASCADE FKs already in the schema.
closes #19
Co-authored-by: user <user@Mac.lan guest wan>
Reviewed-on: #46
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
The `uploads` table's foreign key on `snapshot_id` did not cascade deletes, unlike `snapshot_files` and `snapshot_blobs`. This caused FK violations when deleting snapshots with associated upload records (if FK enforcement is enabled) unless uploads were manually deleted first.
Adds `ON DELETE CASCADE` to the `snapshot_id` FK in `schema.sql` for consistency with the other snapshot-referencing tables.
`docker build .` passes (fmt-check, lint, all tests, build).
closes #18
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #44
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Renames `internal/vaultik/blob_fetch_stub.go` to `internal/vaultik/blob_fetch.go`.
The file contains production code (`hashVerifyReader`, `FetchAndDecryptBlob`), not stubs. The `_stub` suffix was a misnomer from the original implementation in [PR #39](#39).
Pure rename — no code changes. All tests, linting, and formatting pass.
closes#52
Co-authored-by: user <user@Mac.lan guest wan>
Reviewed-on: #53
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
## Summary
`ListSnapshots()` silently deleted local snapshot records not found in remote storage. A list/read operation should not have destructive side effects.
## Changes
1. **Removed destructive sync from `ListSnapshots()`** — the inline loop that deleted local snapshots not present in remote storage has been removed entirely. `ListSnapshots()` now only reads and displays data.
2. **Improved `syncWithRemote()` cascade cleanup** — updated `syncWithRemote()` to use `deleteSnapshotFromLocalDB()` instead of directly calling `Repositories.Snapshots.Delete()`. This ensures proper cascade deletion of related records (`snapshot_files`, `snapshot_blobs`, `snapshot_uploads`) before deleting the snapshot record itself, matching the thorough cleanup that the removed `ListSnapshots` code was doing.
The explicit sync behavior remains available via `syncWithRemote()`, which is called by `PurgeSnapshots()`.
## Testing
- `docker build .` passes (lint, fmt-check, all tests, compilation)
closes #15
Co-authored-by: clawbot <clawbot@eeqj.de>
Reviewed-on: #49
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
## Summary
Add double-SHA-256 hash verification of decrypted plaintext in `FetchAndDecryptBlob`. This ensures blob integrity during restore operations by comparing the computed hash against the expected blob hash before returning data to the caller.
The blob hash is `SHA256(SHA256(plaintext))` as produced by `blobgen.Writer.Sum256()`. Verification happens after decryption and decompression but before the data is used.
## Test
Added `blob_fetch_hash_test.go` with tests for:
- Correct hash passes verification
- Mismatched hash returns descriptive error
## make test output
```
golangci-lint run
0 issues.
ok git.eeqj.de/sneak/vaultik/internal/blob 4.563s
ok git.eeqj.de/sneak/vaultik/internal/blobgen 3.981s
ok git.eeqj.de/sneak/vaultik/internal/chunker 4.127s
ok git.eeqj.de/sneak/vaultik/internal/cli 1.499s
ok git.eeqj.de/sneak/vaultik/internal/config 1.905s
ok git.eeqj.de/sneak/vaultik/internal/crypto 0.519s
ok git.eeqj.de/sneak/vaultik/internal/database 4.590s
ok git.eeqj.de/sneak/vaultik/internal/globals 0.650s
ok git.eeqj.de/sneak/vaultik/internal/models 0.779s
ok git.eeqj.de/sneak/vaultik/internal/pidlock 2.945s
ok git.eeqj.de/sneak/vaultik/internal/s3 3.286s
ok git.eeqj.de/sneak/vaultik/internal/snapshot 3.979s
ok git.eeqj.de/sneak/vaultik/internal/vaultik 4.418s
```
All tests pass, 0 lint issues.
Co-authored-by: user <user@Mac.lan guest wan>
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #39
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
Co-committed-by: clawbot <sneak+clawbot@sneak.cloud>
Add an interactive progress bar (using schollz/progressbar) to the file restore loop, matching the existing pattern in verify. Shows bytes restored with ETA when output is a terminal.
Fixes#20
Co-authored-by: clawbot <clawbot@eeqj.de>
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #23
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
The disk blob cache now uses 4 * BlobSizeLimit from config instead of a
hardcoded 10 GiB default. This ensures the cache scales with the
configured blob size.
The --prune flag on 'snapshot create' was accepted but silently did nothing
(TODO stub). This connects it to actually:
1. Purge old snapshots (keeping only the latest) via PurgeSnapshots
2. Remove unreferenced blobs from storage via PruneBlobs
The pruning runs after all snapshots complete successfully, not per-snapshot.
Both operations use --force mode (no interactive confirmation) since --prune
is an explicit opt-in flag.
Moved the prune logic from createNamedSnapshot (per-snapshot) to
CreateSnapshot (after all snapshots), which is the correct location.
Adds regression tests for issue #28 (fixed in PR #33) to prevent
reintroduction of the double-close bug in CompressStream.
Tests cover:
- CompressStream with normal input
- CompressStream with large (512KB) input
- CompressStream with empty input
- CompressData close correctness
- Replace bare fmt.Scanln with v.scanStdin() helper in snapshot.go
- Remove duplicate FetchBlob from vaultik.go (canonical version in blob_fetch_stub.go)
- Remove duplicate FetchAndDecryptBlob from restore.go (canonical version in blob_fetch_stub.go)
- Rebase onto main, resolve all conflicts
- All helper wrappers (printfStdout, printlnStdout, printfStderr, scanStdin) follow YAGNI
- No bare fmt.Print*/fmt.Scan* calls remain outside helpers
- make test passes: lint clean, all tests pass
Address all four review concerns on PR #31:
1. Fix missed bare fmt.Println() in VerifySnapshotWithOptions (line 620)
2. Replace all direct fmt.Fprintf(v.Stdout,...) / fmt.Fprintln(v.Stdout,...) /
fmt.Fscanln(v.Stdin,...) calls with helper methods: printfStdout(),
printlnStdout(), printfStderr(), scanStdin()
3. Route progress bar and stderr output through v.Stderr instead of os.Stderr
in restore.go (concern #4: v.Stderr now actually used)
4. Rename exported Outputf to unexported printfStdout (YAGNI: only helpers
actually used are created)
Multiple methods wrote directly to os.Stdout instead of using the injectable
v.Stdout writer, breaking the TestVaultik testing infrastructure and making
output impossible to capture or redirect.
Fixed in: ListSnapshots, PurgeSnapshots, VerifySnapshotWithOptions,
PruneBlobs, outputPruneBlobsJSON, outputRemoveJSON, ShowInfo, RemoteInfo.
Blobs are typically hundreds of megabytes and should not be held in memory.
The new blobDiskCache writes cached blobs to a temp directory, tracks LRU
order in memory, and evicts least-recently-used files when total disk usage
exceeds a configurable limit (default 10 GiB).
Design:
- Blobs written to os.TempDir()/vaultik-blobcache-*/<hash>
- Doubly-linked list for O(1) LRU promotion/eviction
- ReadAt support for reading chunk slices without loading full blob
- Temp directory cleaned up on Close()
- Oversized entries (> maxBytes) silently skipped
Also adds blob_fetch_stub.go with stub implementations for
FetchAndDecryptBlob/FetchBlob to fix pre-existing compile errors.
Previously, deleteSnapshotFromLocalDB logged errors but always returned nil,
causing callers to believe deletion succeeded even when it failed. This could
lead to data inconsistency where remote metadata is deleted while local
records persist.
Now returns the first error encountered, allowing callers to handle failures
appropriately.
Restore previously logged errors for individual files but returned
success even if files failed. Now tracks failed files in RestoreResult,
reports them in the summary output, and returns an error if any files
failed to restore.
Fixes#21
These methods were referenced in main but never defined, causing compilation
failures. They were introduced by merges that assumed dependent PRs were
already merged.
