All user-facing output now goes through a single ui.Writer with a
uniform style:
》 (white) for begin / info / notice
》 (green) for complete / success
Warning: for warnings (orange)
ERROR: for errors (red)
》 (indented) for progress heartbeats
Color is enabled when stdout is a TTY and NO_COLOR is unset.
Standards:
- Complete-sentence messages with fully qualified terms ("backup
destination store", "local index database", "snapshot source
files enumeration").
- Every Complete has a matching Begin.
- Natural verb tense conveys state ("Uploading" -> "Uploaded"). The
words "begin"/"complete" never appear in message bodies; the marker
color carries that information.
- ETA means clock time, not duration. Progress lines say "estimated
remaining time (<dur>), finish at <time>" with both labeled.
Adds globals.CommitDate (populated by Makefile/Dockerfile/goreleaser
via ldflags from `git show -s --format=%cI HEAD`) and a startup banner
printed once per invocation.
Strips fx call-chain noise from startup errors so users see the actual
underlying error (e.g. "creating base path: mkdir /Volumes/BACKUPS:
permission denied" instead of three layers of "could not build
arguments for function ...").
README documents the output style and the ui package conventions.
Long-running uploads (multi-GB blobs over slow links) previously
produced silence between the start of the upload and the "Blob
stored" line at the end. Now we print:
Uploading blob: <hash> (<size>)
before the upload starts, and a heartbeat line at most every 15s:
uploading <hash>: <done>/<total> (NN%), <speed>/sec, <elapsed> elapsed, ETA <eta>
This gives the user visible progress on large uploads, especially
over SMB or remote storage where 10+ second stalls are normal.
ResolveConfigPath now stats explicit paths from --config and
$VAULTIK_CONFIG and produces an actionable error naming the bad
path and suggesting 'vaultik config init' (with the right path
in the --config case). The default-search failure message lists
the paths it tried.
The scanner no longer hard-codes os.Stdout vs io.Discard based on
EnableProgress. ScannerConfig and ScannerParams take an explicit
Output io.Writer, and the Vaultik caller passes v.Stdout — which
itself is set to io.Discard in --cron mode. One knob controls
both scanner-level and Vaultik-level user-facing output.
The version command prints a hint when Version == "dev" telling
the user this is a development build without embedded version
metadata.
When the scanner hits a permission-denied error (TCC-protected
directories on macOS without Full Disk Access, or any other EPERM),
the error now names the offending path and includes platform-specific
remediation instructions. On macOS it points the user at System
Settings -> Privacy & Security -> Full Disk Access. On other
platforms it suggests --skip-errors.
The error wraps os.ErrPermission so errors.Is still works for callers
that care about the underlying error.
README quickstart and snapshot create docs now mention the macOS FDA
requirement.
Module path changed from git.eeqj.de/sneak/vaultik to
sneak.berlin/go/vaultik (vanity redirect). All imports, ldflags,
Dockerfile, goreleaser config, and docs updated. App data/config
directories now use plain "vaultik" instead of the reverse-DNS name.
README:
- New copy-pasteable quickstart at top: go install, config init,
age keypair, config set for key + file:// destination, home backup
- All command names in command details are code-quoted
- config set/get gained sequence index support (age_recipients.0)
so lists are settable from the CLI
- Dockerfile build is CGO_ENABLED=0 to match the pure-Go build
These test files existed locally and ran in the suite but were never
committed due to the old .gitignore 'vaultik' pattern matching the
internal/vaultik/ directory.
The config command group manages the config file:
config init - write default config (moved from top-level init)
config edit - open the config in $EDITOR (falls back to vi)
config get - print a value by dotted YAML path (s3.bucket)
config set - set a scalar value by dotted YAML path
get/set operate on the yaml.Node tree so comments and formatting in
the config file are preserved across edits. set creates intermediate
maps as needed.
Removed /.docker (small, contains registry auth config worth keeping)
and /Library/Parallels (small app support; the actual VM disks live in
~/Parallels) from the default excludes.
Added recommended excludes for data that should never be backed up:
- Language/toolchain caches (npm, cargo, rustup, go modules, maven,
vagrant boxes, node_modules, __pycache__, .venv)
- VM disk images (Parallels, VMware Fusion, VirtualBox, OrbStack, UTM)
- Downloaded LLM models (ollama, LM Studio)
- Cloud-synced storage (~/Library/CloudStorage, iCloud Drive) — synced
elsewhere, and dataless placeholder files would be force-downloaded
- Android SDK and emulator images
The init-generated config now ships with a comprehensive home snapshot
exclude list (caches, trash, cloud-synced data, rebuildable app state,
device backups) derived from a battle-tested rsync backup script, plus
an apps snapshot for /Applications excluding Apple-redownloadable apps
(Safari, GarageBand, iWork, iMovie) and large third-party installs.
Obsolete pre-Catalina app entries (Dashboard, iTunes, DVD Player, etc.)
were dropped — OS apps live in /System/Applications on modern macOS and
never appear in /Applications.
Adds a test asserting the template parses as valid YAML with the
expected snapshot structure.
New init command writes a default config file with commented
explanations for every setting. Uses XDG config directory via
github.com/adrg/xdg for platform-appropriate paths:
macOS: ~/Library/Application Support/vaultik/config.yml
Linux: ~/.config/vaultik/config.yml
root: /etc/vaultik/config.yml
Config resolution now searches the XDG path before /etc/vaultik/.
Refuses to overwrite an existing file. Created with 0600 permissions.
README quickstart rewritten as a single copy-pasteable shell block
walking through install, keygen, init, edit, first backup, verify,
and cron setup.
--cron now sets Vaultik.Stdout to io.Discard so all user-facing output
is suppressed, not just the scanner progress. Errors still go to stderr
via the structured logger.
snapshot list now warns when local snapshot records have no matching
remote metadata, and suggests 'vaultik snapshot cleanup' instead of
silently deleting them.
snapshot cleanup is a new subcommand that explicitly removes stale
local snapshot records. syncWithRemote (used by purge) still does
this automatically since purge is already destructive.
.gitignore changed from 'vaultik' to '/vaultik' so it only matches
the binary at the repo root, not the internal/vaultik/ directory.
snapshot create --prune now accepts --keep-newer-than <duration> (e.g.
4w, 30d, 6mo) to keep a rolling window of snapshots instead of only
the latest. Supports d/w/mo/y units and combinations (2w3d).
Without --keep-newer-than, --prune still defaults to keep-latest-only.
README now covers: storage backends (s3/file/rclone), all CLI commands
with full flag docs, configuration reference table, architecture overview,
roadmap (post-1.0 only), and development workflow.
TODO.md removed — completed items dropped, remaining roadmap items
merged into README.
ARCHITECTURE.md updated: correct snapshot ID format, storage.Storer
instead of s3.Client, binary SQLite export instead of SQL dump.
Scanner now records symlinks (with their target) and directories
during the walk phase instead of skipping them. processFileStreaming
detects non-regular entries and writes the DB record without chunking.
The e2e test (TestEndToEndFileStorage) now verifies:
- Symlink target preserved through backup→restore
- Empty directory survives round-trip
- File permissions (0600) restored correctly
Scanner now writes all user-facing output to an io.Writer (os.Stdout
when progress is enabled, io.Discard in --cron mode). This fixes the
long-standing issue where --cron still printed progress lines.
S3 HeadObject now properly distinguishes not-found from other errors
instead of swallowing all errors as not-found.
Config/CLI error messages include actionable hints (where to find the
config, how to generate keys, what storage options exist).
- Adopt origin's SnapshotPurgeOptions naming and PurgeSnapshotsWithOptions
method, but extend with Names []string (repeatable --snapshot flag) and
Quiet bool for use by --prune.
- Adopt origin's parseSnapshotName helper.
- Fold the duplicate post-backup prune block into one runPostBackupPrune
call that filters retention to the snapshot names just backed up.
- Keep the shallow-verify timestamp parsing fix and the dead deep-verify
branch removal; use origin's printVerifyHeader/verifyManifestBlobsExist
helper extraction.
- Drop top-level vaultik purge and verify (duplicates of snapshot purge
and snapshot verify).
- Drop the resurrected daemon block from info.go (config fields no
longer exist).
- Combine Makefile targets: gofmt -l for fmt-check, -race for tests,
release/release-snapshot/docker/hooks/deps/test-coverage all included.
make targets each do one thing now: lint, fmt, fmt-check, test. Use
'make check' for combined lint + fmt-check + test (the standard
pre-commit gate).
Release builds are pure-Go (CGO_ENABLED=0) cross-compiling to
linux/darwin × amd64/arm64.
The --daemon flag, BackupInterval, FullScanInterval, MinTimeBetweenRun
config fields, and DirtyPath model were placeholders for a never-shipped
daemon mode and have been removed. Daemon mode is out of scope for 1.0.
Closes #57
Adopts the [pixa migration pattern](sneak/pixa#36) for schema management. Replaces the monolithic `schema.sql` embed with a numbered migration system.
## Changes
### New: `schema/000.sql` — Bootstrap migration
- Creates `schema_migrations` table with `INTEGER PRIMARY KEY` version column
- Self-contained: includes both `CREATE TABLE IF NOT EXISTS` and `INSERT OR IGNORE` for version 0
- Go code does zero INSERTs for bootstrap — just reads and executes 000.sql
### Renamed: `schema.sql` → `schema/001.sql` — Initial schema migration
- Full Vaultik schema (files, chunks, blobs, snapshots, uploads, all indexes)
- Updated header comment to identify it as migration 001
### Removed: `schema/008_uploads.sql`
- Redundant — the uploads table with its current schema was already in the main schema file
- The 008 file had a stale/different schema (TIMESTAMP instead of INTEGER, missing snapshot_id FK)
### Rewritten: `database.go` — Migration engine
- `//go:embed schema/*.sql` replaces `//go:embed schema.sql`
- `bootstrapMigrationsTable()`: checks if `schema_migrations` table exists, applies 000.sql if missing
- `applyMigrations()`: iterates through numbered .sql files, checks `schema_migrations` for each version, applies and records pending ones
- `collectMigrations()`: reads embedded schema dir, returns sorted filenames
- `ParseMigrationVersion()`: extracts numeric version from filenames like `001.sql` or `001_description.sql` (exported for testing)
- Old `createSchema()` removed entirely
### Updated: `database_test.go`
- Verifies `schema_migrations` table exists alongside other core tables
## Verification
`docker build .` passes — formatting, linting, all tests green.
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #58
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
## Summary
`PurgeSnapshots` now applies `--keep-latest` retention per snapshot name instead of globally across all names.
### Problem
Previously, `--keep-latest` would keep only the single most recent snapshot across ALL snapshot names. For example, with snapshots:
- `system_2024-01-15`
- `home_2024-01-14`
- `system_2024-01-13`
`--keep-latest` would keep only `system_2024-01-15` and delete the latest `home` snapshot too.
### Solution
1. **Per-name retention**: `--keep-latest` now groups snapshots by name and keeps the latest of each group. In the example above, both `system_2024-01-15` and `home_2024-01-14` would be kept.
2. **`--name` flag**: New flag to filter purge operations to a specific snapshot name. `--name home --keep-latest` only purges `home` snapshots, leaving all `system` snapshots untouched.
### Changes
- `internal/vaultik/helpers.go`: Add `parseSnapshotName()` to extract the snapshot name from a snapshot ID (`hostname_name_timestamp` format)
- `internal/vaultik/snapshot.go`: Add `SnapshotPurgeOptions` struct with `Name` field, add `PurgeSnapshotsWithOptions()` method, modify `--keep-latest` logic to group by name
- `internal/cli/purge.go` and `internal/cli/snapshot.go`: Add `--name` flag to both purge CLI surfaces
- `README.md`: Update CLI documentation
### Tests
- `helpers_test.go`: Unit tests for `parseSnapshotName()` and `parseSnapshotTimestamp()`
- `purge_per_name_test.go`: Integration tests covering:
- Per-name retention with multiple names
- Single-name retention
- `--name` filter with `--keep-latest`
- `--name` filter with `--older-than`
- No-match name filter (all snapshots retained)
- Legacy snapshots without name component
- Mixed named and legacy snapshots
- Three different snapshot names
### Backward Compatibility
The existing `PurgeSnapshots(keepLatest, olderThan, force)` signature is preserved as a wrapper around the new `PurgeSnapshotsWithOptions()`. The `--prune` flag in `snapshot create` continues to work unchanged.
`docker build .` passes (lint, fmt-check, all tests).
closes [#9](#9)
Co-authored-by: user <user@Mac.lan guest wan>
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #51
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
## Summary
Replace serial `getManifestSize()` calls in `ListSnapshots` with bounded concurrent downloads using `errgroup`. For each remote snapshot not in the local DB, manifest downloads now run in parallel (up to 10 concurrent goroutines) instead of one at a time.
## Changes
- Use `errgroup` with `SetLimit(10)` for bounded concurrency
- Collect remote-only snapshot IDs first, pre-add entries with zero size
- Download manifests concurrently, patch sizes from results
- Remove now-unused `getManifestSize` helper (logic inlined into goroutines)
- Promote `golang.org/x/sync` from indirect to direct dependency
## Testing
- `make check` passes (fmt-check, lint, tests)
- `docker build .` passes
closes #8
Co-authored-by: user <user@Mac.lan guest wan>
Reviewed-on: #50
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Replace linear scan deduplication of snapshot IDs in `RemoveAllSnapshots()` and `PruneBlobs()` with `map[string]bool` for O(1) lookups.
Previously, each new snapshot ID was checked against the entire collected slice via a linear scan, resulting in O(n²) overall complexity. Now a `seen` map provides constant-time membership checks while preserving insertion order in the slice.
**Changes:**
- `internal/vaultik/snapshot.go` (`RemoveAllSnapshots`): replaced linear `for` loop dedup with `seen` map
- `internal/vaultik/prune.go` (`PruneBlobs`): replaced linear `for` loop dedup with `seen` map
closes #12
Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #45
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Remove all ctime from the codebase per sneak's decision on [PR #48](#48).
## Rationale
- ctime means different things on macOS (birth time) vs Linux (inode change time) — ambiguous cross-platform
- Vaultik never uses ctime operationally (scanning triggers on mtime change)
- Cannot be restored on either platform
- Write-only forensic data with no consumer
## Changes
- **Schema** (`internal/database/schema.sql`): Removed `ctime` column from `files` table
- **Model** (`internal/database/models.go`): Removed `CTime` field from `File` struct
- **Database layer** (`internal/database/files.go`): Removed ctime from all INSERT/SELECT queries, ON CONFLICT updates, and scan targets in both `scanFile` and `scanFileRows` helpers; updated `CreateBatch` accordingly
- **Scanner** (`internal/snapshot/scanner.go`): Removed `CTime: info.ModTime()` assignment in `checkFileInMemory()`
- **Tests**: Removed all `CTime` field assignments from 8 test files
- **Documentation**: Removed ctime references from `ARCHITECTURE.md` and `docs/DATAMODEL.md`
`docker build .` passes clean (lint, fmt-check, all tests).
closes#54
Co-authored-by: user <user@Mac.lan guest wan>
Reviewed-on: #55
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Add `ON DELETE CASCADE` to the two foreign keys that were missing it:
- `snapshot_files.file_id` → `files(id)`
- `snapshot_blobs.blob_id` → `blobs(id)`
This ensures that when a file or blob row is deleted, the corresponding snapshot junction rows are automatically cleaned up, consistent with the other CASCADE FKs already in the schema.
closes #19
Co-authored-by: user <user@Mac.lan guest wan>
Reviewed-on: #46
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
