Files
vaultik/internal/vaultik/snapshot.go
sneak be786fe25e
All checks were successful
check / check (pull_request) Successful in 2m29s
Gate prune's local-cleanup prose on --json and give make build a rule (closes #108)
CleanupLocalSnapshots wrote three prose lines to stdout with no --json
awareness, and they covered every branch of the function, so no input
avoided them: `vaultik prune --json | jq` failed even after the banner
fix. -q never helped either, because printlnStdout and stdoutf write
straight to Vaultik.Stdout and never consult Vaultik.UI, which is what
SetQuiet affects.

The issue offered three fixes and asked for a decision. Taken: thread
*PruneOptions into the function and gate each write on !opts.JSON,
matching PruneBlobs -- its sibling phase, which already takes the same
struct -- along with RemoveSnapshot and remote info, so the package has
one pattern rather than two. Rejected: moving the lines to log.Info,
because the logger's default level is slog.LevelWarn, so that would not
relocate them to stderr, it would delete them from a plain `vaultik
prune`, and the removal of rows from the local index is not something to
narrate only under --verbose. Also rejected: putting the stale-record
count into PruneBlobsResult, whose every field is blob-scoped and which
is produced by the later phase; a prune document covering both phases is
a reasonable thing to want, but that is a schema design question, not a
stream-hygiene fix. The two events are duplicated as log.Info records,
which PruneBlobs already does alongside its own prints, so they survive
on stderr under --verbose.

Also closes #110. `make build` printed "Nothing to be done for 'build'"
and exited 0 without producing a binary: build was listed in .PHONY with
no build: rule anywhere, and declaring a name phony is exactly what
converts make's "No rule to make target" error into a silent success.
Fixed with `build: vaultik`, keeping vaultik: as the file rule. Audited
all 19 .PHONY names: build was the only one without a rule, and vaultik
is correctly absent from .PHONY, being a real file target.

Tests, each verified to fail with the fix reverted rather than assumed
to. CleanupLocalSnapshots leaves stdout untouched under --json in all
three branches (stale records, none, empty index) and still emits every
line without it, so the guard cannot be satisfied by deleting the
output. prune --json runs end to end through Entry, cobra and fx over
the process's real stdout descriptor against a file:// store, asserting
exactly one JSON document, in both the stale and non-stale branches. And
a parse of the Makefile asserts every .PHONY name has a rule and that
build reaches the rule producing the binary, which keeps the audit true
for names added later; it is a parse rather than an invocation of make
because `make test` is what runs it, so shelling back into `make build`
would nest a build inside the test run. The property a parse cannot
establish -- that the recipe still fails when the build fails -- was
verified by hand against a deliberately broken tree: make build exits 2
and produces nothing.

cmd/vaultik gains its first test file, so `make test` now reports 16
packages ok where it reported 15. flagConfig and programName constants
are extracted in the CLI tests because the new argument vector pushed
"--config" and "vaultik" over goconst's threshold.

README's stdout/stderr section described the banner as "the other thing
that writes to stdout", which this defect contradicted; it now states
the contract that holds, which is that stdout under --json is the
document and nothing else, for prune as well as for the other four.
2026-08-09 17:33:06 +00:00

1646 lines
46 KiB
Go

package vaultik
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"time"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// Sentinel errors for snapshot management.
var (
errSnapshotNotInConfig = errors.New("snapshot not found in config")
errNoSnapshotsInConfig = errors.New("no snapshots configured")
errBlobsMissing = errors.New("blobs are missing")
errSnapshotVerifyFailed = errors.New("verification failed")
errRemoveAllNeedsForce = errors.New("--all requires --force")
errInvalidTableName = errors.New("invalid table name")
)
// listRecentLimit caps how many snapshot rows are fetched from the
// local index for listing and purge operations.
const listRecentLimit = 10000
// SnapshotCreateOptions contains options for the snapshot create command
type SnapshotCreateOptions struct {
Cron bool
Prune bool
// KeepNewerThan is used with --prune: keep snapshots newer than this
// duration (e.g. "4w"); default: keep only latest.
KeepNewerThan string
SkipErrors bool // Skip file read errors (log them loudly but continue)
Snapshots []string // Optional list of snapshot names to process (empty = all)
}
// CreateSnapshot executes the snapshot creation operation
func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
overallStartTime := time.Now()
log.Info("Starting snapshot creation",
"version", v.Globals.Version,
"commit", v.Globals.Commit,
"index_path", v.Config.IndexPath,
)
err := v.EnsureStorageBinding()
if err != nil {
return err
}
// Clean up incomplete snapshots FIRST, before any scanning
// This is critical for data safety - see CleanupIncompleteSnapshots for details
hostname := v.Config.Hostname
if hostname == "" {
hostname, _ = os.Hostname()
}
// CRITICAL: This MUST succeed. If we fail to clean up incomplete snapshots,
// the deduplication logic will think files from the incomplete snapshot were
// already backed up and skip them, resulting in data loss.
//
// Prune the database before starting: delete incomplete snapshots and orphaned data.
// This ensures the database is consistent before we start a new snapshot.
// Since we use locking, only one vaultik instance accesses the DB at a time.
_, err = v.PruneDatabase()
if err != nil {
return fmt.Errorf("prune database: %w", err)
}
// Determine which snapshots to process
snapshotNames := opts.Snapshots
if len(snapshotNames) == 0 {
snapshotNames = v.Config.SnapshotNames()
} else {
// Validate requested snapshot names exist
for _, name := range snapshotNames {
if _, ok := v.Config.Snapshots[name]; !ok {
return fmt.Errorf("%w: %q", errSnapshotNotInConfig, name)
}
}
}
if len(snapshotNames) == 0 {
return errNoSnapshotsInConfig
}
// Process each named snapshot
for snapIdx, snapName := range snapshotNames {
err = v.createNamedSnapshot(
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
if err != nil {
return err
}
}
// Print overall summary if multiple snapshots
if len(snapshotNames) > 1 {
v.UI.Completef("All %d snapshots completed in %s.",
len(snapshotNames), v.UI.Duration(time.Since(overallStartTime)))
}
if opts.Prune {
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
if err != nil {
return fmt.Errorf("post-backup prune: %w", err)
}
}
// Terminus must obey the --cron invariant: silent on total
// success only. UI.Complete is dropped in cron/quiet mode (that's
// the success path), but if any warnings fired during the run we
// emit the summary via UI.Warning so cron actually delivers
// something for the user to look at.
if v.UI.WarningCount() > 0 {
v.UI.Warningf("Finished with %d warning(s) — review the output above.",
v.UI.WarningCount())
} else {
v.UI.Completef("Finished successfully.")
}
return nil
}
// runPostBackupPrune drops older snapshots of the given names and removes
// orphan blobs from remote storage. If keepNewerThan is set (e.g. "4w"),
// snapshots newer than that duration are kept. Otherwise only the latest
// snapshot of each name is kept.
func (v *Vaultik) runPostBackupPrune(
snapshotNames []string, keepNewerThan string,
) error {
log.Info("Running post-backup prune",
"snapshots", snapshotNames, "keep_newer_than", keepNewerThan)
v.UI.Beginf("Running post-backup prune.")
purgeOpts := &SnapshotPurgeOptions{
Force: true,
Names: snapshotNames,
Quiet: true,
}
if keepNewerThan != "" {
purgeOpts.OlderThan = keepNewerThan
} else {
purgeOpts.KeepLatest = true
}
err := v.PurgeSnapshotsWithOptions(purgeOpts)
if err != nil {
return fmt.Errorf("purging old snapshots: %w", err)
}
err = v.PruneBlobs(&PruneOptions{Force: true})
if err != nil {
return fmt.Errorf("pruning orphaned blobs: %w", err)
}
return nil
}
// snapshotStats tracks aggregate statistics across directory scans
type snapshotStats struct {
totalFiles int
totalBytes int64
totalChunks int
totalBlobs int
totalBytesSkipped int64
totalFilesSkipped int
totalFilesDeleted int
totalBytesDeleted int64
totalBytesUploaded int64
totalBlobsUploaded int
uploadDuration time.Duration
}
// createNamedSnapshot creates a single named snapshot
func (v *Vaultik) createNamedSnapshot(
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
) error {
snapshotStartTime := time.Now()
if total > 1 {
v.UI.Infof("Snapshot %d/%d: %s.", idx, total, snapName)
}
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
if err != nil {
return err
}
scanner := v.ScannerFactory(snapshot.ScannerParams{
EnableProgress: !opts.Cron,
UI: v.UI,
Fs: v.Fs,
Exclude: v.Config.GetExcludes(snapName),
SkipErrors: opts.SkipErrors,
})
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
if err != nil {
return fmt.Errorf("creating snapshot: %w", err)
}
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
v.UI.Beginf("Creating snapshot %s.", v.UI.Snapshot(snapshotID))
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
if err != nil {
return err
}
v.collectUploadStats(scanner, stats)
err = v.finalizeSnapshotMetadata(snapshotID, stats)
if err != nil {
return err
}
log.Info("Snapshot complete",
"snapshot_id", snapshotID,
"name", snapName,
"files", stats.totalFiles,
"blobs_uploaded", stats.totalBlobsUploaded,
"bytes_uploaded", stats.totalBytesUploaded,
"duration", time.Since(snapshotStartTime))
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
return nil
}
// resolveSnapshotPaths resolves source directories to absolute paths
// with symlink resolution.
func (v *Vaultik) resolveSnapshotPaths(snapName string) ([]string, error) {
snapConfig := v.Config.Snapshots[snapName]
resolvedDirs := make([]string, 0, len(snapConfig.Paths))
for _, dir := range snapConfig.Paths {
absPath, err := filepath.Abs(dir)
if err != nil {
return nil, fmt.Errorf("failed to resolve absolute path for %s: %w", dir, err)
}
resolvedPath, err := filepath.EvalSymlinks(absPath)
if err != nil {
if os.IsNotExist(err) {
resolvedPath = absPath
} else {
return nil, fmt.Errorf(
"failed to resolve symlinks for %s: %w", absPath, err)
}
}
resolvedDirs = append(resolvedDirs, resolvedPath)
}
return resolvedDirs, nil
}
// scanAllDirectories runs the scanner on each resolved directory and
// accumulates stats.
func (v *Vaultik) scanAllDirectories(
scanner *snapshot.Scanner, resolvedDirs []string, snapshotID string,
) (*snapshotStats, error) {
stats := &snapshotStats{}
for i, dir := range resolvedDirs {
select {
case <-v.ctx.Done():
log.Info("Snapshot creation cancelled")
return nil, v.ctx.Err()
default:
}
log.Info("Scanning directory", "path", dir)
v.UI.Beginf("Enumerating snapshot source files in %s (%d of %d).",
v.UI.Path(dir), i+1, len(resolvedDirs))
result, err := scanner.Scan(v.ctx, dir, snapshotID)
if err != nil {
return nil, fmt.Errorf("failed to scan %s: %w", dir, err)
}
stats.totalFiles += result.FilesScanned
stats.totalBytes += result.BytesScanned
stats.totalChunks += result.ChunksCreated
stats.totalBlobs += result.BlobsCreated
stats.totalFilesSkipped += result.FilesSkipped
stats.totalBytesSkipped += result.BytesSkipped
stats.totalFilesDeleted += result.FilesDeleted
stats.totalBytesDeleted += result.BytesDeleted
log.Info("Directory scan complete",
"path", dir,
"files", result.FilesScanned,
"files_skipped", result.FilesSkipped,
"bytes", result.BytesScanned,
"bytes_skipped", result.BytesSkipped,
"chunks", result.ChunksCreated,
"blobs", result.BlobsCreated,
"duration", result.EndTime.Sub(result.StartTime))
}
return stats, nil
}
// collectUploadStats gathers upload statistics from the scanner's
// progress reporter.
func (v *Vaultik) collectUploadStats(scanner *snapshot.Scanner, stats *snapshotStats) {
if s := scanner.GetProgress(); s != nil {
progressStats := s.GetStats()
stats.totalBytesUploaded = progressStats.BytesUploaded.Load()
stats.totalBlobsUploaded = int(progressStats.BlobsUploaded.Load())
stats.uploadDuration = time.Duration(
progressStats.UploadDurationMs.Load()) * time.Millisecond
}
}
// finalizeSnapshotMetadata updates stats, marks complete, and exports metadata
func (v *Vaultik) finalizeSnapshotMetadata(
snapshotID string, stats *snapshotStats,
) error {
extStats := snapshot.ExtendedBackupStats{
BackupStats: snapshot.BackupStats{
FilesScanned: stats.totalFiles,
BytesScanned: stats.totalBytes,
ChunksCreated: stats.totalChunks,
BlobsCreated: stats.totalBlobs,
BytesUploaded: stats.totalBytesUploaded,
},
BlobUncompressedSize: 0,
CompressionLevel: v.Config.CompressionLevel,
UploadDurationMs: stats.uploadDuration.Milliseconds(),
}
err := v.SnapshotManager.UpdateSnapshotStatsExtended(v.ctx, snapshotID, extStats)
if err != nil {
return fmt.Errorf("updating snapshot stats: %w", err)
}
err = v.SnapshotManager.CompleteSnapshot(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("completing snapshot: %w", err)
}
err = v.SnapshotManager.ExportSnapshotMetadata(
v.ctx, v.Config.IndexPath, snapshotID)
if err != nil {
return fmt.Errorf("exporting snapshot metadata: %w", err)
}
return nil
}
// uploadSpeed returns the average network upload rate as a colorized
// bits/sec string, or "N/A" when there's no usable data.
func (v *Vaultik) uploadSpeed(bytesUploaded int64, duration time.Duration) string {
if bytesUploaded <= 0 || duration <= 0 {
return v.UI.Speed(0)
}
return v.UI.Speed(float64(bytesUploaded) / duration.Seconds())
}
// printSnapshotSummary prints the comprehensive snapshot completion summary
func (v *Vaultik) printSnapshotSummary(
snapshotID string, startTime time.Time, stats *snapshotStats,
) {
snapshotDuration := time.Since(startTime)
totalFilesChanged := stats.totalFiles - stats.totalFilesSkipped
totalBytesAll := stats.totalBytes + stats.totalBytesSkipped
// Get total blob sizes from database
compressedSize, uncompressedSize := v.getSnapshotBlobSizes(snapshotID)
var compressionRatio float64
if uncompressedSize > 0 {
compressionRatio = float64(compressedSize) / float64(uncompressedSize)
} else {
compressionRatio = 1.0
}
v.UI.Completef("Created snapshot %s.", v.UI.Snapshot(snapshotID))
filesMsg := fmt.Sprintf("Files: %s examined, %s backed up, %s unchanged",
v.UI.Count(stats.totalFiles),
v.UI.Count(totalFilesChanged),
v.UI.Count(stats.totalFilesSkipped))
if stats.totalFilesDeleted > 0 {
filesMsg += fmt.Sprintf(", %s deleted", v.UI.Count(stats.totalFilesDeleted))
}
v.UI.Detailf("%s.", filesMsg)
dataMsg := fmt.Sprintf("Data: %s total (%s backed up)",
v.UI.Size(totalBytesAll),
v.UI.Size(stats.totalBytes))
if stats.totalBytesDeleted > 0 {
dataMsg += fmt.Sprintf(", %s deleted", v.UI.Size(stats.totalBytesDeleted))
}
v.UI.Detailf("%s.", dataMsg)
if stats.totalBlobsUploaded > 0 {
v.UI.Detailf("Storage: %s compressed from %s (%.2fx ratio).",
v.UI.Size(compressedSize),
v.UI.Size(uncompressedSize),
compressionRatio)
v.UI.Detailf("Upload: %d blobs, %s in %s (%s).",
stats.totalBlobsUploaded,
v.UI.Size(stats.totalBytesUploaded),
v.UI.Duration(stats.uploadDuration),
v.uploadSpeed(stats.totalBytesUploaded, stats.uploadDuration))
}
v.UI.Detailf("Snapshot create duration: %s.", v.UI.Duration(snapshotDuration))
}
// getSnapshotBlobSizes returns total compressed and uncompressed blob
// sizes for a snapshot.
func (v *Vaultik) getSnapshotBlobSizes(snapshotID string) (int64, int64) {
var compressed, uncompressed int64
blobHashes, err := v.Repositories.Snapshots.GetBlobHashes(v.ctx, snapshotID)
if err != nil {
return 0, 0
}
for _, hash := range blobHashes {
blob, err := v.Repositories.Blobs.GetByHash(v.ctx, hash)
if err == nil && blob != nil {
compressed += blob.CompressedSize
uncompressed += blob.UncompressedSize
}
}
return compressed, uncompressed
}
// SnapshotPurgeOptions contains options for the snapshot purge command.
type SnapshotPurgeOptions struct {
KeepLatest bool // Keep only the most recent snapshot per name
// OlderThan drops snapshots older than this duration (e.g. "30d",
// "6m", "1y").
OlderThan string
Force bool // Skip confirmation prompt
// Names restricts the operation to snapshots with one of these
// names when non-empty.
Names []string
Quiet bool // Suppress informational output (used by --prune flag)
}
// PurgeSnapshotsWithOptions removes old snapshots based on criteria.
// Retention is per-snapshot-name: KeepLatest keeps the latest of EACH configured
// snapshot name, not the latest globally. This prevents `home` and `system`
// snapshots from cannibalizing each other.
func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
err := v.EnsureStorageBinding()
if err != nil {
return err
}
// Sync with remote first
err = v.syncWithRemote()
if err != nil {
return fmt.Errorf("syncing with remote: %w", err)
}
// Get snapshots from local database
dbSnapshots, err := v.Repositories.Snapshots.ListRecent(v.ctx, listRecentLimit)
if err != nil {
return fmt.Errorf("listing snapshots: %w", err)
}
// Build name filter set if --snapshot was specified.
nameFilter := make(map[string]struct{}, len(opts.Names))
for _, n := range opts.Names {
nameFilter[n] = struct{}{}
}
// Collect completed snapshots, applying the name filter.
snapshots := make([]SnapshotInfo, 0, len(dbSnapshots))
for _, s := range dbSnapshots {
if s.CompletedAt == nil {
continue
}
if len(nameFilter) > 0 {
if _, ok := nameFilter[parseSnapshotName(s.ID.String())]; !ok {
continue
}
}
snapshots = append(snapshots, SnapshotInfo{
ID: s.ID,
Timestamp: s.StartedAt,
CompressedSize: s.BlobSize,
})
}
// Sort by timestamp (newest first)
sort.Slice(snapshots, func(i, j int) bool {
return snapshots[i].Timestamp.After(snapshots[j].Timestamp)
})
toDelete, err := selectSnapshotsToPurge(snapshots, opts)
if err != nil {
return err
}
if len(toDelete) == 0 {
if !opts.Quiet {
v.printlnStdout("No snapshots to delete")
}
return nil
}
return v.confirmAndExecutePurge(toDelete, opts.Force, opts.Quiet)
}
// selectSnapshotsToPurge applies the purge retention criteria to the
// newest-first sorted snapshot list and returns the deletion
// candidates.
func selectSnapshotsToPurge(
snapshots []SnapshotInfo, opts *SnapshotPurgeOptions,
) ([]SnapshotInfo, error) {
var toDelete []SnapshotInfo
switch {
case opts.KeepLatest:
// Keep the latest snapshot per snapshot name. Snapshots are sorted
// newest-first, so the first occurrence of each name is kept.
seen := make(map[string]bool)
for _, snap := range snapshots {
name := parseSnapshotName(snap.ID.String())
if seen[name] {
toDelete = append(toDelete, snap)
continue
}
seen[name] = true
}
case opts.OlderThan != "":
duration, err := parseDuration(opts.OlderThan)
if err != nil {
return nil, fmt.Errorf("invalid duration: %w", err)
}
cutoff := time.Now().UTC().Add(-duration)
for _, snap := range snapshots {
if snap.Timestamp.Before(cutoff) {
toDelete = append(toDelete, snap)
}
}
}
return toDelete, nil
}
// confirmAndExecutePurge shows deletion candidates, confirms with the
// user, and deletes snapshots.
func (v *Vaultik) confirmAndExecutePurge(
toDelete []SnapshotInfo, force, quiet bool,
) error {
if !quiet {
v.stdoutf("The following snapshots will be deleted:\n\n")
for _, snap := range toDelete {
v.stdoutf(" %s (%s, %s)\n",
snap.ID,
snap.Timestamp.Format("2006-01-02 15:04:05"),
formatBytes(snap.CompressedSize))
}
}
// Confirm unless --force is used
if !force {
v.stdoutf("\nDelete %d snapshot(s)? [y/N] ", len(toDelete))
var confirm string
_, err := v.scanStdin(&confirm)
if err != nil {
v.printlnStdout("Cancelled")
return nil //nolint:nilerr // treat EOF or read error as "no"
}
if strings.ToLower(confirm) != "y" {
v.printlnStdout("Cancelled")
return nil
}
} else if !quiet {
v.stdoutf("\nDeleting %d snapshot(s) (--force specified)\n", len(toDelete))
}
// Delete snapshots (both local and remote)
for _, snap := range toDelete {
snapshotID := snap.ID.String()
log.Info("Deleting snapshot", "id", snapshotID)
err := v.deleteSnapshotFromLocalDB(snapshotID)
if err != nil {
log.Error("Failed to delete from local database",
"snapshot_id", snapshotID, "error", err)
}
err = v.deleteRemoteSnapshotByKey(snapshot.RemoteSnapshotKey(snapshotID))
if err != nil {
return fmt.Errorf("deleting snapshot %s from remote: %w", snapshotID, err)
}
}
// Tidy up local DB orphans now so users don't have to run a
// separate command after a purge. Guarded against nil for tests
// that don't wire up a SnapshotManager.
if v.SnapshotManager != nil {
err := v.SnapshotManager.CleanupOrphanedData(v.ctx)
if err != nil {
log.Warn("Failed to clean up orphaned local data after purge",
"error", err)
}
}
if !quiet {
v.stdoutf("Deleted %d snapshot(s)\n", len(toDelete))
v.printlnStdout(
"\nNote: Run 'vaultik prune' to clean up unreferenced remote blobs.")
}
return nil
}
// VerifySnapshot checks snapshot integrity
func (v *Vaultik) VerifySnapshot(snapshotID string, deep bool) error {
opts := &VerifyOptions{Deep: deep}
if deep {
return v.RunDeepVerify(snapshotID, opts)
}
return v.VerifySnapshotWithOptions(snapshotID, opts)
}
// VerifySnapshotWithOptions checks snapshot integrity with full options.
// Deep verification is delegated to RunDeepVerify so this function only
// implements the shallow (existence-only) path.
func (v *Vaultik) VerifySnapshotWithOptions(
snapshotID string, opts *VerifyOptions,
) error {
if opts.Deep {
return v.RunDeepVerify(snapshotID, opts)
}
result := &VerifyResult{
SnapshotID: snapshotID,
Mode: "shallow",
}
v.printVerifyHeader(snapshotID, opts)
// Download and parse manifest. The caller supplies a human
// snapshot ID; we hash it to address remote storage.
manifest, err := v.downloadManifestByKey(snapshot.RemoteSnapshotKey(snapshotID))
if err != nil {
if opts.JSON {
result.Status = verifyStatusFailed
result.ErrorMessage = fmt.Sprintf("downloading manifest: %v", err)
return v.outputVerifyJSON(result)
}
return fmt.Errorf("downloading manifest: %w", err)
}
result.BlobCount = manifest.BlobCount
result.TotalSize = manifest.TotalCompressedSize
if !opts.JSON {
v.stdoutf("Snapshot information:\n")
v.stdoutf(" Blob count: %d\n", manifest.BlobCount)
v.stdoutf(" Total size: %s\n", ubytes(manifest.TotalCompressedSize))
if manifest.Timestamp != "" {
t, terr := time.Parse(time.RFC3339, manifest.Timestamp)
if terr == nil {
v.stdoutf(" Created: %s\n",
t.Format("2006-01-02 15:04:05 MST"))
}
}
v.printlnStdout()
// Check each blob exists
v.stdoutf("Checking blob existence...\n")
}
result.Verified, result.Missing, result.MissingSize =
v.verifyManifestBlobsExist(manifest, opts)
return v.formatVerifyResult(result, manifest, opts)
}
// printVerifyHeader prints the snapshot ID and parsed timestamp for
// verification output. Snapshot ID format: hostname[_name]_<RFC3339>
func (v *Vaultik) printVerifyHeader(snapshotID string, opts *VerifyOptions) {
var snapshotTime time.Time
t, err := parseSnapshotTimestamp(snapshotID)
if err == nil {
snapshotTime = t
}
if !opts.JSON {
v.stdoutf("Verifying snapshot %s\n", snapshotID)
if !snapshotTime.IsZero() {
v.stdoutf("Snapshot time: %s\n",
snapshotTime.Format("2006-01-02 15:04:05 MST"))
}
v.printlnStdout()
}
}
// verifyManifestBlobsExist checks that each blob in the manifest exists
// in storage, returning the verified count, missing count, and total
// missing bytes.
func (v *Vaultik) verifyManifestBlobsExist(
manifest *snapshot.Manifest, opts *VerifyOptions,
) (int, int, int64) {
var (
verified, missing int
missingSize int64
)
for _, blob := range manifest.Blobs {
blobPath := fmt.Sprintf("blobs/%s/%s/%s",
blob.Hash[:2], blob.Hash[2:4], blob.Hash)
// Shallow: check existence only (deep verification is handled
// by RunDeepVerify).
_, err := v.Storage.Stat(v.ctx, blobPath)
if err != nil {
if !opts.JSON {
v.stdoutf(" Missing: %s (%s)\n",
blob.Hash, ubytes(blob.CompressedSize))
}
missing++
missingSize += blob.CompressedSize
} else {
verified++
}
}
return verified, missing, missingSize
}
// formatVerifyResult outputs the final verification results as JSON or
// human-readable text.
func (v *Vaultik) formatVerifyResult(
result *VerifyResult, manifest *snapshot.Manifest, opts *VerifyOptions,
) error {
if opts.JSON {
if result.Missing > 0 {
result.Status = verifyStatusFailed
result.ErrorMessage = fmt.Sprintf("%d blobs are missing", result.Missing)
} else {
result.Status = "ok"
}
return v.outputVerifyJSON(result)
}
v.stdoutf("\nVerification complete:\n")
v.stdoutf(" Verified: %d blobs (%s)\n", result.Verified,
ubytes(manifest.TotalCompressedSize-result.MissingSize))
if result.Missing > 0 {
v.stdoutf(" Missing: %d blobs (%s)\n",
result.Missing, ubytes(result.MissingSize))
} else {
v.stdoutf(" Missing: 0 blobs\n")
}
v.stdoutf(" Status: ")
if result.Missing > 0 {
v.stdoutf("FAILED - %d blobs are missing\n", result.Missing)
return fmt.Errorf("%d %w", result.Missing, errBlobsMissing)
}
v.stdoutf("OK - All blobs verified\n")
return nil
}
// outputVerifyJSON outputs the verification result as JSON
func (v *Vaultik) outputVerifyJSON(result *VerifyResult) error {
encoder := json.NewEncoder(v.Stdout)
encoder.SetIndent("", " ")
err := encoder.Encode(result)
if err != nil {
return fmt.Errorf("encoding JSON: %w", err)
}
if result.Status == verifyStatusFailed {
return fmt.Errorf("%w: %s", errSnapshotVerifyFailed, result.ErrorMessage)
}
return nil
}
// CleanupLocalSnapshots removes local snapshot records that have no
// corresponding metadata in remote storage. These are typically left
// behind by incomplete or interrupted backups. Each local snapshot's
// human ID is hashed via RemoteSnapshotKey and compared against the
// remote listing.
//
// It takes the whole *PruneOptions, symmetric with PruneBlobs, because
// it is the other half of one command: Prune runs this phase and then
// that one. Only JSON is read here. Under --json every write below is
// suppressed, because stdout carries the PruneBlobsResult document and
// nothing else — prose ahead of it is what made `vaultik prune --json |
// jq` fail (issue #108). The narration is duplicated as log records,
// which go to stderr and so cannot corrupt the document.
func (v *Vaultik) CleanupLocalSnapshots(opts *PruneOptions) error {
err := v.EnsureStorageBinding()
if err != nil {
return err
}
remoteKeys, err := v.listAllRemoteSnapshotKeys()
if err != nil {
return err
}
remoteSet := make(map[string]bool, len(remoteKeys))
for _, k := range remoteKeys {
remoteSet[k] = true
}
localSnapshots, err := v.Repositories.Snapshots.ListRecent(v.ctx, listRecentLimit)
if err != nil {
return fmt.Errorf("listing local snapshots: %w", err)
}
var removed int
for _, snap := range localSnapshots {
id := snap.ID.String()
if !remoteSet[snapshot.RemoteSnapshotKey(id)] {
log.Info("Removing stale local snapshot record", "snapshot_id", id)
if !opts.JSON {
v.stdoutf("Removing stale local record: %s\n", id)
}
err = v.deleteSnapshotFromLocalDB(id)
if err != nil {
log.Error("Failed to delete local snapshot",
"snapshot_id", id, "error", err)
continue
}
removed++
}
}
log.Info("Reconciled local snapshot records against remote metadata",
"removed", removed, "examined", len(localSnapshots))
if opts.JSON {
return nil
}
if removed == 0 {
v.printlnStdout("No stale local snapshots found.")
} else {
v.stdoutf("Removed %d stale local snapshot record(s).\n", removed)
}
return nil
}
// Helper methods that were previously on SnapshotApp
// downloadManifestByKey fetches the manifest at
// metadata/<remoteKey>/manifest.json.zst. The remoteKey is the double-
// SHA256 derivation produced by snapshot.RemoteSnapshotKey, not the
// human snapshot ID. Callers that have a human ID must hash first.
//
// This is the only place vaultik reads a manifest off the destination
// store, deliberately: the manifest is currently stored compressed but
// unencrypted, which is what lets `snapshot list` enumerate the
// destination on a host holding no private key. Whether to encrypt it
// is open (issue #81), and routing every read through here means that
// decision has exactly one call site to change. Keep it that way — do
// not open metadata/<key>/manifest.json.zst directly elsewhere.
func (v *Vaultik) downloadManifestByKey(remoteKey string) (*snapshot.Manifest, error) {
manifestPath := fmt.Sprintf("metadata/%s/manifest.json.zst", remoteKey)
reader, err := v.Storage.Get(v.ctx, manifestPath)
if err != nil {
return nil, err
}
defer func() { _ = reader.Close() }()
manifest, err := snapshot.DecodeManifest(reader)
if err != nil {
return nil, fmt.Errorf("decoding manifest: %w", err)
}
return manifest, nil
}
func (v *Vaultik) syncWithRemote() error {
log.Info("Syncing with remote snapshots")
// Get all remote snapshot IDs
remoteSnapshots := make(map[string]bool)
objectCh := v.Storage.ListStream(v.ctx, "metadata/")
for object := range objectCh {
if object.Err != nil {
return fmt.Errorf("listing remote snapshots: %w", object.Err)
}
// Extract snapshot ID from paths like metadata/hostname-20240115-143052Z/
parts := strings.Split(object.Key, "/")
if len(parts) >= minSnapshotIDParts &&
parts[0] == metadataDirName && parts[1] != "" {
// Skip macOS resource fork files (._*) and other hidden files
if strings.HasPrefix(parts[1], ".") {
continue
}
remoteSnapshots[parts[1]] = true
}
}
log.Debug("Found remote snapshots", "count", len(remoteSnapshots))
// Get all local snapshots (use a high limit to get all)
localSnapshots, err := v.Repositories.Snapshots.ListRecent(v.ctx, listRecentLimit)
if err != nil {
return fmt.Errorf("listing local snapshots: %w", err)
}
// Remove local snapshots that don't exist remotely
removedCount := 0
for _, snap := range localSnapshots {
snapshotIDStr := snap.ID.String()
if !remoteSnapshots[snapshotIDStr] {
log.Info("Removing local snapshot not found in remote",
"snapshot_id", snap.ID)
err = v.deleteSnapshotFromLocalDB(snapshotIDStr)
if err != nil {
log.Error("Failed to delete local snapshot",
"snapshot_id", snap.ID, "error", err)
} else {
removedCount++
}
}
}
if removedCount > 0 {
log.Info("Removed local snapshots not found in remote", "count", removedCount)
}
return nil
}
// RemoveOptions contains options for the snapshot remove command
type RemoveOptions struct {
Force bool
DryRun bool
JSON bool
LocalOnly bool // Skip remote cleanup; only touch the local index
}
// RemoveResult contains the result of a snapshot removal
//
//nolint:tagliatelle // snake_case is the established JSON output format
type RemoveResult struct {
SnapshotID string `json:"snapshot_id,omitempty"`
SnapshotsRemoved []string `json:"snapshots_removed,omitempty"`
RemoteRemoved bool `json:"remote_removed,omitempty"`
DryRun bool `json:"dry_run,omitempty"`
}
// pruneCommandHint is the exact command suggested after a remove
// completes so the user knows how to clean up the blobs that the
// just-removed snapshot left behind on the destination store.
const pruneCommandHint = "vaultik prune"
// RemoveSnapshot removes a snapshot from the local index database and,
// unless LocalOnly is set, also strips the snapshot's metadata from the
// destination store. Blobs are NOT touched: removing a snapshot's
// blobs requires reading every remaining remote manifest (the remote
// may hold snapshots this host doesn't know about), which is what
// `vaultik prune` is for — the command prints the prune invocation to
// run as a follow-up. When the remote is unreachable the command still
// completes the local-DB removal and warns.
func (v *Vaultik) RemoveSnapshot(
snapshotID string, opts *RemoveOptions,
) (*RemoveResult, error) {
result := &RemoveResult{
SnapshotID: snapshotID,
}
err := v.EnsureStorageBinding()
if err != nil {
return result, err
}
if opts.DryRun {
return v.removeSnapshotDryRun(result, snapshotID, opts)
}
if !opts.Force && !opts.JSON && !v.confirmRemoveSnapshot(snapshotID, opts) {
return result, nil
}
log.Info("Removing snapshot from local database", "snapshot_id", snapshotID)
err = v.deleteSnapshotFromLocalDB(snapshotID)
if err != nil {
return result, fmt.Errorf("removing from local database: %w", err)
}
if !opts.LocalOnly {
result.RemoteRemoved = v.removeSnapshotRemote(snapshotID)
}
if v.SnapshotManager != nil {
err = v.SnapshotManager.CleanupOrphanedData(v.ctx)
if err != nil {
log.Warn("Failed to clean up orphaned local data after removal",
"error", err)
}
}
if opts.JSON {
return result, v.outputRemoveJSON(result)
}
v.stdoutf("Removed snapshot '%s' from local database\n", snapshotID)
if !opts.LocalOnly && result.RemoteRemoved {
v.printlnStdout("Removed snapshot metadata from remote storage")
v.stdoutf("\nNote: The removed snapshot's blobs remain on the remote. "+
"Run '%s' to delete any blobs no longer referenced by any "+
"remaining remote snapshot.\n", pruneCommandHint)
}
return result, nil
}
// removeSnapshotDryRun reports what RemoveSnapshot would do without
// making changes.
func (v *Vaultik) removeSnapshotDryRun(
result *RemoveResult, snapshotID string, opts *RemoveOptions,
) (*RemoveResult, error) {
result.DryRun = true
if opts.JSON {
return result, v.outputRemoveJSON(result)
}
v.stdoutf("Would remove snapshot: %s\n", snapshotID)
if !opts.LocalOnly {
v.printlnStdout("Would also remove snapshot metadata from " +
"remote storage (blobs untouched)")
}
v.printlnStdout("[Dry run - no changes made]")
return result, nil
}
// confirmRemoveSnapshot prompts for confirmation before removing a
// snapshot. Returns false (treating EOF/read errors as "no") unless the
// user answers "y".
func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions) bool {
if opts.LocalOnly {
v.stdoutf("Remove snapshot '%s' from local database "+
"(remote untouched)? [y/N] ", snapshotID)
} else {
v.stdoutf("Remove snapshot '%s' from local database AND its metadata "+
"from remote storage? [y/N] ", snapshotID)
}
var confirm string
_, err := v.scanStdin(&confirm)
if err != nil {
v.printlnStdout("Cancelled")
return false
}
if strings.ToLower(confirm) != "y" {
v.printlnStdout("Cancelled")
return false
}
return true
}
// removeSnapshotRemote strips the snapshot's metadata from the
// destination store, warning and proceeding on failure: the local-DB
// removal has already happened, so the user is told the remote half
// didn't finish and can retry with `vaultik prune` once the destination
// store is reachable. Returns true when the remote removal succeeded.
func (v *Vaultik) removeSnapshotRemote(snapshotID string) bool {
log.Info("Removing snapshot metadata from remote storage",
"snapshot_id", snapshotID)
remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
err := v.deleteRemoteSnapshotByKey(remoteKey)
if err != nil {
log.Warn("Could not remove snapshot metadata from remote storage",
"error", err)
if v.UI != nil {
v.UI.Warningf("Could not remove snapshot metadata from remote: "+
"%v. Run '%s' once the remote is reachable to finish cleanup.",
err, pruneCommandHint)
}
return false
}
return true
}
// RemoveAllSnapshots removes every snapshot known to the local
// database from the local index, and (with --remote) every snapshot
// metadata directory in remote storage. Both sides are processed so a
// "remove --all" leaves nothing behind, even when the local DB and
// remote storage have diverged.
func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) {
err := v.EnsureStorageBinding()
if err != nil {
return nil, err
}
localSnaps, err := v.localSnapshotIDs()
if err != nil {
return nil, fmt.Errorf("listing local snapshots: %w", err)
}
// remoteKeys is the set of metadata/<key>/ subdirectories on the
// destination store; failures are downgraded to a warning so a
// permission-denied or unreachable remote can't block a local-only
// remove.
remoteKeys, remoteErr := v.listAllRemoteSnapshotKeys()
if remoteErr != nil {
log.Warn("Could not list remote snapshots", "error", remoteErr)
v.UI.Warningf("Could not list remote snapshots: %v.", remoteErr)
}
// Anything visible on the remote that doesn't correspond to a
// known local human ID is treated as an orphan key — handled only
// when --remote is in effect.
knownLocalKeys := make(map[string]string, len(localSnaps))
for _, id := range localSnaps {
knownLocalKeys[snapshot.RemoteSnapshotKey(id)] = id
}
var orphanRemoteKeys []string
for _, key := range remoteKeys {
if _, known := knownLocalKeys[key]; !known {
orphanRemoteKeys = append(orphanRemoteKeys, key)
}
}
if len(localSnaps) == 0 && len(orphanRemoteKeys) == 0 {
if !opts.JSON {
v.printlnStdout("No snapshots found")
}
return &RemoveResult{}, nil
}
if opts.DryRun {
return v.handleRemoveAllDryRun(localSnaps, orphanRemoteKeys, opts)
}
return v.executeRemoveAll(localSnaps, orphanRemoteKeys, opts)
}
// localSnapshotIDs returns every snapshot ID present in the local
// index database, sorted for deterministic iteration. Empty slice if
// the database has no Repositories (e.g. tests).
func (v *Vaultik) localSnapshotIDs() ([]string, error) {
if v.Repositories == nil {
return nil, nil
}
snaps, err := v.Repositories.Snapshots.ListRecent(v.ctx, listRecentLimit)
if err != nil {
return nil, err
}
ids := make([]string, 0, len(snaps))
for _, s := range snaps {
ids = append(ids, s.ID.String())
}
sort.Strings(ids)
return ids, nil
}
// listAllRemoteSnapshotKeys collects the hashed remote keys
// (subdirectories under metadata/) currently present in the
// destination store. Returns (nil, err) when the store cannot be
// listed; callers must treat that as "no remote info available," not
// fatal.
func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) {
log.Info("Listing all remote snapshots")
objectCh := v.Storage.ListStream(v.ctx, "metadata/")
seen := make(map[string]bool)
var keys []string
for object := range objectCh {
if object.Err != nil {
return nil, fmt.Errorf("listing remote snapshots: %w", object.Err)
}
parts := strings.Split(object.Key, "/")
if len(parts) >= minSnapshotIDParts &&
parts[0] == metadataDirName && parts[1] != "" {
// Skip macOS resource fork files (._*) and other hidden files
if strings.HasPrefix(parts[1], ".") {
continue
}
if strings.HasSuffix(object.Key, "/") ||
strings.Contains(object.Key, "/manifest.json.zst") {
key := parts[1]
if !seen[key] {
seen[key] = true
keys = append(keys, key)
}
}
}
}
return keys, nil
}
// handleRemoveAllDryRun handles the dry-run mode for removing all snapshots
func (v *Vaultik) handleRemoveAllDryRun(
localSnaps, orphanRemoteKeys []string, opts *RemoveOptions,
) (*RemoveResult, error) {
result := &RemoveResult{DryRun: true}
result.SnapshotsRemoved = append(result.SnapshotsRemoved, localSnaps...)
if !opts.LocalOnly {
result.SnapshotsRemoved = append(
result.SnapshotsRemoved, orphanRemoteKeys...)
}
if !opts.JSON {
v.stdoutf("Would remove %d local snapshot(s):\n", len(localSnaps))
for _, id := range localSnaps {
v.stdoutf(" %s\n", id)
}
if !opts.LocalOnly {
if len(orphanRemoteKeys) > 0 {
v.stdoutf("Would also remove %d orphan remote snapshot key(s):\n",
len(orphanRemoteKeys))
for _, key := range orphanRemoteKeys {
v.stdoutf(" %s\n", key)
}
} else {
v.printlnStdout("Would also remove snapshot metadata from " +
"remote storage (blobs untouched)")
}
}
v.printlnStdout("[Dry run - no changes made]")
}
if opts.JSON {
return result, v.outputRemoveJSON(result)
}
return result, nil
}
// executeRemoveAll deletes every local snapshot and, unless LocalOnly
// is set, every corresponding remote metadata directory plus any
// orphan remote keys. Blobs are not touched; the printed prune-command
// hint is the next step.
func (v *Vaultik) executeRemoveAll(
localSnaps, orphanRemoteKeys []string, opts *RemoveOptions,
) (*RemoveResult, error) {
if !opts.Force {
return nil, errRemoveAllNeedsForce
}
log.Info("Removing all snapshots",
"local_count", len(localSnaps),
"orphan_remote_count", len(orphanRemoteKeys))
result := &RemoveResult{}
remoteErrors := v.removeAllLocalSnapshots(localSnaps, opts, result)
if !opts.LocalOnly {
remoteErrors += v.removeAllOrphanRemotes(orphanRemoteKeys, result)
if remoteErrors == 0 {
result.RemoteRemoved = true
} else if v.UI != nil {
v.UI.Warningf("Some remote metadata deletions failed. Run '%s' "+
"once the remote is healthy to clean up unreferenced blobs.",
pruneCommandHint)
}
}
if v.SnapshotManager != nil {
err := v.SnapshotManager.CleanupOrphanedData(v.ctx)
if err != nil {
log.Warn("Failed to clean up orphaned local data after bulk removal", "error", err)
}
}
if opts.JSON {
return result, v.outputRemoveJSON(result)
}
v.stdoutf("Removed %d snapshot(s)\n", len(result.SnapshotsRemoved))
if !opts.LocalOnly && result.RemoteRemoved {
v.printlnStdout("Removed snapshot metadata from remote storage")
v.stdoutf("\nNote: Removed snapshots' blobs remain on the remote. "+
"Run '%s' to delete any blobs no longer referenced by any "+
"remaining remote snapshot.\n", pruneCommandHint)
}
return result, nil
}
// deleteIncompleteSnapshot deletes one incomplete snapshot and its
// association rows, reporting whether the snapshot row itself was
// deleted.
func (v *Vaultik) deleteIncompleteSnapshot(snapshotIDStr string, id any) bool {
log.Info("Deleting incomplete snapshot", "snapshot_id", id)
// Delete related records first
err := v.Repositories.Snapshots.DeleteSnapshotFiles(v.ctx, snapshotIDStr)
if err != nil {
log.Error("Failed to delete snapshot files",
"snapshot_id", id, "error", err)
}
err = v.Repositories.Snapshots.DeleteSnapshotBlobs(v.ctx, snapshotIDStr)
if err != nil {
log.Error("Failed to delete snapshot blobs",
"snapshot_id", id, "error", err)
}
err = v.Repositories.Snapshots.DeleteSnapshotUploads(v.ctx, snapshotIDStr)
if err != nil {
log.Error("Failed to delete snapshot uploads",
"snapshot_id", id, "error", err)
}
err = v.Repositories.Snapshots.Delete(v.ctx, snapshotIDStr)
if err != nil {
log.Error("Failed to delete snapshot",
"snapshot_id", id, "error", err)
return false
}
return true
}
// removeAllLocalSnapshots removes each known local snapshot (and, unless
// --local-only, its remote metadata), returning the number of remote
// deletion failures.
func (v *Vaultik) removeAllLocalSnapshots(
localSnaps []string, opts *RemoveOptions, result *RemoveResult,
) int {
remoteErrors := 0
for _, snapshotID := range localSnaps {
log.Info("Removing snapshot", "snapshot_id", snapshotID)
err := v.deleteSnapshotFromLocalDB(snapshotID)
if err != nil {
log.Error("Failed to remove from local database",
"snapshot_id", snapshotID, "error", err)
continue
}
if !opts.LocalOnly {
err = v.deleteRemoteSnapshotByKey(
snapshot.RemoteSnapshotKey(snapshotID))
if err != nil {
log.Warn("Failed to remove snapshot metadata from remote",
"snapshot_id", snapshotID, "error", err)
remoteErrors++
}
}
result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID)
}
return remoteErrors
}
// removeAllOrphanRemotes deletes remote snapshot metadata that has no
// local counterpart, returning the number of deletion failures.
func (v *Vaultik) removeAllOrphanRemotes(
orphanRemoteKeys []string, result *RemoveResult,
) int {
remoteErrors := 0
for _, key := range orphanRemoteKeys {
log.Info("Removing orphan remote snapshot", "remote_key", key)
err := v.deleteRemoteSnapshotByKey(key)
if err != nil {
log.Warn("Failed to remove orphan from remote",
"remote_key", key, "error", err)
remoteErrors++
continue
}
result.SnapshotsRemoved = append(result.SnapshotsRemoved, key)
}
return remoteErrors
}
// deleteSnapshotFromLocalDB removes a snapshot from the local database only
func (v *Vaultik) deleteSnapshotFromLocalDB(snapshotID string) error {
if v.Repositories == nil {
return nil // No local database
}
// Delete related records first to avoid foreign key constraints
err := v.Repositories.Snapshots.DeleteSnapshotFiles(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("deleting snapshot files for %s: %w", snapshotID, err)
}
err = v.Repositories.Snapshots.DeleteSnapshotBlobs(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("deleting snapshot blobs for %s: %w", snapshotID, err)
}
err = v.Repositories.Snapshots.DeleteSnapshotUploads(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("deleting snapshot uploads for %s: %w", snapshotID, err)
}
err = v.Repositories.Snapshots.Delete(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("deleting snapshot record %s: %w", snapshotID, err)
}
return nil
}
// deleteRemoteSnapshotByKey removes everything under
// metadata/<remoteKey>/ on the destination store. The argument is a
// remote key (double-SHA256 derivation), not a human snapshot ID;
// callers that have a human ID must hash via snapshot.RemoteSnapshotKey
// first.
func (v *Vaultik) deleteRemoteSnapshotByKey(remoteKey string) error {
prefix := fmt.Sprintf("metadata/%s/", remoteKey)
objectCh := v.Storage.ListStream(v.ctx, prefix)
var objectsToDelete []string
for object := range objectCh {
if object.Err != nil {
return fmt.Errorf("listing objects: %w", object.Err)
}
objectsToDelete = append(objectsToDelete, object.Key)
}
for _, key := range objectsToDelete {
err := v.Storage.Delete(v.ctx, key)
if err != nil {
return fmt.Errorf("removing %s: %w", key, err)
}
log.Debug("Deleted remote object", "key", key)
}
return nil
}
// outputRemoveJSON outputs the removal result as JSON
func (v *Vaultik) outputRemoveJSON(result *RemoveResult) error {
encoder := json.NewEncoder(v.Stdout)
encoder.SetIndent("", " ")
return encoder.Encode(result)
}
// PruneResult contains statistics about the prune operation
type PruneResult struct {
SnapshotsDeleted int64
FilesDeleted int64
ChunksDeleted int64
BlobsDeleted int64
}
// PruneDatabase removes incomplete snapshots and orphaned files, chunks,
// and blobs from the local database. This ensures database consistency
// before starting a new backup or on-demand via the prune command.
func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
log.Info("Pruning local database: " +
"removing incomplete snapshots and orphaned data")
v.UI.Beginf("Pruning local index database " +
"(removing incomplete snapshots and orphaned data).")
result := &PruneResult{}
// Snapshot counts before deletion of incompletes.
snapshotCountBefore, _ := v.getTableCount("snapshots")
// First, delete any incomplete snapshots
incompleteSnapshots, err := v.Repositories.Snapshots.GetIncompleteSnapshots(v.ctx)
if err != nil {
return nil, fmt.Errorf("getting incomplete snapshots: %w", err)
}
for _, snap := range incompleteSnapshots {
if v.deleteIncompleteSnapshot(snap.ID.String(), snap.ID) {
result.SnapshotsDeleted++
}
}
// Get counts before cleanup for reporting
fileCountBefore, _ := v.getTableCount("files")
chunkCountBefore, _ := v.getTableCount("chunks")
blobCountBefore, _ := v.getTableCount("blobs")
// Run the cleanup
err = v.SnapshotManager.CleanupOrphanedData(v.ctx)
if err != nil {
return nil, fmt.Errorf("cleanup orphaned data: %w", err)
}
// Get counts after cleanup
fileCountAfter, _ := v.getTableCount("files")
chunkCountAfter, _ := v.getTableCount("chunks")
blobCountAfter, _ := v.getTableCount("blobs")
result.FilesDeleted = fileCountBefore - fileCountAfter
result.ChunksDeleted = chunkCountBefore - chunkCountAfter
result.BlobsDeleted = blobCountBefore - blobCountAfter
log.Info("Local database prune complete",
"incomplete_snapshots", result.SnapshotsDeleted,
"orphaned_files", result.FilesDeleted,
"orphaned_chunks", result.ChunksDeleted,
"orphaned_blobs", result.BlobsDeleted,
)
snapshotCountAfter := snapshotCountBefore - result.SnapshotsDeleted
v.UI.Completef("Pruned local index database.")
v.UI.Detailf("Incomplete snapshots: %d removed (%d remain).",
result.SnapshotsDeleted, snapshotCountAfter)
v.UI.Detailf("Orphaned files: %d removed (%d remain).",
result.FilesDeleted, fileCountAfter)
v.UI.Detailf("Orphaned chunks: %d removed (%d remain).",
result.ChunksDeleted, chunkCountAfter)
v.UI.Detailf("Orphaned blobs: %d removed (%d remain).",
result.BlobsDeleted, blobCountAfter)
return result, nil
}
// validTableNameRe matches table names containing only lowercase
// alphanumeric characters and underscores.
var validTableNameRe = regexp.MustCompile(`^[a-z0-9_]+$`)
// getTableCount returns the count of rows in a table. The tableName is
// sanitized to only allow [a-z0-9_] characters to prevent SQL
// injection.
func (v *Vaultik) getTableCount(tableName string) (int64, error) {
if v.DB == nil {
return 0, nil
}
if !validTableNameRe.MatchString(tableName) {
return 0, fmt.Errorf("%w: %q", errInvalidTableName, tableName)
}
var count int64
query := "SELECT COUNT(*) FROM " + tableName
err := v.DB.Conn().QueryRowContext(v.ctx, query).Scan(&count)
if err != nil {
return 0, err
}
return count, nil
}