sneak/vaultik
1
0
Fork 0
Code Issues 19 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

List remote snapshots without requiring the private key (closes #64)
All checks were successful
check / check (pull_request) Successful in 2m13s
Details

Browse Source 
ListSnapshots built its table entirely from the local SQLite index. The
only remote access, reportRemoteDrift, was gated on AgeSecretKey being
non-empty — so on a correctly configured host, which by design holds no
private key, `snapshot list` never contacted the destination store at
all. A user who lost their local index could not see their own backups,
and the "<remote only>" cell the README documents was unreachable dead
code.

The listing is now the union of the local index and the destination
store, with no age_secret_key gate. The manifest is unencrypted, so a
host holding only the public key can enumerate what it has backed up:
one streamed listing of the metadata/ prefix, then a manifest read per
remote key the local index does not already account for, bounded by
maxRemoteOnlyRows and run with bounded concurrency.

A remote-only snapshot's hostname and name are deliberately NOT
recovered. RemoteSnapshotKey is one-way and the manifest stores the
hash rather than the human ID, so they are recoverable only from the
encrypted per-snapshot database; making them readable from remote
storage would undo a deliberate privacy property (#81). Such rows are
labelled "<remote only:<12 hex chars>>", carry the real timestamp and
compressed size from the manifest, and show "<remote only>" in the two
columns that genuinely require the local index. --json carries the full
64-character key in remote_key, and remote_present distinguishes seen
(true), missing (false) and not-listable (null).

Local records with no counterpart on the destination store are still
surfaced as drift, and the remediation hint now names `vaultik prune`,
which exists, rather than `vaultik snapshot cleanup`, which does not:
CleanupLocalSnapshots is already wired as prune's first pass, and
re-adding a second entry point would undo the CLI consolidation.

reportRemoteDrift collapses. Its remote-only half is subsumed by the
table — those snapshots are rows now, not a footnote count — and its
local-only half reads the merge ListSnapshots already computed, so the
command lists the destination exactly once per invocation.

An unreachable destination stays a warning plus local-only output and a
zero exit code, as the doc comment always claimed. In --json mode that
warning goes to stderr, because the logger and the UI writer both emit
on stdout and would otherwise corrupt the document.

Tests cover remote-only rendering, the no-private-key property (a
storer that counts prefix listings and records fetched keys, asserting
the destination is read and nothing encrypted is touched), graceful
degradation on an unreachable destination in both output modes,
local-only drift, and an unreadable manifest not hiding other
snapshots.
This commit is contained in:
Jeffrey Paul
2026-08-09 03:15:52 +00:00
parent 0952925453
commit 0e2929d75d
6 changed files with 1101 additions and 248 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
README.md
View File

@@ -175,12 +175,40 @@ needed.
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`)
**`snapshot list`**: Show every snapshot known to the destination
store with timestamps and three sizes per snapshot (compressed
remote size; total uncompressed chunk size; size of chunks newly
referenced by that snapshot). The uncompressed and "new chunk"
columns show `<remote only>` for snapshots not in the local index.
* `--json`: Output in JSON format
**`snapshot list`**: Show every snapshot known to this host — the union
of the local index and the backup destination store — with timestamps
and three sizes per snapshot (compressed remote size; total
uncompressed chunk size; size of chunks newly referenced by that
snapshot).
Listing the destination store does **not** require the age secret key,
so it works in vaultik's intended configuration, where the backed-up
host holds only the public key. A host that has lost its local index
can still see what it has backed up.
What that host cannot see is a remote-only snapshot's name. The
snapshot ID is hashed at the storage boundary and the manifest records
only the hash, so hostname and snapshot name exist solely in the local
index and in the encrypted per-snapshot database. Snapshots found only
on the destination store are therefore listed as
`<remote only:<abbreviated remote key>>` and show `<remote only>` in
the uncompressed and "new chunk" columns, which can only be computed
from the local index. Their timestamp and compressed size are real,
read from the manifest.
Snapshots in the local index with no counterpart on the destination
store are reported below the table as drift, with the `vaultik prune`
invocation that reconciles them.
If the destination store cannot be listed (unmounted volume,
permission denied, network down), the command warns, falls back to the
local index alone, and still exits zero.
* `--json`: Output in JSON format. Each entry carries `locally_tracked`
(whether the snapshot is in the local index), `remote_key` (the full
64-character storage key), and `remote_present` (whether it was seen
on the destination store, or `null` if the destination could not be
listed). The warning about an unlistable destination goes to stderr
so stdout stays a single parseable document.
**`snapshot verify`**: Verify snapshot integrity.
* Default (shallow): checks that all blobs referenced in the manifest exist in storage

19
TODO.md
View File

@@ -19,6 +19,25 @@ or delete the branch.
# Completed Steps
- 2026-08-09: Made `snapshot list` list the destination store without
the private key (issue #64). The listing is now the union of the
local index and a single streamed listing of the `metadata/` prefix,
with no `age_secret_key` gate — the manifest is unencrypted, so a
host holding only the public key can enumerate its own backups and a
host that lost its local index can still see them. A remote-only
snapshot's hostname and name are deliberately not recovered (they are
not recoverable without the private key, and making them so would
undo the privacy property tracked in issue #81); such rows are
labelled by an abbreviation of their remote key and carry the real
timestamp and compressed size from the manifest, with `<remote only>`
in the two columns that require the local index. Local-only snapshots
are reported as drift, and the hint now names `vaultik prune`, which
exists, instead of `vaultik snapshot cleanup`, which does not.
`reportRemoteDrift` collapsed into the merged view. Every remote
manifest read in the codebase now goes through
`downloadManifestByKey`, so issue #81 has one call site to change.
Verified with `script/cibuild` and end to end against a `file://`
destination with no secret key present.
- 2026-08-09: Closed the gap between `make lint` and CI (issue #78).
`script/lint` now runs the digest-pinned `golangci-lint` image taken
from the `Dockerfile` lint stage, which is the single source of truth

24
internal/vaultik/helpers.go
View File

@@ -53,18 +53,36 @@ const (
)
// SnapshotInfo contains information about a snapshot.
// UncompressedSize and NewChunkSize are populated only when the snapshot
// is present in the local database; LocallyTracked indicates whether
// those values are meaningful.
//
// LocallyTracked says which of the two sources this row came from, and
// therefore which fields are meaningful:
//
// - true: the snapshot is in the local index. ID is its human
// snapshot ID and UncompressedSize/NewChunkSize are populated.
// - false: the snapshot was found only on the destination store. ID
// is empty, because the human ID cannot be recovered from remote
// storage without the age secret key (see RemoteKey), and
// UncompressedSize/NewChunkSize are zero because they are computed
// from local index rows that do not exist.
//
// RemoteKey is always populated: for a locally tracked snapshot it is
// the key the snapshot would occupy on the destination store, and for a
// remote-only snapshot it is the only identifier available.
//
// RemotePresent reports whether the snapshot's metadata was seen on the
// destination store. It is nil when the destination could not be
// listed, so "absent" and "unknown" stay distinguishable.
//
//nolint:tagliatelle // snake_case is the established output format
type SnapshotInfo struct {
ID types.SnapshotID `json:"id"`
RemoteKey string `json:"remote_key"`
Timestamp time.Time `json:"timestamp"`
CompressedSize int64 `json:"compressed_size"`
UncompressedSize int64 `json:"uncompressed_size,omitempty"`
NewChunkSize int64 `json:"new_chunk_size,omitempty"`
LocallyTracked bool `json:"locally_tracked"`
RemotePresent *bool `json:"remote_present"`
}
// formatBytes formats bytes in a human-readable format

247
internal/vaultik/snapshot.go
View File

@@ -9,10 +9,8 @@ import (
"regexp"
"sort"
"strings"
"text/tabwriter"
"time"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
)
@@ -447,243 +445,6 @@ func (v *Vaultik) getSnapshotBlobSizes(snapshotID string) (int64, int64) {
return compressed, uncompressed
}
// ListSnapshots prints the table of snapshots, plus any reconciliation
// warnings/notes between the local index and the backup destination
// store.
//
// The local index database is always the primary source for the
// table — it has the human snapshot IDs, timestamps, and per-snapshot
// stats.
//
// If an age secret key is configured AND remote listing succeeds, we
// cross-reference: any local snapshot whose hashed key isn't visible
// remotely gets a "local-only" cleanup hint, and any remote key that
// doesn't correspond to a known local snapshot gets reported in a
// NOTE.
//
// If no age key is set the local machine is assumed write-only
// (backup-only), so we skip remote listing entirely — there's no
// value showing keys the user couldn't restore anyway.
//
// If remote listing fails (unmounted volume, permission denied,
// network), we degrade to local-only with a warning. List never
// fails just because the destination is unreachable.
func (v *Vaultik) ListSnapshots(jsonOutput bool) error {
log.Info("Listing snapshots")
localSnaps, err := v.Repositories.Snapshots.ListRecent(v.ctx, listRecentLimit)
if err != nil {
return fmt.Errorf("listing local snapshots: %w", err)
}
snapshots := make([]SnapshotInfo, 0, len(localSnaps))
for _, ls := range localSnaps {
if ls.CompletedAt == nil {
continue
}
snapshots = append(snapshots, v.snapshotInfoFromLocal(ls))
}
sort.Slice(snapshots, func(i, j int) bool {
return snapshots[i].Timestamp.After(snapshots[j].Timestamp)
})
if jsonOutput {
encoder := json.NewEncoder(v.Stdout)
encoder.SetIndent("", " ")
return encoder.Encode(snapshots)
}
err = v.printSnapshotTable(snapshots)
if err != nil {
return err
}
if v.Config.AgeSecretKey == "" {
return nil
}
v.reportRemoteDrift(localSnaps)
return nil
}
// reportRemoteDrift cross-references local snapshot records against the
// remote metadata keys and reports local-only records and unknown
// remote keys. Never fails: remote listing errors degrade to a warning.
func (v *Vaultik) reportRemoteDrift(localSnaps []*database.Snapshot) {
remoteKeys, err := v.listAllRemoteSnapshotKeys()
if err != nil {
v.UI.Warningf("Could not list backup destination store: %v.", err)
return
}
localKeys := make(map[string]string, len(localSnaps))
for _, ls := range localSnaps {
if ls.CompletedAt == nil {
continue
}
localKeys[snapshot.RemoteSnapshotKey(ls.ID.String())] = ls.ID.String()
}
remoteSet := make(map[string]bool, len(remoteKeys))
for _, k := range remoteKeys {
remoteSet[k] = true
}
var localOnly []string
for key, humanID := range localKeys {
if !remoteSet[key] {
localOnly = append(localOnly, humanID)
}
}
var remoteOnlyCount int
for key := range remoteSet {
if _, ok := localKeys[key]; !ok {
remoteOnlyCount++
}
}
if len(localOnly) > 0 {
v.UI.Warningf("%d local snapshot record(s) not found in backup "+
"destination store:", len(localOnly))
for _, id := range localOnly {
v.UI.Infof("%s", v.UI.Snapshot(id))
}
v.UI.Infof("Run 'vaultik snapshot cleanup' to remove stale local records.")
}
if remoteOnlyCount > 0 {
v.UI.Noticef("NOTE: %d remote snapshot(s) found in backup destination "+
"store but not in local database.", remoteOnlyCount)
}
}
// snapshotInfoFromLocal builds a SnapshotInfo row from a local snapshot
// record. Failures from any per-snapshot stat query degrade that
// column to its snapshot-row fallback but never fail the listing.
func (v *Vaultik) snapshotInfoFromLocal(ls *database.Snapshot) SnapshotInfo {
idStr := ls.ID.String()
totalSize, err := v.Repositories.Snapshots.GetSnapshotTotalCompressedSize(
v.ctx, idStr)
if err != nil {
log.Warn("Failed to get total compressed size", "id", idStr, "error", err)
totalSize = ls.BlobSize
}
uncompressedSize, err := v.Repositories.Snapshots.GetSnapshotUncompressedChunkSize(
v.ctx, idStr)
if err != nil {
log.Warn("Failed to get uncompressed chunk size", "id", idStr, "error", err)
}
newChunkSize, err := v.Repositories.Snapshots.GetSnapshotNewChunkSize(v.ctx, idStr)
if err != nil {
log.Warn("Failed to get new chunk size", "id", idStr, "error", err)
}
return SnapshotInfo{
ID: ls.ID,
Timestamp: ls.StartedAt,
CompressedSize: totalSize,
UncompressedSize: uncompressedSize,
NewChunkSize: newChunkSize,
LocallyTracked: true,
}
}
// tabPadding is the tabwriter cell padding for the snapshot table.
const tabPadding = 3
// printSnapshotTable renders the snapshot list as a formatted table
func (v *Vaultik) printSnapshotTable(snapshots []SnapshotInfo) error {
w := tabwriter.NewWriter(v.Stdout, 0, 0, tabPadding, ' ', 0)
_, err := fmt.Fprintln(w, "CONFIGURED SNAPSHOTS:")
if err != nil {
return err
}
_, err = fmt.Fprintln(w, "NAME\tPATHS")
if err != nil {
return err
}
_, err = fmt.Fprintln(w, "────\t─────")
if err != nil {
return err
}
for _, name := range v.Config.SnapshotNames() {
snap := v.Config.Snapshots[name]
paths := strings.Join(snap.Paths, ", ")
_, err = fmt.Fprintf(w, "%s\t%s\n", name, paths)
if err != nil {
return err
}
}
_, err = fmt.Fprintln(w)
if err != nil {
return err
}
_, err = fmt.Fprintln(w, "REMOTE SNAPSHOTS:")
if err != nil {
return err
}
_, err = fmt.Fprintln(w,
"SNAPSHOT ID\tTIMESTAMP\tCOMPRESSED SIZE\t"+
"UNCOMPRESSED SIZE\tNEW CHUNK SIZE")
if err != nil {
return err
}
_, err = fmt.Fprintln(w,
"───────────\t─────────\t───────────────\t"+
"─────────────────\t──────────────")
if err != nil {
return err
}
const remoteOnlyCell = "<remote only>"
for _, snap := range snapshots {
uncompressed := remoteOnlyCell
newChunks := remoteOnlyCell
if snap.LocallyTracked {
uncompressed = formatBytes(snap.UncompressedSize)
newChunks = formatBytes(snap.NewChunkSize)
}
_, err = fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n",
snap.ID,
snap.Timestamp.Format("2006-01-02 15:04:05"),
formatBytes(snap.CompressedSize),
uncompressed,
newChunks)
if err != nil {
return err
}
}
return w.Flush()
}
// SnapshotPurgeOptions contains options for the snapshot purge command.
type SnapshotPurgeOptions struct {
KeepLatest bool // Keep only the most recent snapshot per name
@@ -1123,6 +884,14 @@ func (v *Vaultik) CleanupLocalSnapshots() error {
// metadata/<remoteKey>/manifest.json.zst. The remoteKey is the double-
// SHA256 derivation produced by snapshot.RemoteSnapshotKey, not the
// human snapshot ID. Callers that have a human ID must hash first.
//
// This is the only place vaultik reads a manifest off the destination
// store, deliberately: the manifest is currently stored compressed but
// unencrypted, which is what lets `snapshot list` enumerate the
// destination on a host holding no private key. Whether to encrypt it
// is open (issue #81), and routing every read through here means that
// decision has exactly one call site to change. Keep it that way — do
// not open metadata/<key>/manifest.json.zst directly elsewhere.
func (v *Vaultik) downloadManifestByKey(remoteKey string) (*snapshot.Manifest, error) {
manifestPath := fmt.Sprintf("metadata/%s/manifest.json.zst", remoteKey)

486
internal/vaultik/snapshot_list.go Normal file
View File

@@ -0,0 +1,486 @@
package vaultik
import (
"encoding/json"
"fmt"
"sort"
"strings"
"text/tabwriter"
"time"
"golang.org/x/sync/errgroup"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// remoteOnlyCell fills the table columns that can only be derived from
// the local index. A snapshot present only on the destination store has
// no local rows to derive them from.
const remoteOnlyCell = "<remote only>"
// remoteKeyDisplayLen is how many hex characters of a remote key are
// shown in the identifier column for a remote-only snapshot. Twelve
// matches the abbreviation length used elsewhere in the UI and is far
// past the point of ambiguity for a SHA256 digest.
const remoteKeyDisplayLen = 12
// maxRemoteOnlyRows caps how many remote-only snapshots a single
// `snapshot list` will describe. Each one costs a manifest read, so an
// uncapped listing against a destination holding many thousands of
// unknown snapshots would be both slow and unbounded in memory. Beyond
// the cap the table is truncated and the count of omitted snapshots is
// reported.
const maxRemoteOnlyRows = 1000
// remoteManifestFetchConcurrency bounds how many manifest reads are in
// flight at once while describing remote-only snapshots. The listing
// itself is a single streamed prefix request; only the per-snapshot
// manifest reads need throttling.
const remoteManifestFetchConcurrency = 8
// tabPadding is the tabwriter cell padding for the snapshot table.
const tabPadding = 3
// ListSnapshots prints the table of snapshots known to this host: the
// union of the local index database and the backup destination store.
//
// Remote listing needs no age secret key. A snapshot's manifest
// (metadata/<remote-key>/manifest.json.zst) is compressed but not
// encrypted, so a host holding only the public key — the configuration
// vaultik is designed for — can still enumerate what it has backed up
// and see each snapshot's timestamp and compressed size.
//
// What that host cannot see is a remote-only snapshot's human ID.
// snapshot.RemoteSnapshotKey is one-way and the manifest stores the
// hashed key rather than the ID, so hostname and snapshot name live
// only in the local index and in the encrypted db.zst.age. Remote-only
// rows are therefore identified by an abbreviation of their remote key,
// and the two columns that genuinely require the local index
// (uncompressed size, new chunk size) render as "<remote only>". No
// attempt is made to recover or fabricate the human ID.
//
// Snapshots in the local index with no counterpart on the destination
// store are reported as drift below the table.
//
// If remote listing fails (unmounted volume, permission denied,
// network), we degrade to local-only with a warning. List never fails
// just because the destination is unreachable.
func (v *Vaultik) ListSnapshots(jsonOutput bool) error {
log.Info("Listing snapshots")
localSnaps, err := v.Repositories.Snapshots.ListRecent(v.ctx, listRecentLimit)
if err != nil {
return fmt.Errorf("listing local snapshots: %w", err)
}
snapshots := make([]SnapshotInfo, 0, len(localSnaps))
localKeys := make(map[string]bool, len(localSnaps))
for _, ls := range localSnaps {
if ls.CompletedAt == nil {
continue
}
info := v.snapshotInfoFromLocal(ls)
localKeys[info.RemoteKey] = true
snapshots = append(snapshots, info)
}
listing, remoteErr := v.collectRemoteSnapshots(localKeys)
if remoteErr != nil {
v.warnRemoteListingFailed(remoteErr, jsonOutput)
} else {
snapshots = append(snapshots, listing.remoteOnly...)
markRemotePresence(snapshots, listing.keys)
}
sort.Slice(snapshots, func(i, j int) bool {
return snapshots[i].Timestamp.After(snapshots[j].Timestamp)
})
if jsonOutput {
encoder := json.NewEncoder(v.Stdout)
encoder.SetIndent("", " ")
return encoder.Encode(snapshots)
}
err = v.printSnapshotTable(snapshots)
if err != nil {
return err
}
if remoteErr == nil {
v.reportListDrift(snapshots, listing)
}
return nil
}
// warnRemoteListingFailed reports an unreachable or unreadable
// destination store without failing the command: the local index is
// still worth printing, and `snapshot list` exiting non-zero because a
// volume is unmounted would be worse than useless.
//
// In --json mode the warning goes to stderr rather than through the
// logger or the UI writer, both of which emit on stdout — the JSON
// document has to be the only thing on stdout for `snapshot list --json
// | jq` to work. The failure is also representable in the document
// itself: every row's remote_present is null when the destination could
// not be listed.
func (v *Vaultik) warnRemoteListingFailed(err error, jsonOutput bool) {
if jsonOutput {
_, _ = fmt.Fprintf(v.Stderr,
"Warning: could not list backup destination store: %v. "+
"Showing snapshots from the local index only.\n", err)
return
}
log.Warn("Could not list backup destination store", "error", err)
v.UI.Warningf("Could not list backup destination store: %v.", err)
v.UI.Infof("Showing snapshots from the local index only.")
}
// remoteSnapshotListing is the result of one pass over the destination
// store's metadata/ prefix.
type remoteSnapshotListing struct {
// keys holds every remote snapshot key present on the destination
// store, whether or not it is known locally.
keys map[string]bool
// remoteOnly holds one row per remote key with no local
// counterpart, built from that snapshot's manifest.
remoteOnly []SnapshotInfo
// omitted counts remote-only keys dropped because describing them
// all would have exceeded maxRemoteOnlyRows.
omitted int
// unreadable counts remote-only keys whose manifest could not be
// read or decoded.
unreadable int
}
// collectRemoteSnapshots enumerates the destination store and describes
// every snapshot on it that localKeys does not already account for.
//
// The enumeration is a single streamed listing of the metadata/ prefix,
// so the request count does not scale with the number of snapshots.
// Manifest reads scale only with the number of snapshots the local
// index does not already know about, and are capped at
// maxRemoteOnlyRows.
func (v *Vaultik) collectRemoteSnapshots(
localKeys map[string]bool,
) (*remoteSnapshotListing, error) {
keys, err := v.listAllRemoteSnapshotKeys()
if err != nil {
return nil, err
}
listing := &remoteSnapshotListing{
keys: make(map[string]bool, len(keys)),
}
unknown := make([]string, 0, len(keys))
for _, key := range keys {
listing.keys[key] = true
if !localKeys[key] {
unknown = append(unknown, key)
}
}
// Sorted so both the truncation point and the fetch order are
// deterministic run to run.
sort.Strings(unknown)
if len(unknown) > maxRemoteOnlyRows {
listing.omitted = len(unknown) - maxRemoteOnlyRows
unknown = unknown[:maxRemoteOnlyRows]
}
listing.remoteOnly, listing.unreadable = v.describeRemoteOnlySnapshots(unknown)
return listing, nil
}
// describeRemoteOnlySnapshots reads the manifest for each supplied
// remote key and turns it into a table row, returning the rows and the
// number of keys whose manifest could not be read.
//
// A key whose manifest is missing or corrupt is skipped rather than
// failing the listing: one bad snapshot directory must not hide every
// other snapshot the user has.
func (v *Vaultik) describeRemoteOnlySnapshots(keys []string) ([]SnapshotInfo, int) {
found := make([]SnapshotInfo, len(keys))
ok := make([]bool, len(keys))
var group errgroup.Group
group.SetLimit(remoteManifestFetchConcurrency)
for i, key := range keys {
group.Go(func() error {
info, err := v.remoteSnapshotInfo(key)
if err != nil {
log.Warn("Could not describe remote snapshot",
"remote_key", key, "error", err)
return nil
}
found[i] = info
ok[i] = true
return nil
})
}
// No goroutine above ever returns an error; failures are recorded
// in ok and reported as a count.
_ = group.Wait()
infos := make([]SnapshotInfo, 0, len(keys))
unreadable := 0
for i := range keys {
if !ok[i] {
unreadable++
continue
}
infos = append(infos, found[i])
}
return infos, unreadable
}
// remoteSnapshotInfo builds a table row for a snapshot that exists on
// the destination store but not in the local index, from the only
// source available without the private key: the unencrypted manifest.
//
// ID is deliberately left zero. Recovering it would mean inverting
// snapshot.RemoteSnapshotKey, which is not possible, or writing the
// human ID somewhere unencrypted on the destination, which would undo
// the privacy property that hashing the key exists to provide (see
// issue #81). The renderer marks the row as unnamed rather than
// guessing.
func (v *Vaultik) remoteSnapshotInfo(remoteKey string) (SnapshotInfo, error) {
manifest, err := v.downloadManifestByKey(remoteKey)
if err != nil {
return SnapshotInfo{}, err
}
timestamp, err := time.Parse(time.RFC3339, manifest.Timestamp)
if err != nil {
// The snapshot is really there; an unparseable timestamp is not
// reason enough to hide it. It sorts to the bottom as the zero
// time.
log.Warn("Remote manifest has an unparseable timestamp",
"remote_key", remoteKey, "timestamp", manifest.Timestamp, "error", err)
timestamp = time.Time{}
}
return SnapshotInfo{
RemoteKey: remoteKey,
Timestamp: timestamp.UTC(),
CompressedSize: manifest.TotalCompressedSize,
LocallyTracked: false,
}, nil
}
// markRemotePresence records, for every row, whether its remote key was
// seen on the destination store during this listing. Only called when
// the listing succeeded: when it did not, presence stays nil ("not
// known") rather than being reported as absence.
func markRemotePresence(snapshots []SnapshotInfo, remoteKeys map[string]bool) {
for i := range snapshots {
present := remoteKeys[snapshots[i].RemoteKey]
snapshots[i].RemotePresent = &present
}
}
// snapshotInfoFromLocal builds a SnapshotInfo row from a local snapshot
// record. Failures from any per-snapshot stat query degrade that
// column to its snapshot-row fallback but never fail the listing.
func (v *Vaultik) snapshotInfoFromLocal(ls *database.Snapshot) SnapshotInfo {
idStr := ls.ID.String()
totalSize, err := v.Repositories.Snapshots.GetSnapshotTotalCompressedSize(
v.ctx, idStr)
if err != nil {
log.Warn("Failed to get total compressed size", "id", idStr, "error", err)
totalSize = ls.BlobSize
}
uncompressedSize, err := v.Repositories.Snapshots.GetSnapshotUncompressedChunkSize(
v.ctx, idStr)
if err != nil {
log.Warn("Failed to get uncompressed chunk size", "id", idStr, "error", err)
}
newChunkSize, err := v.Repositories.Snapshots.GetSnapshotNewChunkSize(v.ctx, idStr)
if err != nil {
log.Warn("Failed to get new chunk size", "id", idStr, "error", err)
}
return SnapshotInfo{
ID: ls.ID,
RemoteKey: snapshot.RemoteSnapshotKey(idStr),
Timestamp: ls.StartedAt,
CompressedSize: totalSize,
UncompressedSize: uncompressedSize,
NewChunkSize: newChunkSize,
LocallyTracked: true,
}
}
// reportListDrift prints the reconciliation notes the merged table
// cannot express on its own: local records with no counterpart on the
// destination store, plus counts of remote snapshots that were
// unreadable or omitted.
//
// This is what remains of the old reportRemoteDrift, and it no longer
// touches the destination store. Its remote-only half collapsed into
// the table — those snapshots are rows now, not a footnote count — and
// its local-only half reads the merge ListSnapshots already computed,
// so `snapshot list` lists the destination exactly once per invocation.
func (v *Vaultik) reportListDrift(
snapshots []SnapshotInfo, listing *remoteSnapshotListing,
) {
var localOnly []string
for _, snap := range snapshots {
if snap.LocallyTracked && !listing.keys[snap.RemoteKey] {
localOnly = append(localOnly, snap.ID.String())
}
}
if len(localOnly) > 0 {
v.UI.Warningf("%d local snapshot record(s) not found in backup "+
"destination store:", len(localOnly))
for _, id := range localOnly {
v.UI.Infof("%s", v.UI.Snapshot(id))
}
v.UI.Infof("Run '%s' to remove stale local records.", pruneCommandHint)
}
if len(listing.remoteOnly) > 0 {
v.UI.Noticef("NOTE: %d snapshot(s) on the backup destination store "+
"are not in the local index. Their hostname and snapshot name "+
"cannot be recovered without the age secret key, so they are "+
"listed by remote key.", len(listing.remoteOnly))
}
if listing.unreadable > 0 {
v.UI.Warningf("%d remote snapshot(s) could not be described: "+
"manifest missing or unreadable.", listing.unreadable)
}
if listing.omitted > 0 {
v.UI.Warningf("%d further remote-only snapshot(s) not shown "+
"(limit %d per listing).", listing.omitted, maxRemoteOnlyRows)
}
}
// formatRemoteOnlyID renders the identifier cell for a snapshot absent
// from the local index. Its human ID cannot be recovered without the
// private key, so the cell shows an abbreviation of the remote key
// instead. The angle brackets make it obvious this is not a snapshot
// name, which matters more than compactness: a bare hex string would
// read as a name the user simply doesn't recognize.
func formatRemoteOnlyID(remoteKey string) string {
short := remoteKey
if len(short) > remoteKeyDisplayLen {
short = short[:remoteKeyDisplayLen]
}
return "<remote only:" + short + ">"
}
// printSnapshotTable renders the snapshot list as a formatted table
func (v *Vaultik) printSnapshotTable(snapshots []SnapshotInfo) error {
w := tabwriter.NewWriter(v.Stdout, 0, 0, tabPadding, ' ', 0)
_, err := fmt.Fprintln(w, "CONFIGURED SNAPSHOTS:")
if err != nil {
return err
}
_, err = fmt.Fprintln(w, "NAME\tPATHS")
if err != nil {
return err
}
_, err = fmt.Fprintln(w, "────\t─────")
if err != nil {
return err
}
for _, name := range v.Config.SnapshotNames() {
snap := v.Config.Snapshots[name]
paths := strings.Join(snap.Paths, ", ")
_, err = fmt.Fprintf(w, "%s\t%s\n", name, paths)
if err != nil {
return err
}
}
_, err = fmt.Fprintln(w)
if err != nil {
return err
}
_, err = fmt.Fprintln(w, "REMOTE SNAPSHOTS:")
if err != nil {
return err
}
_, err = fmt.Fprintln(w,
"SNAPSHOT ID\tTIMESTAMP\tCOMPRESSED SIZE\t"+
"UNCOMPRESSED SIZE\tNEW CHUNK SIZE")
if err != nil {
return err
}
_, err = fmt.Fprintln(w,
"───────────\t─────────\t───────────────\t"+
"─────────────────\t──────────────")
if err != nil {
return err
}
for _, snap := range snapshots {
id := formatRemoteOnlyID(snap.RemoteKey)
uncompressed := remoteOnlyCell
newChunks := remoteOnlyCell
if snap.LocallyTracked {
id = snap.ID.String()
uncompressed = formatBytes(snap.UncompressedSize)
newChunks = formatBytes(snap.NewChunkSize)
}
_, err = fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n",
id,
snap.Timestamp.Format("2006-01-02 15:04:05"),
formatBytes(snap.CompressedSize),
uncompressed,
newChunks)
if err != nil {
return err
}
}
return w.Flush()
}

533
internal/vaultik/snapshot_list_test.go Normal file
View File

@@ -0,0 +1,533 @@
package vaultik_test
import (
"bytes"
"context"
"database/sql"
"encoding/json"
"errors"
"io"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/types"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// errRemoteUnreachable stands in for the real-world reasons a
// destination store cannot be listed: unmounted volume, permission
// denied, network down.
var errRemoteUnreachable = errors.New("permission denied")
// observingStorer wraps testStorer to record how the destination store
// was used: how many prefix listings were issued (the merged listing
// must not scale requests with snapshot count) and which object keys
// were fetched (nothing encrypted may be fetched during a listing).
// Setting listErr makes every listing fail, simulating an unreachable
// destination.
type observingStorer struct {
*testStorer
mu sync.Mutex
listCalls int
fetched []string
listErr error
}
func newObservingStorer() *observingStorer {
return &observingStorer{testStorer: newTestStorer()}
}
func (s *observingStorer) ListStream(
ctx context.Context, prefix string,
) <-chan storage.ObjectInfo {
s.mu.Lock()
s.listCalls++
failure := s.listErr
s.mu.Unlock()
if failure != nil {
ch := make(chan storage.ObjectInfo, 1)
ch <- storage.ObjectInfo{Err: failure}
close(ch)
return ch
}
return s.testStorer.ListStream(ctx, prefix)
}
func (s *observingStorer) Get(
ctx context.Context, key string,
) (io.ReadCloser, error) {
s.mu.Lock()
s.fetched = append(s.fetched, key)
s.mu.Unlock()
return s.testStorer.Get(ctx, key)
}
// listStreamCalls returns how many prefix listings were issued.
func (s *observingStorer) listStreamCalls() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.listCalls
}
// fetchedKeys returns a copy of every object key that was read.
func (s *observingStorer) fetchedKeys() []string {
s.mu.Lock()
defer s.mu.Unlock()
return append([]string(nil), s.fetched...)
}
// listEnv is a Vaultik wired for exercising ListSnapshots: an in-memory
// index database, an observable in-memory destination store, and
// captured output.
//
// The configuration deliberately has no age secret key. That is the
// production configuration vaultik is designed for — the backed-up host
// holds only the public key — and every assertion in this file has to
// hold in it.
type listEnv struct {
v *vaultik.Vaultik
store *observingStorer
stdout *bytes.Buffer
stderr *bytes.Buffer
}
func newListEnv(t *testing.T) *listEnv {
t.Helper()
ctx := context.Background()
db, err := database.New(ctx, ":memory:")
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
store := newObservingStorer()
stdout := &bytes.Buffer{}
stderr := &bytes.Buffer{}
v := &vaultik.Vaultik{
Config: &config.Config{
AgeSecretKey: "",
Snapshots: map[string]config.SnapshotConfig{
listConfiguredName: {Paths: []string{"/" + listConfiguredName}},
},
},
Storage: store,
Repositories: database.NewRepositories(db),
DB: db,
Stdout: stdout,
Stderr: stderr,
Stdin: &bytes.Buffer{},
UI: ui.NewWithColor(stdout, false),
}
v.SetContext(ctx)
return &listEnv{v: v, store: store, stdout: stdout, stderr: stderr}
}
// addLocal inserts a completed snapshot into the local index.
func (e *listEnv) addLocal(t *testing.T, id string, startedAt time.Time) {
t.Helper()
completedAt := startedAt.Add(time.Minute)
snap := &database.Snapshot{
ID: types.SnapshotID(id),
Hostname: "testhost",
VaultikVersion: testLabel,
StartedAt: startedAt,
CompletedAt: &completedAt,
}
ctx := context.Background()
err := e.v.Repositories.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
return e.v.Repositories.Snapshots.Create(ctx, tx, snap)
})
require.NoError(t, err, "creating local snapshot %s", id)
}
// addRemote writes a manifest to the destination store at the hashed
// path the production code uses, exactly as a real backup would. Every
// fixture snapshot has the same compressed size (fiveMegabytes); the
// tests care about which columns are populated, not about size variety.
func (e *listEnv) addRemote(
t *testing.T, snapshotID string, timestamp time.Time,
) string {
t.Helper()
remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
manifest := &snapshot.Manifest{
// Note: the hashed key, never the human ID. That is precisely
// why a remote-only snapshot cannot be named.
SnapshotID: remoteKey,
Timestamp: timestamp.UTC().Format(time.RFC3339),
BlobCount: 1,
TotalCompressedSize: fiveMegabytes,
Blobs: []snapshot.BlobInfo{
{Hash: testBlobHashA, CompressedSize: fiveMegabytes},
},
}
data, err := snapshot.EncodeManifest(manifest, 3)
require.NoError(t, err)
err = e.store.Put(context.Background(),
"metadata/"+remoteKey+"/manifest.json.zst", bytes.NewReader(data))
require.NoError(t, err)
return remoteKey
}
// Fixtures shared across the listing tests.
const (
// listConfiguredName is the one snapshot name in the test config.
listConfiguredName = "home"
listLocalID = "testhost_home_2026-03-01T10:00:00Z"
listRemoteID = "otherhost_media_2026-03-02T11:22:33Z"
// fiveMegabytes formats as "5.0 MB" through formatBytes.
fiveMegabytes = 5 * 1024 * 1024
)
// TestListSnapshots_RemoteWithoutSecretKey is the regression guard for
// issue #64: `snapshot list` must read the destination store on a host
// that holds no private key. If the remote listing is ever gated on
// age_secret_key again, this fails.
func TestListSnapshots_RemoteWithoutSecretKey(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
require.Empty(t, env.v.Config.AgeSecretKey,
"this test is meaningless unless the host has no private key")
timestamp := time.Date(2026, 3, 2, 11, 22, 33, 0, time.UTC)
remoteKey := env.addRemote(t, listRemoteID, timestamp)
err := env.v.ListSnapshots(false)
require.NoError(t, err)
// The destination store was actually read, with a single prefix
// listing rather than one request per snapshot.
assert.Equal(t, 1, env.store.listStreamCalls(),
"expected exactly one prefix listing of the destination store")
// Nothing encrypted was touched: enumerating snapshots must never
// need the age secret key.
for _, key := range env.store.fetchedKeys() {
assert.NotContains(t, key, ".age",
"listing must not read encrypted objects")
}
out := env.stdout.String()
// The snapshot is identified by an abbreviation of its remote key.
assert.Contains(t, out, "<remote only:"+remoteKey[:12]+">")
// Its human ID is not recoverable and must not be invented.
assert.NotContains(t, out, "otherhost")
assert.NotContains(t, out, "media")
// Manifest-derived columns carry real values.
assert.Contains(t, out, "2026-03-02 11:22:33")
assert.Contains(t, out, "5.0 MB")
// The two columns that require the local index are marked, not
// blank and not zero. ("<remote only:" does not match this needle,
// so the count is exactly the two marker cells.)
assert.Equal(t, 2, strings.Count(out, remoteOnlyCellText),
"expected the uncompressed and new-chunk cells to be marked")
}
// remoteOnlyCellText is the marker the table puts in columns that can
// only be computed from the local index.
const remoteOnlyCellText = "<remote only>"
// TestListSnapshots_RemoteOnlyRowRendering pins the exact row a
// remote-only snapshot produces, so the "<remote only>" cells and the
// LocallyTracked == false branch are verified rather than assumed.
func TestListSnapshots_RemoteOnlyRowRendering(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
timestamp := time.Date(2026, 3, 2, 11, 22, 33, 0, time.UTC)
remoteKey := env.addRemote(t, listRemoteID, timestamp)
err := env.v.ListSnapshots(false)
require.NoError(t, err)
label := "<remote only:" + remoteKey[:12] + ">"
row := findTableRow(t, env.stdout.String(), label)
// Identifier column: the abbreviated remote key, never blank and
// visibly not a snapshot name.
assert.True(t, strings.HasPrefix(row, label),
"identifier column must lead the row: %q", row)
// Manifest-derived columns: real values, not placeholders.
assert.Contains(t, row, "2026-03-02 11:22:33")
assert.Contains(t, row, "5.0 MB")
// Exactly the two local-index-derived columns are marked.
assert.Equal(t, 2, strings.Count(row, remoteOnlyCellText),
"uncompressed and new-chunk cells must both be marked: %q", row)
// And the note explaining why the row has no name.
assert.Contains(t, env.stdout.String(),
"are not in the local index")
}
// findTableRow returns the single output line containing needle.
func findTableRow(t *testing.T, out, needle string) string {
t.Helper()
var found []string
for line := range strings.SplitSeq(out, "\n") {
if strings.Contains(line, needle) {
found = append(found, line)
}
}
require.Len(t, found, 1, "expected exactly one line containing %q", needle)
return found[0]
}
// TestListSnapshots_MergesLocalAndRemote checks that both sources land
// in one table and that a locally tracked snapshot keeps its human ID
// and its local-index-derived columns.
func TestListSnapshots_MergesLocalAndRemote(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
localStart := time.Date(2026, 3, 1, 10, 0, 0, 0, time.UTC)
env.addLocal(t, listLocalID, localStart)
env.addRemote(t, listLocalID, localStart)
remoteKey := env.addRemote(t, listRemoteID,
time.Date(2026, 3, 2, 11, 22, 33, 0, time.UTC))
err := env.v.ListSnapshots(false)
require.NoError(t, err)
out := env.stdout.String()
assert.Contains(t, out, listLocalID)
assert.Contains(t, out, "<remote only:"+remoteKey[:12]+">")
// The locally tracked row is not marked as remote-only anywhere.
localRow := findTableRow(t, out, listLocalID)
assert.NotContains(t, localRow, "<remote only>")
// The local snapshot is present remotely, so no drift is reported.
assert.NotContains(t, out, "not found in backup destination store")
}
// TestListSnapshots_LocalOnlyReportedAsDrift covers a snapshot in the
// local index with no counterpart on the destination store, and checks
// the remediation hint names a command that actually exists.
func TestListSnapshots_LocalOnlyReportedAsDrift(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
env.addLocal(t, listLocalID, time.Date(2026, 3, 1, 10, 0, 0, 0, time.UTC))
err := env.v.ListSnapshots(false)
require.NoError(t, err)
out := env.stdout.String()
assert.Contains(t, out, listLocalID)
assert.Contains(t, out, "1 local snapshot record(s) not found in backup")
assert.Contains(t, out, "vaultik prune")
// There is no `vaultik snapshot cleanup` command; the hint must not
// name one.
assert.NotContains(t, out, "snapshot cleanup")
}
// TestListSnapshots_UnreachableRemoteDegrades covers the promise in the
// doc comment: an unreachable destination is a warning plus local-only
// output, never a failure.
func TestListSnapshots_UnreachableRemoteDegrades(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
env.addLocal(t, listLocalID, time.Date(2026, 3, 1, 10, 0, 0, 0, time.UTC))
env.store.listErr = errRemoteUnreachable
// Zero exit code: the CLI turns a nil return into exit 0.
err := env.v.ListSnapshots(false)
require.NoError(t, err)
out := env.stdout.String()
assert.Contains(t, out, "Could not list backup destination store")
assert.Contains(t, out, "permission denied")
assert.Contains(t, out, "Showing snapshots from the local index only.")
// The local index is still shown.
assert.Contains(t, out, listLocalID)
// With no remote listing there is no basis for a drift claim, so
// none must be made.
assert.NotContains(t, out, "not found in backup destination store")
}
// TestListSnapshots_UnreadableManifestDoesNotHideOthers checks that one
// corrupt remote snapshot directory cannot suppress every other
// snapshot on the destination store.
func TestListSnapshots_UnreadableManifestDoesNotHideOthers(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
goodKey := env.addRemote(t, listRemoteID,
time.Date(2026, 3, 2, 11, 22, 33, 0, time.UTC))
badKey := snapshot.RemoteSnapshotKey("testhost_broken_2026-03-03T00:00:00Z")
err := env.store.Put(context.Background(),
"metadata/"+badKey+"/manifest.json.zst",
strings.NewReader("this is not a zstd stream"))
require.NoError(t, err)
err = env.v.ListSnapshots(false)
require.NoError(t, err)
out := env.stdout.String()
assert.Contains(t, out, "<remote only:"+goodKey[:12]+">")
assert.NotContains(t, out, "<remote only:"+badKey[:12]+">")
assert.Contains(t, out, "1 remote snapshot(s) could not be described")
}
// listJSONRow mirrors the JSON shape ListSnapshots emits, so the test
// asserts against the wire format rather than the Go struct.
//
//nolint:tagliatelle // snake_case is the established output format
type listJSONRow struct {
ID string `json:"id"`
RemoteKey string `json:"remote_key"`
CompressedSize int64 `json:"compressed_size"`
LocallyTracked bool `json:"locally_tracked"`
RemotePresent *bool `json:"remote_present"`
}
// decodeListJSON parses the command's stdout, which must contain
// nothing but the JSON document.
func decodeListJSON(t *testing.T, out string) []listJSONRow {
t.Helper()
var rows []listJSONRow
err := json.Unmarshal([]byte(out), &rows)
require.NoError(t, err, "stdout must be parseable JSON: %q", out)
return rows
}
// TestListSnapshots_JSONMergedView covers the --json view of all three
// cases at once: tracked-and-present, tracked-but-missing remotely, and
// remote-only.
func TestListSnapshots_JSONMergedView(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
syncedStart := time.Date(2026, 3, 1, 10, 0, 0, 0, time.UTC)
env.addLocal(t, listLocalID, syncedStart)
env.addRemote(t, listLocalID, syncedStart)
driftedID := "testhost_home_2026-02-01T10:00:00Z"
env.addLocal(t, driftedID, time.Date(2026, 2, 1, 10, 0, 0, 0, time.UTC))
remoteKey := env.addRemote(t, listRemoteID,
time.Date(2026, 3, 2, 11, 22, 33, 0, time.UTC))
err := env.v.ListSnapshots(true)
require.NoError(t, err)
rows := decodeListJSON(t, env.stdout.String())
require.Len(t, rows, 3)
byKey := make(map[string]listJSONRow, len(rows))
for _, row := range rows {
byKey[row.RemoteKey] = row
}
synced := byKey[snapshot.RemoteSnapshotKey(listLocalID)]
assert.Equal(t, listLocalID, synced.ID)
assert.True(t, synced.LocallyTracked)
require.NotNil(t, synced.RemotePresent)
assert.True(t, *synced.RemotePresent)
drifted := byKey[snapshot.RemoteSnapshotKey(driftedID)]
assert.Equal(t, driftedID, drifted.ID)
assert.True(t, drifted.LocallyTracked)
require.NotNil(t, drifted.RemotePresent)
assert.False(t, *drifted.RemotePresent,
"a local-only snapshot must be visible as drift in --json too")
remoteOnly := byKey[remoteKey]
assert.False(t, remoteOnly.LocallyTracked)
assert.Empty(t, remoteOnly.ID,
"the human ID is unrecoverable and must not be fabricated")
assert.Len(t, remoteOnly.RemoteKey, 64,
"--json carries the full remote key, not the truncated form")
assert.Equal(t, int64(fiveMegabytes), remoteOnly.CompressedSize)
require.NotNil(t, remoteOnly.RemotePresent)
assert.True(t, *remoteOnly.RemotePresent)
}
// TestListSnapshots_JSONUnreachableRemote checks that a failed listing
// does not corrupt the JSON document with warning text, and that
// "unknown" is reported as null rather than as absence.
func TestListSnapshots_JSONUnreachableRemote(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
env.addLocal(t, listLocalID, time.Date(2026, 3, 1, 10, 0, 0, 0, time.UTC))
env.store.listErr = errRemoteUnreachable
err := env.v.ListSnapshots(true)
require.NoError(t, err)
// stdout must be nothing but the JSON document, so the warning has
// to go to stderr.
rows := decodeListJSON(t, env.stdout.String())
require.Len(t, rows, 1)
assert.Equal(t, listLocalID, rows[0].ID)
assert.True(t, rows[0].LocallyTracked)
assert.Nil(t, rows[0].RemotePresent,
"remote state is unknown when the destination cannot be listed")
assert.Contains(t, env.stderr.String(),
"could not list backup destination store")
assert.Contains(t, env.stderr.String(), "permission denied")
}