3 Commits
Author SHA1 Message Date
sneak 2045476039 Warn about restored ownership only when an entry's owner or group differs (closes #316)
check / check (push) Successful in 11m25s
A restore by a normal user always warned that file ownership was not
preserved, even when every entry came out with its recorded owner and
group. The owner and group each restored entry has on disk are now
compared with the recorded ones, and the warning is printed once if any
differ. The comparison is made on disk because on macOS, and in a setgid
directory on Linux, a new entry takes the group of its directory, not
the user's.

Judgement call: an entry that is not on disk after the restore is not compared.
Unverified: not run on a Mac; the directory-group test runs only as root.

Model: opus-5-5
2026-10-14 02:51:48 +00:00
clawbot 65036bba1b Make snapshot list report a changed destination instead of advising prune (closes #317)
check / check (push) Failing after 6s
When the local index is bound to a different destination than the
configured one, snapshot list now gives the warning prune, snapshot
purge and snapshot remove refuse with: both destinations and the two
ways out (revert the configuration, or run vaultik database delete).
Below the table it replaces the local-only note and its prune advice.
With --json it goes to stderr through the logger, so stdout stays the
document, whose local rows show remote_present false.

Reading the binding moves out of EnsureStorageBinding into
storageBinding, which records nothing, so the read-only list can call
it without binding a fresh index.

Model: opus-5-5
2026-10-14 04:29:36 +02:00
clawbot 54ad279dd6 Count a file as deleted only if the previous snapshot of its name had it (closes #304)
check / check (push) Successful in 11m35s
A backup counted as deleted every file in the local index under the
source path that the walk did not find: files the snapshot excludes
but another snapshot backs up, and files removed long ago, which every
later backup, and the first backup of another name over the same
path, reported as deleted again.

The backup now counts only files that the latest completed snapshot of
the same name lists, the walk did not find, and are gone from disk,
including a file whose directory became a regular file. Excluded files
and files the walk could not reach under --skip-errors are not
counted.

Only the backup's output changes; the snapshots row stores no deleted
figure.

Model: opus-5-5
2026-10-14 04:24:30 +02:00
14 changed files with 788 additions and 73 deletions
+4 -4
View File
@@ -279,8 +279,8 @@ Key methods:
- `Scan(ctx, path, snapshotID)` → Main scan operation
Scan phases:
1. **Phase 0**: Detect deleted files from previous snapshots
2. **Phase 1**: Walk directory, identify files needing processing
1. **Phase 1**: Walk directory, identify files needing processing
2. **Phase 1b**: Count as deleted the files that the previous snapshot of the same name lists, the walk did not find, and are gone from disk; a file that is excluded, or inside an excluded directory, is not counted
3. **Phase 2**: Process files (chunk → pack → upload)
#### SnapshotManager
@@ -320,13 +320,13 @@ CreateSnapshot(opts)
│ │
│ ├─► scanner.Scan(ctx, path, snapshotID)
│ │ │
│ │ ├─► Phase 0: detectDeletedFiles()
│ │ │
│ │ ├─► Phase 1: scanPhase()
│ │ │ Walk directory
│ │ │ Check file metadata changes
│ │ │ Build list of files to process
│ │ │
│ │ ├─► Phase 1b: detectDeletedFiles()
│ │ │
│ │ └─► Phase 2: processPhase()
│ │ For each file:
│ │ chunker.ChunkReaderStreaming()
+10 -4
View File
@@ -323,7 +323,11 @@ read from the manifest.
Snapshots in the local index with no counterpart on the destination
store are reported below the table as drift, with the `vaultik prune`
invocation that reconciles them.
invocation that reconciles them. If the local index is bound to a
different destination than the configured one (after a change to
`storage_url`, or `s3.endpoint`, `s3.bucket` or `s3.prefix`), the
command says so instead and gives the same two ways out that `prune`
gives: revert the configuration, or run `vaultik database delete`.
If the destination store cannot be listed (unmounted volume,
permission denied, network down), the command warns, falls back to the
@@ -332,9 +336,11 @@ local index alone, and still exits zero.
(whether the snapshot is in the local index), `remote_key` (the full
64-character storage key), and `remote_present` (whether it was seen
on the destination store, or `null` if the destination could not be
listed). Warnings about an unlistable destination, unreadable
manifests, and a truncated listing all go to stderr through the
logger, so stdout stays a single parseable document.
listed). Warnings about an unlistable destination, a local index
bound to a different destination (with both destinations and the
same two ways out), unreadable manifests, and a truncated listing all
go to stderr through the logger, so stdout stays a single parseable
document.
**`snapshot verify`**: Verify snapshot integrity.
* Default (shallow): checks that every blob the manifest lists is present in
+39
View File
@@ -267,6 +267,45 @@ func (r *FileRepository) ListUnderPath(
return files, rows.Err()
}
// ListUnderPathInSnapshot returns the files that ListUnderPath returns
// for path and that the snapshot lists, in no particular order.
func (r *FileRepository) ListUnderPathInSnapshot(
ctx context.Context, path, snapshotID string,
) ([]*File, error) {
path = strings.TrimRight(path, "/")
dirPrefix := path + "/"
query := `
SELECT f.id, f.path, f.source_path, f.mtime, f.mtime_nsec, f.size,
f.mode, f.uid, f.gid, f.link_target
FROM files f
JOIN snapshot_files sf ON sf.file_id = f.id
WHERE sf.snapshot_id = ?
AND (f.path = ? OR substr(f.path, 1, length(?)) = ?)
`
rows, err := r.db.conn.QueryContext(ctx, query,
snapshotID, path, dirPrefix, dirPrefix)
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
defer func() { _ = rows.Close() }()
var files []*File
for rows.Next() {
file, err := r.scanFileRows(rows)
if err != nil {
return nil, fmt.Errorf("scanning file: %w", err)
}
files = append(files, file)
}
return files, rows.Err()
}
// ListIDsWithChunksNotInUploadedBlobs returns the IDs of the files whose
// path starts with prefix and that list at least one chunk held by no
// blob whose upload has completed (uploaded_ts set). A new snapshot
+59
View File
@@ -212,6 +212,65 @@ func TestFileRepositoryListUnderPath(t *testing.T) {
}
}
// ListUnderPathInSnapshot leaves out files outside path and files the
// snapshot does not list, though another snapshot lists them.
func TestFileRepositoryListUnderPathInSnapshot(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
files := database.NewFileRepository(db)
snapshots := database.NewSnapshotRepository(db)
listed := map[string][]string{
"one": {"/src/a.txt", "/src2/c.txt"},
"two": {"/src/b.txt"},
}
for snapshotID, paths := range listed {
err := snapshots.Create(ctx, nil, &database.Snapshot{
ID: types.SnapshotID(snapshotID),
Hostname: "host",
StartedAt: time.Now(),
})
if err != nil {
t.Fatalf("failed to create snapshot %s: %v", snapshotID, err)
}
for _, path := range paths {
err := files.Create(ctx, nil, &database.File{
Path: types.FilePath(path),
MTime: time.Now().Truncate(time.Second),
Mode: 0644,
})
if err != nil {
t.Fatalf("failed to create %s: %v", path, err)
}
err = snapshots.AddFile(ctx, nil, snapshotID, path)
if err != nil {
t.Fatalf("failed to add %s to %s: %v", path, snapshotID, err)
}
}
}
found, err := files.ListUnderPathInSnapshot(ctx, "/src", "one")
if err != nil {
t.Fatalf("failed to list files: %v", err)
}
got := make([]string, 0, len(found))
for _, f := range found {
got = append(got, f.Path.String())
}
if !slices.Equal(got, []string{"/src/a.txt"}) {
t.Errorf("got %q, want only /src/a.txt", got)
}
}
func TestFileRepositorySymlink(t *testing.T) {
t.Parallel()
+15 -11
View File
@@ -16,6 +16,9 @@ type ScannerParams struct {
Fs afero.Fs
Exclude []string // Exclude patterns (combined global + snapshot-specific)
SkipErrors bool // Skip file read errors (log loudly but continue)
// PreviousSnapshotID is the most recent completed snapshot of the
// same name, "" for none.
PreviousSnapshotID string
}
// Module exports backup functionality as an fx module.
@@ -44,17 +47,18 @@ func provideScannerFactory(
}
return NewScanner(ScannerConfig{
FS: params.Fs,
ChunkSize: cfg.ChunkSize.Int64(),
Repositories: repos,
Storage: storer,
MaxBlobSize: cfg.BlobSizeLimit.Int64(),
CompressionLevel: cfg.CompressionLevel,
AgeRecipients: cfg.AgeRecipients,
EnableProgress: params.EnableProgress,
UI: params.UI,
Exclude: excludes,
SkipErrors: params.SkipErrors,
FS: params.Fs,
ChunkSize: cfg.ChunkSize.Int64(),
Repositories: repos,
Storage: storer,
MaxBlobSize: cfg.BlobSizeLimit.Int64(),
CompressionLevel: cfg.CompressionLevel,
AgeRecipients: cfg.AgeRecipients,
EnableProgress: params.EnableProgress,
UI: params.UI,
Exclude: excludes,
SkipErrors: params.SkipErrors,
PreviousSnapshotID: params.PreviousSnapshotID,
})
}
}
+74 -19
View File
@@ -58,6 +58,10 @@ type Scanner struct {
compressionLevel int
ageRecipient string
snapshotID string // Current snapshot being processed
// previousSnapshotID is the most recent completed snapshot of the
// same name, "" for none. Only a file it lists can be counted as
// deleted.
previousSnapshotID string
// currentSourcePath is the source directory being scanned, stored with
// each file record.
currentSourcePath string
@@ -129,6 +133,9 @@ type ScannerConfig struct {
// SkipErrors skips files that cannot be opened or read (log loudly but
// continue); packer, database, encryption, and upload errors still abort.
SkipErrors bool
// PreviousSnapshotID is the most recent completed snapshot of the
// same name, "" for none.
PreviousSnapshotID string
}
// ScanResult contains the results of a scan operation. Files and bytes
@@ -207,6 +214,7 @@ func NewScanner(cfg ScannerConfig) *Scanner {
compiledExclude: compiledExclude,
progress: progress,
skipErrors: cfg.SkipErrors,
previousSnapshotID: cfg.PreviousSnapshotID,
ui: uiw,
pendingChunkHashes: make(map[string]struct{}),
}
@@ -258,8 +266,9 @@ func (s *Scanner) Scan(
filesToProcess := scanResult.FilesToProcess
// Phase 1b: Detect deleted files by comparing DB against scanned files
err = s.detectDeletedFilesFromMap(ctx, knownFiles, existingFiles, result)
// Phase 1b: Detect deleted files by comparing the previous snapshot
// of this name against scanned files
err = s.detectDeletedFiles(ctx, path, existingFiles, result)
if err != nil {
return nil, fmt.Errorf("detecting deleted files: %w", err)
}
@@ -439,9 +448,8 @@ func (s *Scanner) summarizeScanPhase(
}
// loadKnownFiles loads the known files at and beneath path from the
// database into a map for fast lookup. Every loaded file the scan does
// not find is counted as deleted. This avoids per-file database queries
// during the scan phase.
// database into a map for fast lookup. This avoids per-file database
// queries during the scan phase.
func (s *Scanner) loadKnownFiles(
ctx context.Context, path string,
) (map[string]*database.File, error) {
@@ -1954,21 +1962,29 @@ func (s *Scanner) queueFileForBatchInsert(
})
}
// detectDeletedFilesFromMap finds files that existed in previous snapshots
// but no longer exist. Uses pre-loaded maps to avoid any filesystem or
// database access.
func (s *Scanner) detectDeletedFilesFromMap(
// detectDeletedFiles counts as deleted each file at or beneath path that
// the previous snapshot of this name lists, the walk did not find, and
// is gone from disk. A file that is excluded, or inside an excluded
// directory, is not counted: the walk skips it whether or not it is
// still on disk.
func (s *Scanner) detectDeletedFiles(
ctx context.Context,
knownFiles map[string]*database.File,
path string,
existingFiles map[string]struct{},
result *ScanResult,
) error {
if len(knownFiles) == 0 {
if s.previousSnapshotID == "" {
return nil
}
// Check each known file against the enumerated set (no filesystem access needed)
for path, file := range knownFiles {
previousFiles, err := s.repos.Files.ListUnderPathInSnapshot(
ctx, path, s.previousSnapshotID)
if err != nil {
return fmt.Errorf("listing files of snapshot %s: %w",
s.previousSnapshotID, err)
}
for _, file := range previousFiles {
// Check context cancellation periodically
select {
case <-ctx.Done():
@@ -1976,13 +1992,22 @@ func (s *Scanner) detectDeletedFilesFromMap(
default:
}
// Check if the file exists in our enumerated set
if _, exists := existingFiles[path]; !exists {
// File has been deleted
result.FilesDeleted++
result.BytesDeleted += file.Size
log.Debug("Detected deleted file", "path", path, "size", file.Size)
filePath := file.Path.String()
_, exists := existingFiles[filePath]
if exists || s.excludedOrInExcludedDir(filePath, path) {
continue
}
// Under --skip-errors the walk also misses what it cannot reach,
// such as the files in a directory it cannot open.
if !s.goneFromDisk(filePath) {
continue
}
result.FilesDeleted++
result.BytesDeleted += file.Size
log.Debug("Detected deleted file", "path", filePath, "size", file.Size)
}
if result.FilesDeleted > 0 {
@@ -1993,6 +2018,36 @@ func (s *Scanner) detectDeletedFilesFromMap(
return nil
}
// excludedOrInExcludedDir reports whether filePath, which is at or
// beneath rootPath, or any directory between rootPath and it, matches an
// exclude pattern. A pattern such as */.local/share/Trash matches the
// directory but not the files inside it, which the walk skips too.
func (s *Scanner) excludedOrInExcludedDir(filePath, rootPath string) bool {
for p := filePath; len(p) > len(rootPath); p = filepath.Dir(p) {
if s.shouldExclude(p, rootPath) {
return true
}
}
return false
}
// goneFromDisk reports whether looking filePath up on disk finds that
// it does not exist. Like the walk, it does not follow a symlink, so a
// symlink whose target is gone is still on disk. A file whose directory
// was replaced by a regular file is gone too, though the lookup reports
// "not a directory".
func (s *Scanner) goneFromDisk(filePath string) bool {
var err error
if lstater, ok := s.fs.(afero.Lstater); ok {
_, _, err = lstater.LstatIfPossible(filePath)
} else {
_, err = s.fs.Stat(filePath)
}
return errors.Is(err, os.ErrNotExist) || errors.Is(err, syscall.ENOTDIR)
}
// wrapPermissionError augments permission errors with platform-specific
// remediation instructions. On macOS, TCC-protected directories (Calendars,
// Reminders, Photos, etc.) return EPERM unless the running application has
+12 -11
View File
@@ -661,17 +661,18 @@ func faultScannerFactory(
) snapshot.ScannerFactory {
return func(params snapshot.ScannerParams) *snapshot.Scanner {
return snapshot.NewScanner(snapshot.ScannerConfig{
FS: params.Fs,
Storage: storer,
ChunkSize: faultChunkSize,
MaxBlobSize: faultMaxBlobSize,
CompressionLevel: cfg.CompressionLevel,
AgeRecipients: cfg.AgeRecipients,
Repositories: repos,
EnableProgress: params.EnableProgress,
UI: params.UI,
Exclude: params.Exclude,
SkipErrors: params.SkipErrors,
FS: params.Fs,
Storage: storer,
ChunkSize: faultChunkSize,
MaxBlobSize: faultMaxBlobSize,
CompressionLevel: cfg.CompressionLevel,
AgeRecipients: cfg.AgeRecipients,
Repositories: repos,
EnableProgress: params.EnableProgress,
UI: params.UI,
Exclude: params.Exclude,
SkipErrors: params.SkipErrors,
PreviousSnapshotID: params.PreviousSnapshotID,
})
}
}
+42 -5
View File
@@ -12,6 +12,7 @@ import (
"path/filepath"
"slices"
"strings"
"syscall"
"time"
"filippo.io/age"
@@ -219,9 +220,7 @@ func (v *Vaultik) finishRestore(
)
if os.Geteuid() != 0 {
v.UI.Warningf("Restore did not preserve file ownership: chown(2) " +
"requires root. Re-run as root (e.g. with sudo) if you need " +
"original UID/GID preserved.")
v.warnIfOwnershipNotPreserved(files, opts.TargetDir)
}
if result.FilesFailed > 0 {
@@ -247,6 +246,43 @@ func (v *Vaultik) finishRestore(
return unmatchedErr
}
// warnIfOwnershipNotPreserved warns once if some entry restored under
// targetDir has another owner or group on disk than the snapshot
// records. It is meant for a restore by a normal user, which changes no
// owner or group: a new entry is owned by that user, and its group is
// that user's, or the group of the directory it is created in on macOS
// and in a setgid directory on Linux. An entry that is not on disk, or
// whose filesystem reports no owner, is skipped.
func (v *Vaultik) warnIfOwnershipNotPreserved(
files []*database.File, targetDir string,
) {
for _, file := range files {
targetPath, err := containedRestorePath(
v.Fs, targetDir, file.Path.String())
if err != nil {
continue
}
info, err := lstatIfPossible(v.Fs, targetPath)
if err != nil {
continue
}
owner, ok := info.Sys().(*syscall.Stat_t)
if !ok {
continue
}
if owner.Uid != file.UID || owner.Gid != file.GID {
v.UI.Warningf("Restore did not preserve file ownership: chown(2) " +
"requires root. Re-run as root (e.g. with sudo) if you need " +
"original UID/GID preserved.")
return
}
}
}
// reportUnmatchedPaths warns, by name, about each requested path that
// matched nothing in the snapshot, and returns an error counting them,
// or nil when there are none.
@@ -965,8 +1001,9 @@ type restoreSession struct {
// root can chown a file to an arbitrary UID/GID — non-root chown
// always fails with EPERM. Attempting it anyway produces N
// guaranteed-failed syscalls + N noisy debug lines, so we skip
// the call entirely as non-root and emit one warning at the end
// of the restore explaining that ownership was not preserved.
// the call entirely as non-root; warnIfOwnershipNotPreserved warns
// once at the end of the restore if that left any owner or group
// other than recorded.
runningAsRoot bool
// directories holds every restored directory, for
// applyDirectoryMetadata to finish after the restore loop.
@@ -0,0 +1,131 @@
package vaultik //nolint:testpackage // calls the unexported ownership check
import (
"bytes"
"context"
"os"
"strings"
"syscall"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/ui"
)
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/316: a
// restore by a normal user warns that ownership was not preserved only
// when some restored entry has another owner or group on disk than the
// snapshot records.
const ownershipWarning = "did not preserve file ownership"
// TestOwnershipWarningSilentWhenEntriesMatch checks that a restore whose
// entries all end up with their recorded owner and group prints no
// warning.
func TestOwnershipWarningSilentWhenEntriesMatch(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
targetDir := t.TempDir()
owner := ownerOf(t, targetDir)
out := restoreAsNormalUser(t, targetDir, []*database.File{
{Path: "/a", Mode: plainFileMode, UID: owner.Uid, GID: owner.Gid},
{Path: "/b", Mode: plainFileMode, UID: owner.Uid, GID: owner.Gid},
})
assert.Empty(t, out)
}
// TestOwnershipWarningWhenOwnerDiffers checks that the warning is printed
// when an entry recorded with another owner is restored owned by the
// restoring user, even though its group matches the recorded one.
func TestOwnershipWarningWhenOwnerDiffers(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
targetDir := t.TempDir()
owner := ownerOf(t, targetDir)
out := restoreAsNormalUser(t, targetDir, []*database.File{
{Path: "/a", Mode: plainFileMode, UID: otherOwnerID, GID: owner.Gid},
})
assert.Equal(t, 1, strings.Count(out, ownershipWarning), out)
}
// TestOwnershipWarningWhenDirectoryGivesItsGroup checks that the warning
// is printed, once, when entries recorded with the restoring user's
// owner and group take another group from the setgid directory they are
// restored into, as every new entry takes its directory's group on
// macOS.
func TestOwnershipWarningWhenDirectoryGivesItsGroup(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
if os.Geteuid() != 0 {
t.Skip("giving a directory to another group needs root")
}
targetDir := t.TempDir()
user := ownerOf(t, targetDir)
require.NoError(t, os.Chown(targetDir, -1, int(otherOwnerID)))
require.NoError(t, os.Chmod(targetDir, os.ModeSetgid|writableTestMode))
out := restoreAsNormalUser(t, targetDir, []*database.File{
{Path: "/a", Mode: plainFileMode, UID: user.Uid, GID: user.Gid},
{Path: "/b", Mode: plainFileMode, UID: user.Uid, GID: user.Gid},
})
assert.Equal(t, 1, strings.Count(out, ownershipWarning), out)
}
// ownerOf returns the owner and group path has on disk.
func ownerOf(t *testing.T, path string) *syscall.Stat_t {
t.Helper()
info, err := os.Lstat(path)
require.NoError(t, err)
owner, ok := info.Sys().(*syscall.Stat_t)
require.True(t, ok)
return owner
}
// restoreAsNormalUser restores rows, all empty regular files, into
// targetDir without changing any owner or group, as restore does for a
// normal user, and returns what warnIfOwnershipNotPreserved prints.
func restoreAsNormalUser(
t *testing.T, targetDir string, rows []*database.File,
) string {
t.Helper()
ctx := context.Background()
rows, repos := makeFiles(ctx, t, rows)
var out bytes.Buffer
v := newContainmentVaultik(ctx, afero.NewOsFs())
v.UI = ui.NewWithColor(&out, false)
session := &restoreSession{
v: v,
ctx: ctx,
repos: repos,
opts: &RestoreOptions{TargetDir: targetDir},
result: &RestoreResult{},
}
for _, row := range rows {
require.NoError(t, session.restoreFile(row))
}
v.warnIfOwnershipNotPreserved(rows, targetDir)
return out.String()
}
+29 -5
View File
@@ -244,12 +244,18 @@ func (v *Vaultik) createNamedSnapshot(
return err
}
previousSnapshotID, err := v.previousSnapshotID(snapName)
if err != nil {
return err
}
scanner := v.ScannerFactory(snapshot.ScannerParams{
EnableProgress: !opts.Cron,
UI: v.UI,
Fs: v.Fs,
Exclude: v.Config.GetExcludes(snapName),
SkipErrors: opts.SkipErrors,
EnableProgress: !opts.Cron,
UI: v.UI,
Fs: v.Fs,
Exclude: v.Config.GetExcludes(snapName),
SkipErrors: opts.SkipErrors,
PreviousSnapshotID: previousSnapshotID,
})
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
@@ -284,6 +290,24 @@ func (v *Vaultik) createNamedSnapshot(
return nil
}
// previousSnapshotID returns the ID of the most recent completed snapshot
// named snapName in the local index, or "" when there is none.
func (v *Vaultik) previousSnapshotID(snapName string) (string, error) {
snapshots, err := v.Repositories.Snapshots.ListRecent(v.ctx, listRecentLimit)
if err != nil {
return "", fmt.Errorf("listing snapshots: %w", err)
}
for _, s := range snapshots {
if s.CompletedAt != nil &&
parseSnapshotName(s.ID.String(), s.Hostname.String()) == snapName {
return s.ID.String(), nil
}
}
return "", nil
}
// resolveSnapshotPaths resolves source directories to absolute paths
// with symlink resolution.
func (v *Vaultik) resolveSnapshotPaths(snapName string) ([]string, error) {
@@ -0,0 +1,249 @@
package vaultik_test
import (
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/304: a
// backup counts a file as deleted only when the previous snapshot of
// the same name had it and it is gone, and never counts an excluded
// file.
// deletedCountEnv is a backup setup over the source directory src whose
// user-facing output is kept in out. Each test configures its snapshots
// with setSnapshot.
type deletedCountEnv struct {
v *vaultik.Vaultik
out *bytes.Buffer
src string
}
// newDeletedCountEnv writes files, keyed by path relative to src.
func newDeletedCountEnv(t *testing.T, files map[string][]byte) *deletedCountEnv {
t.Helper()
ctx := context.Background()
fs := afero.NewOsFs()
tempDir := t.TempDir()
srcDir := filepath.Join(tempDir, "src")
for rel, content := range files {
path := filepath.Join(srcDir, rel)
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o755))
require.NoError(t, afero.WriteFile(fs, path, content, 0o644))
}
cfg := faultTestConfig()
cfg.IndexPath = filepath.Join(tempDir, "index.sqlite")
cfg.Snapshots = map[string]config.SnapshotConfig{}
store, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
require.NoError(t, err)
db, err := database.New(ctx, cfg.IndexPath)
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
out := &bytes.Buffer{}
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
v.UI = ui.NewWithColor(out, false)
return &deletedCountEnv{v: v, out: out, src: srcDir}
}
// setSnapshot configures the snapshot name to back up the path rel
// beneath src, with the given excludes.
func (e *deletedCountEnv) setSnapshot(name, rel string, exclude ...string) {
e.v.Config.Snapshots[name] = config.SnapshotConfig{
Paths: []string{filepath.Join(e.src, rel)},
Exclude: exclude,
}
}
// backUp runs a backup of the named snapshot and returns its output.
func (e *deletedCountEnv) backUp(t *testing.T, name string) string {
t.Helper()
e.out.Reset()
require.NoError(t, e.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Snapshots: []string{name},
}))
return e.out.String()
}
// A file this snapshot excludes is not counted as deleted: neither one
// that another snapshot backs up, nor one that the previous snapshot of
// this name had before the exclude was added.
func TestBackupDoesNotCountExcludedFileAsDeleted(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newDeletedCountEnv(t, map[string][]byte{
"top.txt": []byte("top"),
"cache/tmp.bin": []byte("cache"),
"photos/a.jpg": []byte("photo a"),
"photos/b.jpg": []byte("photo b"),
"photos/c.jpg": []byte("photo c"),
})
env.setSnapshot("photos", "photos")
env.setSnapshot("home", "", "/photos")
env.backUp(t, "photos")
assert.Contains(t, env.backUp(t, "home"), filesLine(2, 2, 0))
env.setSnapshot("home", "", "/photos", "/cache")
assert.Contains(t, env.backUp(t, "home"), filesLine(1, 0, 1))
}
// A file inside a directory that an exclude added after a backup
// matches is not counted as deleted, though the pattern matches only the
// directory and the directory is gone from disk.
func TestBackupDoesNotCountFileInExcludedDirectoryAsDeleted(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newDeletedCountEnv(t, map[string][]byte{
"top.txt": []byte("top"),
"user/.local/share/Trash/old.txt": []byte("old"),
})
env.setSnapshot("home", "")
assert.Contains(t, env.backUp(t, "home"), filesLine(2, 2, 0))
require.NoError(t, os.RemoveAll(
filepath.Join(env.src, "user/.local/share/Trash")))
env.setSnapshot("home", "", "*/.local/share/Trash")
assert.Contains(t, env.backUp(t, "home"), filesLine(1, 0, 1))
}
// A file whose directory was replaced by a regular file of the same name
// is counted as deleted.
func TestBackupCountsFileInDirectoryReplacedByFileAsDeleted(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newDeletedCountEnv(t, map[string][]byte{
"dir/inside.txt": []byte("inside"),
})
env.setSnapshot("home", "")
assert.Contains(t, env.backUp(t, "home"), filesLine(1, 1, 0))
dir := filepath.Join(env.src, "dir")
require.NoError(t, os.RemoveAll(dir))
require.NoError(t, os.WriteFile(dir, []byte("now a file"), 0o600))
assert.Contains(t, env.backUp(t, "home"),
"Files: 1 examined, 1 backed up, 0 unchanged, 1 deleted.")
}
// Under --skip-errors a file inside a directory that the walk cannot
// open is not counted as deleted: it is still on disk.
func TestBackupDoesNotCountFileInUnopenableDirectoryAsDeleted(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newDeletedCountEnv(t, map[string][]byte{
"open.txt": []byte("open"),
"locked/inside.txt": []byte("inside"),
})
env.setSnapshot("work", "")
assert.Contains(t, env.backUp(t, "work"), filesLine(2, 2, 0))
// A directory's mode does not lock out root, who runs the tests, so
// opening the directory is made to fail instead.
env.v.Fs = &openFailFs{
path: filepath.Join(env.src, "locked"),
err: os.ErrPermission,
}
env.out.Reset()
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
SkipErrors: true,
Snapshots: []string{"work"},
}))
assert.Contains(t, env.out.String(), filesLine(1, 0, 1))
}
// A backup counts only files that the previous snapshot of its own name
// had, not one that a later snapshot of another name over the same path
// had and that is since removed.
func TestBackupIgnoresSnapshotOfAnotherName(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newDeletedCountEnv(t, map[string][]byte{
"kept.txt": []byte("kept"),
})
env.setSnapshot("home", "")
env.setSnapshot("other", "")
env.backUp(t, "home")
// Snapshot start times are kept in whole seconds: wait so that other's
// snapshot is strictly the most recent.
time.Sleep(time.Second)
extra := filepath.Join(env.src, "extra.txt")
require.NoError(t, os.WriteFile(extra, []byte("extra"), 0o600))
env.backUp(t, "other")
require.NoError(t, os.Remove(extra))
assert.Contains(t, env.backUp(t, "home"), filesLine(1, 0, 1))
}
// A file removed from disk is counted as deleted by the next backup of
// the snapshot that had it, and not by any later backup of that
// snapshot, nor by the first backup of another name over the same path.
func TestBackupCountsDeletedFileOnce(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
aContent := bytes.Repeat([]byte("a"), 1000)
bContent := bytes.Repeat([]byte("b"), 2000)
keptContent := bytes.Repeat([]byte("k"), 500)
env := newDeletedCountEnv(t, map[string][]byte{
"a.txt": aContent,
"b.txt": bContent,
"kept.txt": keptContent,
})
env.setSnapshot("home", "")
env.setSnapshot("other", "")
assert.Contains(t, env.backUp(t, "home"), filesLine(3, 3, 0))
require.NoError(t, os.Remove(filepath.Join(env.src, "a.txt")))
require.NoError(t, os.Remove(filepath.Join(env.src, "b.txt")))
summary := env.backUp(t, "home")
assert.Contains(t, summary,
"Files: 1 examined, 0 backed up, 1 unchanged, 2 deleted.")
assert.Contains(t, summary,
fmt.Sprintf("Data: %s total (%s backed up), %s deleted.",
env.v.UI.Size(int64(len(keptContent))), env.v.UI.Size(0),
env.v.UI.Size(int64(len(aContent)+len(bContent)))))
assert.Contains(t, env.backUp(t, "home"), filesLine(1, 0, 1))
assert.Contains(t, env.backUp(t, "other"), filesLine(1, 0, 1))
}
+33 -4
View File
@@ -61,7 +61,9 @@ const tabPadding = 3
// attempt is made to recover or fabricate the human ID.
//
// Snapshots in the local index with no counterpart on the destination
// store are reported as drift below the table.
// store are reported as drift below the table, unless the local index
// is bound to a different destination than the configured one: that is
// reported instead.
//
// If remote listing fails (unmounted volume, permission denied,
// network), we degrade to local-only with a warning. List never fails
@@ -103,7 +105,26 @@ func (v *Vaultik) ListSnapshots(jsonOutput bool) error {
return snapshots[i].Timestamp.After(snapshots[j].Timestamp)
})
stored, configured, err := v.storageBinding()
if err != nil {
return err
}
// A local index bound to another destination describes that one, so
// its records are expected to be missing here, and prune refuses to
// run. The binding is reported instead of advising prune.
boundElsewhere := stored != "" && stored != configured
if jsonOutput {
// stdout holds only the document, so this goes to the logger.
if boundElsewhere {
log.Warn("Local index is bound to a different backup "+
"destination; to proceed, revert storage_url (or the s3.* "+
"endpoint, bucket and prefix) to the bound destination, "+
"or run 'vaultik database delete'",
"bound_to", stored, "configured", configured)
}
if remoteErr == nil {
v.reportJSONListingLimits(listing)
}
@@ -125,8 +146,13 @@ func (v *Vaultik) ListSnapshots(jsonOutput bool) error {
}
}
if boundElsewhere {
v.UI.Warningf("%v\n%s", errStorageBindingMismatch,
buildBindingMismatchMessage(stored, configured))
}
if remoteErr == nil {
v.reportListDrift(snapshots, listing)
v.reportListDrift(snapshots, listing, boundElsewhere)
}
return nil
@@ -432,7 +458,9 @@ func (v *Vaultik) snapshotInfoFromLocal(ls *database.Snapshot) SnapshotInfo {
// reportListDrift prints the reconciliation notes the merged table
// cannot express on its own: local records with no counterpart on the
// destination store, plus counts of remote snapshots that were
// unreadable or omitted.
// unreadable or omitted. Local records are not reported when
// boundElsewhere is set: ListSnapshots has already said why they are
// missing.
//
// This is what remains of the old reportRemoteDrift, and it no longer
// touches the destination store. Its remote-only half collapsed into
@@ -441,6 +469,7 @@ func (v *Vaultik) snapshotInfoFromLocal(ls *database.Snapshot) SnapshotInfo {
// so `snapshot list` lists the destination exactly once per invocation.
func (v *Vaultik) reportListDrift(
snapshots []SnapshotInfo, listing *remoteSnapshotListing,
boundElsewhere bool,
) {
var localOnly []string
@@ -450,7 +479,7 @@ func (v *Vaultik) reportListDrift(
}
}
if len(localOnly) > 0 {
if len(localOnly) > 0 && !boundElsewhere {
v.UI.Warningf("%d local snapshot record(s) not found in backup "+
"destination store:", len(localOnly))
+68
View File
@@ -421,6 +421,74 @@ func TestListSnapshots_LocalOnlyReportedAsDrift(t *testing.T) {
assert.NotContains(t, out, "snapshot cleanup")
}
// TestListSnapshots_BoundElsewhereDoesNotAdvisePrune covers a changed
// storage_url: the local index is still bound to the old destination,
// so its records are missing from the new one. prune refuses to run in
// that state, so the listing must give the choices prune gives instead
// of advising it.
func TestListSnapshots_BoundElsewhereDoesNotAdvisePrune(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
const (
oldURL = "file:///mnt/backups/old"
newURL = "file:///mnt/backups/new"
)
env := newListEnv(t)
env.v.Config.StorageURL = newURL
require.NoError(t, env.v.Repositories.LocalMeta.Set(context.Background(),
database.LocalMetaKeyStorageURL, oldURL))
env.addLocal(t, listLocalID, time.Date(2026, 3, 1, 10, 0, 0, 0, time.UTC))
err := env.v.ListSnapshots(false)
require.NoError(t, err)
out := env.stdout.String()
assert.Contains(t, out, listLocalID)
assert.Contains(t, out,
"local index is bound to a different backup destination")
assert.Contains(t, out, oldURL)
assert.Contains(t, out, newURL)
assert.Contains(t, out, "revert storage_url")
assert.Contains(t, out, "vaultik database delete")
assert.NotContains(t, out, "vaultik prune")
}
// TestListSnapshots_JSONReportsBoundElsewhere is the --json side of the
// test above. Every local row comes out with remote_present false, which
// a script reads as stale local records unless it is told, on stderr,
// that the local index is bound to a different destination.
//
//nolint:paralleltest // captureProcessStderr replaces os.Stderr
func TestListSnapshots_JSONReportsBoundElsewhere(t *testing.T) {
const (
oldURL = "file:///mnt/backups/old"
newURL = "file:///mnt/backups/new"
)
env := newListEnv(t)
env.v.Config.StorageURL = newURL
require.NoError(t, env.v.Repositories.LocalMeta.Set(context.Background(),
database.LocalMetaKeyStorageURL, oldURL))
env.addLocal(t, listLocalID, time.Date(2026, 3, 1, 10, 0, 0, 0, time.UTC))
stderr := captureProcessStderr(t, func() {
require.NoError(t, env.v.ListSnapshots(true))
})
rows := decodeListJSON(t, env.stdout.String())
require.Len(t, rows, 1)
assert.Equal(t, listLocalID, rows[0].ID)
assert.Contains(t, stderr, "bound to a different backup destination")
assert.Contains(t, stderr, oldURL)
assert.Contains(t, stderr, newURL)
assert.Contains(t, stderr, "revert storage_url")
assert.Contains(t, stderr, "vaultik database delete")
}
// TestListSnapshots_UnreachableRemoteDegrades covers the promise in the
// doc comment: an unreachable destination is a warning plus local-only
// output, never a failure.
+23 -10
View File
@@ -45,17 +45,9 @@ func (v *Vaultik) EnsureStorageBinding() error {
return nil
}
configured := v.Config.StorageURL
if configured == "" {
// The s3.* form: bind on its endpoint, bucket and prefix,
// written the way storage_url would name the same destination.
s3 := v.Config.S3
configured = "s3://" + s3.Bucket + "/" + s3.Prefix + "?endpoint=" + s3.Endpoint
}
stored, err := v.Repositories.LocalMeta.Get(v.ctx, database.LocalMetaKeyStorageURL)
stored, configured, err := v.storageBinding()
if err != nil {
return fmt.Errorf("reading local storage binding: %w", err)
return err
}
if stored == "" {
@@ -78,6 +70,27 @@ func (v *Vaultik) EnsureStorageBinding() error {
errStorageBindingMismatch, buildBindingMismatchMessage(stored, configured))
}
// storageBinding returns the destination the local index is bound to
// ("" when no binding is recorded yet), then the configured destination
// written the same way. It records nothing, so a read-only command can
// call it.
func (v *Vaultik) storageBinding() (string, string, error) {
configured := v.Config.StorageURL
if configured == "" {
// The s3.* form: bind on its endpoint, bucket and prefix,
// written the way storage_url would name the same destination.
s3 := v.Config.S3
configured = "s3://" + s3.Bucket + "/" + s3.Prefix + "?endpoint=" + s3.Endpoint
}
stored, err := v.Repositories.LocalMeta.Get(v.ctx, database.LocalMetaKeyStorageURL)
if err != nil {
return "", "", fmt.Errorf("reading local storage binding: %w", err)
}
return stored, configured, nil
}
// buildBindingMismatchMessage assembles the multi-line explanation
// shown when the local index is bound to a different destination than
// the currently-configured one (the first line lives in the