1 Commits
Author SHA1 Message Date
clawbot cea4b0dd05 Do not prune after a backup that found a volume not mounted (closes #395)
check / check (push) Canceled after 0s
snapshot create --prune purged the older snapshots holding a volume's
files when the volume, mounted below a configured path, was not
mounted: its empty mount point read as deleted files.

The scan now records in the local index each directory at or below a
configured path that is on a different device from its parent
directory. Under --prune, a run that does not find such a directory
recorded by an earlier snapshot of that name skips the prune, keeps
every older snapshot, names the directory and its snapshot, and exits
non-zero, as for an empty configured path. A volume retired on purpose
is cleared by removing the older snapshots with snapshot purge.

Model: opus-5-5
2026-10-15 01:51:28 +00:00
11 changed files with 107 additions and 427 deletions
+8 -11
View File
@@ -337,17 +337,14 @@ value where a list belongs or a word where a number belongs), is refused.
path that is empty on purpose out of the config, or remove the older path that is empty on purpose out of the config, or remove the older
snapshots by hand with `snapshot purge`. When an earlier snapshot of the snapshots by hand with `snapshot purge`. When an earlier snapshot of the
same name in the local index found a volume mounted at or below a same name in the local index found a volume mounted at or below a
configured path (a directory on a different filesystem from its parent, configured path (a directory on a different filesystem from its parent)
with an entry in it that the backup does not exclude) and this backup does and this backup finds none mounted there, the prune does not run either:
not, the prune does not run either: every older snapshot is kept, and the every older snapshot is kept, and the command names the directory and
command names the directory and exits non-zero, on every run until the exits non-zero, on every run until the volume is mounted again. The same
volume is mounted again. A backup also does not find the volume when it happens when that directory is no longer backed up, because it is excluded
finds no such entry in its directory, as when an automounted volume cannot or its configured path was removed. For a volume retired on purpose, remove
be mounted, or when that directory is no longer backed up, because it is the older snapshots by hand with `snapshot purge`. A file removed during
excluded or its configured path was removed. If any of this is on purpose, the backup does not stop the prune.
as for a retired volume, remove the older snapshots by hand with
`snapshot purge`. A file removed during the backup does not stop the
prune.
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than * `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
Without `--prune`, or with a value that is not a valid duration, the Without `--prune`, or with a value that is not a valid duration, the
+1 -2
View File
@@ -30,8 +30,7 @@ owner's; a `v*` tag runs the release build in
([issue #395](https://git.eeqj.de/sneak/vaultik/issues/395)). Such a ([issue #395](https://git.eeqj.de/sneak/vaultik/issues/395)). Such a
prune could delete every snapshot still holding the volume's files. prune could delete every snapshot still holding the volume's files.
Each snapshot now records in the local index the directories it found Each snapshot now records in the local index the directories it found
on a different filesystem from their parent and found an entry in that on a different filesystem from their parent.
the backup does not exclude.
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup - 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
that found nothing in a configured path that found nothing in a configured path
+3 -3
View File
@@ -154,14 +154,14 @@ Maps snapshots to the blobs they reference.
**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation. **Purpose:** Tracks blob dependencies for snapshots and enables manifest generation.
### 10. `snapshot_mount_points` ### 10. `snapshot_mount_points`
The directories at or below a configured path, the configured path itself included, that a snapshot's backup found an entry in and, once it had listed them, found on a different filesystem from their parent directory (a different device number), as a mount point is. An automount point whose volume cannot be mounted is not recorded, because the backup finds no entry in it. The directories at or below a configured path that a snapshot's backup found on a different filesystem from their parent directory (a different device number), as a mount point is, the configured path itself included.
**Columns:** **Columns:**
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id) - `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
- `path` (TEXT) - Absolute path of the directory - `path` (TEXT) - Absolute path of the directory
- PRIMARY KEY (`snapshot_id`, `path`) - PRIMARY KEY (`snapshot_id`, `path`)
**Purpose:** Lets `snapshot create --prune` skip the prune when a backup does not find a mount point that an earlier snapshot of the same name found, as when the volume is not mounted. **Purpose:** Lets `snapshot create --prune` skip the prune when a volume an earlier snapshot of the same name found mounted is not mounted now.
### 11. `uploads` ### 11. `uploads`
Tracks blob upload metrics. Tracks blob upload metrics.
@@ -183,7 +183,7 @@ Tracks blob upload metrics.
- `INSERT OR REPLACE INTO files` - Update file metadata - `INSERT OR REPLACE INTO files` - Update file metadata
- `SELECT * FROM files WHERE path = ?` - Check if file has changed - `SELECT * FROM files WHERE path = ?` - Check if file has changed
- `INSERT INTO snapshot_files` - Add file to current snapshot - `INSERT INTO snapshot_files` - Add file to current snapshot
- `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a different filesystem from its parent that the walk found an entry in - `INSERT INTO snapshot_mount_points` - Record each directory on a different filesystem from its parent
2. **Chunking** (for changed files) 2. **Chunking** (for changed files)
- `INSERT OR IGNORE INTO chunks` - Store new chunks - `INSERT OR IGNORE INTO chunks` - Store new chunks
@@ -1,7 +1,6 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import ( import (
"context"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -9,7 +8,6 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
) )
// Flags of `snapshot create` used to build argument vectors below. // Flags of `snapshot create` used to build argument vectors below.
@@ -119,51 +117,6 @@ func TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty(t *testing.T) {
assert.Len(t, snapshots, 2, "the older snapshot is kept") assert.Len(t, snapshots, 2, "the older snapshot is kept")
} }
// TestEntrySnapshotCreatePruneExitsNonZeroWhenMountPointIsMissing backs
// up, records in the local index that this backup found the source
// directory as a mount point, and runs `snapshot create --prune`, which
// does not find it as one. The command exits non-zero, reports the
// backup as completed, not as a failed snapshot creation, and keeps the
// older snapshot. See https://git.eeqj.de/sneak/vaultik/issues/395.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreatePruneExitsNonZeroWhenMountPointIsMissing(
t *testing.T,
) {
configPath, storeDir := writeBackupConfig(t)
code, _, _ := runEntry(t, flagConfig, configPath, cmdSnapshot, cmdCreate)
require.Equal(t, 0, code)
// writeBackupConfig puts the source directory and the local index
// beside the config file.
dir := filepath.Dir(configPath)
ctx := context.Background()
db, err := database.New(ctx, filepath.Join(dir, "index.sqlite"))
require.NoError(t, err)
repos := database.NewRepositories(db)
backedUp, err := repos.Snapshots.GetOnlySnapshot(ctx)
require.NoError(t, err)
require.NoError(t, repos.Snapshots.AddMountPoint(ctx, nil,
backedUp.ID.String(), filepath.Join(dir, "source")))
require.NoError(t, db.Close())
code, _, stderr := runEntry(t,
flagConfig, configPath, cmdSnapshot, cmdCreate, flagPrune)
assert.Equal(t, 1, code)
assert.Contains(t, stderr, "Backup completed, but 1 mount point(s) "+
"an earlier snapshot recorded were not found")
assert.NotContains(t, stderr, "Snapshot creation failed")
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
require.NoError(t, err)
assert.Len(t, snapshots, 2, "the older snapshot is kept")
}
// writeBackupConfig builds a hermetic config with one small file in its // writeBackupConfig builds a hermetic config with one small file in its
// source directory and an empty destination store directory. Returns the // source directory and an empty destination store directory. Returns the
// config path and the store directory. // config path and the store directory.
+2 -2
View File
@@ -138,8 +138,8 @@ on Linux), then /etc/vaultik/config.yml.`,
"directory under metadata/ with no manifest once all its "+ "directory under metadata/ with no manifest once all its "+
"objects are older than prune_grace_period and its snapshot is "+ "objects are older than prune_grace_period and its snapshot is "+
"not in the local index (not after a backup that skipped "+ "not in the local index (not after a backup that skipped "+
"unreadable files, found a configured path empty, or did not "+ "unreadable files, found a configured path empty, or found "+
"find a mount point an earlier snapshot found)") "no volume where an earlier snapshot found one mounted)")
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "", cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
"With --prune: keep snapshots newer than this duration "+ "With --prune: keep snapshots newer than this duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+ "(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
+1 -1
View File
@@ -125,7 +125,7 @@ CREATE INDEX IF NOT EXISTS idx_snapshot_blobs_blob_id ON snapshot_blobs(blob_id)
-- Snapshot mount points table: the directories at or below a configured -- Snapshot mount points table: the directories at or below a configured
-- path that the snapshot's backup found on a different filesystem from -- path that the snapshot's backup found on a different filesystem from
-- their parent directory, as a mount point is, and found an entry in -- their parent directory, as a mount point is
CREATE TABLE IF NOT EXISTS snapshot_mount_points ( CREATE TABLE IF NOT EXISTS snapshot_mount_points (
snapshot_id TEXT NOT NULL, snapshot_id TEXT NOT NULL,
path TEXT NOT NULL, path TEXT NOT NULL,
-40
View File
@@ -1,40 +0,0 @@
//nolint:testpackage // exercises the unexported deviceNumberFromStat
package snapshot
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// The device number a real backup reads is the same for a directory and
// a file in it, and differs between the temp directory and /dev.
// Without it no mount point is ever found. See
// https://git.eeqj.de/sneak/vaultik/issues/395.
func TestDeviceNumberFromStat(t *testing.T) {
t.Parallel()
dir := t.TempDir()
file := filepath.Join(dir, "file.txt")
require.NoError(t, os.WriteFile(file, nil, 0o600))
deviceOf := func(path string) uint64 {
t.Helper()
info, err := os.Stat(path)
require.NoError(t, err)
device, ok := deviceNumberFromStat(info)
require.True(t, ok, "no device number read for %s", path)
return device
}
assert.Equal(t, deviceOf(dir), deviceOf(file))
// /dev is a different filesystem from the temp dir on Linux, macOS and in Docker.
assert.NotEqual(t, deviceOf(dir), deviceOf("/dev"))
}
+3 -151
View File
@@ -5,8 +5,6 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"slices" "slices"
"strings"
"syscall"
"testing" "testing"
"github.com/spf13/afero" "github.com/spf13/afero"
@@ -17,10 +15,9 @@ import (
) )
// A scan records with the snapshot each directory on a different device // A scan records with the snapshot each directory on a different device
// from its parent directory that the walk found an entry in, as a mount // from its parent directory, as a mount point is, the scanned path
// point is, the scanned path included. A filesystem that gives no device // included. A filesystem that gives no device numbers, as the in-memory
// numbers, as the in-memory one, records none. See // one, records none. See https://git.eeqj.de/sneak/vaultik/issues/395.
// https://git.eeqj.de/sneak/vaultik/issues/395.
func TestScanRecordsMountPoints(t *testing.T) { func TestScanRecordsMountPoints(t *testing.T) {
t.Parallel() t.Parallel()
@@ -98,148 +95,3 @@ func TestScanRecordsMountPoints(t *testing.T) {
}) })
} }
} }
// automountFs is an in-memory filesystem with a volume at mountPoint
// that is mounted on first access: opening mountPoint to list it mounts
// the volume or, if it cannot be mounted, fails with "no such file or
// directory", as with the autofs daemon. Stat records the device an
// entry is on at that moment, as a real stat does: unmountedDevice for
// mountPoint until the volume is mounted; the volume's, 2, for
// mountPoint once it is and for everything in it; 1 for everything else.
type automountFs struct {
afero.Fs
mountPoint string
unmountedDevice uint64
canMount bool
mounted bool
}
//nolint:ireturn // afero.Fs.Open is defined to return the interface.
func (fs *automountFs) Open(name string) (afero.File, error) {
if name == fs.mountPoint && !fs.canMount {
return nil, &os.PathError{Op: "open", Path: name, Err: syscall.ENOENT}
}
if name == fs.mountPoint {
fs.mounted = true
}
return fs.Fs.Open(name)
}
func (fs *automountFs) Stat(name string) (os.FileInfo, error) {
info, err := fs.Fs.Stat(name)
if err != nil {
return nil, err
}
device := uint64(1)
switch {
case name == fs.mountPoint && !fs.mounted:
device = fs.unmountedDevice
case name == fs.mountPoint || strings.HasPrefix(name, fs.mountPoint+"/"):
device = 2
}
return statWithDevice{FileInfo: info, device: device}, nil
}
// statWithDevice is an entry automountFs.Stat looked up, with the device
// it was on then.
type statWithDevice struct {
os.FileInfo
device uint64
}
// deviceAtLookup returns the device automountFs.Stat recorded for info.
func deviceAtLookup(info os.FileInfo) (uint64, bool) {
stat, ok := info.(statWithDevice)
return stat.device, ok
}
// A volume mounted on first access is mounted when the walk lists its
// mount point. Until then the mount point is on the automounter's own
// device (a systemd automount) or on its parent's (an autofs map that
// shows its mount points before mounting them). Either way only the
// mount point is recorded, by the backup that mounts the volume and by a
// later one that finds it mounted. Listing an automount point whose
// volume cannot be mounted fails; it is not recorded, because the walk
// finds no entry in it. See https://git.eeqj.de/sneak/vaultik/issues/395.
func TestScanRecordsOnlyTheMountPointOfAnAutomountedVolume(t *testing.T) {
t.Parallel()
const scannedPath = "/data/source"
mountPoint := scannedPath + "/nas"
tests := []struct {
name string
unmountedDevice uint64
canMount bool
want []string
}{
{
name: "on the automounter's device until listed",
unmountedDevice: 3, canMount: true, want: []string{mountPoint},
},
{
name: "on its parent's device until listed",
unmountedDevice: 1, canMount: true, want: []string{mountPoint},
},
{
name: "volume that cannot be mounted",
unmountedDevice: 1, canMount: false, want: nil,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx := context.Background()
fs := &automountFs{
Fs: afero.NewMemMapFs(), mountPoint: mountPoint,
unmountedDevice: tt.unmountedDevice, canMount: tt.canMount,
}
require.NoError(t, fs.MkdirAll(mountPoint+"/photos", 0o755))
for _, path := range []string{
scannedPath + "/notes.txt", mountPoint + "/photos/a.jpg",
} {
require.NoError(t, afero.WriteFile(fs, path, []byte(path), 0o644))
}
db, err := database.NewTestDB()
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
for _, snapshotID := range []string{"first", "later"} {
createTestSnapshotRecord(ctx, t, repos, snapshotID)
scanner := snapshot.NewScanner(snapshot.ScannerConfig{
FS: fs,
ChunkSize: int64(1024 * 16),
Repositories: repos,
MaxBlobSize: int64(1024 * 1024),
CompressionLevel: 3,
AgeRecipients: []string{testAgePublicKey},
DeviceNumber: deviceAtLookup,
})
_, err = scanner.Scan(ctx, scannedPath, snapshotID)
require.NoError(t, err)
got, err := repos.Snapshots.GetMountPoints(ctx, snapshotID)
require.NoError(t, err)
assert.Equal(t, tt.want, got, "mount points of the %s backup", snapshotID)
}
})
}
}
+57 -78
View File
@@ -310,7 +310,7 @@ func (s *Scanner) Scan(
} }
} }
err = s.recordDirectories(ctx, scanResult) err = s.recordDirectories(ctx, scanResult.Directories)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -888,11 +888,6 @@ type ScanPhaseResult struct {
FilesToProcess []*FileToProcess FilesToProcess []*FileToProcess
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
Directories []*FileToProcess // Every directory found, to record Directories []*FileToProcess // Every directory found, to record
// MountPoints are the directories, the scanned path included, that
// the walk found an entry in and, once it had listed them, on a
// different device from their parent directory (see
// recordIfMountPoint).
MountPoints []string
} }
// scanPhase performs the initial directory scan to identify files to process // scanPhase performs the initial directory scan to identify files to process
@@ -912,7 +907,7 @@ func (s *Scanner) scanPhase(
// estimate leaves them out. // estimate leaves them out.
estimatedTotal := countNonDirectories(knownFiles) estimatedTotal := countNonDirectories(knownFiles)
collector := &scanCollector{dirsWithEntries: make(map[string]struct{})} collector := &scanCollector{}
// Set up periodic status output // Set up periodic status output
startTime := time.Now() startTime := time.Now()
@@ -957,8 +952,6 @@ func (s *Scanner) scanPhase(
result.EntriesFound++ result.EntriesFound++
} }
s.recordIfMountPoint(filePath, path, collector)
// Handle symlinks and directories // Handle symlinks and directories
handled, err := s.recordSpecialEntry( handled, err := s.recordSpecialEntry(
filePath, info, existingFiles, knownFiles, collector, result) filePath, info, existingFiles, knownFiles, collector, result)
@@ -993,7 +986,6 @@ func (s *Scanner) scanPhase(
FilesToProcess: collector.filesToProcess, FilesToProcess: collector.filesToProcess,
UnchangedFileIDs: collector.unchangedFileIDs, UnchangedFileIDs: collector.unchangedFileIDs,
Directories: collector.directories, Directories: collector.directories,
MountPoints: collector.mountPoints,
}, nil }, nil
} }
@@ -1120,29 +1112,6 @@ type scanCollector struct {
unchangedFileIDs []types.FileID // Just IDs - no new records needed unchangedFileIDs []types.FileID // Just IDs - no new records needed
directories []*FileToProcess directories []*FileToProcess
filesScanned int64 filesScanned int64
// dirsWithEntries holds each directory the walk has found an entry in.
dirsWithEntries map[string]struct{}
mountPoints []string
}
// firstEntryIn notes that the walk found an entry in directory dir and
// reports whether it is the first one.
func (c *scanCollector) firstEntryIn(dir string) bool {
c.mu.Lock()
defer c.mu.Unlock()
_, seen := c.dirsWithEntries[dir]
c.dirsWithEntries[dir] = struct{}{}
return !seen
}
// addMountPoint sets a mount point aside to be recorded.
func (c *scanCollector) addMountPoint(dir string) {
c.mu.Lock()
defer c.mu.Unlock()
c.mountPoints = append(c.mountPoints, dir)
} }
// addDirectory sets a directory aside to be recorded. It is not counted // addDirectory sets a directory aside to be recorded. It is not counted
@@ -1338,23 +1307,29 @@ func (s *Scanner) buildDirectoryEntry(path string, info os.FileInfo) *database.F
// mount points (see recordMountPoints). A directory has no data to back // mount points (see recordMountPoints). A directory has no data to back
// up, so it is not among the files counted and backed up. // up, so it is not among the files counted and backed up.
func (s *Scanner) recordDirectories( func (s *Scanner) recordDirectories(
ctx context.Context, scanned *ScanPhaseResult, ctx context.Context, directories []*FileToProcess,
) error { ) error {
for _, dir := range scanned.Directories { for _, dir := range directories {
err := s.recordNonRegularFile(ctx, dir) err := s.recordNonRegularFile(ctx, dir)
if err != nil { if err != nil {
return fmt.Errorf("recording directory %s: %w", dir.Path, err) return fmt.Errorf("recording directory %s: %w", dir.Path, err)
} }
} }
return s.recordMountPoints(ctx, scanned.MountPoints) return s.recordMountPoints(ctx, directories)
} }
// recordMountPoints records with the snapshot the mount points the walk // recordMountPoints records with the snapshot each directory the walk
// found (see ScanPhaseResult.MountPoints). // found on a different device from its parent directory, as a mount
// point is, the scanned path included.
func (s *Scanner) recordMountPoints( func (s *Scanner) recordMountPoints(
ctx context.Context, mountPoints []string, ctx context.Context, directories []*FileToProcess,
) error { ) error {
mountPoints, err := s.findMountPoints(directories)
if err != nil {
return err
}
return s.repos.WithTx(ctx, func(txCtx context.Context, tx *sql.Tx) error { return s.repos.WithTx(ctx, func(txCtx context.Context, tx *sql.Tx) error {
for _, dir := range mountPoints { for _, dir := range mountPoints {
log.Info("Found a mount point", "path", dir) log.Info("Found a mount point", "path", dir)
@@ -1369,53 +1344,57 @@ func (s *Scanner) recordMountPoints(
}) })
} }
// recordIfMountPoint sets the directory that holds entry, a path the // findMountPoints returns the directories, among those the walk found,
// walk found, aside to be recorded as a mount point if it is on a // that are on a different device from their parent directory. The
// different device from its parent directory. The walk calls it for each // scanned path is compared with the directory that holds it, which the
// entry it finds, and it decides on the first entry in a directory, so a // walk did not reach.
// directory the walk found no entry in is never recorded. By then the func (s *Scanner) findMountPoints(
// walk has listed the directory, which mounts a volume mounted there on directories []*FileToProcess,
// first access, so the directory and its parent are looked up again: ) ([]string, error) {
// before the listing, the directory can be on the automounter's own walked := make(map[string]os.FileInfo, len(directories))
// device (a systemd automount) or on its parent's (an autofs map that for _, dir := range directories {
// shows its mount points before mounting them). The scanned path is in walked[dir.Path] = dir.FileInfo
// no listing the walk made, so it is skipped as an entry. A directory
// that cannot be looked up then, or whose parent cannot, is not
// recorded, so this snapshot does not stop the prune of a later backup
// that misses its volume.
func (s *Scanner) recordIfMountPoint(
entry, scannedPath string, collector *scanCollector,
) {
dir := filepath.Dir(entry)
if entry == scannedPath || !collector.firstEntryIn(dir) {
return
} }
dirDevice, ok := s.deviceNumberAt(dir) var mountPoints []string
if !ok {
return for _, dir := range directories {
parentPath := filepath.Dir(dir.Path)
parent, ok := walked[parentPath]
if !ok {
var err error
parent, err = s.fs.Stat(parentPath)
if err != nil {
return nil, fmt.Errorf(
"looking up the parent of %s: %w", dir.Path, err)
}
}
if s.onOtherDevice(dir.FileInfo, parent) {
mountPoints = append(mountPoints, dir.Path)
}
} }
parentDevice, ok := s.deviceNumberAt(filepath.Dir(dir)) return mountPoints, nil
if ok && dirDevice != parentDevice {
collector.addMountPoint(dir)
}
} }
// deviceNumberAt looks path up now and returns the number of the device // onOtherDevice reports whether dir is on a different device from
// that holds it, and false when the lookup fails (logged) or gives no // parent. A file whose device number cannot be read counts as on the
// device number. // same device as its parent.
func (s *Scanner) deviceNumberAt(path string) (uint64, bool) { func (s *Scanner) onOtherDevice(dir, parent os.FileInfo) bool {
info, err := s.fs.Stat(path) dirDevice, ok := s.deviceNumber(dir)
if err != nil { if !ok {
log.Warn("Could not look up a directory to check for a mount point", return false
"path", path, "error", err)
return 0, false
} }
return s.deviceNumber(info) parentDevice, ok := s.deviceNumber(parent)
if !ok {
return false
}
return dirDevice != parentDevice
} }
// deviceNumberFromStat returns the number of the device that holds the // deviceNumberFromStat returns the number of the device that holds the
+10 -14
View File
@@ -55,14 +55,12 @@ var ErrSourcePathEmpty = errors.New("source path(s) were empty")
// ErrVolumeNotMounted is returned when --prune skipped the post-backup // ErrVolumeNotMounted is returned when --prune skipped the post-backup
// prune because a directory that an earlier snapshot of the same name // prune because a directory that an earlier snapshot of the same name
// found as a mount point was not found as one in this run: the volume // found as a mount point was not one in this run, as when the volume
// mounted there is not mounted, the run found no entry in it, or the // mounted there is not mounted. The snapshots are complete and kept, so
// directory is no longer backed up. The snapshots are complete and // the caller can report it apart from a failed backup. A run that also
// kept, so the caller can report it apart from a failed backup. A run // skipped unreadable files or found a configured path empty returns
// that also skipped unreadable files or found a configured path empty // ErrFilesUnreadable or ErrSourcePathEmpty instead.
// returns ErrFilesUnreadable or ErrSourcePathEmpty instead. var ErrVolumeNotMounted = errors.New("volume(s) were not mounted")
var ErrVolumeNotMounted = errors.New(
"mount point(s) an earlier snapshot recorded were not found")
// listRecentLimit caps how many snapshot rows are fetched from the // listRecentLimit caps how many snapshot rows are fetched from the
// local index for listing and purge operations. // local index for listing and purge operations.
@@ -235,12 +233,10 @@ func (v *Vaultik) skipPostBackupPrune(
} }
if len(unmountedPaths) > 0 { if len(unmountedPaths) > 0 {
v.UI.Warningf("Skipped the post-backup prune because this backup "+ v.UI.Warningf("Skipped the post-backup prune because no volume "+
"did not find a mount point at %s, where an earlier snapshot "+ "is mounted at %s, where an earlier snapshot found one; older "+
"found one: the volume is not mounted, this backup found no "+ "snapshots are kept. If a volume was retired on purpose, "+
"entry in it, or the directory is no longer backed up. Older "+ "remove the older snapshots by hand with "+
"snapshots are kept. Mount the volume again, or, if that is on "+
"purpose, remove the older snapshots by hand with "+
"'vaultik snapshot purge'.", "'vaultik snapshot purge'.",
strings.Join(unmountedPaths, ", ")) strings.Join(unmountedPaths, ", "))
} }
+22 -78
View File
@@ -4,10 +4,8 @@ import (
"context" "context"
"os" "os"
"path/filepath" "path/filepath"
"syscall"
"testing" "testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
@@ -263,69 +261,15 @@ func TestBackupWithNoEmptySourcePathPrunes(t *testing.T) {
} }
} }
// automountFs is the real filesystem with a volume at volumeDir that is
// mounted on first access, as by an autofs map that shows its mount
// points before mounting them: listing volumeDir mounts the volume or,
// while cannotMount, fails with "no such file or directory".
type automountFs struct {
afero.OsFs
volumeDir string
cannotMount bool
mounted bool
}
//nolint:ireturn // afero.Fs.Open is defined to return the interface.
func (f *automountFs) Open(name string) (afero.File, error) {
if name == f.volumeDir && f.cannotMount {
return nil, &os.PathError{Op: "open", Path: name, Err: syscall.ENOENT}
}
if name == f.volumeDir {
f.mounted = true
}
return f.OsFs.Open(name)
}
// A backup under --prune in which a volume that an earlier snapshot found // A backup under --prune in which a volume that an earlier snapshot found
// mounted below the configured path is missing does not prune: run after // mounted below the configured path is not mounted does not prune: run
// run, the snapshot that holds the volume's files is kept with everything // after run, the snapshot that holds the volume's files is kept with
// it stored, and the run fails. Once the volume is back, the prune runs. // everything it stored, and the run fails. Once the volume is mounted
// The volume is mounted on first access and every backup starts with it // again, the prune runs. See https://git.eeqj.de/sneak/vaultik/issues/395.
// not mounted, so its directory is on its parent's device until the walk
// lists it. While the volume is missing, its directory is left empty, or
// listing it fails with "no such file or directory", as when an autofs
// volume cannot be mounted, which the walk counts as a directory removed
// during the backup. See https://git.eeqj.de/sneak/vaultik/issues/395.
func TestBackupWithVolumeNotMountedDoesNotPrune(t *testing.T) { func TestBackupWithVolumeNotMountedDoesNotPrune(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
tests := []struct {
name string
listingFails bool
}{
{name: "directory left empty", listingFails: false},
{name: "listing fails", listingFails: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
backUpWithVolumeNotMounted(t, tt.listingFails)
})
}
}
// backUpWithVolumeNotMounted runs the backups of
// TestBackupWithVolumeNotMountedDoesNotPrune. While the volume is
// missing, listing its directory fails if listingFails; otherwise the
// directory is empty.
func backUpWithVolumeNotMounted(t *testing.T, listingFails bool) {
t.Helper()
// volume is the directory, below the configured path, where the // volume is the directory, below the configured path, where the
// volume is mounted. // volume is mounted.
const ( const (
@@ -338,21 +282,20 @@ func backUpWithVolumeNotMounted(t *testing.T, listingFails bool) {
volume + "/a.txt": []byte("file a"), volume + "/a.txt": []byte("file a"),
"notes.txt": []byte("notes"), "notes.txt": []byte("notes"),
}) })
env.setSnapshot(snapshotName, "") env.v.Config.Snapshots[snapshotName] = config.SnapshotConfig{
Paths: []string{env.src},
}
prune := &vaultik.SnapshotCreateOptions{ prune := &vaultik.SnapshotCreateOptions{
Prune: true, Snapshots: []string{snapshotName}, Prune: true, Snapshots: []string{snapshotName},
} }
// The volume's directory is on the volume's device, 2, while the // While the volume is mounted, its directory is on a device of its
// volume is mounted. Everything else is on device 1. // own.
volumeDir := filepath.Join(env.src, volume) mounted := true
fs := &automountFs{volumeDir: volumeDir}
newScanner := env.v.ScannerFactory newScanner := env.v.ScannerFactory
env.v.ScannerFactory = func(params snapshot.ScannerParams) *snapshot.Scanner { env.v.ScannerFactory = func(params snapshot.ScannerParams) *snapshot.Scanner {
fs.mounted = false
params.Fs = fs
params.DeviceNumber = func(info os.FileInfo) (uint64, bool) { params.DeviceNumber = func(info os.FileInfo) (uint64, bool) {
if info.Name() == volume && fs.mounted { if mounted && info.Name() == volume {
return 2, true return 2, true
} }
@@ -368,13 +311,13 @@ func backUpWithVolumeNotMounted(t *testing.T, listingFails bool) {
stored, err := env.v.Storage.List(ctx, "") stored, err := env.v.Storage.List(ctx, "")
require.NoError(t, err) require.NoError(t, err)
// The volume goes missing. // A volume that is not mounted leaves its mount point behind as an
fs.cannotMount = listingFails // empty directory on its parent's device.
volumeDir := filepath.Join(env.src, volume)
require.NoError(t, os.RemoveAll(volumeDir))
require.NoError(t, os.Mkdir(volumeDir, 0o750))
if !listingFails { mounted = false
require.NoError(t, os.RemoveAll(volumeDir))
require.NoError(t, os.Mkdir(volumeDir, 0o750))
}
for range 2 { for range 2 {
env.out.Reset() env.out.Reset()
@@ -382,7 +325,7 @@ func backUpWithVolumeNotMounted(t *testing.T, listingFails bool) {
err := env.v.CreateSnapshot(prune) err := env.v.CreateSnapshot(prune)
require.ErrorIs(t, err, vaultik.ErrVolumeNotMounted) require.ErrorIs(t, err, vaultik.ErrVolumeNotMounted)
assert.Contains(t, env.out.String(), assert.Contains(t, env.out.String(),
"did not find a mount point at "+volumeDir+" of snapshot "+snapshotName) "no volume is mounted at "+volumeDir+" of snapshot "+snapshotName)
assert.Contains(t, env.out.String(), "'vaultik snapshot purge'") assert.Contains(t, env.out.String(), "'vaultik snapshot purge'")
assert.NotContains(t, env.out.String(), "Running post-backup prune.") assert.NotContains(t, env.out.String(), "Running post-backup prune.")
} }
@@ -399,11 +342,12 @@ func backUpWithVolumeNotMounted(t *testing.T, listingFails bool) {
require.NoError(t, err, "%s is kept", key) require.NoError(t, err, "%s is kept", key)
} }
// The volume is back. // The volume is mounted again.
fs.cannotMount = false
require.NoError(t, os.WriteFile( require.NoError(t, os.WriteFile(
filepath.Join(volumeDir, "a.txt"), []byte("file a"), 0o600)) filepath.Join(volumeDir, "a.txt"), []byte("file a"), 0o600))
mounted = true
require.NoError(t, env.v.CreateSnapshot(prune)) require.NoError(t, env.v.CreateSnapshot(prune))
snapshots, err = env.v.Repositories.Snapshots.ListRecent( snapshots, err = env.v.Repositories.Snapshots.ListRecent(