Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a68a9ae0bd | ||
|
|
5abacbc5c1 |
No files matched your search
@@ -338,15 +338,16 @@ value where a list belongs or a word where a number belongs), is refused.
|
||||
snapshots by hand with `snapshot purge`. When an earlier snapshot of the
|
||||
same name in the local index found a volume mounted at or below a
|
||||
configured path (a directory on a different filesystem from its parent,
|
||||
with something in it) and this backup finds none mounted there, the prune
|
||||
does not run either: every older snapshot is kept, and the command names
|
||||
the directory and exits non-zero, on every run until the volume is mounted
|
||||
again. The same happens when the volume is empty, as an automounted volume
|
||||
that could not be mounted looks, or when that directory is no longer
|
||||
backed up, because it is excluded or its configured path was removed. If
|
||||
any of this is on purpose, as for a retired volume, remove the older
|
||||
snapshots by hand with `snapshot purge`. A file removed during the backup
|
||||
does not stop the prune.
|
||||
with an entry in it that the backup does not exclude) and this backup does
|
||||
not, the prune does not run either: every older snapshot is kept, and the
|
||||
command names the directory and exits non-zero, on every run until the
|
||||
volume is mounted again. A backup also does not find the volume when it
|
||||
finds no such entry in its directory, as when an automounted volume cannot
|
||||
be mounted, or when that directory is no longer backed up, because it is
|
||||
excluded or its configured path was removed. If any of this is on purpose,
|
||||
as for a retired volume, remove the older snapshots by hand with
|
||||
`snapshot purge`. A file removed during the backup does not stop the
|
||||
prune.
|
||||
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
||||
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
||||
Without `--prune`, or with a value that is not a valid duration, the
|
||||
@@ -445,6 +446,9 @@ once the store is reachable to remove the snapshot's metadata from it
|
||||
(`vaultik prune` does not). For a snapshot that is only on the
|
||||
destination store, that failure is an error. To wipe everything on the
|
||||
destination in one go, use `vaultik remote nuke --force`.
|
||||
* Accepts a snapshot ID, or — for a snapshot only on the destination
|
||||
store — its remote key (or an unambiguous leading part of it) as shown
|
||||
by `snapshot list`.
|
||||
* `--local-only`: Skip remote cleanup; only touch the local index
|
||||
* `--dry-run`: Show what would be deleted without deleting
|
||||
* `--force`: Skip confirmation prompt
|
||||
|
||||
@@ -30,7 +30,8 @@ owner's; a `v*` tag runs the release build in
|
||||
([issue #395](https://git.eeqj.de/sneak/vaultik/issues/395)). Such a
|
||||
prune could delete every snapshot still holding the volume's files.
|
||||
Each snapshot now records in the local index the directories it found
|
||||
on a different filesystem from their parent, with something in them.
|
||||
on a different filesystem from their parent and found an entry in that
|
||||
the backup does not exclude.
|
||||
|
||||
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
||||
that found nothing in a configured path
|
||||
|
||||
+2
-2
@@ -154,7 +154,7 @@ Maps snapshots to the blobs they reference.
|
||||
**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation.
|
||||
|
||||
### 10. `snapshot_mount_points`
|
||||
The directories at or below a configured path, the configured path itself included, that a snapshot's backup found on a different filesystem from their parent directory (a different device number), as a mount point is, and found an entry in. An empty one, as an automount point whose volume could not be mounted, is left out.
|
||||
The directories at or below a configured path, the configured path itself included, that a snapshot's backup found an entry in and, once it had listed them, found on a different filesystem from their parent directory (a different device number), as a mount point is. An automount point whose volume cannot be mounted is not recorded, because the backup finds no entry in it.
|
||||
|
||||
**Columns:**
|
||||
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
|
||||
@@ -183,7 +183,7 @@ Tracks blob upload metrics.
|
||||
- `INSERT OR REPLACE INTO files` - Update file metadata
|
||||
- `SELECT * FROM files WHERE path = ?` - Check if file has changed
|
||||
- `INSERT INTO snapshot_files` - Add file to current snapshot
|
||||
- `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a different filesystem from its parent that holds an entry
|
||||
- `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a different filesystem from its parent that the walk found an entry in
|
||||
|
||||
2. **Chunking** (for changed files)
|
||||
- `INSERT OR IGNORE INTO chunks` - Store new chunks
|
||||
|
||||
@@ -288,6 +288,10 @@ know about), which is what 'vaultik prune' does. The command prints the
|
||||
'vaultik prune' invocation to run as a follow-up only when it removed the
|
||||
snapshot's metadata from the destination store, and never under --json.
|
||||
|
||||
The snapshot may be named by its ID or, on a host with no local index, by
|
||||
the remote key that 'snapshot list' prints for a remote-only snapshot (an
|
||||
unambiguous leading part is enough).
|
||||
|
||||
An ID that names no snapshot in the local index or on the destination
|
||||
store is an error.
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"syscall"
|
||||
"testing"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
@@ -16,9 +17,10 @@ import (
|
||||
)
|
||||
|
||||
// A scan records with the snapshot each directory on a different device
|
||||
// from its parent directory that holds an entry, as a mount point is,
|
||||
// the scanned path included. A filesystem that gives no device numbers,
|
||||
// as the in-memory one, records none. See https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||
// from its parent directory that the walk found an entry in, as a mount
|
||||
// point is, the scanned path included. A filesystem that gives no device
|
||||
// numbers, as the in-memory one, records none. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||
func TestScanRecordsMountPoints(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -97,23 +99,29 @@ func TestScanRecordsMountPoints(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// automountFs is an in-memory filesystem in which opening mountPoint to
|
||||
// list it mounts a volume there, if canMount, as a systemd automount
|
||||
// does on first access. Stat records the device an entry is on at that
|
||||
// moment, as a real stat does: the automounter's, 3, for mountPoint
|
||||
// until the volume is mounted; the volume's, 2, for mountPoint once it
|
||||
// is and for everything in it; 1 for everything else.
|
||||
// automountFs is an in-memory filesystem with a volume at mountPoint
|
||||
// that is mounted on first access: opening mountPoint to list it mounts
|
||||
// the volume or, if it cannot be mounted, fails with "no such file or
|
||||
// directory", as with the autofs daemon. Stat records the device an
|
||||
// entry is on at that moment, as a real stat does: unmountedDevice for
|
||||
// mountPoint until the volume is mounted; the volume's, 2, for
|
||||
// mountPoint once it is and for everything in it; 1 for everything else.
|
||||
type automountFs struct {
|
||||
afero.Fs
|
||||
|
||||
mountPoint string
|
||||
canMount bool
|
||||
mounted bool
|
||||
mountPoint string
|
||||
unmountedDevice uint64
|
||||
canMount bool
|
||||
mounted bool
|
||||
}
|
||||
|
||||
//nolint:ireturn // afero.Fs.Open is defined to return the interface.
|
||||
func (fs *automountFs) Open(name string) (afero.File, error) {
|
||||
if name == fs.mountPoint && fs.canMount {
|
||||
if name == fs.mountPoint && !fs.canMount {
|
||||
return nil, &os.PathError{Op: "open", Path: name, Err: syscall.ENOENT}
|
||||
}
|
||||
|
||||
if name == fs.mountPoint {
|
||||
fs.mounted = true
|
||||
}
|
||||
|
||||
@@ -130,7 +138,7 @@ func (fs *automountFs) Stat(name string) (os.FileInfo, error) {
|
||||
|
||||
switch {
|
||||
case name == fs.mountPoint && !fs.mounted:
|
||||
device = 3
|
||||
device = fs.unmountedDevice
|
||||
case name == fs.mountPoint || strings.HasPrefix(name, fs.mountPoint+"/"):
|
||||
device = 2
|
||||
}
|
||||
@@ -154,12 +162,13 @@ func deviceAtLookup(info os.FileInfo) (uint64, bool) {
|
||||
}
|
||||
|
||||
// A volume mounted on first access is mounted when the walk lists its
|
||||
// mount point; until then the mount point is on the automounter's own
|
||||
// device. Only the mount point is recorded, by the backup that mounts
|
||||
// the volume and by a later one. An automount point whose volume cannot
|
||||
// be mounted stays on the automounter's device and lists nothing; it is
|
||||
// not recorded, so it does not count as the volume found mounted. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||
// mount point. Until then the mount point is on the automounter's own
|
||||
// device (a systemd automount) or on its parent's (an autofs map that
|
||||
// shows its mount points before mounting them). Either way only the
|
||||
// mount point is recorded, by the backup that mounts the volume and by a
|
||||
// later one that finds it mounted. Listing an automount point whose
|
||||
// volume cannot be mounted fails; it is not recorded, because the walk
|
||||
// finds no entry in it. See https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||
func TestScanRecordsOnlyTheMountPointOfAnAutomountedVolume(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -168,12 +177,23 @@ func TestScanRecordsOnlyTheMountPointOfAnAutomountedVolume(t *testing.T) {
|
||||
mountPoint := scannedPath + "/nas"
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
canMount bool
|
||||
want []string
|
||||
name string
|
||||
unmountedDevice uint64
|
||||
canMount bool
|
||||
want []string
|
||||
}{
|
||||
{name: "volume that is mounted", canMount: true, want: []string{mountPoint}},
|
||||
{name: "volume that cannot be mounted", canMount: false, want: nil},
|
||||
{
|
||||
name: "on the automounter's device until listed",
|
||||
unmountedDevice: 3, canMount: true, want: []string{mountPoint},
|
||||
},
|
||||
{
|
||||
name: "on its parent's device until listed",
|
||||
unmountedDevice: 1, canMount: true, want: []string{mountPoint},
|
||||
},
|
||||
{
|
||||
name: "volume that cannot be mounted",
|
||||
unmountedDevice: 1, canMount: false, want: nil,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -182,17 +202,16 @@ func TestScanRecordsOnlyTheMountPointOfAnAutomountedVolume(t *testing.T) {
|
||||
|
||||
ctx := context.Background()
|
||||
fs := &automountFs{
|
||||
Fs: afero.NewMemMapFs(), mountPoint: mountPoint, canMount: tt.canMount,
|
||||
Fs: afero.NewMemMapFs(), mountPoint: mountPoint,
|
||||
unmountedDevice: tt.unmountedDevice, canMount: tt.canMount,
|
||||
}
|
||||
|
||||
require.NoError(t, fs.MkdirAll(mountPoint, 0o755))
|
||||
require.NoError(t, afero.WriteFile(
|
||||
fs, scannedPath+"/notes.txt", []byte("notes"), 0o644))
|
||||
require.NoError(t, fs.MkdirAll(mountPoint+"/photos", 0o755))
|
||||
|
||||
if tt.canMount {
|
||||
require.NoError(t, fs.MkdirAll(mountPoint+"/photos", 0o755))
|
||||
require.NoError(t, afero.WriteFile(
|
||||
fs, mountPoint+"/photos/a.jpg", []byte("photo"), 0o644))
|
||||
for _, path := range []string{
|
||||
scannedPath + "/notes.txt", mountPoint + "/photos/a.jpg",
|
||||
} {
|
||||
require.NoError(t, afero.WriteFile(fs, path, []byte(path), 0o644))
|
||||
}
|
||||
|
||||
db, err := database.NewTestDB()
|
||||
|
||||
@@ -888,9 +888,10 @@ type ScanPhaseResult struct {
|
||||
FilesToProcess []*FileToProcess
|
||||
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
|
||||
Directories []*FileToProcess // Every directory found, to record
|
||||
// MountPoints are the directories found on a different device from
|
||||
// their parent directory, the scanned path included, that the walk
|
||||
// found an entry in.
|
||||
// MountPoints are the directories, the scanned path included, that
|
||||
// the walk found an entry in and, once it had listed them, on a
|
||||
// different device from their parent directory (see
|
||||
// recordIfMountPoint).
|
||||
MountPoints []string
|
||||
}
|
||||
|
||||
@@ -911,7 +912,7 @@ func (s *Scanner) scanPhase(
|
||||
// estimate leaves them out.
|
||||
estimatedTotal := countNonDirectories(knownFiles)
|
||||
|
||||
collector := &scanCollector{mountPoints: make(map[string]bool)}
|
||||
collector := &scanCollector{dirsWithEntries: make(map[string]struct{})}
|
||||
|
||||
// Set up periodic status output
|
||||
startTime := time.Now()
|
||||
@@ -954,10 +955,10 @@ func (s *Scanner) scanPhase(
|
||||
// The scanned directory itself is not an entry found in it.
|
||||
if filePath != path || !info.IsDir() {
|
||||
result.EntriesFound++
|
||||
|
||||
collector.foundEntryIn(filepath.Dir(filePath))
|
||||
}
|
||||
|
||||
s.recordIfMountPoint(filePath, path, collector)
|
||||
|
||||
// Handle symlinks and directories
|
||||
handled, err := s.recordSpecialEntry(
|
||||
filePath, info, existingFiles, knownFiles, collector, result)
|
||||
@@ -992,7 +993,7 @@ func (s *Scanner) scanPhase(
|
||||
FilesToProcess: collector.filesToProcess,
|
||||
UnchangedFileIDs: collector.unchangedFileIDs,
|
||||
Directories: collector.directories,
|
||||
MountPoints: collector.mountPointsWithEntries(),
|
||||
MountPoints: collector.mountPoints,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -1012,10 +1013,8 @@ func countNonDirectories(files map[string]*database.File) int64 {
|
||||
// recordSpecialEntry records symlinks and directories (which have no
|
||||
// data to chunk) and reports whether it handled the entry. A symlink is
|
||||
// counted and checked for changes as a regular file is; a directory is
|
||||
// set aside to be recorded, also as a mount point when it is on a
|
||||
// different device from its parent directory, and is not counted. For a
|
||||
// symlink whose target cannot be read it returns handleWalkError's
|
||||
// result.
|
||||
// set aside to be recorded and is not counted. For a symlink whose
|
||||
// target cannot be read it returns handleWalkError's result.
|
||||
func (s *Scanner) recordSpecialEntry(
|
||||
filePath string, info os.FileInfo,
|
||||
existingFiles map[string]struct{},
|
||||
@@ -1044,10 +1043,6 @@ func (s *Scanner) recordSpecialEntry(
|
||||
existingFiles[filePath] = struct{}{}
|
||||
collector.addDirectory(filePath, info, file)
|
||||
|
||||
if s.onOtherDeviceThanParent(filePath, info) {
|
||||
collector.addMountPoint(filePath)
|
||||
}
|
||||
|
||||
return true, nil
|
||||
}
|
||||
|
||||
@@ -1125,46 +1120,29 @@ type scanCollector struct {
|
||||
unchangedFileIDs []types.FileID // Just IDs - no new records needed
|
||||
directories []*FileToProcess
|
||||
filesScanned int64
|
||||
// mountPoints maps each directory found on a different device from
|
||||
// its parent directory to whether the walk found an entry in it.
|
||||
mountPoints map[string]bool
|
||||
// dirsWithEntries holds each directory the walk has found an entry in.
|
||||
dirsWithEntries map[string]struct{}
|
||||
mountPoints []string
|
||||
}
|
||||
|
||||
// addMountPoint sets aside a directory found on a different device from
|
||||
// its parent directory, to be recorded if the walk finds an entry in it.
|
||||
func (c *scanCollector) addMountPoint(path string) {
|
||||
// firstEntryIn notes that the walk found an entry in directory dir and
|
||||
// reports whether it is the first one.
|
||||
func (c *scanCollector) firstEntryIn(dir string) bool {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
c.mountPoints[path] = false
|
||||
_, seen := c.dirsWithEntries[dir]
|
||||
c.dirsWithEntries[dir] = struct{}{}
|
||||
|
||||
return !seen
|
||||
}
|
||||
|
||||
// foundEntryIn notes that the walk found an entry in directory dir.
|
||||
func (c *scanCollector) foundEntryIn(dir string) {
|
||||
// addMountPoint sets a mount point aside to be recorded.
|
||||
func (c *scanCollector) addMountPoint(dir string) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
if _, ok := c.mountPoints[dir]; ok {
|
||||
c.mountPoints[dir] = true
|
||||
}
|
||||
}
|
||||
|
||||
// mountPointsWithEntries returns the mount points the walk found an
|
||||
// entry in. An empty one, as an automount point whose volume could not
|
||||
// be mounted, does not show a volume mounted there.
|
||||
func (c *scanCollector) mountPointsWithEntries() []string {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
var mountPoints []string
|
||||
|
||||
for dir, foundEntry := range c.mountPoints {
|
||||
if foundEntry {
|
||||
mountPoints = append(mountPoints, dir)
|
||||
}
|
||||
}
|
||||
|
||||
return mountPoints
|
||||
c.mountPoints = append(c.mountPoints, dir)
|
||||
}
|
||||
|
||||
// addDirectory sets a directory aside to be recorded. It is not counted
|
||||
@@ -1391,32 +1369,53 @@ func (s *Scanner) recordMountPoints(
|
||||
})
|
||||
}
|
||||
|
||||
// onOtherDeviceThanParent reports whether the directory at path, which
|
||||
// info describes, is on a different device from its parent directory,
|
||||
// looked up now. The walk calls it on reaching the directory, by when it
|
||||
// has listed the parent (the scanned path aside). Listing a directory
|
||||
// mounts a volume mounted there on first access (a systemd automount),
|
||||
// so the walk's own entry for the parent, looked up before the listing,
|
||||
// can show the automounter's device instead. A directory whose device
|
||||
// number, or whose parent's, cannot be read counts as on its parent's
|
||||
// device: a mount point missed can only stop a prune, never let one run.
|
||||
func (s *Scanner) onOtherDeviceThanParent(path string, info os.FileInfo) bool {
|
||||
dirDevice, ok := s.deviceNumber(info)
|
||||
// recordIfMountPoint sets the directory that holds entry, a path the
|
||||
// walk found, aside to be recorded as a mount point if it is on a
|
||||
// different device from its parent directory. The walk calls it for each
|
||||
// entry it finds, and it decides on the first entry in a directory, so a
|
||||
// directory the walk found no entry in is never recorded. By then the
|
||||
// walk has listed the directory, which mounts a volume mounted there on
|
||||
// first access, so the directory and its parent are looked up again:
|
||||
// before the listing, the directory can be on the automounter's own
|
||||
// device (a systemd automount) or on its parent's (an autofs map that
|
||||
// shows its mount points before mounting them). The scanned path is in
|
||||
// no listing the walk made, so it is skipped as an entry. A directory
|
||||
// that cannot be looked up then, or whose parent cannot, is not
|
||||
// recorded, so this snapshot does not stop the prune of a later backup
|
||||
// that misses its volume.
|
||||
func (s *Scanner) recordIfMountPoint(
|
||||
entry, scannedPath string, collector *scanCollector,
|
||||
) {
|
||||
dir := filepath.Dir(entry)
|
||||
|
||||
if entry == scannedPath || !collector.firstEntryIn(dir) {
|
||||
return
|
||||
}
|
||||
|
||||
dirDevice, ok := s.deviceNumberAt(dir)
|
||||
if !ok {
|
||||
return false
|
||||
return
|
||||
}
|
||||
|
||||
parent, err := s.fs.Stat(filepath.Dir(path))
|
||||
parentDevice, ok := s.deviceNumberAt(filepath.Dir(dir))
|
||||
if ok && dirDevice != parentDevice {
|
||||
collector.addMountPoint(dir)
|
||||
}
|
||||
}
|
||||
|
||||
// deviceNumberAt looks path up now and returns the number of the device
|
||||
// that holds it, and false when the lookup fails (logged) or gives no
|
||||
// device number.
|
||||
func (s *Scanner) deviceNumberAt(path string) (uint64, bool) {
|
||||
info, err := s.fs.Stat(path)
|
||||
if err != nil {
|
||||
log.Warn("Could not look up the parent of a directory "+
|
||||
"to check for a mount point", "path", path, "error", err)
|
||||
log.Warn("Could not look up a directory to check for a mount point",
|
||||
"path", path, "error", err)
|
||||
|
||||
return false
|
||||
return 0, false
|
||||
}
|
||||
|
||||
parentDevice, ok := s.deviceNumber(parent)
|
||||
|
||||
return ok && dirDevice != parentDevice
|
||||
return s.deviceNumber(info)
|
||||
}
|
||||
|
||||
// deviceNumberFromStat returns the number of the device that holds the
|
||||
|
||||
@@ -451,6 +451,100 @@ func TestRemoveSnapshot_NotInLocalIndex(t *testing.T) {
|
||||
assert.Contains(t, out, "Removed snapshot metadata from remote storage")
|
||||
}
|
||||
|
||||
// TestRemoveSnapshot_RemoteOnlyByRemoteKey checks that a snapshot only
|
||||
// on the destination store, on a host with no local index, is removed by
|
||||
// the remote key that 'snapshot list' prints for such a snapshot, and by
|
||||
// an unambiguous leading part of it — the same identifiers verify and
|
||||
// restore accept. Snapshot remove used to hash the argument again, so
|
||||
// the hashed key never matched and only the plain snapshot ID worked.
|
||||
func TestRemoveSnapshot_RemoteOnlyByRemoteKey(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
remoteKey := snapshot.RemoteSnapshotKey("snapshot-001")
|
||||
|
||||
identifiers := map[string]string{
|
||||
"full key": remoteKey,
|
||||
"leading part": remoteKey[:8],
|
||||
}
|
||||
|
||||
for name, identifier := range identifiers {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
store := newTestStorer()
|
||||
addManifest(t, store, "snapshot-001", nil)
|
||||
|
||||
// No local index: a host that sees the snapshot only remotely.
|
||||
tv := vaultik.NewForTesting(store)
|
||||
tv.Repositories = newLocalIndex(t)
|
||||
|
||||
result, err := tv.RemoveSnapshot(identifier,
|
||||
&vaultik.RemoveOptions{Force: true})
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.True(t, result.RemoteRemoved)
|
||||
assert.False(t, store.hasKey(remoteKeyPath("snapshot-001")))
|
||||
|
||||
out := tv.Stdout.String()
|
||||
assert.NotContains(t, out, "from local database")
|
||||
assert.Contains(t, out, "Removed snapshot metadata from remote storage")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRemoveSnapshot_LocalIndexByRemoteKey checks that a remote key (or
|
||||
// its unambiguous leading part) that names a snapshot the local index
|
||||
// holds removes that snapshot's local row and its remote metadata
|
||||
// together. `snapshot list --json` prints a remote key for every
|
||||
// snapshot, including local ones.
|
||||
func TestRemoveSnapshot_LocalIndexByRemoteKey(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
remoteKey := snapshot.RemoteSnapshotKey("snapshot-001")
|
||||
|
||||
identifiers := map[string]string{
|
||||
"full key": remoteKey,
|
||||
"leading part": remoteKey[:8],
|
||||
}
|
||||
|
||||
for name, identifier := range identifiers {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
store := newTestStorer()
|
||||
addManifest(t, store, "snapshot-001", nil)
|
||||
|
||||
// A backup wrote to this destination, so it carries the
|
||||
// marker, and the local index holds the snapshot.
|
||||
require.NoError(t, store.Put(context.Background(), markerFile,
|
||||
bytes.NewReader(nil)))
|
||||
|
||||
tv := vaultik.NewForTesting(store)
|
||||
repos := newLocalIndex(t, "snapshot-001")
|
||||
tv.Repositories = repos
|
||||
|
||||
result, err := tv.RemoveSnapshot(identifier,
|
||||
&vaultik.RemoveOptions{Force: true})
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.True(t, result.RemoteRemoved)
|
||||
assert.False(t, store.hasKey(remoteKeyPath("snapshot-001")))
|
||||
|
||||
row, err := repos.Snapshots.GetByID(
|
||||
context.Background(), "snapshot-001")
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, row)
|
||||
|
||||
out := tv.Stdout.String()
|
||||
assert.Contains(t, out,
|
||||
"Removed snapshot 'snapshot-001' from local database")
|
||||
assert.Contains(t, out, "Removed snapshot metadata from remote storage")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRemoveSnapshot_NotInLocalIndexDeleteFails checks that removing a
|
||||
// snapshot only the destination store holds fails when its metadata
|
||||
// cannot be deleted there, under --force and under --json: nothing was
|
||||
|
||||
@@ -56,11 +56,11 @@ var ErrSourcePathEmpty = errors.New("source path(s) were empty")
|
||||
// ErrVolumeNotMounted is returned when --prune skipped the post-backup
|
||||
// prune because a directory that an earlier snapshot of the same name
|
||||
// found as a mount point was not found as one in this run: the volume
|
||||
// mounted there is not mounted or is empty, or the directory is no
|
||||
// longer backed up. The snapshots are complete and kept, so the caller
|
||||
// can report it apart from a failed backup. A run that also skipped
|
||||
// unreadable files or found a configured path empty returns
|
||||
// ErrFilesUnreadable or ErrSourcePathEmpty instead.
|
||||
// mounted there is not mounted, the run found no entry in it, or the
|
||||
// directory is no longer backed up. The snapshots are complete and
|
||||
// kept, so the caller can report it apart from a failed backup. A run
|
||||
// that also skipped unreadable files or found a configured path empty
|
||||
// returns ErrFilesUnreadable or ErrSourcePathEmpty instead.
|
||||
var ErrVolumeNotMounted = errors.New(
|
||||
"mount point(s) an earlier snapshot recorded were not found")
|
||||
|
||||
@@ -237,10 +237,11 @@ func (v *Vaultik) skipPostBackupPrune(
|
||||
if len(unmountedPaths) > 0 {
|
||||
v.UI.Warningf("Skipped the post-backup prune because this backup "+
|
||||
"did not find a mount point at %s, where an earlier snapshot "+
|
||||
"found one: the volume is not mounted or is empty, or the "+
|
||||
"directory is no longer backed up. Older snapshots are kept. "+
|
||||
"Mount the volume again, or, if that is on purpose, remove the "+
|
||||
"older snapshots by hand with 'vaultik snapshot purge'.",
|
||||
"found one: the volume is not mounted, this backup found no "+
|
||||
"entry in it, or the directory is no longer backed up. Older "+
|
||||
"snapshots are kept. Mount the volume again, or, if that is on "+
|
||||
"purpose, remove the older snapshots by hand with "+
|
||||
"'vaultik snapshot purge'.",
|
||||
strings.Join(unmountedPaths, ", "))
|
||||
}
|
||||
|
||||
@@ -1479,7 +1480,7 @@ func (v *Vaultik) RemoveSnapshot(
|
||||
return result, err
|
||||
}
|
||||
|
||||
inLocalIndex, err := v.findSnapshotToRemove(snapshotID, opts)
|
||||
inLocalIndex, localID, remoteKey, err := v.findSnapshotToRemove(snapshotID, opts)
|
||||
if err != nil {
|
||||
return result, err
|
||||
}
|
||||
@@ -1501,16 +1502,16 @@ func (v *Vaultik) RemoveSnapshot(
|
||||
}
|
||||
|
||||
if inLocalIndex {
|
||||
log.Info("Removing snapshot from local database", "snapshot_id", snapshotID)
|
||||
log.Info("Removing snapshot from local database", "snapshot_id", localID)
|
||||
|
||||
err = v.deleteSnapshotFromLocalDB(snapshotID)
|
||||
err = v.deleteSnapshotFromLocalDB(localID)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("removing from local database: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
result.RemoteRemoved, err = v.removeSnapshotRemote(
|
||||
snapshotID, inLocalIndex, markerErr, opts)
|
||||
snapshotID, remoteKey, inLocalIndex, markerErr, opts)
|
||||
if err != nil {
|
||||
return result, err
|
||||
}
|
||||
@@ -1529,7 +1530,7 @@ func (v *Vaultik) RemoveSnapshot(
|
||||
|
||||
if inLocalIndex {
|
||||
v.UI.Infof("Removed snapshot '%s' from local database.",
|
||||
v.UI.Snapshot(snapshotID))
|
||||
v.UI.Snapshot(localID))
|
||||
}
|
||||
|
||||
if result.RemoteRemoved {
|
||||
@@ -1542,50 +1543,68 @@ func (v *Vaultik) RemoveSnapshot(
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// findSnapshotToRemove reports whether the local index has a row for
|
||||
// snapshotID, and returns errSnapshotNotFound when there is nothing to
|
||||
// remove: no row and, unless LocalOnly is set, no metadata on the
|
||||
// destination store either. The destination store is consulted only
|
||||
// when the local index has no row, so a snapshot in the local index can
|
||||
// still be removed from it while the destination store is unreachable,
|
||||
// and only after checkDestinationMarker passes, so an unplugged volume's
|
||||
// empty mount point is not taken for a store without the snapshot.
|
||||
// findSnapshotToRemove resolves snapshotID, the identifier remove was
|
||||
// given, and returns whether the local index holds that snapshot, the
|
||||
// human ID of its row when it does, and the remote key to strip from
|
||||
// the destination store. A human ID is looked up in the local index
|
||||
// literally and hashed without consulting the store; a remote key, or
|
||||
// an unambiguous leading part of one, is resolved against the store's
|
||||
// metadata listing the way verify and restore do, and then matched back
|
||||
// to the local row whose hashed ID equals it, so removing by such a key
|
||||
// removes the local row too. The destination store is consulted only
|
||||
// after checkDestinationMarker, so an unplugged volume's empty mount
|
||||
// point is not taken for a store without the snapshot. The identifier
|
||||
// naming nothing, on the local index or, unless LocalOnly is set, on
|
||||
// the store, is errSnapshotNotFound; a remote-key form that matches
|
||||
// nothing or more than one directory surfaces
|
||||
// resolveSnapshotRemoteKey's errSnapshotKeyNotFound or
|
||||
// errSnapshotKeyAmbiguous.
|
||||
func (v *Vaultik) findSnapshotToRemove(
|
||||
snapshotID string, opts *RemoveOptions,
|
||||
) (bool, error) {
|
||||
) (bool, string, string, error) {
|
||||
if v.Repositories != nil {
|
||||
snap, err := v.Repositories.Snapshots.GetByID(v.ctx, snapshotID)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("looking up snapshot in local index: %w", err)
|
||||
return false, "", "", fmt.Errorf("looking up snapshot in local index: %w", err)
|
||||
}
|
||||
|
||||
if snap != nil {
|
||||
return true, nil
|
||||
return true, snapshotID, snapshot.RemoteSnapshotKey(snapshotID), nil
|
||||
}
|
||||
}
|
||||
|
||||
if opts.LocalOnly {
|
||||
return false, fmt.Errorf("%w in the local index: %s",
|
||||
return false, "", "", fmt.Errorf("%w in the local index: %s",
|
||||
errSnapshotNotFound, snapshotID)
|
||||
}
|
||||
|
||||
err := v.checkDestinationMarker()
|
||||
if err != nil {
|
||||
return false, err
|
||||
return false, "", "", err
|
||||
}
|
||||
|
||||
objects, err := v.remoteSnapshotObjects(snapshot.RemoteSnapshotKey(snapshotID))
|
||||
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("looking up snapshot on destination store: %w", err)
|
||||
return false, "", "", err
|
||||
}
|
||||
|
||||
objects, err := v.remoteSnapshotObjects(remoteKey)
|
||||
if err != nil {
|
||||
return false, "", "", fmt.Errorf("looking up snapshot on destination store: %w", err)
|
||||
}
|
||||
|
||||
if len(objects) == 0 {
|
||||
return false, fmt.Errorf(
|
||||
return false, "", "", fmt.Errorf(
|
||||
"%w in the local index or on the destination store: %s",
|
||||
errSnapshotNotFound, snapshotID)
|
||||
}
|
||||
|
||||
return false, nil
|
||||
localID, err := v.localSnapshotIDForRemoteKey(remoteKey)
|
||||
if err != nil {
|
||||
return false, "", "", err
|
||||
}
|
||||
|
||||
return localID != "", localID, remoteKey, nil
|
||||
}
|
||||
|
||||
// removeSnapshotDryRun reports what RemoveSnapshot would do without
|
||||
@@ -1648,19 +1667,20 @@ func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions)
|
||||
}
|
||||
|
||||
// removeSnapshotRemote strips the snapshot's metadata from the
|
||||
// destination store. It returns true only when the delete finished
|
||||
// without an error and removed at least one object; under LocalOnly it
|
||||
// does nothing. markerErr is the result of checkDestinationMarker; when
|
||||
// it is set, nothing is deleted and it is the failure. When the snapshot
|
||||
// is only on the destination store, a failure is returned as an error,
|
||||
// because that removal is all the command does. When the snapshot was in
|
||||
// the local index, its removal there has already happened, so a failure
|
||||
// is only a warning telling the user to run `vaultik snapshot remove`
|
||||
// for the snapshot again once the destination store is reachable
|
||||
// (`vaultik prune` never removes snapshot metadata that has its
|
||||
// manifest).
|
||||
// destination store. remoteKey is the resolved metadata directory name
|
||||
// to delete, as returned by findSnapshotToRemove. It returns true only
|
||||
// when the delete finished without an error and removed at least one
|
||||
// object; under LocalOnly it does nothing. markerErr is the result of
|
||||
// checkDestinationMarker; when it is set, nothing is deleted and it is
|
||||
// the failure. When the snapshot is only on the destination store, a
|
||||
// failure is returned as an error, because that removal is all the
|
||||
// command does. When the snapshot was in the local index, its removal
|
||||
// there has already happened, so a failure is only a warning telling
|
||||
// the user to run `vaultik snapshot remove` for the snapshot again once
|
||||
// the destination store is reachable (`vaultik prune` never removes
|
||||
// snapshot metadata that has its manifest).
|
||||
func (v *Vaultik) removeSnapshotRemote(
|
||||
snapshotID string, inLocalIndex bool, markerErr error,
|
||||
snapshotID, remoteKey string, inLocalIndex bool, markerErr error,
|
||||
opts *RemoveOptions,
|
||||
) (bool, error) {
|
||||
if opts.LocalOnly {
|
||||
@@ -1674,8 +1694,7 @@ func (v *Vaultik) removeSnapshotRemote(
|
||||
|
||||
err := markerErr
|
||||
if err == nil {
|
||||
deleted, err = v.deleteRemoteSnapshotByKey(
|
||||
snapshot.RemoteSnapshotKey(snapshotID))
|
||||
deleted, err = v.deleteRemoteSnapshotByKey(remoteKey)
|
||||
}
|
||||
|
||||
if err != nil && !inLocalIndex {
|
||||
@@ -1789,6 +1808,23 @@ func (v *Vaultik) localSnapshotIDs() ([]string, error) {
|
||||
return ids, nil
|
||||
}
|
||||
|
||||
// localSnapshotIDForRemoteKey returns the human snapshot ID of the local
|
||||
// index row whose RemoteSnapshotKey is key, or "" when no row has it.
|
||||
func (v *Vaultik) localSnapshotIDForRemoteKey(key string) (string, error) {
|
||||
ids, err := v.localSnapshotIDs()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("listing local snapshots: %w", err)
|
||||
}
|
||||
|
||||
for _, id := range ids {
|
||||
if snapshot.RemoteSnapshotKey(id) == key {
|
||||
return id, nil
|
||||
}
|
||||
}
|
||||
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// listAllRemoteSnapshotKeys collects the hashed remote keys
|
||||
// (subdirectories under metadata/) currently present in the
|
||||
// destination store. Returns (nil, err) when the store cannot be
|
||||
|
||||
@@ -4,8 +4,10 @@ import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"syscall"
|
||||
"testing"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/config"
|
||||
@@ -261,42 +263,67 @@ func TestBackupWithNoEmptySourcePathPrunes(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// automountFs is the real filesystem with a volume at volumeDir that is
|
||||
// mounted on first access, as by an autofs map that shows its mount
|
||||
// points before mounting them: listing volumeDir mounts the volume or,
|
||||
// while cannotMount, fails with "no such file or directory".
|
||||
type automountFs struct {
|
||||
afero.OsFs
|
||||
|
||||
volumeDir string
|
||||
cannotMount bool
|
||||
mounted bool
|
||||
}
|
||||
|
||||
//nolint:ireturn // afero.Fs.Open is defined to return the interface.
|
||||
func (f *automountFs) Open(name string) (afero.File, error) {
|
||||
if name == f.volumeDir && f.cannotMount {
|
||||
return nil, &os.PathError{Op: "open", Path: name, Err: syscall.ENOENT}
|
||||
}
|
||||
|
||||
if name == f.volumeDir {
|
||||
f.mounted = true
|
||||
}
|
||||
|
||||
return f.OsFs.Open(name)
|
||||
}
|
||||
|
||||
// A backup under --prune in which a volume that an earlier snapshot found
|
||||
// mounted below the configured path is not mounted does not prune: run
|
||||
// after run, the snapshot that holds the volume's files is kept with
|
||||
// everything it stored, and the run fails. Once the volume is mounted
|
||||
// again, the prune runs. The volume's directory is left either as a
|
||||
// plain directory or as an automount point whose volume cannot be
|
||||
// mounted, which stays empty on the automounter's own device. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||
// mounted below the configured path is missing does not prune: run after
|
||||
// run, the snapshot that holds the volume's files is kept with everything
|
||||
// it stored, and the run fails. Once the volume is back, the prune runs.
|
||||
// The volume is mounted on first access and every backup starts with it
|
||||
// not mounted, so its directory is on its parent's device until the walk
|
||||
// lists it. While the volume is missing, its directory is left empty, or
|
||||
// listing it fails with "no such file or directory", as when an autofs
|
||||
// volume cannot be mounted, which the walk counts as a directory removed
|
||||
// during the backup. See https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||
func TestBackupWithVolumeNotMountedDoesNotPrune(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
// unmountedDevice is the device number of the volume's directory
|
||||
// while the volume is not mounted. Its parent directory is on device
|
||||
// 1, the mounted volume on device 2.
|
||||
tests := []struct {
|
||||
name string
|
||||
unmountedDevice uint64
|
||||
name string
|
||||
listingFails bool
|
||||
}{
|
||||
{name: "plain directory", unmountedDevice: 1},
|
||||
{name: "automount point that cannot be mounted", unmountedDevice: 3},
|
||||
{name: "directory left empty", listingFails: false},
|
||||
{name: "listing fails", listingFails: true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
backUpWithVolumeNotMounted(t, tt.unmountedDevice)
|
||||
backUpWithVolumeNotMounted(t, tt.listingFails)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// backUpWithVolumeNotMounted runs the backups of
|
||||
// TestBackupWithVolumeNotMountedDoesNotPrune, with the volume's directory
|
||||
// on device unmountedDevice while the volume is not mounted.
|
||||
func backUpWithVolumeNotMounted(t *testing.T, unmountedDevice uint64) {
|
||||
// TestBackupWithVolumeNotMountedDoesNotPrune. While the volume is
|
||||
// missing, listing its directory fails if listingFails; otherwise the
|
||||
// directory is empty.
|
||||
func backUpWithVolumeNotMounted(t *testing.T, listingFails bool) {
|
||||
t.Helper()
|
||||
|
||||
// volume is the directory, below the configured path, where the
|
||||
@@ -311,21 +338,22 @@ func backUpWithVolumeNotMounted(t *testing.T, unmountedDevice uint64) {
|
||||
volume + "/a.txt": []byte("file a"),
|
||||
"notes.txt": []byte("notes"),
|
||||
})
|
||||
env.v.Config.Snapshots[snapshotName] = config.SnapshotConfig{
|
||||
Paths: []string{env.src},
|
||||
}
|
||||
env.setSnapshot(snapshotName, "")
|
||||
prune := &vaultik.SnapshotCreateOptions{
|
||||
Prune: true, Snapshots: []string{snapshotName},
|
||||
}
|
||||
|
||||
// volumeDevice is the device of the volume's directory, 2 while the
|
||||
// The volume's directory is on the volume's device, 2, while the
|
||||
// volume is mounted. Everything else is on device 1.
|
||||
volumeDevice := uint64(2)
|
||||
volumeDir := filepath.Join(env.src, volume)
|
||||
fs := &automountFs{volumeDir: volumeDir}
|
||||
newScanner := env.v.ScannerFactory
|
||||
env.v.ScannerFactory = func(params snapshot.ScannerParams) *snapshot.Scanner {
|
||||
fs.mounted = false
|
||||
params.Fs = fs
|
||||
params.DeviceNumber = func(info os.FileInfo) (uint64, bool) {
|
||||
if info.Name() == volume {
|
||||
return volumeDevice, true
|
||||
if info.Name() == volume && fs.mounted {
|
||||
return 2, true
|
||||
}
|
||||
|
||||
return 1, true
|
||||
@@ -340,13 +368,13 @@ func backUpWithVolumeNotMounted(t *testing.T, unmountedDevice uint64) {
|
||||
stored, err := env.v.Storage.List(ctx, "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// A volume that is not mounted leaves its mount point behind as an
|
||||
// empty directory.
|
||||
volumeDir := filepath.Join(env.src, volume)
|
||||
require.NoError(t, os.RemoveAll(volumeDir))
|
||||
require.NoError(t, os.Mkdir(volumeDir, 0o750))
|
||||
// The volume goes missing.
|
||||
fs.cannotMount = listingFails
|
||||
|
||||
volumeDevice = unmountedDevice
|
||||
if !listingFails {
|
||||
require.NoError(t, os.RemoveAll(volumeDir))
|
||||
require.NoError(t, os.Mkdir(volumeDir, 0o750))
|
||||
}
|
||||
|
||||
for range 2 {
|
||||
env.out.Reset()
|
||||
@@ -371,12 +399,11 @@ func backUpWithVolumeNotMounted(t *testing.T, unmountedDevice uint64) {
|
||||
require.NoError(t, err, "%s is kept", key)
|
||||
}
|
||||
|
||||
// The volume is mounted again.
|
||||
// The volume is back.
|
||||
fs.cannotMount = false
|
||||
|
||||
require.NoError(t, os.WriteFile(
|
||||
filepath.Join(volumeDir, "a.txt"), []byte("file a"), 0o600))
|
||||
|
||||
volumeDevice = 2
|
||||
|
||||
require.NoError(t, env.v.CreateSnapshot(prune))
|
||||
|
||||
snapshots, err = env.v.Repositories.Snapshots.ListRecent(
|
||||
|
||||
Reference in new issue
Block a user