Do not prune when a backup misses an earlier mount point (closes #395)
snapshot create --prune purged the older snapshots holding a volume's files when the volume, mounted below a configured path, was not mounted: its empty mount point read as deleted files. The scan now records in the local index each directory at or below a configured path that the walk found an entry in and that, once listed, is on a different device from its parent; listing mounts an automounted volume. Under --prune, when an earlier snapshot of that name recorded such a directory and this run does not, the run skips the prune, keeps every older snapshot and exits non-zero, naming the directory. For a volume retired on purpose, snapshot purge removes the older snapshots. Model: opus-5-5
This commit is contained in:
@@ -335,8 +335,19 @@ value where a list belongs or a word where a number belongs), is refused.
|
|||||||
either: every older snapshot is kept, and the command names the path and
|
either: every older snapshot is kept, and the command names the path and
|
||||||
exits non-zero, on every run until the path holds something again. Take a
|
exits non-zero, on every run until the path holds something again. Take a
|
||||||
path that is empty on purpose out of the config, or remove the older
|
path that is empty on purpose out of the config, or remove the older
|
||||||
snapshots by hand with `snapshot purge`. A file removed during the backup
|
snapshots by hand with `snapshot purge`. When an earlier snapshot of the
|
||||||
does not stop the prune.
|
same name in the local index found a volume mounted at or below a
|
||||||
|
configured path (a directory on a different filesystem from its parent,
|
||||||
|
with an entry in it that the backup does not exclude) and this backup does
|
||||||
|
not, the prune does not run either: every older snapshot is kept, and the
|
||||||
|
command names the directory and exits non-zero, on every run until the
|
||||||
|
volume is mounted again. A backup also does not find the volume when it
|
||||||
|
finds no such entry in its directory, as when an automounted volume cannot
|
||||||
|
be mounted, or when that directory is no longer backed up, because it is
|
||||||
|
excluded or its configured path was removed. If any of this is on purpose,
|
||||||
|
as for a retired volume, remove the older snapshots by hand with
|
||||||
|
`snapshot purge`. A file removed during the backup does not stop the
|
||||||
|
prune.
|
||||||
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
||||||
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
||||||
Without `--prune`, or with a value that is not a valid duration, the
|
Without `--prune`, or with a value that is not a valid duration, the
|
||||||
|
|||||||
@@ -24,6 +24,15 @@ owner's; a `v*` tag runs the release build in
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-15: Stopped `snapshot create --prune` pruning after a backup
|
||||||
|
in which a volume mounted below a configured path in an earlier
|
||||||
|
snapshot of the same name was not mounted
|
||||||
|
([issue #395](https://git.eeqj.de/sneak/vaultik/issues/395)). Such a
|
||||||
|
prune could delete every snapshot still holding the volume's files.
|
||||||
|
Each snapshot now records in the local index the directories it found
|
||||||
|
on a different filesystem from their parent and found an entry in that
|
||||||
|
the backup does not exclude.
|
||||||
|
|
||||||
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
||||||
that found nothing in a configured path
|
that found nothing in a configured path
|
||||||
([issue #388](https://git.eeqj.de/sneak/vaultik/issues/388)), as in
|
([issue #388](https://git.eeqj.de/sneak/vaultik/issues/388)), as in
|
||||||
|
|||||||
+12
-1
@@ -153,7 +153,17 @@ Maps snapshots to the blobs they reference.
|
|||||||
|
|
||||||
**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation.
|
**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation.
|
||||||
|
|
||||||
### 10. `uploads`
|
### 10. `snapshot_mount_points`
|
||||||
|
The directories at or below a configured path, the configured path itself included, that a snapshot's backup found an entry in and, once it had listed them, found on a different filesystem from their parent directory (a different device number), as a mount point is. An automount point whose volume cannot be mounted is not recorded, because the backup finds no entry in it.
|
||||||
|
|
||||||
|
**Columns:**
|
||||||
|
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
|
||||||
|
- `path` (TEXT) - Absolute path of the directory
|
||||||
|
- PRIMARY KEY (`snapshot_id`, `path`)
|
||||||
|
|
||||||
|
**Purpose:** Lets `snapshot create --prune` skip the prune when a backup does not find a mount point that an earlier snapshot of the same name found, as when the volume is not mounted.
|
||||||
|
|
||||||
|
### 11. `uploads`
|
||||||
Tracks blob upload metrics.
|
Tracks blob upload metrics.
|
||||||
|
|
||||||
**Columns:**
|
**Columns:**
|
||||||
@@ -173,6 +183,7 @@ Tracks blob upload metrics.
|
|||||||
- `INSERT OR REPLACE INTO files` - Update file metadata
|
- `INSERT OR REPLACE INTO files` - Update file metadata
|
||||||
- `SELECT * FROM files WHERE path = ?` - Check if file has changed
|
- `SELECT * FROM files WHERE path = ?` - Check if file has changed
|
||||||
- `INSERT INTO snapshot_files` - Add file to current snapshot
|
- `INSERT INTO snapshot_files` - Add file to current snapshot
|
||||||
|
- `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a different filesystem from its parent that the walk found an entry in
|
||||||
|
|
||||||
2. **Chunking** (for changed files)
|
2. **Chunking** (for changed files)
|
||||||
- `INSERT OR IGNORE INTO chunks` - Store new chunks
|
- `INSERT OR IGNORE INTO chunks` - Store new chunks
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
|
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -8,6 +9,7 @@ import (
|
|||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Flags of `snapshot create` used to build argument vectors below.
|
// Flags of `snapshot create` used to build argument vectors below.
|
||||||
@@ -117,6 +119,51 @@ func TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty(t *testing.T) {
|
|||||||
assert.Len(t, snapshots, 2, "the older snapshot is kept")
|
assert.Len(t, snapshots, 2, "the older snapshot is kept")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestEntrySnapshotCreatePruneExitsNonZeroWhenMountPointIsMissing backs
|
||||||
|
// up, records in the local index that this backup found the source
|
||||||
|
// directory as a mount point, and runs `snapshot create --prune`, which
|
||||||
|
// does not find it as one. The command exits non-zero, reports the
|
||||||
|
// backup as completed, not as a failed snapshot creation, and keeps the
|
||||||
|
// older snapshot. See https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||||
|
func TestEntrySnapshotCreatePruneExitsNonZeroWhenMountPointIsMissing(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
configPath, storeDir := writeBackupConfig(t)
|
||||||
|
|
||||||
|
code, _, _ := runEntry(t, flagConfig, configPath, cmdSnapshot, cmdCreate)
|
||||||
|
require.Equal(t, 0, code)
|
||||||
|
|
||||||
|
// writeBackupConfig puts the source directory and the local index
|
||||||
|
// beside the config file.
|
||||||
|
dir := filepath.Dir(configPath)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
db, err := database.New(ctx, filepath.Join(dir, "index.sqlite"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
repos := database.NewRepositories(db)
|
||||||
|
|
||||||
|
backedUp, err := repos.Snapshots.GetOnlySnapshot(ctx)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, repos.Snapshots.AddMountPoint(ctx, nil,
|
||||||
|
backedUp.ID.String(), filepath.Join(dir, "source")))
|
||||||
|
require.NoError(t, db.Close())
|
||||||
|
|
||||||
|
code, _, stderr := runEntry(t,
|
||||||
|
flagConfig, configPath, cmdSnapshot, cmdCreate, flagPrune)
|
||||||
|
|
||||||
|
assert.Equal(t, 1, code)
|
||||||
|
assert.Contains(t, stderr, "Backup completed, but 1 mount point(s) "+
|
||||||
|
"an earlier snapshot recorded were not found")
|
||||||
|
assert.NotContains(t, stderr, "Snapshot creation failed")
|
||||||
|
|
||||||
|
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Len(t, snapshots, 2, "the older snapshot is kept")
|
||||||
|
}
|
||||||
|
|
||||||
// writeBackupConfig builds a hermetic config with one small file in its
|
// writeBackupConfig builds a hermetic config with one small file in its
|
||||||
// source directory and an empty destination store directory. Returns the
|
// source directory and an empty destination store directory. Returns the
|
||||||
// config path and the store directory.
|
// config path and the store directory.
|
||||||
|
|||||||
@@ -116,7 +116,8 @@ on Linux), then /etc/vaultik/config.yml.`,
|
|||||||
}
|
}
|
||||||
|
|
||||||
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
|
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
|
||||||
errors.Is(err, vaultik.ErrSourcePathEmpty) {
|
errors.Is(err, vaultik.ErrSourcePathEmpty) ||
|
||||||
|
errors.Is(err, vaultik.ErrVolumeNotMounted) {
|
||||||
log.Error("Backup completed with an error", "error", err)
|
log.Error("Backup completed with an error", "error", err)
|
||||||
ReportErrorf("Backup completed, but %v", err)
|
ReportErrorf("Backup completed, but %v", err)
|
||||||
|
|
||||||
@@ -137,7 +138,8 @@ on Linux), then /etc/vaultik/config.yml.`,
|
|||||||
"directory under metadata/ with no manifest once all its "+
|
"directory under metadata/ with no manifest once all its "+
|
||||||
"objects are older than prune_grace_period and its snapshot is "+
|
"objects are older than prune_grace_period and its snapshot is "+
|
||||||
"not in the local index (not after a backup that skipped "+
|
"not in the local index (not after a backup that skipped "+
|
||||||
"unreadable files or found a configured path empty)")
|
"unreadable files, found a configured path empty, or did not "+
|
||||||
|
"find a mount point an earlier snapshot found)")
|
||||||
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
||||||
"With --prune: keep snapshots newer than this duration "+
|
"With --prune: keep snapshots newer than this duration "+
|
||||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
||||||
|
|||||||
@@ -123,6 +123,16 @@ CREATE TABLE IF NOT EXISTS snapshot_blobs (
|
|||||||
-- Index for efficient blob lookups (used in orphan detection)
|
-- Index for efficient blob lookups (used in orphan detection)
|
||||||
CREATE INDEX IF NOT EXISTS idx_snapshot_blobs_blob_id ON snapshot_blobs(blob_id);
|
CREATE INDEX IF NOT EXISTS idx_snapshot_blobs_blob_id ON snapshot_blobs(blob_id);
|
||||||
|
|
||||||
|
-- Snapshot mount points table: the directories at or below a configured
|
||||||
|
-- path that the snapshot's backup found on a different filesystem from
|
||||||
|
-- their parent directory, as a mount point is, and found an entry in
|
||||||
|
CREATE TABLE IF NOT EXISTS snapshot_mount_points (
|
||||||
|
snapshot_id TEXT NOT NULL,
|
||||||
|
path TEXT NOT NULL,
|
||||||
|
PRIMARY KEY (snapshot_id, path),
|
||||||
|
FOREIGN KEY (snapshot_id) REFERENCES snapshots(id) ON DELETE CASCADE
|
||||||
|
);
|
||||||
|
|
||||||
-- Uploads table: tracks blob upload metrics
|
-- Uploads table: tracks blob upload metrics
|
||||||
CREATE TABLE IF NOT EXISTS uploads (
|
CREATE TABLE IF NOT EXISTS uploads (
|
||||||
blob_hash TEXT PRIMARY KEY,
|
blob_hash TEXT PRIMARY KEY,
|
||||||
|
|||||||
@@ -24,7 +24,8 @@ var (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// SnapshotRepository provides access to the snapshots table and its
|
// SnapshotRepository provides access to the snapshots table and its
|
||||||
// snapshot_files / snapshot_blobs association tables.
|
// snapshot_files / snapshot_blobs association tables, and to the mount
|
||||||
|
// points each snapshot's backup found (snapshot_mount_points).
|
||||||
type SnapshotRepository struct {
|
type SnapshotRepository struct {
|
||||||
db *DB
|
db *DB
|
||||||
}
|
}
|
||||||
@@ -494,33 +495,63 @@ func (r *SnapshotRepository) GetBlobHashes(
|
|||||||
ctx context.Context, snapshotID string,
|
ctx context.Context, snapshotID string,
|
||||||
) ([]string, error) {
|
) ([]string, error) {
|
||||||
query := `
|
query := `
|
||||||
SELECT sb.blob_hash
|
SELECT sb.blob_hash
|
||||||
FROM snapshot_blobs sb
|
FROM snapshot_blobs sb
|
||||||
WHERE sb.snapshot_id = ?
|
WHERE sb.snapshot_id = ?
|
||||||
ORDER BY sb.blob_hash
|
ORDER BY sb.blob_hash
|
||||||
`
|
`
|
||||||
|
|
||||||
rows, err := r.db.conn.QueryContext(ctx, query, snapshotID)
|
blobs, err := r.queryStrings(ctx, query, snapshotID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("querying blob hashes: %w", err)
|
return nil, fmt.Errorf("querying blob hashes: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
defer func() { _ = rows.Close() }()
|
return blobs, nil
|
||||||
|
}
|
||||||
|
|
||||||
var blobs []string
|
// AddMountPoint records dir as a directory that the backup of a snapshot
|
||||||
|
// found on a different filesystem from its parent directory, as a mount
|
||||||
|
// point is, using tx when non-nil.
|
||||||
|
func (r *SnapshotRepository) AddMountPoint(
|
||||||
|
ctx context.Context, tx *sql.Tx, snapshotID string, dir string,
|
||||||
|
) error {
|
||||||
|
query := `
|
||||||
|
INSERT OR IGNORE INTO snapshot_mount_points (snapshot_id, path)
|
||||||
|
VALUES (?, ?)
|
||||||
|
`
|
||||||
|
|
||||||
for rows.Next() {
|
var err error
|
||||||
var blobHash string
|
if tx != nil {
|
||||||
|
_, err = tx.ExecContext(ctx, query, snapshotID, dir)
|
||||||
err := rows.Scan(&blobHash)
|
} else {
|
||||||
if err != nil {
|
_, err = r.db.ExecWithLog(ctx, query, snapshotID, dir)
|
||||||
return nil, fmt.Errorf("scanning blob hash: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
blobs = append(blobs, blobHash)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return blobs, rows.Err()
|
if err != nil {
|
||||||
|
return fmt.Errorf("adding mount point to snapshot: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetMountPoints returns the directories AddMountPoint recorded for a
|
||||||
|
// snapshot, sorted.
|
||||||
|
func (r *SnapshotRepository) GetMountPoints(
|
||||||
|
ctx context.Context, snapshotID string,
|
||||||
|
) ([]string, error) {
|
||||||
|
query := `
|
||||||
|
SELECT path
|
||||||
|
FROM snapshot_mount_points
|
||||||
|
WHERE snapshot_id = ?
|
||||||
|
ORDER BY path
|
||||||
|
`
|
||||||
|
|
||||||
|
dirs, err := r.queryStrings(ctx, query, snapshotID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("querying mount points: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return dirs, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetSnapshotTotalCompressedSize returns the total compressed size of all
|
// GetSnapshotTotalCompressedSize returns the total compressed size of all
|
||||||
@@ -709,6 +740,34 @@ func (r *SnapshotRepository) DeleteSnapshotUploads(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// queryStrings runs query, which selects one text column, and returns
|
||||||
|
// its values in the order of the rows.
|
||||||
|
func (r *SnapshotRepository) queryStrings(
|
||||||
|
ctx context.Context, query string, args ...any,
|
||||||
|
) ([]string, error) {
|
||||||
|
rows, err := r.db.conn.QueryContext(ctx, query, args...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = rows.Close() }()
|
||||||
|
|
||||||
|
var values []string
|
||||||
|
|
||||||
|
for rows.Next() {
|
||||||
|
var value string
|
||||||
|
|
||||||
|
err := rows.Scan(&value)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
values = append(values, value)
|
||||||
|
}
|
||||||
|
|
||||||
|
return values, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
// extendedCompressionRatio computes the compression ratio for a snapshot
|
// extendedCompressionRatio computes the compression ratio for a snapshot
|
||||||
// from its stored blob_size and the given uncompressed size. Returns 1.0
|
// from its stored blob_size and the given uncompressed size. Returns 1.0
|
||||||
// when the uncompressed size is zero.
|
// when the uncompressed size is zero.
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"math"
|
"math"
|
||||||
|
"slices"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -422,3 +423,52 @@ func TestSnapshotRepositoryDuplicate(t *testing.T) {
|
|||||||
t.Error("expected error for duplicate snapshot")
|
t.Error("expected error for duplicate snapshot")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Deleting a snapshot from the local index deletes the mount points its
|
||||||
|
// backup recorded, and no other snapshot's. See
|
||||||
|
// https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||||
|
func TestSnapshotRepositoryDeleteRemovesMountPoints(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
db, cleanup := setupTestDB(t)
|
||||||
|
defer cleanup()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
repo := database.NewSnapshotRepository(db)
|
||||||
|
|
||||||
|
for _, id := range []types.SnapshotID{"kept", "deleted"} {
|
||||||
|
err := repo.Create(ctx, nil, &database.Snapshot{
|
||||||
|
ID: id,
|
||||||
|
Hostname: testHostname,
|
||||||
|
VaultikVersion: testVersion,
|
||||||
|
StartedAt: time.Now().Truncate(time.Second),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create snapshot: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = repo.AddMountPoint(ctx, nil, id.String(), "/data/nas")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to add mount point: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
err := repo.Delete(ctx, "deleted")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to delete snapshot: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for id, want := range map[string][]string{
|
||||||
|
"kept": {"/data/nas"},
|
||||||
|
"deleted": nil,
|
||||||
|
} {
|
||||||
|
got, err := repo.GetMountPoints(ctx, id)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to read mount points: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("mount points of %s: got %v, want %v", id, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,40 @@
|
|||||||
|
//nolint:testpackage // exercises the unexported deviceNumberFromStat
|
||||||
|
package snapshot
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The device number a real backup reads is the same for a directory and
|
||||||
|
// a file in it, and differs between the temp directory and /dev.
|
||||||
|
// Without it no mount point is ever found. See
|
||||||
|
// https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||||
|
func TestDeviceNumberFromStat(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
file := filepath.Join(dir, "file.txt")
|
||||||
|
require.NoError(t, os.WriteFile(file, nil, 0o600))
|
||||||
|
|
||||||
|
deviceOf := func(path string) uint64 {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
info, err := os.Stat(path)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
device, ok := deviceNumberFromStat(info)
|
||||||
|
require.True(t, ok, "no device number read for %s", path)
|
||||||
|
|
||||||
|
return device
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.Equal(t, deviceOf(dir), deviceOf(file))
|
||||||
|
|
||||||
|
// /dev is a different filesystem from the temp dir on Linux, macOS and in Docker.
|
||||||
|
assert.NotEqual(t, deviceOf(dir), deviceOf("/dev"))
|
||||||
|
}
|
||||||
@@ -1,6 +1,8 @@
|
|||||||
package snapshot
|
package snapshot
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"os"
|
||||||
|
|
||||||
"github.com/spf13/afero"
|
"github.com/spf13/afero"
|
||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
"sneak.berlin/go/vaultik/internal/config"
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
@@ -19,6 +21,9 @@ type ScannerParams struct {
|
|||||||
// PreviousSnapshotID is the most recent completed snapshot of the
|
// PreviousSnapshotID is the most recent completed snapshot of the
|
||||||
// same name, "" for none.
|
// same name, "" for none.
|
||||||
PreviousSnapshotID string
|
PreviousSnapshotID string
|
||||||
|
// DeviceNumber is ScannerConfig.DeviceNumber; nil reads it from the
|
||||||
|
// file's stat.
|
||||||
|
DeviceNumber func(info os.FileInfo) (uint64, bool)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Module exports backup functionality as an fx module.
|
// Module exports backup functionality as an fx module.
|
||||||
@@ -59,6 +64,7 @@ func provideScannerFactory(
|
|||||||
Exclude: excludes,
|
Exclude: excludes,
|
||||||
SkipErrors: params.SkipErrors,
|
SkipErrors: params.SkipErrors,
|
||||||
PreviousSnapshotID: params.PreviousSnapshotID,
|
PreviousSnapshotID: params.PreviousSnapshotID,
|
||||||
|
DeviceNumber: params.DeviceNumber,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,245 @@
|
|||||||
|
package snapshot_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"syscall"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/spf13/afero"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
|
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A scan records with the snapshot each directory on a different device
|
||||||
|
// from its parent directory that the walk found an entry in, as a mount
|
||||||
|
// point is, the scanned path included. A filesystem that gives no device
|
||||||
|
// numbers, as the in-memory one, records none. See
|
||||||
|
// https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||||
|
func TestScanRecordsMountPoints(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
scannedPath = "/data/source"
|
||||||
|
snapshotID = "mount-points"
|
||||||
|
)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
// onVolume names the directories on the volume's device; every
|
||||||
|
// other directory is on another one. nil leaves the device
|
||||||
|
// numbers to the in-memory filesystem, which has none.
|
||||||
|
onVolume []string
|
||||||
|
want []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "volume below the scanned path",
|
||||||
|
onVolume: []string{"nas"},
|
||||||
|
want: []string{scannedPath + "/nas"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "volume at the scanned path",
|
||||||
|
onVolume: []string{"source", "nas"},
|
||||||
|
want: []string{scannedPath},
|
||||||
|
},
|
||||||
|
{name: "no device numbers", want: nil},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
fs := afero.NewMemMapFs()
|
||||||
|
|
||||||
|
for _, path := range []string{
|
||||||
|
scannedPath + "/notes.txt", scannedPath + "/nas/a.txt",
|
||||||
|
} {
|
||||||
|
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o755))
|
||||||
|
require.NoError(t, afero.WriteFile(fs, path, []byte(path), 0o644))
|
||||||
|
}
|
||||||
|
|
||||||
|
db, err := database.NewTestDB()
|
||||||
|
require.NoError(t, err)
|
||||||
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
|
||||||
|
repos := database.NewRepositories(db)
|
||||||
|
createTestSnapshotRecord(ctx, t, repos, snapshotID)
|
||||||
|
|
||||||
|
cfg := snapshot.ScannerConfig{
|
||||||
|
FS: fs,
|
||||||
|
ChunkSize: int64(1024 * 16),
|
||||||
|
Repositories: repos,
|
||||||
|
MaxBlobSize: int64(1024 * 1024),
|
||||||
|
CompressionLevel: 3,
|
||||||
|
AgeRecipients: []string{testAgePublicKey},
|
||||||
|
}
|
||||||
|
if tt.onVolume != nil {
|
||||||
|
cfg.DeviceNumber = func(info os.FileInfo) (uint64, bool) {
|
||||||
|
if slices.Contains(tt.onVolume, info.Name()) {
|
||||||
|
return 2, true
|
||||||
|
}
|
||||||
|
|
||||||
|
return 1, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = snapshot.NewScanner(cfg).Scan(ctx, scannedPath, snapshotID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got, err := repos.Snapshots.GetMountPoints(ctx, snapshotID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, tt.want, got)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// automountFs is an in-memory filesystem with a volume at mountPoint
|
||||||
|
// that is mounted on first access: opening mountPoint to list it mounts
|
||||||
|
// the volume or, if it cannot be mounted, fails with "no such file or
|
||||||
|
// directory", as with the autofs daemon. Stat records the device an
|
||||||
|
// entry is on at that moment, as a real stat does: unmountedDevice for
|
||||||
|
// mountPoint until the volume is mounted; the volume's, 2, for
|
||||||
|
// mountPoint once it is and for everything in it; 1 for everything else.
|
||||||
|
type automountFs struct {
|
||||||
|
afero.Fs
|
||||||
|
|
||||||
|
mountPoint string
|
||||||
|
unmountedDevice uint64
|
||||||
|
canMount bool
|
||||||
|
mounted bool
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:ireturn // afero.Fs.Open is defined to return the interface.
|
||||||
|
func (fs *automountFs) Open(name string) (afero.File, error) {
|
||||||
|
if name == fs.mountPoint && !fs.canMount {
|
||||||
|
return nil, &os.PathError{Op: "open", Path: name, Err: syscall.ENOENT}
|
||||||
|
}
|
||||||
|
|
||||||
|
if name == fs.mountPoint {
|
||||||
|
fs.mounted = true
|
||||||
|
}
|
||||||
|
|
||||||
|
return fs.Fs.Open(name)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (fs *automountFs) Stat(name string) (os.FileInfo, error) {
|
||||||
|
info, err := fs.Fs.Stat(name)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
device := uint64(1)
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case name == fs.mountPoint && !fs.mounted:
|
||||||
|
device = fs.unmountedDevice
|
||||||
|
case name == fs.mountPoint || strings.HasPrefix(name, fs.mountPoint+"/"):
|
||||||
|
device = 2
|
||||||
|
}
|
||||||
|
|
||||||
|
return statWithDevice{FileInfo: info, device: device}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// statWithDevice is an entry automountFs.Stat looked up, with the device
|
||||||
|
// it was on then.
|
||||||
|
type statWithDevice struct {
|
||||||
|
os.FileInfo
|
||||||
|
|
||||||
|
device uint64
|
||||||
|
}
|
||||||
|
|
||||||
|
// deviceAtLookup returns the device automountFs.Stat recorded for info.
|
||||||
|
func deviceAtLookup(info os.FileInfo) (uint64, bool) {
|
||||||
|
stat, ok := info.(statWithDevice)
|
||||||
|
|
||||||
|
return stat.device, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
// A volume mounted on first access is mounted when the walk lists its
|
||||||
|
// mount point. Until then the mount point is on the automounter's own
|
||||||
|
// device (a systemd automount) or on its parent's (an autofs map that
|
||||||
|
// shows its mount points before mounting them). Either way only the
|
||||||
|
// mount point is recorded, by the backup that mounts the volume and by a
|
||||||
|
// later one that finds it mounted. Listing an automount point whose
|
||||||
|
// volume cannot be mounted fails; it is not recorded, because the walk
|
||||||
|
// finds no entry in it. See https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||||
|
func TestScanRecordsOnlyTheMountPointOfAnAutomountedVolume(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const scannedPath = "/data/source"
|
||||||
|
|
||||||
|
mountPoint := scannedPath + "/nas"
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
unmountedDevice uint64
|
||||||
|
canMount bool
|
||||||
|
want []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "on the automounter's device until listed",
|
||||||
|
unmountedDevice: 3, canMount: true, want: []string{mountPoint},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "on its parent's device until listed",
|
||||||
|
unmountedDevice: 1, canMount: true, want: []string{mountPoint},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "volume that cannot be mounted",
|
||||||
|
unmountedDevice: 1, canMount: false, want: nil,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
fs := &automountFs{
|
||||||
|
Fs: afero.NewMemMapFs(), mountPoint: mountPoint,
|
||||||
|
unmountedDevice: tt.unmountedDevice, canMount: tt.canMount,
|
||||||
|
}
|
||||||
|
|
||||||
|
require.NoError(t, fs.MkdirAll(mountPoint+"/photos", 0o755))
|
||||||
|
|
||||||
|
for _, path := range []string{
|
||||||
|
scannedPath + "/notes.txt", mountPoint + "/photos/a.jpg",
|
||||||
|
} {
|
||||||
|
require.NoError(t, afero.WriteFile(fs, path, []byte(path), 0o644))
|
||||||
|
}
|
||||||
|
|
||||||
|
db, err := database.NewTestDB()
|
||||||
|
require.NoError(t, err)
|
||||||
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
|
||||||
|
repos := database.NewRepositories(db)
|
||||||
|
|
||||||
|
for _, snapshotID := range []string{"first", "later"} {
|
||||||
|
createTestSnapshotRecord(ctx, t, repos, snapshotID)
|
||||||
|
|
||||||
|
scanner := snapshot.NewScanner(snapshot.ScannerConfig{
|
||||||
|
FS: fs,
|
||||||
|
ChunkSize: int64(1024 * 16),
|
||||||
|
Repositories: repos,
|
||||||
|
MaxBlobSize: int64(1024 * 1024),
|
||||||
|
CompressionLevel: 3,
|
||||||
|
AgeRecipients: []string{testAgePublicKey},
|
||||||
|
DeviceNumber: deviceAtLookup,
|
||||||
|
})
|
||||||
|
|
||||||
|
_, err = scanner.Scan(ctx, scannedPath, snapshotID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got, err := repos.Snapshots.GetMountPoints(ctx, snapshotID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, tt.want, got, "mount points of the %s backup", snapshotID)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -73,6 +73,9 @@ type Scanner struct {
|
|||||||
skipErrors bool
|
skipErrors bool
|
||||||
// ui is the user-facing output; never nil (defaults to a discarding writer).
|
// ui is the user-facing output; never nil (defaults to a discarding writer).
|
||||||
ui *ui.Writer
|
ui *ui.Writer
|
||||||
|
// deviceNumber returns the number of the device that holds a file;
|
||||||
|
// never nil (defaults to deviceNumberFromStat).
|
||||||
|
deviceNumber func(info os.FileInfo) (uint64, bool)
|
||||||
|
|
||||||
// In-memory cache of known chunk hashes for fast existence checks
|
// In-memory cache of known chunk hashes for fast existence checks
|
||||||
knownChunks map[string]struct{}
|
knownChunks map[string]struct{}
|
||||||
@@ -136,6 +139,10 @@ type ScannerConfig struct {
|
|||||||
// PreviousSnapshotID is the most recent completed snapshot of the
|
// PreviousSnapshotID is the most recent completed snapshot of the
|
||||||
// same name, "" for none.
|
// same name, "" for none.
|
||||||
PreviousSnapshotID string
|
PreviousSnapshotID string
|
||||||
|
// DeviceNumber returns the number of the device that holds a file,
|
||||||
|
// and false when there is none to read. nil reads it from the file's
|
||||||
|
// stat; tests replace it, since the in-memory test filesystem has none.
|
||||||
|
DeviceNumber func(info os.FileInfo) (uint64, bool)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ScanResult contains the results of a scan operation. Files and bytes
|
// ScanResult contains the results of a scan operation. Files and bytes
|
||||||
@@ -214,6 +221,11 @@ func NewScanner(cfg ScannerConfig) *Scanner {
|
|||||||
uiw = ui.NewWithColor(io.Discard, false)
|
uiw = ui.NewWithColor(io.Discard, false)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
deviceNumber := cfg.DeviceNumber
|
||||||
|
if deviceNumber == nil {
|
||||||
|
deviceNumber = deviceNumberFromStat
|
||||||
|
}
|
||||||
|
|
||||||
return &Scanner{
|
return &Scanner{
|
||||||
fs: cfg.FS,
|
fs: cfg.FS,
|
||||||
chunker: chunker.NewChunker(cfg.ChunkSize),
|
chunker: chunker.NewChunker(cfg.ChunkSize),
|
||||||
@@ -229,6 +241,7 @@ func NewScanner(cfg ScannerConfig) *Scanner {
|
|||||||
skipErrors: cfg.SkipErrors,
|
skipErrors: cfg.SkipErrors,
|
||||||
previousSnapshotID: cfg.PreviousSnapshotID,
|
previousSnapshotID: cfg.PreviousSnapshotID,
|
||||||
ui: uiw,
|
ui: uiw,
|
||||||
|
deviceNumber: deviceNumber,
|
||||||
pendingChunkHashes: make(map[string]struct{}),
|
pendingChunkHashes: make(map[string]struct{}),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -297,7 +310,7 @@ func (s *Scanner) Scan(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
err = s.recordDirectories(ctx, scanResult.Directories)
|
err = s.recordDirectories(ctx, scanResult)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -875,6 +888,11 @@ type ScanPhaseResult struct {
|
|||||||
FilesToProcess []*FileToProcess
|
FilesToProcess []*FileToProcess
|
||||||
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
|
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
|
||||||
Directories []*FileToProcess // Every directory found, to record
|
Directories []*FileToProcess // Every directory found, to record
|
||||||
|
// MountPoints are the directories, the scanned path included, that
|
||||||
|
// the walk found an entry in and, once it had listed them, on a
|
||||||
|
// different device from their parent directory (see
|
||||||
|
// recordIfMountPoint).
|
||||||
|
MountPoints []string
|
||||||
}
|
}
|
||||||
|
|
||||||
// scanPhase performs the initial directory scan to identify files to process
|
// scanPhase performs the initial directory scan to identify files to process
|
||||||
@@ -894,7 +912,7 @@ func (s *Scanner) scanPhase(
|
|||||||
// estimate leaves them out.
|
// estimate leaves them out.
|
||||||
estimatedTotal := countNonDirectories(knownFiles)
|
estimatedTotal := countNonDirectories(knownFiles)
|
||||||
|
|
||||||
collector := &scanCollector{}
|
collector := &scanCollector{dirsWithEntries: make(map[string]struct{})}
|
||||||
|
|
||||||
// Set up periodic status output
|
// Set up periodic status output
|
||||||
startTime := time.Now()
|
startTime := time.Now()
|
||||||
@@ -939,6 +957,8 @@ func (s *Scanner) scanPhase(
|
|||||||
result.EntriesFound++
|
result.EntriesFound++
|
||||||
}
|
}
|
||||||
|
|
||||||
|
s.recordIfMountPoint(filePath, path, collector)
|
||||||
|
|
||||||
// Handle symlinks and directories
|
// Handle symlinks and directories
|
||||||
handled, err := s.recordSpecialEntry(
|
handled, err := s.recordSpecialEntry(
|
||||||
filePath, info, existingFiles, knownFiles, collector, result)
|
filePath, info, existingFiles, knownFiles, collector, result)
|
||||||
@@ -973,6 +993,7 @@ func (s *Scanner) scanPhase(
|
|||||||
FilesToProcess: collector.filesToProcess,
|
FilesToProcess: collector.filesToProcess,
|
||||||
UnchangedFileIDs: collector.unchangedFileIDs,
|
UnchangedFileIDs: collector.unchangedFileIDs,
|
||||||
Directories: collector.directories,
|
Directories: collector.directories,
|
||||||
|
MountPoints: collector.mountPoints,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1099,6 +1120,29 @@ type scanCollector struct {
|
|||||||
unchangedFileIDs []types.FileID // Just IDs - no new records needed
|
unchangedFileIDs []types.FileID // Just IDs - no new records needed
|
||||||
directories []*FileToProcess
|
directories []*FileToProcess
|
||||||
filesScanned int64
|
filesScanned int64
|
||||||
|
// dirsWithEntries holds each directory the walk has found an entry in.
|
||||||
|
dirsWithEntries map[string]struct{}
|
||||||
|
mountPoints []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// firstEntryIn notes that the walk found an entry in directory dir and
|
||||||
|
// reports whether it is the first one.
|
||||||
|
func (c *scanCollector) firstEntryIn(dir string) bool {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
|
_, seen := c.dirsWithEntries[dir]
|
||||||
|
c.dirsWithEntries[dir] = struct{}{}
|
||||||
|
|
||||||
|
return !seen
|
||||||
|
}
|
||||||
|
|
||||||
|
// addMountPoint sets a mount point aside to be recorded.
|
||||||
|
func (c *scanCollector) addMountPoint(dir string) {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
|
c.mountPoints = append(c.mountPoints, dir)
|
||||||
}
|
}
|
||||||
|
|
||||||
// addDirectory sets a directory aside to be recorded. It is not counted
|
// addDirectory sets a directory aside to be recorded. It is not counted
|
||||||
@@ -1290,20 +1334,100 @@ func (s *Scanner) buildDirectoryEntry(path string, info os.FileInfo) *database.F
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// recordDirectories records every directory the walk found. A directory
|
// recordDirectories records every directory the walk found, and its
|
||||||
// has no data to back up, so it is not among the files counted and
|
// mount points (see recordMountPoints). A directory has no data to back
|
||||||
// backed up.
|
// up, so it is not among the files counted and backed up.
|
||||||
func (s *Scanner) recordDirectories(
|
func (s *Scanner) recordDirectories(
|
||||||
ctx context.Context, directories []*FileToProcess,
|
ctx context.Context, scanned *ScanPhaseResult,
|
||||||
) error {
|
) error {
|
||||||
for _, dir := range directories {
|
for _, dir := range scanned.Directories {
|
||||||
err := s.recordNonRegularFile(ctx, dir)
|
err := s.recordNonRegularFile(ctx, dir)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("recording directory %s: %w", dir.Path, err)
|
return fmt.Errorf("recording directory %s: %w", dir.Path, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return s.recordMountPoints(ctx, scanned.MountPoints)
|
||||||
|
}
|
||||||
|
|
||||||
|
// recordMountPoints records with the snapshot the mount points the walk
|
||||||
|
// found (see ScanPhaseResult.MountPoints).
|
||||||
|
func (s *Scanner) recordMountPoints(
|
||||||
|
ctx context.Context, mountPoints []string,
|
||||||
|
) error {
|
||||||
|
return s.repos.WithTx(ctx, func(txCtx context.Context, tx *sql.Tx) error {
|
||||||
|
for _, dir := range mountPoints {
|
||||||
|
log.Info("Found a mount point", "path", dir)
|
||||||
|
|
||||||
|
err := s.repos.Snapshots.AddMountPoint(txCtx, tx, s.snapshotID, dir)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("recording mount point %s: %w", dir, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// recordIfMountPoint sets the directory that holds entry, a path the
|
||||||
|
// walk found, aside to be recorded as a mount point if it is on a
|
||||||
|
// different device from its parent directory. The walk calls it for each
|
||||||
|
// entry it finds, and it decides on the first entry in a directory, so a
|
||||||
|
// directory the walk found no entry in is never recorded. By then the
|
||||||
|
// walk has listed the directory, which mounts a volume mounted there on
|
||||||
|
// first access, so the directory and its parent are looked up again:
|
||||||
|
// before the listing, the directory can be on the automounter's own
|
||||||
|
// device (a systemd automount) or on its parent's (an autofs map that
|
||||||
|
// shows its mount points before mounting them). The scanned path is in
|
||||||
|
// no listing the walk made, so it is skipped as an entry. A directory
|
||||||
|
// that cannot be looked up then, or whose parent cannot, is not
|
||||||
|
// recorded, so this snapshot does not stop the prune of a later backup
|
||||||
|
// that misses its volume.
|
||||||
|
func (s *Scanner) recordIfMountPoint(
|
||||||
|
entry, scannedPath string, collector *scanCollector,
|
||||||
|
) {
|
||||||
|
dir := filepath.Dir(entry)
|
||||||
|
|
||||||
|
if entry == scannedPath || !collector.firstEntryIn(dir) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
dirDevice, ok := s.deviceNumberAt(dir)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
parentDevice, ok := s.deviceNumberAt(filepath.Dir(dir))
|
||||||
|
if ok && dirDevice != parentDevice {
|
||||||
|
collector.addMountPoint(dir)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// deviceNumberAt looks path up now and returns the number of the device
|
||||||
|
// that holds it, and false when the lookup fails (logged) or gives no
|
||||||
|
// device number.
|
||||||
|
func (s *Scanner) deviceNumberAt(path string) (uint64, bool) {
|
||||||
|
info, err := s.fs.Stat(path)
|
||||||
|
if err != nil {
|
||||||
|
log.Warn("Could not look up a directory to check for a mount point",
|
||||||
|
"path", path, "error", err)
|
||||||
|
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
|
||||||
|
return s.deviceNumber(info)
|
||||||
|
}
|
||||||
|
|
||||||
|
// deviceNumberFromStat returns the number of the device that holds the
|
||||||
|
// file info describes, and false when info carries no stat, as from the
|
||||||
|
// in-memory test filesystem.
|
||||||
|
func deviceNumberFromStat(info os.FileInfo) (uint64, bool) {
|
||||||
|
stat, ok := info.Sys().(*syscall.Stat_t)
|
||||||
|
if !ok {
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
|
||||||
|
return uint64(stat.Dev), true //nolint:unconvert // Dev is an int32 on darwin
|
||||||
}
|
}
|
||||||
|
|
||||||
// recordNonRegularFile writes a symlink or directory entry to the database
|
// recordNonRegularFile writes a symlink or directory entry to the database
|
||||||
|
|||||||
@@ -518,8 +518,8 @@ type CleanupStats struct {
|
|||||||
//
|
//
|
||||||
// The cleanup is performed in a specific order to maintain referential integrity:
|
// The cleanup is performed in a specific order to maintain referential integrity:
|
||||||
// 1. Delete other snapshots
|
// 1. Delete other snapshots
|
||||||
// 2. Delete orphaned snapshot associations (snapshot_files, snapshot_blobs)
|
// 2. Delete orphaned snapshot associations (snapshot_files, snapshot_blobs,
|
||||||
// for deleted snapshots
|
// snapshot_mount_points) for deleted snapshots
|
||||||
// 3. Delete orphaned files (not in the current snapshot)
|
// 3. Delete orphaned files (not in the current snapshot)
|
||||||
// 4. Delete orphaned chunk-to-file mappings (references to deleted files)
|
// 4. Delete orphaned chunk-to-file mappings (references to deleted files)
|
||||||
// 5. Delete orphaned blobs (not in the current snapshot)
|
// 5. Delete orphaned blobs (not in the current snapshot)
|
||||||
@@ -948,8 +948,8 @@ func (sm *SnapshotManager) deleteOtherSnapshots(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// deleteOrphanedSnapshotAssociations deletes snapshot_files and
|
// deleteOrphanedSnapshotAssociations deletes snapshot_files,
|
||||||
// snapshot_blobs for deleted snapshots
|
// snapshot_blobs and snapshot_mount_points for deleted snapshots
|
||||||
func (sm *SnapshotManager) deleteOrphanedSnapshotAssociations(
|
func (sm *SnapshotManager) deleteOrphanedSnapshotAssociations(
|
||||||
ctx context.Context, tx *sql.Tx, currentSnapshotID string,
|
ctx context.Context, tx *sql.Tx, currentSnapshotID string,
|
||||||
) error {
|
) error {
|
||||||
@@ -983,6 +983,19 @@ func (sm *SnapshotManager) deleteOrphanedSnapshotAssociations(
|
|||||||
log.Debug("[Temp DB Cleanup] Deleted snapshot_blobs associations",
|
log.Debug("[Temp DB Cleanup] Deleted snapshot_blobs associations",
|
||||||
"count", rowsAffected)
|
"count", rowsAffected)
|
||||||
|
|
||||||
|
// Delete the mount points other snapshots recorded
|
||||||
|
query := "DELETE FROM snapshot_mount_points WHERE snapshot_id != ?"
|
||||||
|
database.LogSQL("Execute", query, currentSnapshotID)
|
||||||
|
|
||||||
|
result, err = tx.ExecContext(ctx, query, currentSnapshotID)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("deleting orphaned snapshot_mount_points: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rowsAffected, _ = result.RowsAffected()
|
||||||
|
log.Debug("[Temp DB Cleanup] Deleted snapshot_mount_points rows",
|
||||||
|
"count", rowsAffected)
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -7,12 +7,14 @@ import (
|
|||||||
"database/sql"
|
"database/sql"
|
||||||
"io"
|
"io"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/spf13/afero"
|
"github.com/spf13/afero"
|
||||||
"sneak.berlin/go/vaultik/internal/config"
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
"sneak.berlin/go/vaultik/internal/database"
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
|
"sneak.berlin/go/vaultik/internal/types"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -387,6 +389,77 @@ func TestCleanSnapshotDBNonExistentSnapshot(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The exported metadata database keeps only the mount points that the
|
||||||
|
// exported snapshot recorded. See
|
||||||
|
// https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||||
|
func TestCleanSnapshotDBKeepsOnlyTheSnapshotsMountPoints(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
dbPath := filepath.Join(t.TempDir(), "export.db")
|
||||||
|
|
||||||
|
db, err := database.New(ctx, dbPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create database: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
repos := database.NewRepositories(db)
|
||||||
|
|
||||||
|
for _, id := range []types.SnapshotID{"exported", "other"} {
|
||||||
|
err = repos.Snapshots.Create(ctx, nil, &database.Snapshot{ID: id})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create snapshot: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = repos.Snapshots.AddMountPoint(ctx, nil, id.String(), "/data/"+id.String())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to add mount point: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
err = db.Close()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to close database: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
sm := &SnapshotManager{
|
||||||
|
config: &config.Config{
|
||||||
|
CompressionLevel: 3,
|
||||||
|
AgeRecipients: []string{testAgeRecipient},
|
||||||
|
},
|
||||||
|
fs: afero.NewOsFs(),
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = sm.cleanSnapshotDB(ctx, dbPath, "exported")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to clean snapshot database: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cleaned, err := database.New(ctx, dbPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to open cleaned database: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = cleaned.Close() }()
|
||||||
|
|
||||||
|
cleanedRepos := database.NewRepositories(cleaned)
|
||||||
|
|
||||||
|
for id, want := range map[string][]string{
|
||||||
|
"exported": {"/data/exported"},
|
||||||
|
"other": nil,
|
||||||
|
} {
|
||||||
|
got, err := cleanedRepos.Snapshots.GetMountPoints(ctx, id)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to read mount points: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("mount points of %s: got %v, want %v", id, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Two creates of one snapshot name back to back start within one second,
|
// Two creates of one snapshot name back to back start within one second,
|
||||||
// the resolution of the timestamp in a snapshot ID. See
|
// the resolution of the timestamp in a snapshot ID. See
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/270.
|
// https://git.eeqj.de/sneak/vaultik/issues/270.
|
||||||
|
|||||||
@@ -679,6 +679,7 @@ func faultScannerFactory(
|
|||||||
Exclude: params.Exclude,
|
Exclude: params.Exclude,
|
||||||
SkipErrors: params.SkipErrors,
|
SkipErrors: params.SkipErrors,
|
||||||
PreviousSnapshotID: params.PreviousSnapshotID,
|
PreviousSnapshotID: params.PreviousSnapshotID,
|
||||||
|
DeviceNumber: params.DeviceNumber,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+103
-14
@@ -7,6 +7,7 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"regexp"
|
"regexp"
|
||||||
|
"slices"
|
||||||
"sort"
|
"sort"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -52,6 +53,17 @@ var ErrFilesUnreadable = errors.New("file(s) could not be read")
|
|||||||
// returns ErrFilesUnreadable instead.
|
// returns ErrFilesUnreadable instead.
|
||||||
var ErrSourcePathEmpty = errors.New("source path(s) were empty")
|
var ErrSourcePathEmpty = errors.New("source path(s) were empty")
|
||||||
|
|
||||||
|
// ErrVolumeNotMounted is returned when --prune skipped the post-backup
|
||||||
|
// prune because a directory that an earlier snapshot of the same name
|
||||||
|
// found as a mount point was not found as one in this run: the volume
|
||||||
|
// mounted there is not mounted, the run found no entry in it, or the
|
||||||
|
// directory is no longer backed up. The snapshots are complete and
|
||||||
|
// kept, so the caller can report it apart from a failed backup. A run
|
||||||
|
// that also skipped unreadable files or found a configured path empty
|
||||||
|
// returns ErrFilesUnreadable or ErrSourcePathEmpty instead.
|
||||||
|
var ErrVolumeNotMounted = errors.New(
|
||||||
|
"mount point(s) an earlier snapshot recorded were not found")
|
||||||
|
|
||||||
// listRecentLimit caps how many snapshot rows are fetched from the
|
// listRecentLimit caps how many snapshot rows are fetched from the
|
||||||
// local index for listing and purge operations.
|
// local index for listing and purge operations.
|
||||||
const listRecentLimit = 10000
|
const listRecentLimit = 10000
|
||||||
@@ -142,7 +154,7 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
filesUnreadable, emptyPaths, err := v.createNamedSnapshots(
|
filesUnreadable, emptyPaths, unmountedPaths, err := v.createNamedSnapshots(
|
||||||
opts, hostname, snapshotNames)
|
opts, hostname, snapshotNames)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -155,8 +167,10 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
switch {
|
switch {
|
||||||
case opts.Prune && (filesUnreadable > 0 || len(emptyPaths) > 0):
|
case opts.Prune && (filesUnreadable > 0 || len(emptyPaths) > 0 ||
|
||||||
return v.skipPostBackupPrune(filesUnreadable, emptyPaths)
|
len(unmountedPaths) > 0):
|
||||||
|
return v.skipPostBackupPrune(
|
||||||
|
filesUnreadable, emptyPaths, unmountedPaths)
|
||||||
case opts.Prune:
|
case opts.Prune:
|
||||||
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -169,20 +183,22 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
|||||||
|
|
||||||
// createNamedSnapshots creates each named snapshot in turn. It returns
|
// createNamedSnapshots creates each named snapshot in turn. It returns
|
||||||
// the number of entries --skip-errors skipped because it could not read
|
// the number of entries --skip-errors skipped because it could not read
|
||||||
// or list them, and each configured path in which the scan found
|
// or list them, each configured path in which the scan found nothing,
|
||||||
// nothing, named with its snapshot.
|
// and, under --prune, each directory where an earlier snapshot found a
|
||||||
|
// volume mounted and this run did not, the paths named with their
|
||||||
|
// snapshot.
|
||||||
func (v *Vaultik) createNamedSnapshots(
|
func (v *Vaultik) createNamedSnapshots(
|
||||||
opts *SnapshotCreateOptions, hostname string, snapshotNames []string,
|
opts *SnapshotCreateOptions, hostname string, snapshotNames []string,
|
||||||
) (int, []string, error) {
|
) (int, []string, []string, error) {
|
||||||
filesUnreadable := 0
|
filesUnreadable := 0
|
||||||
|
|
||||||
var emptyPaths []string
|
var emptyPaths, unmountedPaths []string
|
||||||
|
|
||||||
for snapIdx, snapName := range snapshotNames {
|
for snapIdx, snapName := range snapshotNames {
|
||||||
stats, err := v.createNamedSnapshot(
|
stats, err := v.createNamedSnapshot(
|
||||||
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, nil, err
|
return 0, nil, nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
filesUnreadable += stats.totalFilesUnreadable
|
filesUnreadable += stats.totalFilesUnreadable
|
||||||
@@ -190,21 +206,27 @@ func (v *Vaultik) createNamedSnapshots(
|
|||||||
for _, dir := range stats.emptyPaths {
|
for _, dir := range stats.emptyPaths {
|
||||||
emptyPaths = append(emptyPaths, dir+" of snapshot "+snapName)
|
emptyPaths = append(emptyPaths, dir+" of snapshot "+snapName)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
for _, dir := range stats.unmountedPaths {
|
||||||
|
unmountedPaths = append(unmountedPaths,
|
||||||
|
dir+" of snapshot "+snapName)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return filesUnreadable, emptyPaths, nil
|
return filesUnreadable, emptyPaths, unmountedPaths, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// skipPostBackupPrune says why the post-backup prune does not run, naming
|
// skipPostBackupPrune says why the post-backup prune does not run, naming
|
||||||
// every cause, and returns the error the run ends with. An older snapshot
|
// every cause, and returns the error the run ends with. An older snapshot
|
||||||
// may be the only one that still holds a file this run could not read,
|
// may be the only one that still holds a file this run could not read,
|
||||||
// or what an empty path held. The previous snapshot is no guide: after
|
// or what an empty path or a volume that is not mounted held. The
|
||||||
// one run with an empty path it is empty too.
|
// previous snapshot is no guide: after one such run it lacks them too.
|
||||||
func (v *Vaultik) skipPostBackupPrune(
|
func (v *Vaultik) skipPostBackupPrune(
|
||||||
filesUnreadable int, emptyPaths []string,
|
filesUnreadable int, emptyPaths, unmountedPaths []string,
|
||||||
) error {
|
) error {
|
||||||
log.Warn("Skipping post-backup prune",
|
log.Warn("Skipping post-backup prune",
|
||||||
"files_unreadable", filesUnreadable, "empty_paths", emptyPaths)
|
"files_unreadable", filesUnreadable, "empty_paths", emptyPaths,
|
||||||
|
"unmounted_paths", unmountedPaths)
|
||||||
|
|
||||||
if len(emptyPaths) > 0 {
|
if len(emptyPaths) > 0 {
|
||||||
v.UI.Warningf("Skipped the post-backup prune because nothing was "+
|
v.UI.Warningf("Skipped the post-backup prune because nothing was "+
|
||||||
@@ -212,6 +234,17 @@ func (v *Vaultik) skipPostBackupPrune(
|
|||||||
strings.Join(emptyPaths, ", "))
|
strings.Join(emptyPaths, ", "))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if len(unmountedPaths) > 0 {
|
||||||
|
v.UI.Warningf("Skipped the post-backup prune because this backup "+
|
||||||
|
"did not find a mount point at %s, where an earlier snapshot "+
|
||||||
|
"found one: the volume is not mounted, this backup found no "+
|
||||||
|
"entry in it, or the directory is no longer backed up. Older "+
|
||||||
|
"snapshots are kept. Mount the volume again, or, if that is on "+
|
||||||
|
"purpose, remove the older snapshots by hand with "+
|
||||||
|
"'vaultik snapshot purge'.",
|
||||||
|
strings.Join(unmountedPaths, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
if filesUnreadable > 0 {
|
if filesUnreadable > 0 {
|
||||||
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
|
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
|
||||||
"could not be read; older snapshots are kept.", filesUnreadable)
|
"could not be read; older snapshots are kept.", filesUnreadable)
|
||||||
@@ -219,7 +252,11 @@ func (v *Vaultik) skipPostBackupPrune(
|
|||||||
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
|
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
|
||||||
}
|
}
|
||||||
|
|
||||||
return fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty)
|
if len(emptyPaths) > 0 {
|
||||||
|
return fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty)
|
||||||
|
}
|
||||||
|
|
||||||
|
return fmt.Errorf("%d %w", len(unmountedPaths), ErrVolumeNotMounted)
|
||||||
}
|
}
|
||||||
|
|
||||||
// finishCreateSnapshot ends a run whose snapshots are all complete and
|
// finishCreateSnapshot ends a run whose snapshots are all complete and
|
||||||
@@ -329,6 +366,10 @@ type snapshotStats struct {
|
|||||||
// The configured paths in which the scan found nothing.
|
// The configured paths in which the scan found nothing.
|
||||||
emptyPaths []string
|
emptyPaths []string
|
||||||
|
|
||||||
|
// Under --prune, the directories where an earlier snapshot of the
|
||||||
|
// same name found a volume mounted and this one did not.
|
||||||
|
unmountedPaths []string
|
||||||
|
|
||||||
// The sizes of all blobs the snapshot references, set by
|
// The sizes of all blobs the snapshot references, set by
|
||||||
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
||||||
blobSize int64
|
blobSize int64
|
||||||
@@ -394,9 +435,57 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
|
|
||||||
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
||||||
|
|
||||||
|
if opts.Prune {
|
||||||
|
stats.unmountedPaths, err = v.findUnmountedPaths(snapName, snapshotID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return stats, nil
|
return stats, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// findUnmountedPaths returns each directory that an earlier snapshot
|
||||||
|
// named snapName in the local index recorded as a mount point and
|
||||||
|
// snapshot snapshotID did not, as when the volume mounted there is not
|
||||||
|
// mounted.
|
||||||
|
func (v *Vaultik) findUnmountedPaths(
|
||||||
|
snapName, snapshotID string,
|
||||||
|
) ([]string, error) {
|
||||||
|
found, err := v.Repositories.Snapshots.GetMountPoints(v.ctx, snapshotID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
snapshots, err := v.Repositories.Snapshots.ListRecent(v.ctx, listRecentLimit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("listing snapshots: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var unmounted []string
|
||||||
|
|
||||||
|
for _, s := range snapshots {
|
||||||
|
if s.ID.String() == snapshotID ||
|
||||||
|
parseSnapshotName(s.ID.String(), s.Hostname.String()) != snapName {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
recorded, err := v.Repositories.Snapshots.GetMountPoints(
|
||||||
|
v.ctx, s.ID.String())
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, dir := range recorded {
|
||||||
|
if !slices.Contains(found, dir) && !slices.Contains(unmounted, dir) {
|
||||||
|
unmounted = append(unmounted, dir)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return unmounted, nil
|
||||||
|
}
|
||||||
|
|
||||||
// previousSnapshotID returns the ID of the most recent completed snapshot
|
// previousSnapshotID returns the ID of the most recent completed snapshot
|
||||||
// named snapName in the local index, or "" when there is none.
|
// named snapName in the local index, or "" when there is none.
|
||||||
func (v *Vaultik) previousSnapshotID(snapName string) (string, error) {
|
func (v *Vaultik) previousSnapshotID(snapName string) (string, error) {
|
||||||
|
|||||||
@@ -4,12 +4,15 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"syscall"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/spf13/afero"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/vaultik/internal/config"
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
|
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -259,3 +262,152 @@ func TestBackupWithNoEmptySourcePathPrunes(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// automountFs is the real filesystem with a volume at volumeDir that is
|
||||||
|
// mounted on first access, as by an autofs map that shows its mount
|
||||||
|
// points before mounting them: listing volumeDir mounts the volume or,
|
||||||
|
// while cannotMount, fails with "no such file or directory".
|
||||||
|
type automountFs struct {
|
||||||
|
afero.OsFs
|
||||||
|
|
||||||
|
volumeDir string
|
||||||
|
cannotMount bool
|
||||||
|
mounted bool
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:ireturn // afero.Fs.Open is defined to return the interface.
|
||||||
|
func (f *automountFs) Open(name string) (afero.File, error) {
|
||||||
|
if name == f.volumeDir && f.cannotMount {
|
||||||
|
return nil, &os.PathError{Op: "open", Path: name, Err: syscall.ENOENT}
|
||||||
|
}
|
||||||
|
|
||||||
|
if name == f.volumeDir {
|
||||||
|
f.mounted = true
|
||||||
|
}
|
||||||
|
|
||||||
|
return f.OsFs.Open(name)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A backup under --prune in which a volume that an earlier snapshot found
|
||||||
|
// mounted below the configured path is missing does not prune: run after
|
||||||
|
// run, the snapshot that holds the volume's files is kept with everything
|
||||||
|
// it stored, and the run fails. Once the volume is back, the prune runs.
|
||||||
|
// The volume is mounted on first access and every backup starts with it
|
||||||
|
// not mounted, so its directory is on its parent's device until the walk
|
||||||
|
// lists it. While the volume is missing, its directory is left empty, or
|
||||||
|
// listing it fails with "no such file or directory", as when an autofs
|
||||||
|
// volume cannot be mounted, which the walk counts as a directory removed
|
||||||
|
// during the backup. See https://git.eeqj.de/sneak/vaultik/issues/395.
|
||||||
|
func TestBackupWithVolumeNotMountedDoesNotPrune(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
listingFails bool
|
||||||
|
}{
|
||||||
|
{name: "directory left empty", listingFails: false},
|
||||||
|
{name: "listing fails", listingFails: true},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
backUpWithVolumeNotMounted(t, tt.listingFails)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// backUpWithVolumeNotMounted runs the backups of
|
||||||
|
// TestBackupWithVolumeNotMountedDoesNotPrune. While the volume is
|
||||||
|
// missing, listing its directory fails if listingFails; otherwise the
|
||||||
|
// directory is empty.
|
||||||
|
func backUpWithVolumeNotMounted(t *testing.T, listingFails bool) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
// volume is the directory, below the configured path, where the
|
||||||
|
// volume is mounted.
|
||||||
|
const (
|
||||||
|
snapshotName = "data"
|
||||||
|
volume = "nas"
|
||||||
|
)
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
env := newDeletedCountEnv(t, map[string][]byte{
|
||||||
|
volume + "/a.txt": []byte("file a"),
|
||||||
|
"notes.txt": []byte("notes"),
|
||||||
|
})
|
||||||
|
env.setSnapshot(snapshotName, "")
|
||||||
|
prune := &vaultik.SnapshotCreateOptions{
|
||||||
|
Prune: true, Snapshots: []string{snapshotName},
|
||||||
|
}
|
||||||
|
|
||||||
|
// The volume's directory is on the volume's device, 2, while the
|
||||||
|
// volume is mounted. Everything else is on device 1.
|
||||||
|
volumeDir := filepath.Join(env.src, volume)
|
||||||
|
fs := &automountFs{volumeDir: volumeDir}
|
||||||
|
newScanner := env.v.ScannerFactory
|
||||||
|
env.v.ScannerFactory = func(params snapshot.ScannerParams) *snapshot.Scanner {
|
||||||
|
fs.mounted = false
|
||||||
|
params.Fs = fs
|
||||||
|
params.DeviceNumber = func(info os.FileInfo) (uint64, bool) {
|
||||||
|
if info.Name() == volume && fs.mounted {
|
||||||
|
return 2, true
|
||||||
|
}
|
||||||
|
|
||||||
|
return 1, true
|
||||||
|
}
|
||||||
|
|
||||||
|
return newScanner(params)
|
||||||
|
}
|
||||||
|
|
||||||
|
env.backUp(t, snapshotName)
|
||||||
|
|
||||||
|
firstID := localSnapshotID(ctx, t, env.v.Repositories, snapshotName)
|
||||||
|
stored, err := env.v.Storage.List(ctx, "")
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
// The volume goes missing.
|
||||||
|
fs.cannotMount = listingFails
|
||||||
|
|
||||||
|
if !listingFails {
|
||||||
|
require.NoError(t, os.RemoveAll(volumeDir))
|
||||||
|
require.NoError(t, os.Mkdir(volumeDir, 0o750))
|
||||||
|
}
|
||||||
|
|
||||||
|
for range 2 {
|
||||||
|
env.out.Reset()
|
||||||
|
|
||||||
|
err := env.v.CreateSnapshot(prune)
|
||||||
|
require.ErrorIs(t, err, vaultik.ErrVolumeNotMounted)
|
||||||
|
assert.Contains(t, env.out.String(),
|
||||||
|
"did not find a mount point at "+volumeDir+" of snapshot "+snapshotName)
|
||||||
|
assert.Contains(t, env.out.String(), "'vaultik snapshot purge'")
|
||||||
|
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
|
||||||
|
}
|
||||||
|
|
||||||
|
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||||
|
ctx, listRecentTestLimit)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, snapshots, 3)
|
||||||
|
assert.Equal(t, firstID, snapshots[2].ID.String(),
|
||||||
|
"the snapshot that holds the volume's files is kept")
|
||||||
|
|
||||||
|
for _, key := range stored {
|
||||||
|
_, err := env.v.Storage.Stat(ctx, key)
|
||||||
|
require.NoError(t, err, "%s is kept", key)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The volume is back.
|
||||||
|
fs.cannotMount = false
|
||||||
|
|
||||||
|
require.NoError(t, os.WriteFile(
|
||||||
|
filepath.Join(volumeDir, "a.txt"), []byte("file a"), 0o600))
|
||||||
|
require.NoError(t, env.v.CreateSnapshot(prune))
|
||||||
|
|
||||||
|
snapshots, err = env.v.Repositories.Snapshots.ListRecent(
|
||||||
|
ctx, listRecentTestLimit)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, snapshots, 1, "the older snapshots are pruned")
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user