Compare commits

..
Author SHA1 Message Date
sneak 6ed84bf6d8 Fail restore for a requested path that matches nothing (closes #309)
check / check (push) Waiting to run
snapshot restore said nothing about a requested path that matched no
entry in the snapshot (a typo, a relative path, or a path through a
configured snapshot path that is a symlink, whose entries are stored
under the resolved directory) and exited 0, also when nothing matched
at all. Each such path is now named in a warning, and restore returns
an error after restoring the paths that do match.

Making a path through a symlink match is not part of this change.
Judgement call: when files also fail to restore, that error is the one
returned; the unmatched paths are still named.

Model: opus-5-5
2026-10-13 08:34:09 +00:00
45 changed files with 303 additions and 725 deletions
-13
View File
@@ -311,8 +311,6 @@ Repository interfaces:
``` ```
CreateSnapshot(opts) CreateSnapshot(opts)
│
├─► markDestination() // Refuse a destination without the marker, then write it
│ │
├─► PruneDatabase() // Critical: avoid dedup errors ├─► PruneDatabase() // Critical: avoid dedup errors
│ │
@@ -369,8 +367,6 @@ CreateSnapshot(opts)
``` ```
bucket/ bucket/
├── vaultik-destination.txt # Marker written by every backup
│
├── blobs/ ├── blobs/
│ └── {hash[0:2]}/ │ └── {hash[0:2]}/
│ └── {hash[2:4]}/ │ └── {hash[2:4]}/
@@ -382,15 +378,6 @@ bucket/
└── manifest.json.zst # Blob list (for pruning/verification) └── manifest.json.zst # Blob list (for pruning/verification)
``` ```
While the local index holds snapshots, a destination without
`vaultik-destination.txt` is treated as not present
(`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
`snapshot create` refuses to write to it, and listing it fails as for a
destination that cannot be listed. An unplugged volume's empty mount point
is therefore never taken for a destination whose snapshots are all gone.
The `file://` backend never creates its destination directory, so a write
to a missing one fails.
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
written to the store as a directory name. See written to the store as a directory name. See
+15 -47
View File
@@ -24,10 +24,6 @@ grep 'public key' vaultik_backup_private_key.txt
vaultik config set age_recipients.0 age1YOUR_PUBLIC_KEY_HERE vaultik config set age_recipients.0 age1YOUR_PUBLIC_KEY_HERE
vaultik config set storage_url "file:///Volumes/usbstick/mybackup" vaultik config set storage_url "file:///Volumes/usbstick/mybackup"
# create the destination directory; vaultik never creates it, so a backup
# to a volume that is not plugged in fails instead of filling the disk
mkdir /Volumes/usbstick/mybackup
# macOS only: grant your terminal app Full Disk Access first # macOS only: grant your terminal app Full Disk Access first
# (System Settings → Privacy & Security → Full Disk Access), otherwise # (System Settings → Privacy & Security → Full Disk Access), otherwise
# the backup will abort with a permission error on protected directories # the backup will abort with a permission error on protected directories
@@ -293,10 +289,6 @@ needed.
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
Without `--prune`, or with a value that is not a valid duration, the Without `--prune`, or with a value that is not a valid duration, the
command fails before the backup starts. command fails before the backup starts.
* Fails before it writes anything when a `file://` destination directory
does not exist, or when the local index holds snapshots and the
destination has no `vaultik-destination.txt` (see
[remote storage layout](#remote-storage-layout))
**`snapshot list`**: Show every snapshot known to this host — the union **`snapshot list`**: Show every snapshot known to this host — the union
of the local index and the backup destination store — with timestamps of the local index and the backup destination store — with timestamps
@@ -325,10 +317,7 @@ invocation that reconciles them.
If the destination store cannot be listed (unmounted volume, If the destination store cannot be listed (unmounted volume,
permission denied, network down), the command warns, falls back to the permission denied, network down), the command warns, falls back to the
local index alone, and still exits zero. It does the same when the local index alone, and still exits zero.
local index holds snapshots and the destination has no
`vaultik-destination.txt`, as when a volume mounted at the destination
is unplugged and leaves its empty mount point behind.
* `--json`: Output in JSON format. Each entry carries `locally_tracked` * `--json`: Output in JSON format. Each entry carries `locally_tracked`
(whether the snapshot is in the local index), `remote_key` (the full (whether the snapshot is in the local index), `remote_key` (the full
64-character storage key), and `remote_present` (whether it was seen 64-character storage key), and `remote_present` (whether it was seen
@@ -366,13 +355,11 @@ prune` invocation to run as a follow-up. An ID that names no snapshot in
the local index or on the destination store is an error; with the local index or on the destination store is an error; with
`--local-only`, the snapshot must be in the local index. Local row `--local-only`, the snapshot must be in the local index. Local row
cleanup (files, chunks, blobs the snapshot was the last referrer for) cleanup (files, chunks, blobs the snapshot was the last referrer for)
runs automatically. If the destination store is unreachable, or the runs automatically. If the destination store is unreachable, the local-DB
local index holds snapshots and the destination has no removal still completes and a warning is emitted; run `vaultik snapshot
`vaultik-destination.txt`, the local-DB removal still completes and a remove <snapshot-id>` again once the store is reachable to remove the
warning is emitted; run `vaultik snapshot remove <snapshot-id>` again snapshot's metadata from it (`vaultik prune` does not). To wipe everything
once the store is reachable to remove the snapshot's metadata from it on the destination in one go, use `vaultik remote nuke --force`.
(`vaultik prune` does not). To wipe everything on the destination in one
go, use `vaultik remote nuke --force`.
* `--local-only`: Skip remote cleanup; only touch the local index * `--local-only`: Skip remote cleanup; only touch the local index
* `--dry-run`: Show what would be deleted without deleting * `--dry-run`: Show what would be deleted without deleting
* `--force`: Skip confirmation prompt * `--force`: Skip confirmation prompt
@@ -385,7 +372,9 @@ go, use `vaultik remote nuke --force`.
store — its remote key (or an unambiguous leading part of it) as shown store — its remote key (or an unambiguous leading part of it) as shown
by `snapshot list`. See by `snapshot list`. See
[restoring on another machine](#restoring-on-another-machine). [restoring on another machine](#restoring-on-another-machine).
* Optional path arguments to restore specific files/directories (default: all) * Optional path arguments to restore specific files/directories (default: all).
A path that matches nothing in the snapshot is named in a warning, and the
command exits non-zero after restoring the paths that match
* Preserves file permissions, timestamps, ownership (ownership requires root), * Preserves file permissions, timestamps, ownership (ownership requires root),
symlinks, and empty directories symlinks, and empty directories
* `--verify`: After restoring, verify every file's chunk hashes match * `--verify`: After restoring, verify every file's chunk hashes match
@@ -399,10 +388,7 @@ manifest on the destination store to compute the still-referenced blob
set and delete any blob not in that set. Step (3) reads all remote set and delete any blob not in that set. Step (3) reads all remote
manifests — network cost scales with the number of snapshots. `snapshot manifests — network cost scales with the number of snapshots. `snapshot
create --prune` runs the same cleanup automatically; this is the create --prune` runs the same cleanup automatically; this is the
manual entry point for the same work. If the destination store cannot manual entry point for the same work.
be listed, or the local index holds snapshots and the destination has
no `vaultik-destination.txt` (an unplugged volume's empty mount point),
prune fails and deletes nothing. `snapshot purge` does the same.
* `--force`: Skip confirmation prompt * `--force`: Skip confirmation prompt
* `--json`: Output stats as JSON. Also skips the confirmation prompt, as * `--json`: Output stats as JSON. Also skips the confirmation prompt, as
`--force` does. `--force` does.
@@ -424,10 +410,10 @@ under `blobs/` whose name is not a blob hash is not counted as a blob,
since `prune` never deletes it. since `prune` never deletes it.
* `--json`: Output as JSON * `--json`: Output as JSON
**`remote nuke`**: Delete every snapshot's metadata, every blob and **`remote nuke`**: Delete every snapshot's metadata and every blob from the
`vaultik-destination.txt` from the backup destination store, leaving the backup destination store, leaving the bucket prefix empty. Destructive and
bucket prefix empty. Destructive and irreversible. This is the single irreversible. This is the single supported way to wipe the entire
supported way to wipe the entire destination store. destination store.
* `--force`: Required to confirm destruction. * `--force`: Required to confirm destruction.
**`database delete`**: Delete the local SQLite state database file **`database delete`**: Delete the local SQLite state database file
@@ -449,15 +435,7 @@ object store. Credentials are read from `s3.access_key_id` and
`s3.secret_access_key` in the config file. `s3.secret_access_key` in the config file.
**Local filesystem** (`file:///path/to/backup`): Stores blobs and metadata on **Local filesystem** (`file:///path/to/backup`): Stores blobs and metadata on
a local or mounted filesystem. Useful for testing or backing up to a NAS. The a local or mounted filesystem. Useful for testing or backing up to a NAS.
path must be absolute; a relative one is refused when the config loads.
Percent-escapes in the path are decoded, so `my%20stick` and `my stick` name
the same directory. The directory must exist before the first backup: vaultik
never creates it. A directory on a removable volume is gone while the volume is
not mounted, so a backup then fails instead of writing to the disk under the
mount point. The mount point itself stays behind when the volume is unplugged:
there only `vaultik-destination.txt` stops such a backup, and only while the
local index holds snapshots.
**Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud **Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud
providers. Requires rclone to be configured separately (`rclone config`). providers. Requires rclone to be configured separately (`rclone config`).
@@ -488,7 +466,6 @@ For an implementation-level view of the internals — the data model, the
``` ```
<bucket>/<prefix>/ <bucket>/<prefix>/
├── vaultik-destination.txt # Marker written by every backup
├── blobs/ ├── blobs/
│ └── <aa>/<bb>/<full_blob_hash> │ └── <aa>/<bb>/<full_blob_hash>
└── metadata/ └── metadata/
@@ -497,15 +474,6 @@ For an implementation-level view of the internals — the data model, the
└── manifest.json.zst # Unencrypted blob list (for pruning) └── manifest.json.zst # Unencrypted blob list (for pruning)
``` ```
* `vaultik-destination.txt` holds one fixed line of text, with no host
name, path or time. While the local index holds snapshots, a
destination without it is treated as not present: `snapshot create`
refuses to write to it, `prune` and `snapshot purge` fail, and
`snapshot list` and `snapshot remove` warn, as for a destination that
cannot be listed. This keeps an unplugged volume's empty mount point
from being taken for a destination whose snapshots are all gone. Mount
the volume if it is removable; if the destination was emptied on
purpose, run `vaultik database delete` and back up again.
* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash * Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
* `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted) * `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted)
containing all file metadata, chunk mappings, and relationships for the snapshot containing all file metadata, chunk mappings, and relationships for the snapshot
+1 -6
View File
@@ -4,7 +4,7 @@ This document describes the structure and organization of data stored in the S3
## Overview ## Overview
Vaultik stores all backup data in an S3-compatible object store. The repository consists of two main components, plus a marker file: Vaultik stores all backup data in an S3-compatible object store. The repository consists of two main components:
1. **Blobs** - The actual backup data (content-addressed, encrypted) 1. **Blobs** - The actual backup data (content-addressed, encrypted)
2. **Metadata** - Snapshot information and manifests (partially encrypted) 2. **Metadata** - Snapshot information and manifests (partially encrypted)
@@ -12,7 +12,6 @@ Vaultik stores all backup data in an S3-compatible object store. The repository
``` ```
<bucket>/<prefix>/ <bucket>/<prefix>/
├── vaultik-destination.txt
├── blobs/ ├── blobs/
│ └── <hash[0:2]>/ │ └── <hash[0:2]>/
│ └── <hash[2:4]>/ │ └── <hash[2:4]>/
@@ -25,10 +24,6 @@ Vaultik stores all backup data in an S3-compatible object store. The repository
The metadata subdirectory is named with the **remote key**, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See [Remote Key Derivation](#remote-key-derivation). The metadata subdirectory is named with the **remote key**, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See [Remote Key Derivation](#remote-key-derivation).
## Marker File (`vaultik-destination.txt`)
Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. While the local index holds snapshots, a destination without it is treated as not present: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove` and `snapshot list` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
## Blobs Directory (`blobs/`) ## Blobs Directory (`blobs/`)
### Structure ### Structure
-11
View File
@@ -245,17 +245,6 @@ func (p *Packer) Flush(ctx context.Context) error {
return nil return nil
} }
// DiscardCurrentBlob closes and removes the temp file of the blob being
// packed, if any, and drops that blob. A caller that stops before Flush
// must call it, or the temp file is left behind. Thread-safe.
func (p *Packer) DiscardCurrentBlob() {
p.mu.Lock()
defer p.mu.Unlock()
p.cleanupTempFile()
p.currentBlob = nil
}
// FinalizeBlob finalizes the current blob being assembled. // FinalizeBlob finalizes the current blob being assembled.
// This compresses the accumulated chunks, encrypts the result, and computes // This compresses the accumulated chunks, encrypts the result, and computes
// the content-addressed hash. The finalized blob is either passed to the // the content-addressed hash. The finalized blob is either passed to the
+1 -7
View File
@@ -101,8 +101,7 @@ func TestEntryPruneJSONStdoutIsExactlyOneDocument(t *testing.T) {
// writeHermeticPruneConfig builds a config over a temp directory with an // writeHermeticPruneConfig builds a config over a temp directory with an
// empty destination directory and, if seedStale is set, creates the // empty destination directory and, if seedStale is set, creates the
// index database up front with one snapshot record that has no // index database up front with one snapshot record that has no
// counterpart on the destination store, and gives the destination the // counterpart on the destination store. Returns the config path.
// marker a backup writes. Returns the config path.
func writeHermeticPruneConfig(t *testing.T, seedStale bool) string { func writeHermeticPruneConfig(t *testing.T, seedStale bool) string {
t.Helper() t.Helper()
@@ -132,11 +131,6 @@ func writeHermeticPruneConfig(t *testing.T, seedStale bool) string {
if seedStale { if seedStale {
seedStaleSnapshotRecord(t, indexPath) seedStaleSnapshotRecord(t, indexPath)
// While the local index holds a snapshot, prune fails on a
// destination without the marker a backup writes.
require.NoError(t, os.WriteFile(
filepath.Join(storeDir, "vaultik-destination.txt"), nil, 0o600))
} }
return configPath return configPath
+2
View File
@@ -30,6 +30,8 @@ This command will restore files from the specified snapshot to the
target directory. target directory.
If no paths are specified, all files are restored. If no paths are specified, all files are restored.
If paths are specified, only matching files/directories are restored. If paths are specified, only matching files/directories are restored.
A path that matches nothing in the snapshot is named in a warning, and
the command exits non-zero after restoring the paths that match.
The snapshot may be named by its ID or, when restoring on a host with no The snapshot may be named by its ID or, when restoring on a host with no
local index, by the remote key that 'snapshot list' prints for a local index, by the remote key that 'snapshot list' prints for a
+2 -12
View File
@@ -61,9 +61,6 @@ var (
errBadS3PartSize = errors.New("s3.part_size must be between 5MiB and 5GiB") errBadS3PartSize = errors.New("s3.part_size must be between 5MiB and 5GiB")
errBadStorageScheme = errors.New( errBadStorageScheme = errors.New(
"storage_url must start with s3://, file://, or rclone://") "storage_url must start with s3://, file://, or rclone://")
errFileURLNotAbsolute = errors.New(
"a file:// storage_url needs an absolute path, " +
"as in file:///path/to/backup")
errStorageNotConfigured = errors.New( errStorageNotConfigured = errors.New(
"storage not configured; set storage_url or provide s3.endpoint + " + "storage not configured; set storage_url or provide s3.endpoint + " +
"s3.bucket + credentials") "s3.bucket + credentials")
@@ -464,18 +461,11 @@ func (c *Config) validateStorage() error {
// validateStorageURL validates URL-based storage configuration. File and // validateStorageURL validates URL-based storage configuration. File and
// rclone URLs need no credentials; S3 URLs require the legacy s3.* // rclone URLs need no credentials; S3 URLs require the legacy s3.*
// credential fields. A file URL must name an absolute path: a relative one // credential fields.
// would resolve against the working directory, so the destination would
// move with it.
func (c *Config) validateStorageURL() error { func (c *Config) validateStorageURL() error {
switch { switch {
case strings.HasPrefix(c.StorageURL, "file://"): case strings.HasPrefix(c.StorageURL, "file://"):
// storage.ParseStorageURL applies the same rule, but config // File storage doesn't need S3 credentials
// cannot import storage, which imports config.
if !strings.HasPrefix(c.StorageURL, "file:///") {
return errFileURLNotAbsolute
}
return nil return nil
case strings.HasPrefix(c.StorageURL, "rclone://"): case strings.HasPrefix(c.StorageURL, "rclone://"):
// Rclone storage uses rclone's own config // Rclone storage uses rclone's own config
-29
View File
@@ -337,35 +337,6 @@ func TestLoadS3PartSize(t *testing.T) {
}) })
} }
// TestLoadRejectsRelativeFileURL checks that a file:// storage_url with a
// relative path fails at load. Accepted, it would resolve against the
// working directory, so a backup run from another directory would write
// to a different destination and still succeed.
func TestLoadRejectsRelativeFileURL(t *testing.T) {
t.Parallel()
const relativeConfig = "snapshots:\n" +
" test:\n" +
" paths: [/tmp/vaultik-test-source]\n" +
"storage_url: file://dest\n"
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(relativeConfig), 0o600)
if err != nil {
t.Fatalf("write config: %v", err)
}
_, err = Load(path)
if !errors.Is(err, errFileURLNotAbsolute) {
t.Fatalf("Load() error = %v, want errFileURLNotAbsolute", err)
}
if !strings.Contains(err.Error(), "absolute path") {
t.Errorf("error %q does not say to use an absolute path", err.Error())
}
}
// TestValidateAgeRecipients checks that recipients are parsed at config load // TestValidateAgeRecipients checks that recipients are parsed at config load
// (a bad entry fails immediately, not mid-backup) and that no invalid entry — // (a bad entry fails immediately, not mid-backup) and that no invalid entry —
// least of all a pasted secret key — is echoed in the error. An empty list // least of all a pasted secret key — is echoed in the error. An empty list
-5
View File
@@ -1295,11 +1295,6 @@ func (s *Scanner) batchAddFilesToSnapshot(
func (s *Scanner) processPhase( func (s *Scanner) processPhase(
ctx context.Context, filesToProcess []*FileToProcess, result *ScanResult, ctx context.Context, filesToProcess []*FileToProcess, result *ScanResult,
) error { ) error {
// On an error or cancellation the blob being packed is never
// finalized, so its temp file must be removed here. After a
// successful flush there is no such blob and this does nothing.
defer s.packer.DiscardCurrentBlob()
// Calculate total bytes to process // Calculate total bytes to process
var totalBytes int64 var totalBytes int64
for _, f := range filesToProcess { for _, f := range filesToProcess {
-57
View File
@@ -381,60 +381,3 @@ func TestScannerCancelStopsSkipErrorsRun(t *testing.T) {
}) })
} }
} }
// TestScannerRemovesBlobTempFileWhenRunStops checks that a run stopped
// part-way, by a file it cannot read or by a cancellation, removes the
// temp file of the blob it was packing.
func TestScannerRemovesBlobTempFileWhenRunStops(t *testing.T) {
t.Parallel()
// a.txt is backed up first and starts a blob; the run stops at b.txt.
const target = "/source/b.txt"
tests := []struct {
name string
newFs func(cancel context.CancelFunc) afero.Fs
}{
{
name: "file cannot be read",
newFs: func(context.CancelFunc) afero.Fs {
return &readFailFs{Fs: afero.NewMemMapFs(), target: target}
},
},
{
name: "cancelled",
newFs: func(cancel context.CancelFunc) afero.Fs {
return &cancelOnOpenFs{
Fs: afero.NewMemMapFs(), target: target, cancel: cancel,
}
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
fs := tt.newFs(cancel)
writeSkipErrorTestFile(t, fs, "/source/a.txt", "first file content")
writeSkipErrorTestFile(t, fs, target, "second file content")
_, err := runSkipErrorScan(ctx, t, fs, "/source", false, nil)
if err == nil {
t.Fatal("expected the scan to stop with an error, got nil")
}
left, err := afero.Glob(fs, filepath.Join(os.TempDir(), "vaultik-blob-*"))
if err != nil {
t.Fatalf("listing blob temp files: %v", err)
}
if len(left) != 0 {
t.Fatalf("expected no blob temp file left, got %v", left)
}
})
}
}
+4 -17
View File
@@ -21,10 +21,9 @@ type FileStorer struct {
// NewFileStorer creates a new filesystem storage backend. // NewFileStorer creates a new filesystem storage backend.
// //
// Construction is intentionally cheap and does not touch the filesystem. // Construction is intentionally cheap and does not touch the filesystem.
// The backend never creates basePath: the user does, so that a write to // The basePath is recorded; the directory is created lazily on first
// an unplugged volume fails instead of landing on the disk under its // write. Write operations (Put/PutWithProgress) call MkdirAll for the
// empty mount point. Put and PutWithProgress fail on a missing basePath // per-blob parent directory, which also covers basePath on first use.
// and create the directories below it as needed.
// Get and Stat report a key under a missing basePath as ErrNotFound. // Get and Stat report a key under a missing basePath as ErrNotFound.
// List and ListStream fail on a missing basePath, because listing it as // List and ListStream fail on a missing basePath, because listing it as
// an empty store would make `prune` drop every local snapshot record. // an empty store would make `prune` drop every local snapshot record.
@@ -298,19 +297,7 @@ func (f *FileStorer) writeAtomic(
path := f.fullPath(key) path := f.fullPath(key)
dir := filepath.Dir(path) dir := filepath.Dir(path)
// Checked on every write, so a destination that disappears part-way err := f.fs.MkdirAll(dir, storageDirPerm)
// through a backup fails it rather than being recreated by MkdirAll.
_, err := f.fs.Stat(f.basePath)
if os.IsNotExist(err) {
return fmt.Errorf("destination directory does not exist; "+
"create it, or mount the volume it is on: %w", err)
}
if err != nil {
return fmt.Errorf("checking destination directory: %w", err)
}
err = f.fs.MkdirAll(dir, storageDirPerm)
if err != nil { if err != nil {
return fmt.Errorf("creating directories: %w", err) return fmt.Errorf("creating directories: %w", err)
} }
-42
View File
@@ -4,9 +4,7 @@ import (
"context" "context"
"errors" "errors"
"io/fs" "io/fs"
"os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"sneak.berlin/go/vaultik/internal/storage" "sneak.berlin/go/vaultik/internal/storage"
@@ -62,43 +60,3 @@ func TestFileStorerListMissingDestination(t *testing.T) {
t.Errorf("ListStream error = %v, want a not-exist error", streamErr) t.Errorf("ListStream error = %v, want a not-exist error", streamErr)
} }
} }
// TestFileStorerPutMissingDestination checks that a write fails when the
// destination directory has gone, even after earlier writes to it
// succeeded, and that it does not create the directory again.
func TestFileStorerPutMissingDestination(t *testing.T) {
t.Parallel()
ctx := context.Background()
base := filepath.Join(t.TempDir(), "usbstick")
err := os.Mkdir(base, 0o750)
if err != nil {
t.Fatalf("Mkdir: %v", err)
}
s, err := storage.NewFileStorer(base)
if err != nil {
t.Fatalf("NewFileStorer: %v", err)
}
err = s.Put(ctx, "blobs/aa/bb/first", strings.NewReader("first"))
if err != nil {
t.Fatalf("first Put: %v", err)
}
err = os.Rename(base, base+"-unplugged")
if err != nil {
t.Fatalf("Rename: %v", err)
}
err = s.Put(ctx, "blobs/aa/bb/second", strings.NewReader("second"))
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("Put = %v, want a not-exist error", err)
}
_, err = os.Stat(base)
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("Put created the destination directory again (Stat = %v)", err)
}
}
+1 -2
View File
@@ -41,8 +41,7 @@ type Info struct {
// directory for filesystem). // directory for filesystem).
type Storer interface { type Storer interface {
// Put stores data at the specified key. // Put stores data at the specified key.
// The filesystem backend fails when its destination directory does // Parent directories are created automatically for filesystem backends.
// not exist, and creates the directories below it as needed.
Put(ctx context.Context, key string, data io.Reader) error Put(ctx context.Context, key string, data io.Reader) error
// PutWithProgress stores data with progress reporting. // PutWithProgress stores data with progress reporting.
+12 -26
View File
@@ -17,10 +17,8 @@ const (
// Sentinel errors for storage URL parsing. // Sentinel errors for storage URL parsing.
var ( var (
ErrEmptyStorageURL = errors.New("storage URL is empty") ErrEmptyStorageURL = errors.New("storage URL is empty")
ErrEmptyFilePath = errors.New("file URL path is empty") ErrEmptyFilePath = errors.New("file URL path is empty")
ErrRelativeFilePath = errors.New(
"file URL needs an absolute path, as in file:///path/to/backup")
ErrMissingBucket = errors.New("s3 URL missing bucket name") ErrMissingBucket = errors.New("s3 URL missing bucket name")
ErrMissingRemote = errors.New("rclone URL missing remote name") ErrMissingRemote = errors.New("rclone URL missing remote name")
ErrUnsupportedScheme = errors.New( ErrUnsupportedScheme = errors.New(
@@ -53,8 +51,17 @@ func ParseStorageURL(rawURL string) (*URL, error) {
return nil, ErrEmptyStorageURL return nil, ErrEmptyStorageURL
} }
// Handle file:// URLs
if after, ok := strings.CutPrefix(rawURL, "file://"); ok { if after, ok := strings.CutPrefix(rawURL, "file://"); ok {
return parseFileURL(after) path := after
if path == "" {
return nil, ErrEmptyFilePath
}
return &URL{
Scheme: schemeFile,
Prefix: path,
}, nil
} }
if strings.HasPrefix(rawURL, "s3://") { if strings.HasPrefix(rawURL, "s3://") {
@@ -68,27 +75,6 @@ func ParseStorageURL(rawURL string) (*URL, error) {
return nil, ErrUnsupportedScheme return nil, ErrUnsupportedScheme
} }
// parseFileURL parses the path that follows file://. The path must be
// absolute: a relative one would resolve against the working directory,
// so the destination would move with it. Percent-escapes are decoded, so
// my%20stick names the directory "my stick"; a literal space is kept.
func parseFileURL(rawPath string) (*URL, error) {
if rawPath == "" {
return nil, ErrEmptyFilePath
}
if !strings.HasPrefix(rawPath, "/") {
return nil, ErrRelativeFilePath
}
path, err := url.PathUnescape(rawPath)
if err != nil {
return nil, fmt.Errorf("invalid file URL path: %w", err)
}
return &URL{Scheme: schemeFile, Prefix: path}, nil
}
// parseS3URL parses an s3://bucket/prefix URL. It rejects credentials in // parseS3URL parses an s3://bucket/prefix URL. It rejects credentials in
// the userinfo and any query parameter other than endpoint, region and // the userinfo and any query parameter other than endpoint, region and
// ssl, so a credential-bearing URL is never stored or echoed. // ssl, so a credential-bearing URL is never stored or echoed.
+2 -16
View File
@@ -14,10 +14,7 @@ import (
func TestParseStorageURLValid(t *testing.T) { func TestParseStorageURLValid(t *testing.T) {
t.Parallel() t.Parallel()
const ( const bucket = "mybucket"
bucket = "mybucket"
fileScheme = "file"
)
cases := []struct { cases := []struct {
name string name string
@@ -27,17 +24,7 @@ func TestParseStorageURLValid(t *testing.T) {
{ {
name: "file absolute path", name: "file absolute path",
raw: "file:///var/backups/vaultik", raw: "file:///var/backups/vaultik",
want: &storage.URL{Scheme: fileScheme, Prefix: "/var/backups/vaultik"}, want: &storage.URL{Scheme: "file", Prefix: "/var/backups/vaultik"},
},
{
name: "file percent-escaped path is decoded",
raw: "file:///Volumes/my%20stick/backup",
want: &storage.URL{Scheme: fileScheme, Prefix: "/Volumes/my stick/backup"},
},
{
name: "file path with a literal space",
raw: "file:///Volumes/my stick/backup",
want: &storage.URL{Scheme: fileScheme, Prefix: "/Volumes/my stick/backup"},
}, },
{ {
name: "s3 bucket and prefix, ssl defaults on", name: "s3 bucket and prefix, ssl defaults on",
@@ -103,7 +90,6 @@ func TestParseStorageURLErrors(t *testing.T) {
}{ }{
{"empty url", "", storage.ErrEmptyStorageURL}, {"empty url", "", storage.ErrEmptyStorageURL},
{"file empty path", "file://", storage.ErrEmptyFilePath}, {"file empty path", "file://", storage.ErrEmptyFilePath},
{"file relative path", "file://dest", storage.ErrRelativeFilePath},
{"s3 missing bucket", "s3://", storage.ErrMissingBucket}, {"s3 missing bucket", "s3://", storage.ErrMissingBucket},
{"s3 missing bucket with path", "s3:///justprefix", storage.ErrMissingBucket}, {"s3 missing bucket with path", "s3:///justprefix", storage.ErrMissingBucket},
{"rclone missing remote", "rclone://", storage.ErrMissingRemote}, {"rclone missing remote", "rclone://", storage.ErrMissingRemote},
@@ -137,7 +137,7 @@ func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) {
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src") dataDir := filepath.Join(tempDir, "src")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
changedPath := filepath.Join(dataDir, "appended.bin") changedPath := filepath.Join(dataDir, "appended.bin")
@@ -184,7 +184,7 @@ func TestBackupAfterInterruptedRunRestoresChangedFile(t *testing.T) {
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src") dataDir := filepath.Join(tempDir, "src")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
changedPath := filepath.Join(dataDir, "appended.bin") changedPath := filepath.Join(dataDir, "appended.bin")
+1 -1
View File
@@ -30,7 +30,7 @@ func TestDeepVerifyAcceptsHealthyAndRejectsCorruptBlob(t *testing.T) {
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source") dataDir := filepath.Join(tempDir, "source")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(64 * 1024) chunkSize := int64(64 * 1024)
-74
View File
@@ -1,74 +0,0 @@
package vaultik
import (
"errors"
"fmt"
"strings"
"sneak.berlin/go/vaultik/internal/storage"
)
// destinationMarkerKey is the object every backup writes at the root of
// the backup destination store. While the local index holds a snapshot,
// a destination without it is treated as not present: an unplugged
// volume leaves an empty directory at its mount point, which would
// otherwise read as a store whose snapshots are all gone.
const destinationMarkerKey = "vaultik-destination.txt"
// destinationMarkerText is the whole content of the marker. It names no
// host, path or time.
const destinationMarkerText = "This is a vaultik backup destination store.\n"
// errDestinationNotPresent is returned when the local index holds a
// snapshot and the backup destination store has no marker.
var errDestinationNotPresent = errors.New("backup destination store is not present")
// checkDestinationMarker returns errDestinationNotPresent when the local
// index holds a snapshot and the backup destination store has no marker.
// With no snapshot in the local index, as on a first backup or on
// another machine, there is nothing to check.
func (v *Vaultik) checkDestinationMarker() error {
ids, err := v.localSnapshotIDs()
if err != nil {
return fmt.Errorf("listing local snapshots: %w", err)
}
if len(ids) == 0 {
return nil
}
_, err = v.Storage.Stat(v.ctx, destinationMarkerKey)
if errors.Is(err, storage.ErrNotFound) {
return fmt.Errorf("%w: %s has no %s, but the local index holds "+
"snapshots stored there; mount the volume if it is removable, "+
"or if the destination was emptied on purpose, run "+
"'vaultik database delete' and back up again",
errDestinationNotPresent, v.Storage.Info().Location,
destinationMarkerKey)
}
if err != nil {
return fmt.Errorf("checking for %s: %w", destinationMarkerKey, err)
}
return nil
}
// markDestination checks the backup destination store with
// checkDestinationMarker and then writes the marker, so a first backup
// marks its destination and a later one writes the same text again.
func (v *Vaultik) markDestination() error {
err := v.checkDestinationMarker()
if err != nil {
return err
}
err = v.Storage.Put(v.ctx, destinationMarkerKey,
strings.NewReader(destinationMarkerText))
if err != nil {
return fmt.Errorf("writing %s to the backup destination store: %w",
destinationMarkerKey, err)
}
return nil
}
-184
View File
@@ -1,184 +0,0 @@
package vaultik_test
import (
"bytes"
"context"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/239: a
// file:// destination at a volume's mount point leaves an empty directory
// behind when the volume is unplugged. While the local index holds
// snapshots, a destination without vaultik-destination.txt is not present,
// so no command may write to it or conclude from it that the local
// snapshots are gone.
// markerFile is the name of the file a backup writes at the root of the
// destination.
const markerFile = "vaultik-destination.txt"
// backUpThenUnplugMountPoint backs up to a file:// destination, then
// moves the destination directory away and leaves an empty directory in
// its place, as unplugging a volume mounted there would. It returns the
// path of that empty directory.
func backUpThenUnplugMountPoint(
ctx context.Context, t *testing.T,
) (*vaultik.Vaultik, *database.Repositories, *bytes.Buffer, string) {
t.Helper()
storeDir := filepath.Join(t.TempDir(), "mnt")
v, repos, out := backUpToFileDestination(ctx, t, storeDir)
require.NoError(t, os.Rename(storeDir, storeDir+"-unplugged"))
require.NoError(t, os.Mkdir(storeDir, 0o750))
return v, repos, out, storeDir
}
// assertUntouched checks that the local index still holds its one
// snapshot and that nothing was written to the empty mount point.
func assertUntouched(
ctx context.Context, t *testing.T,
repos *database.Repositories, storeDir string,
) {
t.Helper()
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err)
assert.Len(t, snapshots, 1, "no local snapshot record may be deleted")
entries, err := os.ReadDir(storeDir)
require.NoError(t, err)
assert.Empty(t, entries, "nothing may be written to the empty mount point")
}
// TestFirstBackupWritesMarker checks that a backup writes the marker
// file at the root of the destination.
func TestFirstBackupWritesMarker(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
storeDir := t.TempDir()
backUpToFileDestination(context.Background(), t, storeDir)
assert.FileExists(t, filepath.Join(storeDir, markerFile))
}
// TestBackupRefusesDestinationWithoutMarker checks that snapshot create
// --prune writes nothing and drops no local snapshot record when the
// destination has no marker.
func TestBackupRefusesDestinationWithoutMarker(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
v, repos, _, storeDir := backUpThenUnplugMountPoint(ctx, t)
err := v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Cron: true,
Prune: true,
})
require.ErrorContains(t, err, markerFile)
assertUntouched(ctx, t, repos, storeDir)
}
// TestPruneRefusesDestinationWithoutMarker checks that prune fails and
// deletes no local snapshot record when the destination has no marker.
func TestPruneRefusesDestinationWithoutMarker(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
v, repos, _, storeDir := backUpThenUnplugMountPoint(ctx, t)
err := v.Prune(&vaultik.PruneOptions{Force: true})
require.ErrorContains(t, err, markerFile)
assertUntouched(ctx, t, repos, storeDir)
}
// TestPurgeRefusesDestinationWithoutMarker checks that snapshot purge
// fails and deletes no local snapshot record when the destination has no
// marker.
func TestPurgeRefusesDestinationWithoutMarker(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
v, repos, _, storeDir := backUpThenUnplugMountPoint(ctx, t)
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
KeepLatest: true,
Force: true,
})
require.ErrorContains(t, err, markerFile)
assertUntouched(ctx, t, repos, storeDir)
}
// TestRemoveSnapshotWarnsWhenDestinationHasNoMarker checks that snapshot
// remove warns that the metadata could not be removed from a destination
// without a marker, instead of reporting that it was.
func TestRemoveSnapshotWarnsWhenDestinationHasNoMarker(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
v, repos, out, _ := backUpThenUnplugMountPoint(ctx, t)
result, err := v.RemoveSnapshot(localSnapshotID(ctx, t, repos, "first"),
&vaultik.RemoveOptions{Force: true})
require.NoError(t, err)
assert.False(t, result.RemoteRemoved)
assert.Contains(t, out.String(),
"Could not remove snapshot metadata from remote")
assert.Contains(t, out.String(), markerFile)
assert.NotContains(t, out.String(),
"Removed snapshot metadata from remote storage")
}
// TestListSnapshotsWarnsWhenDestinationHasNoMarker checks that snapshot
// list warns and shows the local index alone when the destination has no
// marker, without reporting the local snapshot as missing from it.
func TestListSnapshotsWarnsWhenDestinationHasNoMarker(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
v, repos, out, _ := backUpThenUnplugMountPoint(ctx, t)
id := localSnapshotID(ctx, t, repos, "first")
require.NoError(t, v.ListSnapshots(false))
assert.Contains(t, out.String(), "Could not list backup destination store")
assert.Contains(t, out.String(), markerFile)
assert.Contains(t, out.String(), id)
assert.NotContains(t, out.String(), "not found in backup destination store")
}
// TestBackupAfterNukeRemoteSucceeds checks that remote nuke leaves the
// destination ready for a new first backup, which writes the marker
// again.
func TestBackupAfterNukeRemoteSucceeds(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
storeDir := t.TempDir()
v, _, _ := backUpToFileDestination(context.Background(), t, storeDir)
require.NoError(t, v.NukeRemote(true))
assert.NoFileExists(t, filepath.Join(storeDir, markerFile))
require.NoError(t, backUp(v, "second"))
assert.FileExists(t, filepath.Join(storeDir, markerFile))
}
+11 -7
View File
@@ -192,7 +192,7 @@ func assertRestoreRejectsDamagedBlob(
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src") dataDir := filepath.Join(tempDir, "src")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
@@ -238,7 +238,7 @@ func TestDeepVerifyCatchesLyingBackend(t *testing.T) {
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src") dataDir := filepath.Join(tempDir, "src")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background() ctx := context.Background()
@@ -290,7 +290,7 @@ func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src") dataDir := filepath.Join(tempDir, "src")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background() ctx := context.Background()
@@ -298,6 +298,10 @@ func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
writeFaultSourceTree(t, fs, dataDir) writeFaultSourceTree(t, fs, dataDir)
// No upload succeeds, so nothing creates the destination directory.
// It must exist for the listing below to show that no blob survived.
require.NoError(t, os.Mkdir(storeDir, 0o750))
inner, err := storage.NewFileStorer(storeDir) inner, err := storage.NewFileStorer(storeDir)
require.NoError(t, err) require.NoError(t, err)
@@ -361,7 +365,7 @@ func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src") dataDir := filepath.Join(tempDir, "src")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
@@ -425,7 +429,7 @@ func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src") dataDir := filepath.Join(tempDir, "src")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background() ctx := context.Background()
@@ -507,7 +511,7 @@ func TestBackupCompletesOnlyAfterMetadataExport(t *testing.T) {
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src") dataDir := filepath.Join(tempDir, "src")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
@@ -682,7 +686,7 @@ func TestRestoreReportsDiskFull(t *testing.T) {
osFS := afero.NewOsFs() osFS := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src") dataDir := filepath.Join(tempDir, "src")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
+2 -2
View File
@@ -771,7 +771,7 @@ func TestEndToEndFileStorage(t *testing.T) {
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source") dataDir := filepath.Join(tempDir, "source")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
@@ -964,7 +964,7 @@ func TestDedupOnlySnapshotRestores(t *testing.T) {
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source") dataDir := filepath.Join(tempDir, "source")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
+16 -35
View File
@@ -24,11 +24,13 @@ import (
// that is not plugged in, cannot be listed. It is not an empty store, so // that is not plugged in, cannot be listed. It is not an empty store, so
// no command may conclude from it that the local snapshots are gone. // no command may conclude from it that the local snapshots are gone.
// newFileDestinationVaultik returns a Vaultik that backs up the // backUpToFileDestination backs up the snapshot named "first" to a
// snapshot named "first" to a file:// destination at storeDir. // file:// destination at storeDir, which need not exist yet. Everything
func newFileDestinationVaultik( // the returned Vaultik prints after the backup goes to the returned
// buffer.
func backUpToFileDestination(
ctx context.Context, t *testing.T, storeDir string, ctx context.Context, t *testing.T, storeDir string,
) (*vaultik.Vaultik, *database.Repositories) { ) (*vaultik.Vaultik, *database.Repositories, *bytes.Buffer) {
t.Helper() t.Helper()
osFs := afero.NewOsFs() osFs := afero.NewOsFs()
@@ -47,22 +49,8 @@ func newFileDestinationVaultik(
repos := database.NewRepositories(db) repos := database.NewRepositories(db)
cfg := changedFileConfig(dataDir, dbPath) cfg := changedFileConfig(dataDir, dbPath)
v := newBackupVaultik(ctx, cfg, store, repos, db, osFs)
return newBackupVaultik(ctx, cfg, store, repos, db, osFs), repos
}
// backUpToFileDestination creates the directory storeDir, as a user must
// before the first backup, and backs up the snapshot named "first" to a
// file:// destination there. Everything the returned Vaultik prints
// after the backup goes to the returned buffer.
func backUpToFileDestination(
ctx context.Context, t *testing.T, storeDir string,
) (*vaultik.Vaultik, *database.Repositories, *bytes.Buffer) {
t.Helper()
require.NoError(t, os.MkdirAll(storeDir, 0o750))
v, repos := newFileDestinationVaultik(ctx, t, storeDir)
require.NoError(t, backUp(v, "first")) require.NoError(t, backUp(v, "first"))
out := &bytes.Buffer{} out := &bytes.Buffer{}
@@ -87,28 +75,21 @@ func backUpThenUnplug(
return v, repos, out return v, repos, out
} }
// TestBackupFailsWhenDestinationDirectoryMissing checks that a backup to // TestFirstBackupCreatesDestinationDirectory checks that a first backup
// a file:// destination whose directory does not exist fails before it // to a destination directory that does not exist yet creates it, and
// records a snapshot, and does not create the directory. // that the destination can be listed afterwards.
func TestBackupFailsWhenDestinationDirectoryMissing(t *testing.T) { func TestFirstBackupCreatesDestinationDirectory(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
ctx := context.Background() ctx := context.Background()
volume := filepath.Join(t.TempDir(), "volume") storeDir := filepath.Join(t.TempDir(), "volume", "backup")
v, repos := newFileDestinationVaultik(ctx, t, filepath.Join(volume, "backup")) v, _, out := backUpToFileDestination(ctx, t, storeDir)
err := backUp(v, "first") require.NoError(t, v.ListSnapshots(false))
require.ErrorIs(t, err, fs.ErrNotExist)
require.ErrorContains(t, err, "create it, or mount the volume it is on")
_, err = os.Stat(volume) assert.NotContains(t, out.String(), "Could not list backup destination store")
require.ErrorIs(t, err, fs.ErrNotExist, assert.NotContains(t, out.String(), "not found in backup destination store")
"the backup must not create the destination directory")
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err)
assert.Empty(t, snapshots)
} }
// TestListSnapshotsWarnsWhenDestinationMissing checks that snapshot list // TestListSnapshotsWarnsWhenDestinationMissing checks that snapshot list
+2 -10
View File
@@ -29,9 +29,8 @@ const blobKeyParts = 4
// NukeRemote deletes every snapshot's metadata and every blob from remote // NukeRemote deletes every snapshot's metadata and every blob from remote
// storage, along with any object an upload cut off part-way left under a // storage, along with any object an upload cut off part-way left under a
// temporary `.partial` name, and the destination marker. After this // temporary `.partial` name. After this returns successfully the bucket
// returns successfully the bucket prefix is empty and the next backup // prefix is empty and the next backup starts from scratch.
// starts from scratch.
// //
// Refuses to run unless force is true. The caller is responsible for // Refuses to run unless force is true. The caller is responsible for
// confirming with the user. // confirming with the user.
@@ -63,13 +62,6 @@ func (v *Vaultik) NukeRemote(force bool) error {
} }
} }
// Last, once the local snapshot records are gone: the next backup
// then has none and writes the marker again.
err = v.Storage.Delete(v.ctx, destinationMarkerKey)
if err != nil {
return fmt.Errorf("deleting %s: %w", destinationMarkerKey, err)
}
v.UI.Completef("Backup destination store is now empty.") v.UI.Completef("Backup destination store is now empty.")
return nil return nil
@@ -34,9 +34,6 @@ func setupConsistencyTest(
repos := database.NewRepositories(db) repos := database.NewRepositories(db)
mockStorage := NewMockStorer() mockStorage := NewMockStorer()
// A backup wrote to this destination, so it carries the marker.
require.NoError(t, mockStorage.Put(ctx, markerFile, strings.NewReader("")))
for _, id := range snapshotIDs { for _, id := range snapshotIDs {
parts := strings.Split(id, "_") parts := strings.Split(id, "_")
startedAt, err := time.Parse(time.RFC3339, parts[len(parts)-1]) startedAt, err := time.Parse(time.RFC3339, parts[len(parts)-1])
-3
View File
@@ -44,9 +44,6 @@ func setupPurgeTest(
repos := database.NewRepositories(db) repos := database.NewRepositories(db)
mockStorage := NewMockStorer() mockStorage := NewMockStorer()
// A backup wrote to this destination, so it carries the marker.
require.NoError(t, mockStorage.Put(ctx, markerFile, strings.NewReader("")))
// Insert each snapshot into the DB and create remote metadata stubs. // Insert each snapshot into the DB and create remote metadata stubs.
// Use timestamps parsed from snapshot IDs for realistic ordering. // Use timestamps parsed from snapshot IDs for realistic ordering.
for _, id := range snapshotIDs { for _, id := range snapshotIDs {
+1 -10
View File
@@ -303,10 +303,6 @@ func TestRemoveSnapshot_DefaultRemovesMetadataNotBlobs(t *testing.T) {
addBlob(t, store, blobUnique) addBlob(t, store, blobUnique)
addBlob(t, store, blobShared) addBlob(t, store, blobShared)
// A backup wrote to this destination, so it carries the marker.
require.NoError(t, store.Put(context.Background(), markerFile,
bytes.NewReader(nil)))
tv := vaultik.NewForTesting(store) tv := vaultik.NewForTesting(store)
tv.Repositories = newLocalIndex(t, "snapshot-001", "snapshot-002") tv.Repositories = newLocalIndex(t, "snapshot-001", "snapshot-002")
@@ -396,12 +392,7 @@ func TestRemoveSnapshot_NoMetadataOnDestination(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
// A backup wrote to this destination, so it carries the marker. tv := vaultik.NewForTesting(newTestStorer())
store := newTestStorer()
require.NoError(t, store.Put(context.Background(), markerFile,
bytes.NewReader(nil)))
tv := vaultik.NewForTesting(store)
tv.Repositories = newLocalIndex(t, "snapshot-001") tv.Repositories = newLocalIndex(t, "snapshot-001")
result, err := tv.RemoveSnapshot("snapshot-001", result, err := tv.RemoveSnapshot("snapshot-001",
+43 -15
View File
@@ -27,6 +27,7 @@ import (
// Sentinel errors for restore failures. // Sentinel errors for restore failures.
var ( var (
errFilesFailedRestore = errors.New("file(s) failed to restore") errFilesFailedRestore = errors.New("file(s) failed to restore")
errPathsNotInSnapshot = errors.New("requested path(s) not in the snapshot")
errFilesFailedVerify = errors.New("files failed verification") errFilesFailedVerify = errors.New("files failed verification")
errDecryptionKeyRequired = errors.New( errDecryptionKeyRequired = errors.New(
"decryption key required for restore\n\n" + "decryption key required for restore\n\n" +
@@ -149,7 +150,7 @@ func (v *Vaultik) Restore(opts *RestoreOptions) error {
repos := database.NewRepositories(tempDB) repos := database.NewRepositories(tempDB)
// Step 2: Get list of files to restore // Step 2: Get list of files to restore
files, err := v.getFilesToRestore(v.ctx, repos, opts.Paths) files, unmatched, err := v.getFilesToRestore(v.ctx, repos, opts.Paths)
if err != nil { if err != nil {
return fmt.Errorf("getting files to restore: %w", err) return fmt.Errorf("getting files to restore: %w", err)
} }
@@ -158,7 +159,7 @@ func (v *Vaultik) Restore(opts *RestoreOptions) error {
log.Warn("No files found to restore") log.Warn("No files found to restore")
v.UI.Warningf("No files found to restore.") v.UI.Warningf("No files found to restore.")
return nil return v.reportUnmatchedPaths(unmatched)
} }
log.Info("Found files to restore", "count", len(files)) log.Info("Found files to restore", "count", len(files))
@@ -198,14 +199,16 @@ func (v *Vaultik) Restore(opts *RestoreOptions) error {
v.UI.Duration(result.Duration), v.UI.Duration(result.Duration),
) )
return v.finishRestore(repos, files, opts, result) return v.finishRestore(repos, files, unmatched, opts, result)
} }
// finishRestore emits the post-restore warnings, runs optional // finishRestore emits the post-restore warnings, runs optional
// verification, and converts any failed-file count into an error. // verification, and converts any failed-file count or requested path
// that matched nothing into an error.
func (v *Vaultik) finishRestore( func (v *Vaultik) finishRestore(
repos *database.Repositories, repos *database.Repositories,
files []*database.File, files []*database.File,
unmatched []string,
opts *RestoreOptions, opts *RestoreOptions,
result *RestoreResult, result *RestoreResult,
) error { ) error {
@@ -223,6 +226,8 @@ func (v *Vaultik) finishRestore(
} }
} }
unmatchedErr := v.reportUnmatchedPaths(unmatched)
// Run verification if requested // Run verification if requested
if opts.Verify { if opts.Verify {
err := v.handleRestoreVerification(repos, files, opts, result) err := v.handleRestoreVerification(repos, files, opts, result)
@@ -235,7 +240,23 @@ func (v *Vaultik) finishRestore(
return fmt.Errorf("%d %w", result.FilesFailed, errFilesFailedRestore) return fmt.Errorf("%d %w", result.FilesFailed, errFilesFailedRestore)
} }
return nil return unmatchedErr
}
// reportUnmatchedPaths warns, by name, about each requested path that
// matched nothing in the snapshot, and returns an error counting them,
// or nil when there are none.
func (v *Vaultik) reportUnmatchedPaths(unmatched []string) error {
if len(unmatched) == 0 {
return nil
}
for _, path := range unmatched {
log.Warn("Requested path matches nothing in the snapshot", "path", path)
v.UI.Warningf("%s matches nothing in the snapshot.", v.UI.Path(path))
}
return fmt.Errorf("%d %w", len(unmatched), errPathsNotInSnapshot)
} }
// restoreIdentities parses the configured age secret key once into every // restoreIdentities parses the configured age secret key once into every
@@ -812,27 +833,34 @@ func (v *Vaultik) materializeSnapshotDB(
return db, tempDir, nil return db, tempDir, nil
} }
// getFilesToRestore returns the list of files to restore based on path filters // getFilesToRestore returns the list of files to restore based on path
// filters, and the filters, as given, that matched no file.
func (v *Vaultik) getFilesToRestore( func (v *Vaultik) getFilesToRestore(
ctx context.Context, repos *database.Repositories, pathFilters []string, ctx context.Context, repos *database.Repositories, pathFilters []string,
) ([]*database.File, error) { ) ([]*database.File, []string, error) {
// If no filters, get all files // If no filters, get all files
if len(pathFilters) == 0 { if len(pathFilters) == 0 {
return repos.Files.ListAll(ctx) files, err := repos.Files.ListAll(ctx)
return files, nil, err
} }
// Get files matching the path filters // Get files matching the path filters
var result []*database.File var (
result []*database.File
unmatched []string
)
seen := make(map[string]bool) seen := make(map[string]bool)
for _, filter := range pathFilters { for _, filter := range pathFilters {
// Normalize the filter path files, err := repos.Files.ListUnderPath(ctx, filepath.Clean(filter))
filter = filepath.Clean(filter)
files, err := repos.Files.ListUnderPath(ctx, filter)
if err != nil { if err != nil {
return nil, fmt.Errorf("listing files under %s: %w", filter, err) return nil, nil, fmt.Errorf("listing files under %s: %w", filter, err)
}
if len(files) == 0 {
unmatched = append(unmatched, filter)
} }
for _, file := range files { for _, file := range files {
@@ -843,7 +871,7 @@ func (v *Vaultik) getFilesToRestore(
} }
} }
return result, nil return result, unmatched, nil
} }
// buildChunkToBlobMap creates a mapping from chunk hash to blob information // buildChunkToBlobMap creates a mapping from chunk hash to blob information
@@ -42,7 +42,7 @@ func TestRestoreOnAnotherMachine(t *testing.T) {
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source") dataDir := filepath.Join(tempDir, "source")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
+1 -1
View File
@@ -65,7 +65,7 @@ func TestRestoreCleansTempDirOnInterrupt(t *testing.T) {
root := t.TempDir() root := t.TempDir()
dataDir := filepath.Join(root, "source") dataDir := filepath.Join(root, "source")
storeDir := t.TempDir() storeDir := filepath.Join(root, "remote")
restoreDir := filepath.Join(root, "restored") restoreDir := filepath.Join(root, "restored")
dbPath := filepath.Join(root, "index.sqlite") dbPath := filepath.Join(root, "index.sqlite")
+1 -1
View File
@@ -197,7 +197,7 @@ func TestRestoreLocalityAndReadAt(t *testing.T) {
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source") dataDir := filepath.Join(tempDir, "source")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
+1 -1
View File
@@ -107,7 +107,7 @@ func backupOneFile(
require.NoError(t, afero.WriteFile(fs, srcPath, content, mode)) require.NoError(t, afero.WriteFile(fs, srcPath, content, mode))
require.NoError(t, fs.Chmod(srcPath, mode)) require.NoError(t, fs.Chmod(srcPath, mode))
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
storer, err := storage.NewFileStorer(storeDir) storer, err := storage.NewFileStorer(storeDir)
+1 -1
View File
@@ -127,7 +127,7 @@ func backupThenDeleteOneBlob(
require.NoError(t, afero.WriteFile(fs, path, files[path], 0o644)) require.NoError(t, afero.WriteFile(fs, path, files[path], 0o644))
} }
storer, err := storage.NewFileStorer(t.TempDir()) storer, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
require.NoError(t, err) require.NoError(t, err)
db, err := database.New(ctx, dbPath) db, err := database.New(ctx, dbPath)
@@ -29,7 +29,7 @@ func TestRestoreAndDeepVerifyRejectSwappedDatabase(t *testing.T) {
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
chunkSize := int64(64 * 1024) chunkSize := int64(64 * 1024)
@@ -100,7 +100,7 @@ func TestDeepVerifyRejectsSwappedDatabaseWithEmptyManifest(t *testing.T) {
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
chunkSize := int64(64 * 1024) chunkSize := int64(64 * 1024)
@@ -186,7 +186,7 @@ func TestRestoreSweeperEvictsBlobs(t *testing.T) {
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source") dataDir := filepath.Join(tempDir, "source")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
@@ -33,7 +33,7 @@ func TestRestoreRejectsTruncatedMetadataDB(t *testing.T) {
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source") dataDir := filepath.Join(tempDir, "source")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
@@ -0,0 +1,142 @@
package vaultik_test
import (
"bytes"
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/309: a path
// given to restore that matches nothing in the snapshot is named in a
// warning and makes the restore fail, after the paths that do match are
// restored.
func TestRestoreNamesUnmatchedPathAndRestoresTheMatchingOne(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background()
cfg := faultTestConfig()
files := writeFaultSourceTree(t, fs, dataDir)
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
id := fullFaultBackup(ctx, t, fs, store, cfg,
database.NewRepositories(db), dataDir, dbPath, "unmatched")
require.NoError(t, db.Close())
matched := filepath.Join(dataDir, "small.txt")
unmatched := filepath.Join(dataDir, "typo.txt")
var out bytes.Buffer
v := newReaderVaultik(ctx, cfg, store, nil, fs)
v.UI = ui.NewWithColor(&out, false)
err = v.Restore(&vaultik.RestoreOptions{
SnapshotID: id,
TargetDir: restoreDir,
Paths: []string{matched, unmatched},
})
require.Error(t, err, "a path that matches nothing must fail the restore")
assert.Contains(t, out.String(), unmatched,
"the path that matches nothing must be named")
assert.NotContains(t, out.String(), matched,
"the path that matches must not be reported")
got, err := afero.ReadFile(fs, filepath.Join(restoreDir, matched))
require.NoError(t, err, "the path that matches must still be restored")
assert.Equal(t, files[matched], got)
}
// When no requested path matches, nothing is restored and each path is
// named. One path is relative. The other goes through the symlink that
// is the snapshot's configured path, while the entries are stored under
// the directory the symlink points to.
func TestRestoreNamesEveryPathWhenNothingMatches(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
const snapshotName = "linked"
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
linkDir := filepath.Join(tempDir, "srclink")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background()
writeFaultSourceTree(t, fs, dataDir)
require.NoError(t, os.Symlink(dataDir, linkDir))
cfg := faultTestConfig()
cfg.IndexPath = dbPath
cfg.ChunkSize = config.Size(faultChunkSize)
cfg.Snapshots = map[string]config.SnapshotConfig{
snapshotName: {Paths: []string{linkDir}},
}
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
require.NoError(t, backUp(v, snapshotName))
id := localSnapshotID(ctx, t, repos, snapshotName)
require.NoError(t, db.Close())
throughLink := filepath.Join(linkDir, "sub")
relative := strings.TrimPrefix(filepath.Join(dataDir, "small.txt"), "/")
var out bytes.Buffer
reader := newReaderVaultik(ctx, cfg, store, nil, fs)
reader.UI = ui.NewWithColor(&out, false)
err = reader.Restore(&vaultik.RestoreOptions{
SnapshotID: id,
TargetDir: restoreDir,
Paths: []string{throughLink, relative},
})
require.Error(t, err, "a restore that matches nothing must fail")
assert.Contains(t, out.String(), throughLink,
"the path through the symlink must be named")
assert.Contains(t, out.String(), relative,
"the relative path must be named")
_, err = fs.Stat(restoreDir)
assert.True(t, os.IsNotExist(err), "nothing must be restored")
}
+1 -1
View File
@@ -25,7 +25,7 @@ func TestBackupOfSameSecondRewriteRestoresNewContent(t *testing.T) {
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src") dataDir := filepath.Join(tempDir, "src")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
rewrittenPath := filepath.Join(dataDir, "small.txt") rewrittenPath := filepath.Join(dataDir, "small.txt")
+2 -2
View File
@@ -31,7 +31,7 @@ func TestShallowVerifyDetectsWrongBlobSize(t *testing.T) {
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source") dataDir := filepath.Join(tempDir, "source")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(32 * 1024) chunkSize := int64(32 * 1024)
@@ -105,7 +105,7 @@ func TestShallowVerifyDetectsMissingDatabase(t *testing.T) {
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source") dataDir := filepath.Join(tempDir, "source")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(32 * 1024) chunkSize := int64(32 * 1024)
+26 -69
View File
@@ -92,13 +92,6 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
return err return err
} }
// Before PruneDatabase: it drops incomplete snapshots, which could
// leave the local index without one and so skip the marker check.
err = v.markDestination()
if err != nil {
return err
}
// Clean up incomplete snapshots FIRST, before any scanning. // Clean up incomplete snapshots FIRST, before any scanning.
// This is critical for data safety; PruneDatabase below does it. // This is critical for data safety; PruneDatabase below does it.
hostname := v.Config.Hostname hostname := v.Config.Hostname
@@ -121,9 +114,21 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
return fmt.Errorf("prune database: %w", err) return fmt.Errorf("prune database: %w", err)
} }
snapshotNames, err := v.snapshotNamesToCreate(opts.Snapshots) // Determine which snapshots to process
if err != nil { snapshotNames := opts.Snapshots
return err if len(snapshotNames) == 0 {
snapshotNames = v.Config.SnapshotNames()
} else {
// Validate requested snapshot names exist
for _, name := range snapshotNames {
if _, ok := v.Config.Snapshots[name]; !ok {
return fmt.Errorf("%w: %q", errSnapshotNotInConfig, name)
}
}
}
if len(snapshotNames) == 0 {
return errNoSnapshotsInConfig
} }
// Process each named snapshot // Process each named snapshot
@@ -163,28 +168,6 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
return nil return nil
} }
// snapshotNamesToCreate returns the requested snapshot names, each checked
// against the config, or every configured name when none was requested.
func (v *Vaultik) snapshotNamesToCreate(requested []string) ([]string, error) {
snapshotNames := requested
if len(snapshotNames) == 0 {
snapshotNames = v.Config.SnapshotNames()
} else {
// Validate requested snapshot names exist
for _, name := range snapshotNames {
if _, ok := v.Config.Snapshots[name]; !ok {
return nil, fmt.Errorf("%w: %q", errSnapshotNotInConfig, name)
}
}
}
if len(snapshotNames) == 0 {
return nil, errNoSnapshotsInConfig
}
return snapshotNames, nil
}
// runPostBackupPrune drops older snapshots of the given names and removes // runPostBackupPrune drops older snapshots of the given names and removes
// orphan blobs from remote storage. If keepNewerThan is set (e.g. "4w"), // orphan blobs from remote storage. If keepNewerThan is set (e.g. "4w"),
// snapshots newer than that duration are kept. Otherwise only the latest // snapshots newer than that duration are kept. Otherwise only the latest
@@ -1206,18 +1189,10 @@ func (v *Vaultik) RemoveSnapshot(
return v.removeSnapshotDryRun(result, snapshotID, opts) return v.removeSnapshotDryRun(result, snapshotID, opts)
} }
if !v.confirmRemoveSnapshot(snapshotID, opts) { if !opts.Force && !opts.JSON && !v.confirmRemoveSnapshot(snapshotID, opts) {
return result, nil return result, nil
} }
// Checked before the local record is deleted: it may be the last
// snapshot in the local index, and the check applies only while the
// local index holds one.
var markerErr error
if !opts.LocalOnly {
markerErr = v.checkDestinationMarker()
}
if inLocalIndex { if inLocalIndex {
log.Info("Removing snapshot from local database", "snapshot_id", snapshotID) log.Info("Removing snapshot from local database", "snapshot_id", snapshotID)
@@ -1228,7 +1203,7 @@ func (v *Vaultik) RemoveSnapshot(
} }
if !opts.LocalOnly { if !opts.LocalOnly {
result.RemoteRemoved = v.removeSnapshotRemote(snapshotID, markerErr, opts) result.RemoteRemoved = v.removeSnapshotRemote(snapshotID, opts)
} }
if v.SnapshotManager != nil { if v.SnapshotManager != nil {
@@ -1320,13 +1295,9 @@ func (v *Vaultik) removeSnapshotDryRun(
} }
// confirmRemoveSnapshot prompts for confirmation before removing a // confirmRemoveSnapshot prompts for confirmation before removing a
// snapshot, unless --force or --json is set. Returns false (treating // snapshot. Returns false (treating EOF/read errors as "no") unless the
// EOF/read errors as "no") unless the user answers "y". // user answers "y".
func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions) bool { func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions) bool {
if opts.Force || opts.JSON {
return true
}
if opts.LocalOnly { if opts.LocalOnly {
v.stdoutf("Remove snapshot '%s' from local database "+ v.stdoutf("Remove snapshot '%s' from local database "+
"(remote untouched)? [y/N] ", snapshotID) "(remote untouched)? [y/N] ", snapshotID)
@@ -1358,24 +1329,15 @@ func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions)
// removal has already happened, so the user is told the remote half // removal has already happened, so the user is told the remote half
// didn't finish and to run `vaultik snapshot remove` for the snapshot // didn't finish and to run `vaultik snapshot remove` for the snapshot
// again once the destination store is reachable (`vaultik prune` never // again once the destination store is reachable (`vaultik prune` never
// removes snapshot metadata). markerErr is the result of // removes snapshot metadata). Returns true when the remote removal
// checkDestinationMarker; when it is set, nothing is deleted and it is // succeeded and deleted something.
// the failure reported. Returns true when the remote removal succeeded func (v *Vaultik) removeSnapshotRemote(snapshotID string, opts *RemoveOptions) bool {
// and deleted something.
func (v *Vaultik) removeSnapshotRemote(
snapshotID string, markerErr error, opts *RemoveOptions,
) bool {
log.Info("Removing snapshot metadata from remote storage", log.Info("Removing snapshot metadata from remote storage",
"snapshot_id", snapshotID) "snapshot_id", snapshotID)
var deleted int remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
err := markerErr
if err == nil {
deleted, err = v.deleteRemoteSnapshotByKey(
snapshot.RemoteSnapshotKey(snapshotID))
}
deleted, err := v.deleteRemoteSnapshotByKey(remoteKey)
if err != nil { if err != nil {
log.Warn("Could not remove snapshot metadata from remote storage; "+ log.Warn("Could not remove snapshot metadata from remote storage; "+
"run '"+snapshotRemoveCommandHint+"' with the snapshot's ID "+ "run '"+snapshotRemoveCommandHint+"' with the snapshot's ID "+
@@ -1479,8 +1441,8 @@ func (v *Vaultik) localSnapshotIDs() ([]string, error) {
// listAllRemoteSnapshotKeys collects the hashed remote keys // listAllRemoteSnapshotKeys collects the hashed remote keys
// (subdirectories under metadata/) currently present in the // (subdirectories under metadata/) currently present in the
// destination store. Returns (nil, err) when the store cannot be // destination store. Returns (nil, err) when the store cannot be
// listed, or is not present by checkDestinationMarker; callers must // listed; callers must treat that as "no remote info available," not
// treat that as "no remote info available," not fatal. // fatal.
func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) { func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) {
log.Info("Listing all remote snapshots") log.Info("Listing all remote snapshots")
@@ -1529,11 +1491,6 @@ func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) {
} }
} }
err := v.checkDestinationMarker()
if err != nil {
return nil, err
}
return keys, nil return keys, nil
} }
@@ -87,7 +87,7 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
"src": {Paths: []string{srcDir}}, "src": {Paths: []string{srcDir}},
} }
store, err := storage.NewFileStorer(t.TempDir()) store, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
require.NoError(t, err) require.NoError(t, err)
db, err := database.New(ctx, cfg.IndexPath) db, err := database.New(ctx, cfg.IndexPath)
-3
View File
@@ -122,9 +122,6 @@ func newListEnv(t *testing.T) *listEnv {
store := newObservingStorer() store := newObservingStorer()
stdout := &bytes.Buffer{} stdout := &bytes.Buffer{}
// A backup wrote to this destination, so it carries the marker.
require.NoError(t, store.Put(ctx, markerFile, strings.NewReader("")))
v := &vaultik.Vaultik{ v := &vaultik.Vaultik{
Config: &config.Config{ Config: &config.Config{
AgeSecretKey: "", AgeSecretKey: "",
@@ -45,7 +45,7 @@ func TestSkipErrorsBackupStopsOnIndexErrorRecordingDirectory(t *testing.T) {
"tree": {Paths: []string{srcDir}}, "tree": {Paths: []string{srcDir}},
} }
store, err := storage.NewFileStorer(t.TempDir()) store, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
require.NoError(t, err) require.NoError(t, err)
db, err := database.New(ctx, cfg.IndexPath) db, err := database.New(ctx, cfg.IndexPath)
+1 -1
View File
@@ -84,7 +84,7 @@ func newSummaryEnv(t *testing.T) *summaryEnv {
"split": {Paths: []string{dirOne, dirTwo}}, "split": {Paths: []string{dirOne, dirTwo}},
} }
inner, err := storage.NewFileStorer(t.TempDir()) inner, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
require.NoError(t, err) require.NoError(t, err)
store := faultstore.New(inner) store := faultstore.New(inner)
+1 -1
View File
@@ -28,7 +28,7 @@ func TestBackupWithoutCronOfTwoPathSnapshotRestoresBothPaths(t *testing.T) {
tempDir := t.TempDir() tempDir := t.TempDir()
firstDir := filepath.Join(tempDir, "first") firstDir := filepath.Join(tempDir, "first")
secondDir := filepath.Join(tempDir, "second") secondDir := filepath.Join(tempDir, "second")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored") restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
+1 -1
View File
@@ -28,7 +28,7 @@ func TestVerify_QuietSuppressesReport(t *testing.T) {
tempDir := t.TempDir() tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source") dataDir := filepath.Join(tempDir, "source")
storeDir := t.TempDir() storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite") dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(32 * 1024) chunkSize := int64(32 * 1024)