Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
b3cf02c817 |
+5
-6
@@ -385,12 +385,11 @@ bucket/
|
||||
A destination without `vaultik-destination.txt` is treated as not present
|
||||
while the local index holds snapshots stored there (`checkDestinationMarker`
|
||||
in `internal/vaultik/destination_marker.go`): `snapshot create` refuses to
|
||||
write to it, `prune`, `snapshot purge` and `remote nuke` fail, and
|
||||
`snapshot list` and `snapshot remove` behave as for a destination that cannot
|
||||
be listed. This keeps those commands from taking an unplugged volume's empty
|
||||
mount point for a destination whose snapshots are all gone. The `file://`
|
||||
backend never creates its destination directory, so a write to a missing one
|
||||
fails.
|
||||
write to it, `prune` and `snapshot purge` fail, and `snapshot list` and
|
||||
`snapshot remove` behave as for a destination that cannot be listed. This
|
||||
keeps those commands from taking an unplugged volume's empty mount point for
|
||||
a destination whose snapshots are all gone. The `file://` backend never
|
||||
creates its destination directory, so a write to a missing one fails.
|
||||
|
||||
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
|
||||
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
|
||||
|
||||
@@ -182,7 +182,7 @@ vaultik version
|
||||
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
|
||||
* `--debug`: Enable debug output (on stderr — see below)
|
||||
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
|
||||
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run. A run that skipped anything still finishes but exits non-zero: `snapshot create` keeps the snapshot and counts each file or directory it skipped as failed. A file removed during the backup is also counted as failed but does not change the exit status.
|
||||
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run.
|
||||
|
||||
### locking
|
||||
|
||||
@@ -297,11 +297,7 @@ value where a list belongs or a word where a number belongs), is refused.
|
||||
(for crontab)
|
||||
* `--prune`: After backup, drop older snapshots of each backed-up name and
|
||||
remove orphaned blobs from remote storage. By default keeps only the latest
|
||||
snapshot per name; use `--keep-newer-than` for a rolling window. When
|
||||
`--skip-errors` skipped a file or directory it could not read or list, the
|
||||
prune does not run and every older snapshot is kept, since one of them may
|
||||
be the only one that holds what was skipped; the command says so and exits
|
||||
non-zero. A file removed during the backup does not stop the prune.
|
||||
snapshot per name; use `--keep-newer-than` for a rolling window.
|
||||
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
||||
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
||||
Without `--prune`, or with a value that is not a valid duration, the
|
||||
@@ -458,14 +454,9 @@ since `prune` never deletes it.
|
||||
the backup destination store, whatever its name, and then
|
||||
`vaultik-destination.txt`, leaving the bucket prefix empty. Destructive
|
||||
and irreversible. This is the single supported way to wipe the entire
|
||||
destination store. A snapshot is removed from the local index only once
|
||||
its metadata is deleted from the destination store. If the destination
|
||||
store cannot be listed, or the local index holds snapshots and the
|
||||
destination has no `vaultik-destination.txt` (an unplugged volume's
|
||||
empty mount point), the command fails before it removes anything. If a
|
||||
listing afterwards still finds anything under `metadata/` or `blobs/`,
|
||||
the command names each object left and exits non-zero; running it again
|
||||
deletes what is left.
|
||||
destination store. If a listing afterwards still finds anything under
|
||||
`metadata/` or `blobs/`, the command names each object left and exits
|
||||
non-zero; running it again deletes what is left.
|
||||
* `--force`: Required to confirm destruction.
|
||||
|
||||
**`database delete`**: Delete the local SQLite state database file
|
||||
@@ -538,11 +529,10 @@ For an implementation-level view of the internals — the data model, the
|
||||
* `vaultik-destination.txt` holds one fixed line of text, with no host
|
||||
name, path or time. A destination without it is treated as not
|
||||
present while the local index holds snapshots stored there:
|
||||
`snapshot create` refuses to write to it, `prune`, `snapshot purge`
|
||||
and `remote nuke` fail, and `snapshot list` and `snapshot remove`
|
||||
behave as for a destination that cannot be listed. This keeps an
|
||||
unplugged volume's empty mount point from being taken for a
|
||||
destination whose snapshots are all gone. Mount
|
||||
`snapshot create` refuses to write to it, `prune` and `snapshot purge`
|
||||
fail, and `snapshot list` and `snapshot remove` behave as for a
|
||||
destination that cannot be listed. This keeps an unplugged volume's empty mount point
|
||||
from being taken for a destination whose snapshots are all gone. Mount
|
||||
the volume if it is removable; if the destination was emptied on
|
||||
purpose, run `vaultik database delete` and back up again.
|
||||
* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
|
||||
|
||||
@@ -15,229 +15,13 @@ pre-1.0
|
||||
# Next Step
|
||||
|
||||
The 1.0 work is complete on `next`: the scope settled on
|
||||
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) has landed,
|
||||
and so has every fix from the defect hunt before the release candidate
|
||||
([issue #288](https://git.eeqj.de/sneak/vaultik/issues/288)). What is
|
||||
left is merging `next` to `main` and tagging the release, both the
|
||||
owner's; a `v*` tag runs the release build in
|
||||
`.gitea/workflows/release.yml`.
|
||||
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) was the work
|
||||
already planned for 1.0, and all of it has landed. The mechanism to cut
|
||||
the tag exists and is exercised; what is left is merging `next` to
|
||||
`main` and tagging, both the owner's.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
||||
that skipped files it could not read
|
||||
([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a
|
||||
prune could delete the only snapshot still holding those files.
|
||||
|
||||
- 2026-10-14: Made `snapshot create --skip-errors` exit non-zero,
|
||||
instead of 0, when it skips anything it cannot read
|
||||
([issue #310](https://git.eeqj.de/sneak/vaultik/issues/310)).
|
||||
|
||||
- 2026-10-14: Kept the local index when `remote nuke` cannot list or
|
||||
find the destination store
|
||||
([issue #292](https://git.eeqj.de/sneak/vaultik/issues/292)). Nuke
|
||||
had cleared the local snapshot records anyway; it now fails before
|
||||
removing anything.
|
||||
|
||||
- 2026-10-14: Stopped counting directories and unchanged symlinks as
|
||||
files to back up
|
||||
([issue #305](https://git.eeqj.de/sneak/vaultik/issues/305)).
|
||||
|
||||
- 2026-10-14: Refused a backup destination that an unplugged volume
|
||||
left empty, as the owner ruled
|
||||
([issue #239](https://git.eeqj.de/sneak/vaultik/issues/239)). Every
|
||||
backup now writes `vaultik-destination.txt` at the destination root,
|
||||
and while the local index holds snapshots stored there, a destination
|
||||
without it is treated as not present.
|
||||
|
||||
- 2026-10-14: Warned about restored ownership only when an entry's
|
||||
owner or group differs from the recorded one
|
||||
([issue #316](https://git.eeqj.de/sneak/vaultik/issues/316)); a
|
||||
restore by a non-root user had warned even when none did.
|
||||
|
||||
- 2026-10-14: Made `snapshot list` report a changed destination instead
|
||||
of advising `prune`, which refuses in that case
|
||||
([issue #317](https://git.eeqj.de/sneak/vaultik/issues/317)).
|
||||
|
||||
- 2026-10-14: Counted a file as deleted by a backup only if the
|
||||
previous snapshot of the same name listed it
|
||||
([issue #304](https://git.eeqj.de/sneak/vaultik/issues/304)); files
|
||||
that snapshot did not list had been counted too.
|
||||
|
||||
- 2026-10-14: Named each prefix once in restore, verify and purge
|
||||
errors, some of which repeated it
|
||||
([issue #327](https://git.eeqj.de/sneak/vaultik/issues/327)). A deep
|
||||
verify log line that named S3 whatever the destination now names
|
||||
storage.
|
||||
|
||||
- 2026-10-14: Refused credentials in a storage URL's endpoint or rclone
|
||||
remote name, which were accepted and could then be printed
|
||||
([issue #322](https://git.eeqj.de/sneak/vaultik/issues/322)).
|
||||
|
||||
- 2026-10-14: Ran `$EDITOR` through the shell in `config edit`, so a
|
||||
value with arguments, such as `code --wait`, no longer fails
|
||||
([issue #315](https://git.eeqj.de/sneak/vaultik/issues/315)).
|
||||
|
||||
- 2026-10-14: Made `snapshot remove` fail, instead of exiting 0, when
|
||||
it cannot delete the metadata of a snapshot that is only on the
|
||||
destination store
|
||||
([issue #340](https://git.eeqj.de/sneak/vaultik/issues/340)). The
|
||||
README and the help now say when it prints the `vaultik prune`
|
||||
follow-up.
|
||||
|
||||
- 2026-10-14: Labelled the totals in the local index cleanup before
|
||||
each backup as totals
|
||||
([issue #306](https://git.eeqj.de/sneak/vaultik/issues/306)). Each
|
||||
was shown as `(N remain)`, which read as rows left to clean up.
|
||||
|
||||
- 2026-10-13: Freed a cached blob once no file still to restore needs
|
||||
it ([issue #324](https://git.eeqj.de/sneak/vaultik/issues/324)); one
|
||||
also used by a file outside the selected paths had stayed in the
|
||||
temporary cache until the restore ended.
|
||||
|
||||
- 2026-10-13: Kept the whole startup error when it contains `): `
|
||||
([issue #320](https://git.eeqj.de/sneak/vaultik/issues/320)). Such
|
||||
an error was cut at its last `): `, so the `blob_size_limit` one lost
|
||||
the setting's name.
|
||||
|
||||
- 2026-10-13: Silenced the status lines of `prune` and snapshot removal
|
||||
under `--cron` and `--quiet`
|
||||
([issue #294](https://git.eeqj.de/sneak/vaultik/issues/294)).
|
||||
|
||||
- 2026-10-13: Read the rclone config file that `RCLONE_CONFIG` names
|
||||
([issue #321](https://git.eeqj.de/sneak/vaultik/issues/321)); a
|
||||
remote defined only there had been reported as not found.
|
||||
|
||||
- 2026-10-13: Made `/**/` in an exclude pattern match zero directories,
|
||||
as in gitignore
|
||||
([issue #338](https://git.eeqj.de/sneak/vaultik/issues/338));
|
||||
`a/**/*.log` had missed `a/x.log`.
|
||||
|
||||
- 2026-10-13: Made the `chunk_size` and `blob_size_limit` defaults and
|
||||
the `chunk_size` minimum 1000-based, like the size parser and the
|
||||
docs ([issue #313](https://git.eeqj.de/sneak/vaultik/issues/313)).
|
||||
|
||||
- 2026-10-13: Kept a snapshot's remote metadata when `snapshot purge`
|
||||
cannot remove it from the local index
|
||||
([issue #337](https://git.eeqj.de/sneak/vaultik/issues/337)); the
|
||||
purge had deleted the metadata anyway and exited 0.
|
||||
|
||||
- 2026-10-13: Made `snapshot restore` fail for a requested path that
|
||||
matches nothing, which it had ignored
|
||||
([issue #309](https://git.eeqj.de/sneak/vaultik/issues/309)).
|
||||
|
||||
- 2026-10-13: Made `remote nuke` delete every object under `metadata/`
|
||||
and `blobs/`
|
||||
([issue #291](https://git.eeqj.de/sneak/vaultik/issues/291)); it had
|
||||
left some, such as a stray `.DS_Store`, while reporting the store
|
||||
empty.
|
||||
|
||||
- 2026-10-13: Made `config set` create a list for an index under a
|
||||
missing key
|
||||
([issue #308](https://git.eeqj.de/sneak/vaultik/issues/308)); it had
|
||||
written a map, after which no command could load the config.
|
||||
|
||||
- 2026-10-13: Listed a `file://` destination through a symlinked
|
||||
`metadata/` or `blobs/`, which had listed as empty
|
||||
([issue #296](https://git.eeqj.de/sneak/vaultik/issues/296)). A link
|
||||
to a missing target now fails the listing
|
||||
([issue #332](https://git.eeqj.de/sneak/vaultik/issues/332)).
|
||||
|
||||
- 2026-10-13: Made `snapshot verify --deep --json` report its blob
|
||||
counts ([issue #314](https://git.eeqj.de/sneak/vaultik/issues/314));
|
||||
a failed run had reported 0 missing and 0 verified.
|
||||
|
||||
- 2026-10-13: Limited and retried the module download in the Docker
|
||||
build ([issue #351](https://git.eeqj.de/sneak/vaultik/issues/351)),
|
||||
which had hung for as long as a stalled connection to the module
|
||||
proxy lasted.
|
||||
|
||||
- 2026-10-13: Recorded each snapshot's uncompressed size when its
|
||||
backup runs
|
||||
([issue #360](https://git.eeqj.de/sneak/vaultik/issues/360)). A later
|
||||
backup could change an older snapshot's UNCOMPRESSED SIZE in
|
||||
`snapshot list`.
|
||||
|
||||
- 2026-10-13: Stopped `snapshot restore --skip-errors --verify`
|
||||
reporting a file that failed to restore again as a checksum failure
|
||||
([issue #323](https://git.eeqj.de/sneak/vaultik/issues/323)).
|
||||
|
||||
- 2026-10-13: Recorded the stored size of a blob a backup finds already
|
||||
stored, not the size of its local copy, which can differ, so
|
||||
`snapshot verify` could fail on a good snapshot
|
||||
([issue #300](https://git.eeqj.de/sneak/vaultik/issues/300)).
|
||||
|
||||
- 2026-10-13: Stopped the local index and the blob upload step ending
|
||||
the process on a failed close of query results or a failed temp file
|
||||
cleanup ([issue #345](https://git.eeqj.de/sneak/vaultik/issues/345));
|
||||
both are now returned as errors.
|
||||
|
||||
- 2026-10-13: Closed the snapshot database's storage reader too when
|
||||
restore or deep verify is interrupted
|
||||
([issue #356](https://git.eeqj.de/sneak/vaultik/issues/356)).
|
||||
|
||||
- 2026-10-13: Made `prune` and `remote nuke` fail when a delete on the
|
||||
backup destination store fails
|
||||
([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)).
|
||||
|
||||
- 2026-10-13: Made an S3 listing that the server cuts short an error
|
||||
([issue #297](https://git.eeqj.de/sneak/vaultik/issues/297)); `prune`
|
||||
had taken it as complete and could delete blobs still in use.
|
||||
|
||||
- 2026-10-13: Made shallow `snapshot verify` report a blob it could not
|
||||
check apart from a missing one
|
||||
([issue #318](https://git.eeqj.de/sneak/vaultik/issues/318)); any
|
||||
error checking a blob had counted as missing, its cause dropped.
|
||||
|
||||
- 2026-10-13: Restored the other files when a file's chunk is in no
|
||||
blob under `--skip-errors`
|
||||
([issue #326](https://git.eeqj.de/sneak/vaultik/issues/326)); that
|
||||
file had stopped the restore before anything was restored.
|
||||
|
||||
- 2026-10-13: Recorded each snapshot's new chunk size when its backup
|
||||
runs ([issue #311](https://git.eeqj.de/sneak/vaultik/issues/311)).
|
||||
`snapshot list` could show a NEW CHUNK SIZE of 0 B for a snapshot
|
||||
whose change was a modified file.
|
||||
|
||||
- 2026-10-13: Made a backup interrupted during a local index write exit
|
||||
as interrupted
|
||||
([issue #301](https://git.eeqj.de/sneak/vaultik/issues/301)); it could
|
||||
end with `FATAL: failed to rollback transaction` and exit 1 instead of
|
||||
130.
|
||||
|
||||
- 2026-10-13: Printed `snapshot list` and `snapshot purge` sizes like
|
||||
every other command
|
||||
([issue #312](https://git.eeqj.de/sneak/vaultik/issues/312)). They,
|
||||
and the `bytes` value in terminal log lines, were 1024-based but
|
||||
labelled KB or MB, and now use 1000-based units.
|
||||
|
||||
- 2026-10-13: Closed each blob's storage reader when restore or deep
|
||||
verify is interrupted, which ends a stalled read unless it is stuck in
|
||||
the kernel ([issue #325](https://git.eeqj.de/sneak/vaultik/issues/325)).
|
||||
|
||||
- 2026-10-13: Made a mistyped subcommand under `snapshot`, `config`,
|
||||
`remote`, `database` or `completion` an error
|
||||
([issue #319](https://git.eeqj.de/sneak/vaultik/issues/319)); each had
|
||||
printed its help and exited 0.
|
||||
|
||||
- 2026-10-13: Made an exclude pattern starting with `**/` match
|
||||
directly under the snapshot path too, as in gitignore
|
||||
([issue #303](https://git.eeqj.de/sneak/vaultik/issues/303)).
|
||||
|
||||
- 2026-10-13: Refused a relative `file://` storage URL and decoded its
|
||||
percent-escapes
|
||||
([issue #299](https://git.eeqj.de/sneak/vaultik/issues/299)); a
|
||||
relative path had moved with the working directory.
|
||||
|
||||
- 2026-10-13: Removed the blob temp file when a backup stops part-way
|
||||
([issue #302](https://git.eeqj.de/sneak/vaultik/issues/302)); the
|
||||
half-written blob had stayed in `$TMPDIR`.
|
||||
|
||||
- 2026-10-13: Checked `--keep-newer-than` before a backup starts
|
||||
([issue #307](https://git.eeqj.de/sneak/vaultik/issues/307)); an
|
||||
invalid value had been read only by the prune after the backup, so
|
||||
retention never ran.
|
||||
|
||||
- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that
|
||||
is not a blob as an orphaned blob
|
||||
([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray
|
||||
|
||||
@@ -27,7 +27,7 @@ The metadata subdirectory is named with the **remote key**, a one-way hash of th
|
||||
|
||||
## Marker File (`vaultik-destination.txt`)
|
||||
|
||||
Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. A destination without it is treated as not present while the local index holds snapshots stored there: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
|
||||
Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. A destination without it is treated as not present while the local index holds snapshots stored there: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove` and `snapshot list` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
|
||||
|
||||
## Blobs Directory (`blobs/`)
|
||||
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
|
||||
// under --skip-errors, a source directory that does not exist, which the
|
||||
// walk skips. The command exits non-zero and reports the backup as
|
||||
// completed, not as a failed snapshot creation. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
//
|
||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
|
||||
// writeBackupConfig would create the source directory.
|
||||
configPath := writeHermeticPruneConfig(t, false)
|
||||
|
||||
code, stdout, stderr := runEntry(t,
|
||||
flagConfig, configPath, "--skip-errors", cmdSnapshot, cmdCreate)
|
||||
|
||||
assert.Equal(t, 1, code)
|
||||
assert.Contains(t, stdout,
|
||||
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
|
||||
assert.NotContains(t, stdout, "Finished successfully.")
|
||||
assert.Contains(t, stderr,
|
||||
"Backup completed, but 1 file(s) could not be read")
|
||||
assert.NotContains(t, stderr, "Snapshot creation failed")
|
||||
}
|
||||
@@ -115,13 +115,6 @@ on Linux), then /etc/vaultik/config.yml.`,
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, vaultik.ErrFilesUnreadable) {
|
||||
log.Error("Backup completed with unreadable files", "error", err)
|
||||
ReportErrorf("Backup completed, but %v", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
log.Error("Snapshot creation failed", "error", err)
|
||||
ReportErrorf("Snapshot creation failed: %v", err)
|
||||
})
|
||||
@@ -132,8 +125,7 @@ on Linux), then /etc/vaultik/config.yml.`,
|
||||
"Run in cron mode (silent unless warning or error)")
|
||||
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
|
||||
"After backup, drop older snapshots of the same name and remove "+
|
||||
"orphaned blobs (not after a backup that skipped unreadable "+
|
||||
"files)")
|
||||
"orphaned blobs")
|
||||
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
||||
"With --prune: keep snapshots newer than this duration "+
|
||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
||||
|
||||
@@ -141,12 +141,10 @@ type ScannerConfig struct {
|
||||
// ScanResult contains the results of a scan operation. Files and bytes
|
||||
// are counted per file: BytesScanned is the size of the new and changed
|
||||
// files, BytesSkipped that of the unchanged ones. FilesFailed counts the
|
||||
// new and changed files that phase 2 could not store, each entry removed
|
||||
// after the walk listed its directory and, under --skip-errors, each
|
||||
// entry the walk could not read, such as a directory it could not list;
|
||||
// FilesScanned includes them and BytesScanned does not. ChunksCreated
|
||||
// counts the chunks that were not already stored, and NewChunkSize is
|
||||
// their total size.
|
||||
// new and changed files that phase 2 could not store; FilesScanned
|
||||
// includes them and BytesScanned does not. ChunksCreated counts the
|
||||
// chunks that were not already stored, and NewChunkSize is their total
|
||||
// size.
|
||||
type ScanResult struct {
|
||||
FilesScanned int
|
||||
FilesSkipped int
|
||||
@@ -163,11 +161,6 @@ type ScanResult struct {
|
||||
UploadDuration time.Duration
|
||||
StartTime time.Time
|
||||
EndTime time.Time
|
||||
|
||||
// FilesUnreadable is the part of FilesFailed that --skip-errors
|
||||
// skipped because it could not be read or listed; the rest are files
|
||||
// removed after the walk listed them.
|
||||
FilesUnreadable int
|
||||
}
|
||||
|
||||
// NewScanner creates a new scanner instance
|
||||
@@ -897,28 +890,8 @@ func (s *Scanner) scanPhase(
|
||||
log.Debug("Starting directory walk", "path", path)
|
||||
|
||||
walkFn := func(filePath string, info os.FileInfo, err error) error {
|
||||
// Check exclude patterns first, so an excluded entry that is gone
|
||||
// or cannot be looked up is neither counted nor reported. For
|
||||
// directories, skip the entire subtree. info is nil when the
|
||||
// walk could not look the entry up.
|
||||
if s.shouldExclude(filePath, path) {
|
||||
if info != nil && info.IsDir() {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// The source path is in no directory listing, so its absence is
|
||||
// an error rather than a removal.
|
||||
if errors.Is(err, os.ErrNotExist) && filePath != path {
|
||||
countRemovedDuringWalk(filePath, existingFiles, result)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return s.handleWalkError(filePath, err, result)
|
||||
return s.handleWalkError(filePath, err)
|
||||
}
|
||||
|
||||
// Check context cancellation
|
||||
@@ -928,6 +901,15 @@ func (s *Scanner) scanPhase(
|
||||
default:
|
||||
}
|
||||
|
||||
// Check exclude patterns - for directories, skip the entire subtree
|
||||
if s.shouldExclude(filePath, path) {
|
||||
if info.IsDir() {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Handle symlinks and directories
|
||||
handled, err := s.recordSpecialEntry(
|
||||
filePath, info, existingFiles, knownFiles, collector, result)
|
||||
@@ -1003,7 +985,7 @@ func (s *Scanner) recordSpecialEntry(
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
file, err := s.buildSymlinkEntry(filePath, info)
|
||||
if err != nil {
|
||||
return true, s.handleWalkError(filePath, err, result)
|
||||
return true, s.handleWalkError(filePath, err)
|
||||
}
|
||||
|
||||
existingFiles[filePath] = struct{}{}
|
||||
@@ -1027,38 +1009,15 @@ func (s *Scanner) recordSpecialEntry(
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// countRemovedDuringWalk counts an entry that was in its directory's
|
||||
// listing but was gone when the walk looked it up, like a file removed
|
||||
// before phase 2 reads it: as examined and failed, and not as an error,
|
||||
// with or without --skip-errors.
|
||||
func countRemovedDuringWalk(
|
||||
filePath string, existingFiles map[string]struct{}, result *ScanResult,
|
||||
) {
|
||||
log.Warn("File was deleted during the directory walk, skipping",
|
||||
"path", filePath)
|
||||
|
||||
// The walk found it, so it is not also counted as deleted.
|
||||
existingFiles[filePath] = struct{}{}
|
||||
result.FilesScanned++
|
||||
result.FilesFailed++
|
||||
}
|
||||
|
||||
// handleWalkError deals with a filesystem error surfaced by the walk:
|
||||
// under --skip-errors the entry is skipped with a UI notice and counted
|
||||
// as examined, failed and unreadable, otherwise the error is fatal.
|
||||
func (s *Scanner) handleWalkError(
|
||||
filePath string, err error, result *ScanResult,
|
||||
) error {
|
||||
// skipped (with a UI notice) under --skip-errors, fatal otherwise.
|
||||
func (s *Scanner) handleWalkError(filePath string, err error) error {
|
||||
if s.skipErrors {
|
||||
log.Error("Failed to access file (skipping due to --skip-errors)",
|
||||
"path", filePath, "error", err)
|
||||
s.ui.Errorf("Failed to access %s: %v. Skipping (--skip-errors).",
|
||||
s.ui.Path(filePath), err)
|
||||
|
||||
result.FilesScanned++
|
||||
result.FilesFailed++
|
||||
result.FilesUnreadable++
|
||||
|
||||
return nil // Continue scanning
|
||||
}
|
||||
|
||||
@@ -1500,7 +1459,6 @@ func (s *Scanner) processFileWithErrorHandling(
|
||||
s.ui.Path(fileToProcess.Path), err)
|
||||
|
||||
countFailedFile(fileToProcess, result)
|
||||
result.FilesUnreadable++
|
||||
|
||||
return true, nil
|
||||
}
|
||||
|
||||
@@ -152,25 +152,6 @@ func TestPurgeRefusesDestinationWithoutMarker(t *testing.T) {
|
||||
assertUntouched(ctx, t, repos, storeDir)
|
||||
}
|
||||
|
||||
// TestNukeRemoteRefusesDestinationWithoutMarker checks that remote nuke
|
||||
// fails, deletes no local snapshot record, and neither reports the
|
||||
// destination empty nor advises running nuke again when the destination
|
||||
// has no marker.
|
||||
func TestNukeRemoteRefusesDestinationWithoutMarker(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
v, repos, out, storeDir := backUpThenUnplugMountPoint(ctx, t)
|
||||
|
||||
err := v.NukeRemote(true)
|
||||
require.ErrorContains(t, err, markerFile)
|
||||
assert.NotContains(t, out.String(), "now empty")
|
||||
assert.NotContains(t, out.String(), "vaultik remote nuke --force")
|
||||
|
||||
assertUntouched(ctx, t, repos, storeDir)
|
||||
}
|
||||
|
||||
// TestRemoveSnapshotWarnsWhenDestinationHasNoMarker checks that snapshot
|
||||
// remove warns that the metadata could not be removed from a destination
|
||||
// without a marker, instead of reporting that it was.
|
||||
|
||||
@@ -188,41 +188,3 @@ func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
|
||||
assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
|
||||
err.Error())
|
||||
}
|
||||
|
||||
// TestNukeRemoteKeepsLocalIndexWhenDestinationMissing checks that remote
|
||||
// nuke fails on a destination it cannot list, leaves the local index
|
||||
// unchanged, and neither reports the destination empty nor advises
|
||||
// running nuke again.
|
||||
func TestNukeRemoteKeepsLocalIndexWhenDestinationMissing(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
v, repos, out := backUpThenUnplug(ctx, t)
|
||||
|
||||
filesBefore, err := repos.Files.ListAll(ctx)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, filesBefore)
|
||||
|
||||
blobsBefore, err := repos.Blobs.GetAll(ctx)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, blobsBefore)
|
||||
|
||||
err = v.NukeRemote(true)
|
||||
require.ErrorIs(t, err, fs.ErrNotExist)
|
||||
require.ErrorContains(t, err, "listing remote snapshots")
|
||||
assert.NotContains(t, out.String(), "now empty")
|
||||
assert.NotContains(t, out.String(), "vaultik remote nuke --force")
|
||||
|
||||
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 1, "nuke must delete no local snapshot record")
|
||||
|
||||
files, err := repos.Files.ListAll(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, files, len(filesBefore), "nuke must delete no local file record")
|
||||
|
||||
blobs, err := repos.Blobs.GetAll(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, blobs, len(blobsBefore), "nuke must delete no local blob record")
|
||||
}
|
||||
|
||||
@@ -104,32 +104,6 @@ func TestNukeRemoteFailsWhenADeleteFails(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestNukeRemoteKeepsLocalSnapshotWhoseMetadataIsNotDeleted checks that
|
||||
// remote nuke keeps a snapshot in the local index while its metadata
|
||||
// cannot be deleted from the destination store: dropped from the local
|
||||
// index, it would be left there as a remote-only snapshot without its
|
||||
// name.
|
||||
func TestNukeRemoteKeepsLocalSnapshotWhoseMetadataIsNotDeleted(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
storeDir := filepath.Join(t.TempDir(), "store")
|
||||
v, repos, _ := backUpToFileDestination(ctx, t, storeDir)
|
||||
|
||||
store := faultstore.New(v.Storage)
|
||||
store.OnDelete = func(key string) bool {
|
||||
return strings.HasPrefix(key, "metadata/")
|
||||
}
|
||||
v.Storage = store
|
||||
|
||||
require.Error(t, v.NukeRemote(true))
|
||||
|
||||
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 1)
|
||||
}
|
||||
|
||||
// TestNukeRemoteNamesWhatIsLeft checks that remote nuke, when an object
|
||||
// it does not recognise cannot be deleted, does not report the store
|
||||
// empty but names that object and fails.
|
||||
@@ -180,7 +154,8 @@ func TestNukeRemoteRefusesAnotherDestinationWithoutRetryAdvice(t *testing.T) {
|
||||
|
||||
// TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved checks that a
|
||||
// snapshot remote nuke cannot remove from the local index fails the
|
||||
// run, and that nuke then does not report the destination store empty.
|
||||
// run: nuke then leaves that snapshot's metadata on the destination
|
||||
// store.
|
||||
func TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
@@ -55,19 +55,12 @@ func (v *Vaultik) NukeRemote(force bool) error {
|
||||
}
|
||||
|
||||
// Checked before anything is deleted: running nuke again would fail
|
||||
// the same way, so the advice below must not follow these errors.
|
||||
// The listing also fails on a destination without the marker while
|
||||
// the local index holds snapshots.
|
||||
// the same way, so the advice below must not follow this error.
|
||||
err := v.EnsureStorageBinding()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
_, err = v.listAllRemoteSnapshotKeys()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = v.emptyDestinationStore()
|
||||
if err != nil {
|
||||
v.UI.Warningf("Run '%s' again to delete what is left on the "+
|
||||
|
||||
@@ -39,11 +39,6 @@ var (
|
||||
// caller can report it apart from a failed backup.
|
||||
var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
|
||||
|
||||
// ErrFilesUnreadable is returned when --skip-errors skipped files or
|
||||
// directories it could not read or list. The snapshot is complete and
|
||||
// kept, so the caller can report it apart from a failed backup.
|
||||
var ErrFilesUnreadable = errors.New("file(s) could not be read")
|
||||
|
||||
// listRecentLimit caps how many snapshot rows are fetched from the
|
||||
// local index for listing and purge operations.
|
||||
const listRecentLimit = 10000
|
||||
@@ -135,16 +130,12 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
||||
}
|
||||
|
||||
// Process each named snapshot
|
||||
filesUnreadable := 0
|
||||
|
||||
for snapIdx, snapName := range snapshotNames {
|
||||
unreadable, err := v.createNamedSnapshot(
|
||||
err = v.createNamedSnapshot(
|
||||
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
filesUnreadable += unreadable
|
||||
}
|
||||
|
||||
// Print overall summary if multiple snapshots
|
||||
@@ -153,34 +144,13 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
||||
len(snapshotNames), v.UI.Duration(time.Since(overallStartTime)))
|
||||
}
|
||||
|
||||
switch {
|
||||
case opts.Prune && filesUnreadable > 0:
|
||||
// An older snapshot may be the only one that still holds a file
|
||||
// this run could not read.
|
||||
log.Warn("Skipping post-backup prune",
|
||||
"files_unreadable", filesUnreadable)
|
||||
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
|
||||
"could not be read; older snapshots are kept.", filesUnreadable)
|
||||
case opts.Prune:
|
||||
if opts.Prune {
|
||||
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: %w", ErrPostBackupPruneFailed, err)
|
||||
}
|
||||
}
|
||||
|
||||
return v.finishCreateSnapshot(filesUnreadable)
|
||||
}
|
||||
|
||||
// finishCreateSnapshot ends a run whose snapshots are all complete and
|
||||
// kept. Like finishRestore, it returns an error when --skip-errors
|
||||
// skipped anything it could not read or list; otherwise it prints the
|
||||
// closing line. A file removed during the run is counted as failed in
|
||||
// the summary but is not an error, with or without --skip-errors.
|
||||
func (v *Vaultik) finishCreateSnapshot(filesUnreadable int) error {
|
||||
if filesUnreadable > 0 {
|
||||
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
|
||||
}
|
||||
|
||||
// Terminus must obey the --cron invariant: silent on total
|
||||
// success only. UI.Complete is dropped in cron/quiet mode (that's
|
||||
// the success path), but if any warnings fired during the run we
|
||||
@@ -270,22 +240,16 @@ type snapshotStats struct {
|
||||
totalBlobsUploaded int
|
||||
uploadDuration time.Duration
|
||||
|
||||
// The part of totalFilesFailed that --skip-errors skipped because it
|
||||
// could not be read or listed.
|
||||
totalFilesUnreadable int
|
||||
|
||||
// The sizes of all blobs the snapshot references, set by
|
||||
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
||||
blobSize int64
|
||||
blobUncompressedSize int64
|
||||
}
|
||||
|
||||
// createNamedSnapshot creates a single named snapshot and returns the
|
||||
// number of entries --skip-errors skipped because it could not read or
|
||||
// list them.
|
||||
// createNamedSnapshot creates a single named snapshot
|
||||
func (v *Vaultik) createNamedSnapshot(
|
||||
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
|
||||
) (int, error) {
|
||||
) error {
|
||||
snapshotStartTime := time.Now()
|
||||
|
||||
if total > 1 {
|
||||
@@ -294,12 +258,12 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
previousSnapshotID, err := v.previousSnapshotID(snapName)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
scanner := v.ScannerFactory(snapshot.ScannerParams{
|
||||
@@ -314,7 +278,7 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
|
||||
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("creating snapshot: %w", err)
|
||||
return fmt.Errorf("creating snapshot: %w", err)
|
||||
}
|
||||
|
||||
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
|
||||
@@ -322,12 +286,12 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
err = v.finalizeSnapshotMetadata(snapshotID, stats)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
log.Info("Snapshot complete",
|
||||
@@ -340,7 +304,7 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
||||
|
||||
return stats.totalFilesUnreadable, nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// previousSnapshotID returns the ID of the most recent completed snapshot
|
||||
@@ -426,7 +390,6 @@ func (v *Vaultik) scanAllDirectories(
|
||||
stats.totalBlobs += result.BlobsCreated
|
||||
stats.totalFilesSkipped += result.FilesSkipped
|
||||
stats.totalFilesFailed += result.FilesFailed
|
||||
stats.totalFilesUnreadable += result.FilesUnreadable
|
||||
stats.totalBytesSkipped += result.BytesSkipped
|
||||
stats.totalFilesDeleted += result.FilesDeleted
|
||||
stats.totalBytesDeleted += result.BytesDeleted
|
||||
@@ -1553,9 +1516,7 @@ func (v *Vaultik) removeSnapshotRemote(
|
||||
// database from the local index, and (with --remote) every snapshot
|
||||
// metadata directory in remote storage. Both sides are processed so a
|
||||
// "remove --all" leaves nothing behind, even when the local DB and
|
||||
// remote storage have diverged. Unless LocalOnly is set, it fails
|
||||
// before it removes anything when the destination store cannot be
|
||||
// listed or is not present by checkDestinationMarker.
|
||||
// remote storage have diverged.
|
||||
func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) {
|
||||
err := v.EnsureStorageBinding()
|
||||
if err != nil {
|
||||
@@ -1568,15 +1529,11 @@ func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error)
|
||||
}
|
||||
|
||||
// remoteKeys is the set of metadata/<key>/ subdirectories on the
|
||||
// destination store. Only for a local-only remove is a failure
|
||||
// downgraded to a warning, so a permission-denied or unreachable
|
||||
// remote can't block it.
|
||||
// destination store; failures are downgraded to a warning so a
|
||||
// permission-denied or unreachable remote can't block a local-only
|
||||
// remove.
|
||||
remoteKeys, remoteErr := v.listAllRemoteSnapshotKeys()
|
||||
if remoteErr != nil {
|
||||
if !opts.LocalOnly {
|
||||
return nil, remoteErr
|
||||
}
|
||||
|
||||
log.Warn("Could not list remote snapshots", "error", remoteErr)
|
||||
v.UI.Warningf("Could not list remote snapshots: %v.", remoteErr)
|
||||
}
|
||||
@@ -1638,7 +1595,8 @@ func (v *Vaultik) localSnapshotIDs() ([]string, error) {
|
||||
// listAllRemoteSnapshotKeys collects the hashed remote keys
|
||||
// (subdirectories under metadata/) currently present in the
|
||||
// destination store. Returns (nil, err) when the store cannot be
|
||||
// listed, or is not present by checkDestinationMarker.
|
||||
// listed, or is not present by checkDestinationMarker; callers must
|
||||
// treat that as "no remote info available," not fatal.
|
||||
func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) {
|
||||
log.Info("Listing all remote snapshots")
|
||||
|
||||
@@ -1822,12 +1780,10 @@ func (v *Vaultik) deleteIncompleteSnapshot(snapshotIDStr string, id any) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// removeAllLocalSnapshots removes each known local snapshot's remote
|
||||
// metadata, unless --local-only, and then its local record, returning
|
||||
// the number of snapshots it could not remove. The local record goes
|
||||
// only once the metadata is deleted: dropped first, it would leave that
|
||||
// metadata on the destination store as a remote-only snapshot whose name
|
||||
// is lost.
|
||||
// removeAllLocalSnapshots removes each known local snapshot (and, unless
|
||||
// --local-only, its remote metadata), returning the number of snapshots
|
||||
// it could not remove. A snapshot left in the local index keeps its
|
||||
// remote metadata too, since it is not an orphan remote key.
|
||||
func (v *Vaultik) removeAllLocalSnapshots(
|
||||
localSnaps []string, opts *RemoveOptions, result *RemoveResult,
|
||||
) int {
|
||||
@@ -1836,19 +1792,6 @@ func (v *Vaultik) removeAllLocalSnapshots(
|
||||
for _, snapshotID := range localSnaps {
|
||||
log.Info("Removing snapshot", "snapshot_id", snapshotID)
|
||||
|
||||
if !opts.LocalOnly {
|
||||
_, err := v.deleteRemoteSnapshotByKey(
|
||||
snapshot.RemoteSnapshotKey(snapshotID))
|
||||
if err != nil {
|
||||
log.Warn("Failed to remove snapshot metadata from remote",
|
||||
"snapshot_id", snapshotID, "error", err)
|
||||
|
||||
failures++
|
||||
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
err := v.deleteSnapshotFromLocalDB(snapshotID)
|
||||
if err != nil {
|
||||
log.Error("Failed to remove from local database",
|
||||
@@ -1859,6 +1802,17 @@ func (v *Vaultik) removeAllLocalSnapshots(
|
||||
continue
|
||||
}
|
||||
|
||||
if !opts.LocalOnly {
|
||||
_, err = v.deleteRemoteSnapshotByKey(
|
||||
snapshot.RemoteSnapshotKey(snapshotID))
|
||||
if err != nil {
|
||||
log.Warn("Failed to remove snapshot metadata from remote",
|
||||
"snapshot_id", snapshotID, "error", err)
|
||||
|
||||
failures++
|
||||
}
|
||||
}
|
||||
|
||||
result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID)
|
||||
}
|
||||
|
||||
|
||||
@@ -178,13 +178,11 @@ func TestBackupDoesNotCountFileInUnopenableDirectoryAsDeleted(t *testing.T) {
|
||||
}
|
||||
|
||||
env.out.Reset()
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"work"},
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
||||
}))
|
||||
assert.Contains(t, env.out.String(), filesLine(1, 0, 1))
|
||||
}
|
||||
|
||||
// A backup counts only files that the previous snapshot of its own name
|
||||
|
||||
@@ -49,19 +49,13 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
||||
name string
|
||||
openErr error
|
||||
skipErrors bool
|
||||
wantErr error // nil for none
|
||||
}{
|
||||
// What a normal user gets opening a file with mode 000.
|
||||
{name: "unopenable under skip-errors",
|
||||
openErr: os.ErrPermission, skipErrors: true,
|
||||
wantErr: vaultik.ErrFilesUnreadable},
|
||||
openErr: os.ErrPermission, skipErrors: true},
|
||||
// What opening a file removed after phase 1 gives.
|
||||
{name: "removed between the phases",
|
||||
openErr: os.ErrNotExist, skipErrors: false},
|
||||
// A removal is not an error, so --skip-errors does not make the
|
||||
// run fail. See https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
{name: "removed between the phases under skip-errors",
|
||||
openErr: os.ErrNotExist, skipErrors: true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -105,11 +99,10 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
||||
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
|
||||
v.UI = ui.NewWithColor(out, false)
|
||||
|
||||
err = v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
require.NoError(t, v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: tt.skipErrors,
|
||||
Snapshots: []string{"src"},
|
||||
})
|
||||
require.ErrorIs(t, err, tt.wantErr)
|
||||
}))
|
||||
|
||||
summary := out.String()
|
||||
assert.Contains(t, summary,
|
||||
|
||||
@@ -3,7 +3,6 @@ package vaultik_test
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
@@ -74,210 +73,3 @@ func TestSkipErrorsBackupStopsOnIndexErrorRecordingDirectory(t *testing.T) {
|
||||
require.Len(t, snapshots, 1)
|
||||
assert.Nil(t, snapshots[0].CompletedAt)
|
||||
}
|
||||
|
||||
// A backup under --skip-errors that skips a file it cannot open, or a
|
||||
// directory it cannot list, keeps the snapshot, counts what it skipped
|
||||
// as failed and returns an error instead of finishing successfully. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestSkipErrorsBackupKeepsSnapshotAndFailsWhenItSkips(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
skipped string // relative to the source directory
|
||||
}{
|
||||
{name: "file it cannot open", skipped: "skipdir/file.txt"},
|
||||
{name: "directory it cannot list", skipped: "skipdir"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stored.txt": []byte("stored"),
|
||||
"skipdir/file.txt": []byte("in skipdir"),
|
||||
})
|
||||
env.setSnapshot("work", "")
|
||||
|
||||
// A mode does not lock out root, who runs the tests, so
|
||||
// opening the skipped path is made to fail instead.
|
||||
env.v.Fs = &openFailFs{
|
||||
path: filepath.Join(env.src, tt.skipped),
|
||||
err: os.ErrPermission,
|
||||
}
|
||||
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"work"},
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 1 backed up, 0 unchanged, 1 failed.")
|
||||
assert.NotContains(t, env.out.String(), "Finished successfully.")
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, snapshots, 1)
|
||||
assert.NotNil(t, snapshots[0].CompletedAt, "the snapshot is kept")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// removeOnLookupFs is the real filesystem, except that path is removed
|
||||
// just before it is looked up. The walk looks an entry up after listing
|
||||
// its directory, so path is listed but gone when the walk reaches it.
|
||||
type removeOnLookupFs struct {
|
||||
afero.OsFs
|
||||
|
||||
path string
|
||||
}
|
||||
|
||||
func (f *removeOnLookupFs) LstatIfPossible(
|
||||
name string,
|
||||
) (os.FileInfo, bool, error) {
|
||||
if name == f.path {
|
||||
_ = os.Remove(name)
|
||||
}
|
||||
|
||||
return f.OsFs.LstatIfPossible(name)
|
||||
}
|
||||
|
||||
// A file removed before the walk reaches it is counted once, as failed,
|
||||
// like a file removed before its data is read, and the backup finishes
|
||||
// successfully, with or without --skip-errors. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestBackupCountsFileRemovedBeforeWalkReachesItAsFailed(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
skipErrors bool
|
||||
}{
|
||||
{name: "under skip-errors", skipErrors: true},
|
||||
{name: "without skip-errors", skipErrors: false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stays.txt": []byte("stays"),
|
||||
"removed.txt": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("walk", "")
|
||||
|
||||
// The previous snapshot has the file, so a removal the walk
|
||||
// also counted as deleted would show.
|
||||
env.backUp(t, "walk")
|
||||
|
||||
env.v.Fs = &removeOnLookupFs{
|
||||
path: filepath.Join(env.src, "removed.txt"),
|
||||
}
|
||||
|
||||
env.out.Reset()
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: tt.skipErrors,
|
||||
Snapshots: []string{"walk"},
|
||||
}))
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A file an exclude pattern matches is not counted when it is removed
|
||||
// before the walk reaches it. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestBackupIgnoresExcludedFileRemovedBeforeWalkReachesIt(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stays.txt": []byte("stays"),
|
||||
"removed.tmp": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("walk", "", "*.tmp")
|
||||
env.v.Fs = &removeOnLookupFs{
|
||||
path: filepath.Join(env.src, "removed.tmp"),
|
||||
}
|
||||
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"walk"},
|
||||
}))
|
||||
assert.Contains(t, env.out.String(), filesLine(1, 1, 0))
|
||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
||||
}
|
||||
|
||||
// A backup under --skip-errors --prune that skipped a file it could not
|
||||
// read does not prune: the previous snapshot, the only one that holds
|
||||
// the file, is kept, and the output says the prune was skipped. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
||||
func TestSkipErrorsBackupDoesNotPruneWhenItSkips(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"other.txt": []byte("other"),
|
||||
"skipped.txt": []byte("skipped"),
|
||||
})
|
||||
env.setSnapshot("photos", "")
|
||||
env.backUp(t, "photos")
|
||||
|
||||
// An unchanged file is not read again, so the file is changed before
|
||||
// it becomes unreadable.
|
||||
skipped := filepath.Join(env.src, "skipped.txt")
|
||||
require.NoError(t, os.WriteFile(skipped, []byte("changed"), 0o600))
|
||||
env.v.Fs = &openFailFs{path: skipped, err: os.ErrPermission}
|
||||
|
||||
env.out.Reset()
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
Prune: true,
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"photos"},
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Skipped the post-backup prune because 1 file(s) could not be read")
|
||||
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 2, "the previous snapshot is kept")
|
||||
}
|
||||
|
||||
// A file removed during a backup under --skip-errors --prune is not a
|
||||
// skipped file, so the backup prunes as usual. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
||||
func TestSkipErrorsBackupPrunesWhenAFileIsRemovedDuringIt(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"other.txt": []byte("other"),
|
||||
"removed.txt": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("photos", "")
|
||||
env.backUp(t, "photos")
|
||||
|
||||
env.v.Fs = &removeOnLookupFs{path: filepath.Join(env.src, "removed.txt")}
|
||||
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
Prune: true,
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"photos"},
|
||||
}))
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 1, "the previous snapshot is pruned")
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user