Compare commits
3
Commits
99a4df67e6
...
v1.0.0-rc1
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
2590f49583 | ||
|
clawbot
|
5b8d84e38a | ||
|
clawbot
|
7a950ea2aa |
@@ -182,7 +182,7 @@ vaultik version
|
||||
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
|
||||
* `--debug`: Enable debug output (on stderr — see below)
|
||||
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
|
||||
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run.
|
||||
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run. A run that skipped anything still finishes but exits non-zero: `snapshot create` keeps the snapshot and counts each file or directory it skipped as failed. A file removed during the backup is also counted as failed but does not change the exit status.
|
||||
|
||||
### locking
|
||||
|
||||
@@ -297,7 +297,11 @@ value where a list belongs or a word where a number belongs), is refused.
|
||||
(for crontab)
|
||||
* `--prune`: After backup, drop older snapshots of each backed-up name and
|
||||
remove orphaned blobs from remote storage. By default keeps only the latest
|
||||
snapshot per name; use `--keep-newer-than` for a rolling window.
|
||||
snapshot per name; use `--keep-newer-than` for a rolling window. When
|
||||
`--skip-errors` skipped a file or directory it could not read or list, the
|
||||
prune does not run and every older snapshot is kept, since one of them may
|
||||
be the only one that holds what was skipped; the command says so and exits
|
||||
non-zero. A file removed during the backup does not stop the prune.
|
||||
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
||||
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
||||
Without `--prune`, or with a value that is not a valid duration, the
|
||||
|
||||
@@ -15,13 +15,229 @@ pre-1.0
|
||||
# Next Step
|
||||
|
||||
The 1.0 work is complete on `next`: the scope settled on
|
||||
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) was the work
|
||||
already planned for 1.0, and all of it has landed. The mechanism to cut
|
||||
the tag exists and is exercised; what is left is merging `next` to
|
||||
`main` and tagging, both the owner's.
|
||||
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) has landed,
|
||||
and so has every fix from the defect hunt before the release candidate
|
||||
([issue #288](https://git.eeqj.de/sneak/vaultik/issues/288)). What is
|
||||
left is merging `next` to `main` and tagging the release, both the
|
||||
owner's; a `v*` tag runs the release build in
|
||||
`.gitea/workflows/release.yml`.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
||||
that skipped files it could not read
|
||||
([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a
|
||||
prune could delete the only snapshot still holding those files.
|
||||
|
||||
- 2026-10-14: Made `snapshot create --skip-errors` exit non-zero,
|
||||
instead of 0, when it skips anything it cannot read
|
||||
([issue #310](https://git.eeqj.de/sneak/vaultik/issues/310)).
|
||||
|
||||
- 2026-10-14: Kept the local index when `remote nuke` cannot list or
|
||||
find the destination store
|
||||
([issue #292](https://git.eeqj.de/sneak/vaultik/issues/292)). Nuke
|
||||
had cleared the local snapshot records anyway; it now fails before
|
||||
removing anything.
|
||||
|
||||
- 2026-10-14: Stopped counting directories and unchanged symlinks as
|
||||
files to back up
|
||||
([issue #305](https://git.eeqj.de/sneak/vaultik/issues/305)).
|
||||
|
||||
- 2026-10-14: Refused a backup destination that an unplugged volume
|
||||
left empty, as the owner ruled
|
||||
([issue #239](https://git.eeqj.de/sneak/vaultik/issues/239)). Every
|
||||
backup now writes `vaultik-destination.txt` at the destination root,
|
||||
and while the local index holds snapshots stored there, a destination
|
||||
without it is treated as not present.
|
||||
|
||||
- 2026-10-14: Warned about restored ownership only when an entry's
|
||||
owner or group differs from the recorded one
|
||||
([issue #316](https://git.eeqj.de/sneak/vaultik/issues/316)); a
|
||||
restore by a non-root user had warned even when none did.
|
||||
|
||||
- 2026-10-14: Made `snapshot list` report a changed destination instead
|
||||
of advising `prune`, which refuses in that case
|
||||
([issue #317](https://git.eeqj.de/sneak/vaultik/issues/317)).
|
||||
|
||||
- 2026-10-14: Counted a file as deleted by a backup only if the
|
||||
previous snapshot of the same name listed it
|
||||
([issue #304](https://git.eeqj.de/sneak/vaultik/issues/304)); files
|
||||
that snapshot did not list had been counted too.
|
||||
|
||||
- 2026-10-14: Named each prefix once in restore, verify and purge
|
||||
errors, some of which repeated it
|
||||
([issue #327](https://git.eeqj.de/sneak/vaultik/issues/327)). A deep
|
||||
verify log line that named S3 whatever the destination now names
|
||||
storage.
|
||||
|
||||
- 2026-10-14: Refused credentials in a storage URL's endpoint or rclone
|
||||
remote name, which were accepted and could then be printed
|
||||
([issue #322](https://git.eeqj.de/sneak/vaultik/issues/322)).
|
||||
|
||||
- 2026-10-14: Ran `$EDITOR` through the shell in `config edit`, so a
|
||||
value with arguments, such as `code --wait`, no longer fails
|
||||
([issue #315](https://git.eeqj.de/sneak/vaultik/issues/315)).
|
||||
|
||||
- 2026-10-14: Made `snapshot remove` fail, instead of exiting 0, when
|
||||
it cannot delete the metadata of a snapshot that is only on the
|
||||
destination store
|
||||
([issue #340](https://git.eeqj.de/sneak/vaultik/issues/340)). The
|
||||
README and the help now say when it prints the `vaultik prune`
|
||||
follow-up.
|
||||
|
||||
- 2026-10-14: Labelled the totals in the local index cleanup before
|
||||
each backup as totals
|
||||
([issue #306](https://git.eeqj.de/sneak/vaultik/issues/306)). Each
|
||||
was shown as `(N remain)`, which read as rows left to clean up.
|
||||
|
||||
- 2026-10-13: Freed a cached blob once no file still to restore needs
|
||||
it ([issue #324](https://git.eeqj.de/sneak/vaultik/issues/324)); one
|
||||
also used by a file outside the selected paths had stayed in the
|
||||
temporary cache until the restore ended.
|
||||
|
||||
- 2026-10-13: Kept the whole startup error when it contains `): `
|
||||
([issue #320](https://git.eeqj.de/sneak/vaultik/issues/320)). Such
|
||||
an error was cut at its last `): `, so the `blob_size_limit` one lost
|
||||
the setting's name.
|
||||
|
||||
- 2026-10-13: Silenced the status lines of `prune` and snapshot removal
|
||||
under `--cron` and `--quiet`
|
||||
([issue #294](https://git.eeqj.de/sneak/vaultik/issues/294)).
|
||||
|
||||
- 2026-10-13: Read the rclone config file that `RCLONE_CONFIG` names
|
||||
([issue #321](https://git.eeqj.de/sneak/vaultik/issues/321)); a
|
||||
remote defined only there had been reported as not found.
|
||||
|
||||
- 2026-10-13: Made `/**/` in an exclude pattern match zero directories,
|
||||
as in gitignore
|
||||
([issue #338](https://git.eeqj.de/sneak/vaultik/issues/338));
|
||||
`a/**/*.log` had missed `a/x.log`.
|
||||
|
||||
- 2026-10-13: Made the `chunk_size` and `blob_size_limit` defaults and
|
||||
the `chunk_size` minimum 1000-based, like the size parser and the
|
||||
docs ([issue #313](https://git.eeqj.de/sneak/vaultik/issues/313)).
|
||||
|
||||
- 2026-10-13: Kept a snapshot's remote metadata when `snapshot purge`
|
||||
cannot remove it from the local index
|
||||
([issue #337](https://git.eeqj.de/sneak/vaultik/issues/337)); the
|
||||
purge had deleted the metadata anyway and exited 0.
|
||||
|
||||
- 2026-10-13: Made `snapshot restore` fail for a requested path that
|
||||
matches nothing, which it had ignored
|
||||
([issue #309](https://git.eeqj.de/sneak/vaultik/issues/309)).
|
||||
|
||||
- 2026-10-13: Made `remote nuke` delete every object under `metadata/`
|
||||
and `blobs/`
|
||||
([issue #291](https://git.eeqj.de/sneak/vaultik/issues/291)); it had
|
||||
left some, such as a stray `.DS_Store`, while reporting the store
|
||||
empty.
|
||||
|
||||
- 2026-10-13: Made `config set` create a list for an index under a
|
||||
missing key
|
||||
([issue #308](https://git.eeqj.de/sneak/vaultik/issues/308)); it had
|
||||
written a map, after which no command could load the config.
|
||||
|
||||
- 2026-10-13: Listed a `file://` destination through a symlinked
|
||||
`metadata/` or `blobs/`, which had listed as empty
|
||||
([issue #296](https://git.eeqj.de/sneak/vaultik/issues/296)). A link
|
||||
to a missing target now fails the listing
|
||||
([issue #332](https://git.eeqj.de/sneak/vaultik/issues/332)).
|
||||
|
||||
- 2026-10-13: Made `snapshot verify --deep --json` report its blob
|
||||
counts ([issue #314](https://git.eeqj.de/sneak/vaultik/issues/314));
|
||||
a failed run had reported 0 missing and 0 verified.
|
||||
|
||||
- 2026-10-13: Limited and retried the module download in the Docker
|
||||
build ([issue #351](https://git.eeqj.de/sneak/vaultik/issues/351)),
|
||||
which had hung for as long as a stalled connection to the module
|
||||
proxy lasted.
|
||||
|
||||
- 2026-10-13: Recorded each snapshot's uncompressed size when its
|
||||
backup runs
|
||||
([issue #360](https://git.eeqj.de/sneak/vaultik/issues/360)). A later
|
||||
backup could change an older snapshot's UNCOMPRESSED SIZE in
|
||||
`snapshot list`.
|
||||
|
||||
- 2026-10-13: Stopped `snapshot restore --skip-errors --verify`
|
||||
reporting a file that failed to restore again as a checksum failure
|
||||
([issue #323](https://git.eeqj.de/sneak/vaultik/issues/323)).
|
||||
|
||||
- 2026-10-13: Recorded the stored size of a blob a backup finds already
|
||||
stored, not the size of its local copy, which can differ, so
|
||||
`snapshot verify` could fail on a good snapshot
|
||||
([issue #300](https://git.eeqj.de/sneak/vaultik/issues/300)).
|
||||
|
||||
- 2026-10-13: Stopped the local index and the blob upload step ending
|
||||
the process on a failed close of query results or a failed temp file
|
||||
cleanup ([issue #345](https://git.eeqj.de/sneak/vaultik/issues/345));
|
||||
both are now returned as errors.
|
||||
|
||||
- 2026-10-13: Closed the snapshot database's storage reader too when
|
||||
restore or deep verify is interrupted
|
||||
([issue #356](https://git.eeqj.de/sneak/vaultik/issues/356)).
|
||||
|
||||
- 2026-10-13: Made `prune` and `remote nuke` fail when a delete on the
|
||||
backup destination store fails
|
||||
([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)).
|
||||
|
||||
- 2026-10-13: Made an S3 listing that the server cuts short an error
|
||||
([issue #297](https://git.eeqj.de/sneak/vaultik/issues/297)); `prune`
|
||||
had taken it as complete and could delete blobs still in use.
|
||||
|
||||
- 2026-10-13: Made shallow `snapshot verify` report a blob it could not
|
||||
check apart from a missing one
|
||||
([issue #318](https://git.eeqj.de/sneak/vaultik/issues/318)); any
|
||||
error checking a blob had counted as missing, its cause dropped.
|
||||
|
||||
- 2026-10-13: Restored the other files when a file's chunk is in no
|
||||
blob under `--skip-errors`
|
||||
([issue #326](https://git.eeqj.de/sneak/vaultik/issues/326)); that
|
||||
file had stopped the restore before anything was restored.
|
||||
|
||||
- 2026-10-13: Recorded each snapshot's new chunk size when its backup
|
||||
runs ([issue #311](https://git.eeqj.de/sneak/vaultik/issues/311)).
|
||||
`snapshot list` could show a NEW CHUNK SIZE of 0 B for a snapshot
|
||||
whose change was a modified file.
|
||||
|
||||
- 2026-10-13: Made a backup interrupted during a local index write exit
|
||||
as interrupted
|
||||
([issue #301](https://git.eeqj.de/sneak/vaultik/issues/301)); it could
|
||||
end with `FATAL: failed to rollback transaction` and exit 1 instead of
|
||||
130.
|
||||
|
||||
- 2026-10-13: Printed `snapshot list` and `snapshot purge` sizes like
|
||||
every other command
|
||||
([issue #312](https://git.eeqj.de/sneak/vaultik/issues/312)). They,
|
||||
and the `bytes` value in terminal log lines, were 1024-based but
|
||||
labelled KB or MB, and now use 1000-based units.
|
||||
|
||||
- 2026-10-13: Closed each blob's storage reader when restore or deep
|
||||
verify is interrupted, which ends a stalled read unless it is stuck in
|
||||
the kernel ([issue #325](https://git.eeqj.de/sneak/vaultik/issues/325)).
|
||||
|
||||
- 2026-10-13: Made a mistyped subcommand under `snapshot`, `config`,
|
||||
`remote`, `database` or `completion` an error
|
||||
([issue #319](https://git.eeqj.de/sneak/vaultik/issues/319)); each had
|
||||
printed its help and exited 0.
|
||||
|
||||
- 2026-10-13: Made an exclude pattern starting with `**/` match
|
||||
directly under the snapshot path too, as in gitignore
|
||||
([issue #303](https://git.eeqj.de/sneak/vaultik/issues/303)).
|
||||
|
||||
- 2026-10-13: Refused a relative `file://` storage URL and decoded its
|
||||
percent-escapes
|
||||
([issue #299](https://git.eeqj.de/sneak/vaultik/issues/299)); a
|
||||
relative path had moved with the working directory.
|
||||
|
||||
- 2026-10-13: Removed the blob temp file when a backup stops part-way
|
||||
([issue #302](https://git.eeqj.de/sneak/vaultik/issues/302)); the
|
||||
half-written blob had stayed in `$TMPDIR`.
|
||||
|
||||
- 2026-10-13: Checked `--keep-newer-than` before a backup starts
|
||||
([issue #307](https://git.eeqj.de/sneak/vaultik/issues/307)); an
|
||||
invalid value had been read only by the prune after the backup, so
|
||||
retention never ran.
|
||||
|
||||
- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that
|
||||
is not a blob as an orphaned blob
|
||||
([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
|
||||
// under --skip-errors, a source directory that does not exist, which the
|
||||
// walk skips. The command exits non-zero and reports the backup as
|
||||
// completed, not as a failed snapshot creation. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
//
|
||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
|
||||
// writeBackupConfig would create the source directory.
|
||||
configPath := writeHermeticPruneConfig(t, false)
|
||||
|
||||
code, stdout, stderr := runEntry(t,
|
||||
flagConfig, configPath, "--skip-errors", cmdSnapshot, cmdCreate)
|
||||
|
||||
assert.Equal(t, 1, code)
|
||||
assert.Contains(t, stdout,
|
||||
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
|
||||
assert.NotContains(t, stdout, "Finished successfully.")
|
||||
assert.Contains(t, stderr,
|
||||
"Backup completed, but 1 file(s) could not be read")
|
||||
assert.NotContains(t, stderr, "Snapshot creation failed")
|
||||
}
|
||||
@@ -115,6 +115,13 @@ on Linux), then /etc/vaultik/config.yml.`,
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, vaultik.ErrFilesUnreadable) {
|
||||
log.Error("Backup completed with unreadable files", "error", err)
|
||||
ReportErrorf("Backup completed, but %v", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
log.Error("Snapshot creation failed", "error", err)
|
||||
ReportErrorf("Snapshot creation failed: %v", err)
|
||||
})
|
||||
@@ -125,7 +132,8 @@ on Linux), then /etc/vaultik/config.yml.`,
|
||||
"Run in cron mode (silent unless warning or error)")
|
||||
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
|
||||
"After backup, drop older snapshots of the same name and remove "+
|
||||
"orphaned blobs")
|
||||
"orphaned blobs (not after a backup that skipped unreadable "+
|
||||
"files)")
|
||||
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
||||
"With --prune: keep snapshots newer than this duration "+
|
||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
||||
|
||||
@@ -141,10 +141,12 @@ type ScannerConfig struct {
|
||||
// ScanResult contains the results of a scan operation. Files and bytes
|
||||
// are counted per file: BytesScanned is the size of the new and changed
|
||||
// files, BytesSkipped that of the unchanged ones. FilesFailed counts the
|
||||
// new and changed files that phase 2 could not store; FilesScanned
|
||||
// includes them and BytesScanned does not. ChunksCreated counts the
|
||||
// chunks that were not already stored, and NewChunkSize is their total
|
||||
// size.
|
||||
// new and changed files that phase 2 could not store, each entry removed
|
||||
// after the walk listed its directory and, under --skip-errors, each
|
||||
// entry the walk could not read, such as a directory it could not list;
|
||||
// FilesScanned includes them and BytesScanned does not. ChunksCreated
|
||||
// counts the chunks that were not already stored, and NewChunkSize is
|
||||
// their total size.
|
||||
type ScanResult struct {
|
||||
FilesScanned int
|
||||
FilesSkipped int
|
||||
@@ -161,6 +163,11 @@ type ScanResult struct {
|
||||
UploadDuration time.Duration
|
||||
StartTime time.Time
|
||||
EndTime time.Time
|
||||
|
||||
// FilesUnreadable is the part of FilesFailed that --skip-errors
|
||||
// skipped because it could not be read or listed; the rest are files
|
||||
// removed after the walk listed them.
|
||||
FilesUnreadable int
|
||||
}
|
||||
|
||||
// NewScanner creates a new scanner instance
|
||||
@@ -890,8 +897,28 @@ func (s *Scanner) scanPhase(
|
||||
log.Debug("Starting directory walk", "path", path)
|
||||
|
||||
walkFn := func(filePath string, info os.FileInfo, err error) error {
|
||||
// Check exclude patterns first, so an excluded entry that is gone
|
||||
// or cannot be looked up is neither counted nor reported. For
|
||||
// directories, skip the entire subtree. info is nil when the
|
||||
// walk could not look the entry up.
|
||||
if s.shouldExclude(filePath, path) {
|
||||
if info != nil && info.IsDir() {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// The source path is in no directory listing, so its absence is
|
||||
// an error rather than a removal.
|
||||
if errors.Is(err, os.ErrNotExist) && filePath != path {
|
||||
countRemovedDuringWalk(filePath, existingFiles, result)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return s.handleWalkError(filePath, err)
|
||||
return s.handleWalkError(filePath, err, result)
|
||||
}
|
||||
|
||||
// Check context cancellation
|
||||
@@ -901,15 +928,6 @@ func (s *Scanner) scanPhase(
|
||||
default:
|
||||
}
|
||||
|
||||
// Check exclude patterns - for directories, skip the entire subtree
|
||||
if s.shouldExclude(filePath, path) {
|
||||
if info.IsDir() {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Handle symlinks and directories
|
||||
handled, err := s.recordSpecialEntry(
|
||||
filePath, info, existingFiles, knownFiles, collector, result)
|
||||
@@ -985,7 +1003,7 @@ func (s *Scanner) recordSpecialEntry(
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
file, err := s.buildSymlinkEntry(filePath, info)
|
||||
if err != nil {
|
||||
return true, s.handleWalkError(filePath, err)
|
||||
return true, s.handleWalkError(filePath, err, result)
|
||||
}
|
||||
|
||||
existingFiles[filePath] = struct{}{}
|
||||
@@ -1009,15 +1027,38 @@ func (s *Scanner) recordSpecialEntry(
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// countRemovedDuringWalk counts an entry that was in its directory's
|
||||
// listing but was gone when the walk looked it up, like a file removed
|
||||
// before phase 2 reads it: as examined and failed, and not as an error,
|
||||
// with or without --skip-errors.
|
||||
func countRemovedDuringWalk(
|
||||
filePath string, existingFiles map[string]struct{}, result *ScanResult,
|
||||
) {
|
||||
log.Warn("File was deleted during the directory walk, skipping",
|
||||
"path", filePath)
|
||||
|
||||
// The walk found it, so it is not also counted as deleted.
|
||||
existingFiles[filePath] = struct{}{}
|
||||
result.FilesScanned++
|
||||
result.FilesFailed++
|
||||
}
|
||||
|
||||
// handleWalkError deals with a filesystem error surfaced by the walk:
|
||||
// skipped (with a UI notice) under --skip-errors, fatal otherwise.
|
||||
func (s *Scanner) handleWalkError(filePath string, err error) error {
|
||||
// under --skip-errors the entry is skipped with a UI notice and counted
|
||||
// as examined, failed and unreadable, otherwise the error is fatal.
|
||||
func (s *Scanner) handleWalkError(
|
||||
filePath string, err error, result *ScanResult,
|
||||
) error {
|
||||
if s.skipErrors {
|
||||
log.Error("Failed to access file (skipping due to --skip-errors)",
|
||||
"path", filePath, "error", err)
|
||||
s.ui.Errorf("Failed to access %s: %v. Skipping (--skip-errors).",
|
||||
s.ui.Path(filePath), err)
|
||||
|
||||
result.FilesScanned++
|
||||
result.FilesFailed++
|
||||
result.FilesUnreadable++
|
||||
|
||||
return nil // Continue scanning
|
||||
}
|
||||
|
||||
@@ -1459,6 +1500,7 @@ func (s *Scanner) processFileWithErrorHandling(
|
||||
s.ui.Path(fileToProcess.Path), err)
|
||||
|
||||
countFailedFile(fileToProcess, result)
|
||||
result.FilesUnreadable++
|
||||
|
||||
return true, nil
|
||||
}
|
||||
|
||||
@@ -39,6 +39,11 @@ var (
|
||||
// caller can report it apart from a failed backup.
|
||||
var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
|
||||
|
||||
// ErrFilesUnreadable is returned when --skip-errors skipped files or
|
||||
// directories it could not read or list. The snapshot is complete and
|
||||
// kept, so the caller can report it apart from a failed backup.
|
||||
var ErrFilesUnreadable = errors.New("file(s) could not be read")
|
||||
|
||||
// listRecentLimit caps how many snapshot rows are fetched from the
|
||||
// local index for listing and purge operations.
|
||||
const listRecentLimit = 10000
|
||||
@@ -130,12 +135,16 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
||||
}
|
||||
|
||||
// Process each named snapshot
|
||||
filesUnreadable := 0
|
||||
|
||||
for snapIdx, snapName := range snapshotNames {
|
||||
err = v.createNamedSnapshot(
|
||||
unreadable, err := v.createNamedSnapshot(
|
||||
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
filesUnreadable += unreadable
|
||||
}
|
||||
|
||||
// Print overall summary if multiple snapshots
|
||||
@@ -144,13 +153,34 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
||||
len(snapshotNames), v.UI.Duration(time.Since(overallStartTime)))
|
||||
}
|
||||
|
||||
if opts.Prune {
|
||||
switch {
|
||||
case opts.Prune && filesUnreadable > 0:
|
||||
// An older snapshot may be the only one that still holds a file
|
||||
// this run could not read.
|
||||
log.Warn("Skipping post-backup prune",
|
||||
"files_unreadable", filesUnreadable)
|
||||
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
|
||||
"could not be read; older snapshots are kept.", filesUnreadable)
|
||||
case opts.Prune:
|
||||
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: %w", ErrPostBackupPruneFailed, err)
|
||||
}
|
||||
}
|
||||
|
||||
return v.finishCreateSnapshot(filesUnreadable)
|
||||
}
|
||||
|
||||
// finishCreateSnapshot ends a run whose snapshots are all complete and
|
||||
// kept. Like finishRestore, it returns an error when --skip-errors
|
||||
// skipped anything it could not read or list; otherwise it prints the
|
||||
// closing line. A file removed during the run is counted as failed in
|
||||
// the summary but is not an error, with or without --skip-errors.
|
||||
func (v *Vaultik) finishCreateSnapshot(filesUnreadable int) error {
|
||||
if filesUnreadable > 0 {
|
||||
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
|
||||
}
|
||||
|
||||
// Terminus must obey the --cron invariant: silent on total
|
||||
// success only. UI.Complete is dropped in cron/quiet mode (that's
|
||||
// the success path), but if any warnings fired during the run we
|
||||
@@ -240,16 +270,22 @@ type snapshotStats struct {
|
||||
totalBlobsUploaded int
|
||||
uploadDuration time.Duration
|
||||
|
||||
// The part of totalFilesFailed that --skip-errors skipped because it
|
||||
// could not be read or listed.
|
||||
totalFilesUnreadable int
|
||||
|
||||
// The sizes of all blobs the snapshot references, set by
|
||||
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
||||
blobSize int64
|
||||
blobUncompressedSize int64
|
||||
}
|
||||
|
||||
// createNamedSnapshot creates a single named snapshot
|
||||
// createNamedSnapshot creates a single named snapshot and returns the
|
||||
// number of entries --skip-errors skipped because it could not read or
|
||||
// list them.
|
||||
func (v *Vaultik) createNamedSnapshot(
|
||||
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
|
||||
) error {
|
||||
) (int, error) {
|
||||
snapshotStartTime := time.Now()
|
||||
|
||||
if total > 1 {
|
||||
@@ -258,12 +294,12 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
|
||||
if err != nil {
|
||||
return err
|
||||
return 0, err
|
||||
}
|
||||
|
||||
previousSnapshotID, err := v.previousSnapshotID(snapName)
|
||||
if err != nil {
|
||||
return err
|
||||
return 0, err
|
||||
}
|
||||
|
||||
scanner := v.ScannerFactory(snapshot.ScannerParams{
|
||||
@@ -278,7 +314,7 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
|
||||
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating snapshot: %w", err)
|
||||
return 0, fmt.Errorf("creating snapshot: %w", err)
|
||||
}
|
||||
|
||||
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
|
||||
@@ -286,12 +322,12 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
|
||||
if err != nil {
|
||||
return err
|
||||
return 0, err
|
||||
}
|
||||
|
||||
err = v.finalizeSnapshotMetadata(snapshotID, stats)
|
||||
if err != nil {
|
||||
return err
|
||||
return 0, err
|
||||
}
|
||||
|
||||
log.Info("Snapshot complete",
|
||||
@@ -304,7 +340,7 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
||||
|
||||
return nil
|
||||
return stats.totalFilesUnreadable, nil
|
||||
}
|
||||
|
||||
// previousSnapshotID returns the ID of the most recent completed snapshot
|
||||
@@ -390,6 +426,7 @@ func (v *Vaultik) scanAllDirectories(
|
||||
stats.totalBlobs += result.BlobsCreated
|
||||
stats.totalFilesSkipped += result.FilesSkipped
|
||||
stats.totalFilesFailed += result.FilesFailed
|
||||
stats.totalFilesUnreadable += result.FilesUnreadable
|
||||
stats.totalBytesSkipped += result.BytesSkipped
|
||||
stats.totalFilesDeleted += result.FilesDeleted
|
||||
stats.totalBytesDeleted += result.BytesDeleted
|
||||
|
||||
@@ -178,11 +178,13 @@ func TestBackupDoesNotCountFileInUnopenableDirectoryAsDeleted(t *testing.T) {
|
||||
}
|
||||
|
||||
env.out.Reset()
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"work"},
|
||||
}))
|
||||
assert.Contains(t, env.out.String(), filesLine(1, 0, 1))
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
||||
}
|
||||
|
||||
// A backup counts only files that the previous snapshot of its own name
|
||||
|
||||
@@ -49,13 +49,19 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
||||
name string
|
||||
openErr error
|
||||
skipErrors bool
|
||||
wantErr error // nil for none
|
||||
}{
|
||||
// What a normal user gets opening a file with mode 000.
|
||||
{name: "unopenable under skip-errors",
|
||||
openErr: os.ErrPermission, skipErrors: true},
|
||||
openErr: os.ErrPermission, skipErrors: true,
|
||||
wantErr: vaultik.ErrFilesUnreadable},
|
||||
// What opening a file removed after phase 1 gives.
|
||||
{name: "removed between the phases",
|
||||
openErr: os.ErrNotExist, skipErrors: false},
|
||||
// A removal is not an error, so --skip-errors does not make the
|
||||
// run fail. See https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
{name: "removed between the phases under skip-errors",
|
||||
openErr: os.ErrNotExist, skipErrors: true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -99,10 +105,11 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
||||
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
|
||||
v.UI = ui.NewWithColor(out, false)
|
||||
|
||||
require.NoError(t, v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
err = v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: tt.skipErrors,
|
||||
Snapshots: []string{"src"},
|
||||
}))
|
||||
})
|
||||
require.ErrorIs(t, err, tt.wantErr)
|
||||
|
||||
summary := out.String()
|
||||
assert.Contains(t, summary,
|
||||
|
||||
@@ -3,6 +3,7 @@ package vaultik_test
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
@@ -73,3 +74,210 @@ func TestSkipErrorsBackupStopsOnIndexErrorRecordingDirectory(t *testing.T) {
|
||||
require.Len(t, snapshots, 1)
|
||||
assert.Nil(t, snapshots[0].CompletedAt)
|
||||
}
|
||||
|
||||
// A backup under --skip-errors that skips a file it cannot open, or a
|
||||
// directory it cannot list, keeps the snapshot, counts what it skipped
|
||||
// as failed and returns an error instead of finishing successfully. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestSkipErrorsBackupKeepsSnapshotAndFailsWhenItSkips(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
skipped string // relative to the source directory
|
||||
}{
|
||||
{name: "file it cannot open", skipped: "skipdir/file.txt"},
|
||||
{name: "directory it cannot list", skipped: "skipdir"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stored.txt": []byte("stored"),
|
||||
"skipdir/file.txt": []byte("in skipdir"),
|
||||
})
|
||||
env.setSnapshot("work", "")
|
||||
|
||||
// A mode does not lock out root, who runs the tests, so
|
||||
// opening the skipped path is made to fail instead.
|
||||
env.v.Fs = &openFailFs{
|
||||
path: filepath.Join(env.src, tt.skipped),
|
||||
err: os.ErrPermission,
|
||||
}
|
||||
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"work"},
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 1 backed up, 0 unchanged, 1 failed.")
|
||||
assert.NotContains(t, env.out.String(), "Finished successfully.")
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, snapshots, 1)
|
||||
assert.NotNil(t, snapshots[0].CompletedAt, "the snapshot is kept")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// removeOnLookupFs is the real filesystem, except that path is removed
|
||||
// just before it is looked up. The walk looks an entry up after listing
|
||||
// its directory, so path is listed but gone when the walk reaches it.
|
||||
type removeOnLookupFs struct {
|
||||
afero.OsFs
|
||||
|
||||
path string
|
||||
}
|
||||
|
||||
func (f *removeOnLookupFs) LstatIfPossible(
|
||||
name string,
|
||||
) (os.FileInfo, bool, error) {
|
||||
if name == f.path {
|
||||
_ = os.Remove(name)
|
||||
}
|
||||
|
||||
return f.OsFs.LstatIfPossible(name)
|
||||
}
|
||||
|
||||
// A file removed before the walk reaches it is counted once, as failed,
|
||||
// like a file removed before its data is read, and the backup finishes
|
||||
// successfully, with or without --skip-errors. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestBackupCountsFileRemovedBeforeWalkReachesItAsFailed(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
skipErrors bool
|
||||
}{
|
||||
{name: "under skip-errors", skipErrors: true},
|
||||
{name: "without skip-errors", skipErrors: false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stays.txt": []byte("stays"),
|
||||
"removed.txt": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("walk", "")
|
||||
|
||||
// The previous snapshot has the file, so a removal the walk
|
||||
// also counted as deleted would show.
|
||||
env.backUp(t, "walk")
|
||||
|
||||
env.v.Fs = &removeOnLookupFs{
|
||||
path: filepath.Join(env.src, "removed.txt"),
|
||||
}
|
||||
|
||||
env.out.Reset()
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: tt.skipErrors,
|
||||
Snapshots: []string{"walk"},
|
||||
}))
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A file an exclude pattern matches is not counted when it is removed
|
||||
// before the walk reaches it. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestBackupIgnoresExcludedFileRemovedBeforeWalkReachesIt(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stays.txt": []byte("stays"),
|
||||
"removed.tmp": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("walk", "", "*.tmp")
|
||||
env.v.Fs = &removeOnLookupFs{
|
||||
path: filepath.Join(env.src, "removed.tmp"),
|
||||
}
|
||||
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"walk"},
|
||||
}))
|
||||
assert.Contains(t, env.out.String(), filesLine(1, 1, 0))
|
||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
||||
}
|
||||
|
||||
// A backup under --skip-errors --prune that skipped a file it could not
|
||||
// read does not prune: the previous snapshot, the only one that holds
|
||||
// the file, is kept, and the output says the prune was skipped. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
||||
func TestSkipErrorsBackupDoesNotPruneWhenItSkips(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"other.txt": []byte("other"),
|
||||
"skipped.txt": []byte("skipped"),
|
||||
})
|
||||
env.setSnapshot("photos", "")
|
||||
env.backUp(t, "photos")
|
||||
|
||||
// An unchanged file is not read again, so the file is changed before
|
||||
// it becomes unreadable.
|
||||
skipped := filepath.Join(env.src, "skipped.txt")
|
||||
require.NoError(t, os.WriteFile(skipped, []byte("changed"), 0o600))
|
||||
env.v.Fs = &openFailFs{path: skipped, err: os.ErrPermission}
|
||||
|
||||
env.out.Reset()
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
Prune: true,
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"photos"},
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Skipped the post-backup prune because 1 file(s) could not be read")
|
||||
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 2, "the previous snapshot is kept")
|
||||
}
|
||||
|
||||
// A file removed during a backup under --skip-errors --prune is not a
|
||||
// skipped file, so the backup prunes as usual. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
||||
func TestSkipErrorsBackupPrunesWhenAFileIsRemovedDuringIt(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"other.txt": []byte("other"),
|
||||
"removed.txt": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("photos", "")
|
||||
env.backUp(t, "photos")
|
||||
|
||||
env.v.Fs = &removeOnLookupFs{path: filepath.Join(env.src, "removed.txt")}
|
||||
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
Prune: true,
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"photos"},
|
||||
}))
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 1, "the previous snapshot is pruned")
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user