Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
523629f1a8 |
+1
-1
@@ -359,7 +359,7 @@ CreateSnapshot(opts)
|
||||
|
||||
## Deduplication Strategy
|
||||
|
||||
1. **File-level**: Files and symlinks unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked
|
||||
1. **File-level**: Files unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked
|
||||
|
||||
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded.
|
||||
|
||||
|
||||
@@ -182,7 +182,7 @@ vaultik version
|
||||
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
|
||||
* `--debug`: Enable debug output (on stderr — see below)
|
||||
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
|
||||
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run. A run that skipped anything still finishes but exits non-zero: `snapshot create` keeps the snapshot and counts each file or directory it skipped as failed. A file removed during the backup is also counted as failed but does not change the exit status.
|
||||
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run.
|
||||
|
||||
### locking
|
||||
|
||||
@@ -297,11 +297,7 @@ value where a list belongs or a word where a number belongs), is refused.
|
||||
(for crontab)
|
||||
* `--prune`: After backup, drop older snapshots of each backed-up name and
|
||||
remove orphaned blobs from remote storage. By default keeps only the latest
|
||||
snapshot per name; use `--keep-newer-than` for a rolling window. When
|
||||
`--skip-errors` skipped a file or directory it could not read or list, the
|
||||
prune does not run and every older snapshot is kept, since one of them may
|
||||
be the only one that holds what was skipped; the command says so and exits
|
||||
non-zero. A file removed during the backup does not stop the prune.
|
||||
snapshot per name; use `--keep-newer-than` for a rolling window.
|
||||
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
||||
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
||||
Without `--prune`, or with a value that is not a valid duration, the
|
||||
|
||||
@@ -15,229 +15,13 @@ pre-1.0
|
||||
# Next Step
|
||||
|
||||
The 1.0 work is complete on `next`: the scope settled on
|
||||
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) has landed,
|
||||
and so has every fix from the defect hunt before the release candidate
|
||||
([issue #288](https://git.eeqj.de/sneak/vaultik/issues/288)). What is
|
||||
left is merging `next` to `main` and tagging the release, both the
|
||||
owner's; a `v*` tag runs the release build in
|
||||
`.gitea/workflows/release.yml`.
|
||||
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) was the work
|
||||
already planned for 1.0, and all of it has landed. The mechanism to cut
|
||||
the tag exists and is exercised; what is left is merging `next` to
|
||||
`main` and tagging, both the owner's.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
||||
that skipped files it could not read
|
||||
([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a
|
||||
prune could delete the only snapshot still holding those files.
|
||||
|
||||
- 2026-10-14: Made `snapshot create --skip-errors` exit non-zero,
|
||||
instead of 0, when it skips anything it cannot read
|
||||
([issue #310](https://git.eeqj.de/sneak/vaultik/issues/310)).
|
||||
|
||||
- 2026-10-14: Kept the local index when `remote nuke` cannot list or
|
||||
find the destination store
|
||||
([issue #292](https://git.eeqj.de/sneak/vaultik/issues/292)). Nuke
|
||||
had cleared the local snapshot records anyway; it now fails before
|
||||
removing anything.
|
||||
|
||||
- 2026-10-14: Stopped counting directories and unchanged symlinks as
|
||||
files to back up
|
||||
([issue #305](https://git.eeqj.de/sneak/vaultik/issues/305)).
|
||||
|
||||
- 2026-10-14: Refused a backup destination that an unplugged volume
|
||||
left empty, as the owner ruled
|
||||
([issue #239](https://git.eeqj.de/sneak/vaultik/issues/239)). Every
|
||||
backup now writes `vaultik-destination.txt` at the destination root,
|
||||
and while the local index holds snapshots stored there, a destination
|
||||
without it is treated as not present.
|
||||
|
||||
- 2026-10-14: Warned about restored ownership only when an entry's
|
||||
owner or group differs from the recorded one
|
||||
([issue #316](https://git.eeqj.de/sneak/vaultik/issues/316)); a
|
||||
restore by a non-root user had warned even when none did.
|
||||
|
||||
- 2026-10-14: Made `snapshot list` report a changed destination instead
|
||||
of advising `prune`, which refuses in that case
|
||||
([issue #317](https://git.eeqj.de/sneak/vaultik/issues/317)).
|
||||
|
||||
- 2026-10-14: Counted a file as deleted by a backup only if the
|
||||
previous snapshot of the same name listed it
|
||||
([issue #304](https://git.eeqj.de/sneak/vaultik/issues/304)); files
|
||||
that snapshot did not list had been counted too.
|
||||
|
||||
- 2026-10-14: Named each prefix once in restore, verify and purge
|
||||
errors, some of which repeated it
|
||||
([issue #327](https://git.eeqj.de/sneak/vaultik/issues/327)). A deep
|
||||
verify log line that named S3 whatever the destination now names
|
||||
storage.
|
||||
|
||||
- 2026-10-14: Refused credentials in a storage URL's endpoint or rclone
|
||||
remote name, which were accepted and could then be printed
|
||||
([issue #322](https://git.eeqj.de/sneak/vaultik/issues/322)).
|
||||
|
||||
- 2026-10-14: Ran `$EDITOR` through the shell in `config edit`, so a
|
||||
value with arguments, such as `code --wait`, no longer fails
|
||||
([issue #315](https://git.eeqj.de/sneak/vaultik/issues/315)).
|
||||
|
||||
- 2026-10-14: Made `snapshot remove` fail, instead of exiting 0, when
|
||||
it cannot delete the metadata of a snapshot that is only on the
|
||||
destination store
|
||||
([issue #340](https://git.eeqj.de/sneak/vaultik/issues/340)). The
|
||||
README and the help now say when it prints the `vaultik prune`
|
||||
follow-up.
|
||||
|
||||
- 2026-10-14: Labelled the totals in the local index cleanup before
|
||||
each backup as totals
|
||||
([issue #306](https://git.eeqj.de/sneak/vaultik/issues/306)). Each
|
||||
was shown as `(N remain)`, which read as rows left to clean up.
|
||||
|
||||
- 2026-10-13: Freed a cached blob once no file still to restore needs
|
||||
it ([issue #324](https://git.eeqj.de/sneak/vaultik/issues/324)); one
|
||||
also used by a file outside the selected paths had stayed in the
|
||||
temporary cache until the restore ended.
|
||||
|
||||
- 2026-10-13: Kept the whole startup error when it contains `): `
|
||||
([issue #320](https://git.eeqj.de/sneak/vaultik/issues/320)). Such
|
||||
an error was cut at its last `): `, so the `blob_size_limit` one lost
|
||||
the setting's name.
|
||||
|
||||
- 2026-10-13: Silenced the status lines of `prune` and snapshot removal
|
||||
under `--cron` and `--quiet`
|
||||
([issue #294](https://git.eeqj.de/sneak/vaultik/issues/294)).
|
||||
|
||||
- 2026-10-13: Read the rclone config file that `RCLONE_CONFIG` names
|
||||
([issue #321](https://git.eeqj.de/sneak/vaultik/issues/321)); a
|
||||
remote defined only there had been reported as not found.
|
||||
|
||||
- 2026-10-13: Made `/**/` in an exclude pattern match zero directories,
|
||||
as in gitignore
|
||||
([issue #338](https://git.eeqj.de/sneak/vaultik/issues/338));
|
||||
`a/**/*.log` had missed `a/x.log`.
|
||||
|
||||
- 2026-10-13: Made the `chunk_size` and `blob_size_limit` defaults and
|
||||
the `chunk_size` minimum 1000-based, like the size parser and the
|
||||
docs ([issue #313](https://git.eeqj.de/sneak/vaultik/issues/313)).
|
||||
|
||||
- 2026-10-13: Kept a snapshot's remote metadata when `snapshot purge`
|
||||
cannot remove it from the local index
|
||||
([issue #337](https://git.eeqj.de/sneak/vaultik/issues/337)); the
|
||||
purge had deleted the metadata anyway and exited 0.
|
||||
|
||||
- 2026-10-13: Made `snapshot restore` fail for a requested path that
|
||||
matches nothing, which it had ignored
|
||||
([issue #309](https://git.eeqj.de/sneak/vaultik/issues/309)).
|
||||
|
||||
- 2026-10-13: Made `remote nuke` delete every object under `metadata/`
|
||||
and `blobs/`
|
||||
([issue #291](https://git.eeqj.de/sneak/vaultik/issues/291)); it had
|
||||
left some, such as a stray `.DS_Store`, while reporting the store
|
||||
empty.
|
||||
|
||||
- 2026-10-13: Made `config set` create a list for an index under a
|
||||
missing key
|
||||
([issue #308](https://git.eeqj.de/sneak/vaultik/issues/308)); it had
|
||||
written a map, after which no command could load the config.
|
||||
|
||||
- 2026-10-13: Listed a `file://` destination through a symlinked
|
||||
`metadata/` or `blobs/`, which had listed as empty
|
||||
([issue #296](https://git.eeqj.de/sneak/vaultik/issues/296)). A link
|
||||
to a missing target now fails the listing
|
||||
([issue #332](https://git.eeqj.de/sneak/vaultik/issues/332)).
|
||||
|
||||
- 2026-10-13: Made `snapshot verify --deep --json` report its blob
|
||||
counts ([issue #314](https://git.eeqj.de/sneak/vaultik/issues/314));
|
||||
a failed run had reported 0 missing and 0 verified.
|
||||
|
||||
- 2026-10-13: Limited and retried the module download in the Docker
|
||||
build ([issue #351](https://git.eeqj.de/sneak/vaultik/issues/351)),
|
||||
which had hung for as long as a stalled connection to the module
|
||||
proxy lasted.
|
||||
|
||||
- 2026-10-13: Recorded each snapshot's uncompressed size when its
|
||||
backup runs
|
||||
([issue #360](https://git.eeqj.de/sneak/vaultik/issues/360)). A later
|
||||
backup could change an older snapshot's UNCOMPRESSED SIZE in
|
||||
`snapshot list`.
|
||||
|
||||
- 2026-10-13: Stopped `snapshot restore --skip-errors --verify`
|
||||
reporting a file that failed to restore again as a checksum failure
|
||||
([issue #323](https://git.eeqj.de/sneak/vaultik/issues/323)).
|
||||
|
||||
- 2026-10-13: Recorded the stored size of a blob a backup finds already
|
||||
stored, not the size of its local copy, which can differ, so
|
||||
`snapshot verify` could fail on a good snapshot
|
||||
([issue #300](https://git.eeqj.de/sneak/vaultik/issues/300)).
|
||||
|
||||
- 2026-10-13: Stopped the local index and the blob upload step ending
|
||||
the process on a failed close of query results or a failed temp file
|
||||
cleanup ([issue #345](https://git.eeqj.de/sneak/vaultik/issues/345));
|
||||
both are now returned as errors.
|
||||
|
||||
- 2026-10-13: Closed the snapshot database's storage reader too when
|
||||
restore or deep verify is interrupted
|
||||
([issue #356](https://git.eeqj.de/sneak/vaultik/issues/356)).
|
||||
|
||||
- 2026-10-13: Made `prune` and `remote nuke` fail when a delete on the
|
||||
backup destination store fails
|
||||
([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)).
|
||||
|
||||
- 2026-10-13: Made an S3 listing that the server cuts short an error
|
||||
([issue #297](https://git.eeqj.de/sneak/vaultik/issues/297)); `prune`
|
||||
had taken it as complete and could delete blobs still in use.
|
||||
|
||||
- 2026-10-13: Made shallow `snapshot verify` report a blob it could not
|
||||
check apart from a missing one
|
||||
([issue #318](https://git.eeqj.de/sneak/vaultik/issues/318)); any
|
||||
error checking a blob had counted as missing, its cause dropped.
|
||||
|
||||
- 2026-10-13: Restored the other files when a file's chunk is in no
|
||||
blob under `--skip-errors`
|
||||
([issue #326](https://git.eeqj.de/sneak/vaultik/issues/326)); that
|
||||
file had stopped the restore before anything was restored.
|
||||
|
||||
- 2026-10-13: Recorded each snapshot's new chunk size when its backup
|
||||
runs ([issue #311](https://git.eeqj.de/sneak/vaultik/issues/311)).
|
||||
`snapshot list` could show a NEW CHUNK SIZE of 0 B for a snapshot
|
||||
whose change was a modified file.
|
||||
|
||||
- 2026-10-13: Made a backup interrupted during a local index write exit
|
||||
as interrupted
|
||||
([issue #301](https://git.eeqj.de/sneak/vaultik/issues/301)); it could
|
||||
end with `FATAL: failed to rollback transaction` and exit 1 instead of
|
||||
130.
|
||||
|
||||
- 2026-10-13: Printed `snapshot list` and `snapshot purge` sizes like
|
||||
every other command
|
||||
([issue #312](https://git.eeqj.de/sneak/vaultik/issues/312)). They,
|
||||
and the `bytes` value in terminal log lines, were 1024-based but
|
||||
labelled KB or MB, and now use 1000-based units.
|
||||
|
||||
- 2026-10-13: Closed each blob's storage reader when restore or deep
|
||||
verify is interrupted, which ends a stalled read unless it is stuck in
|
||||
the kernel ([issue #325](https://git.eeqj.de/sneak/vaultik/issues/325)).
|
||||
|
||||
- 2026-10-13: Made a mistyped subcommand under `snapshot`, `config`,
|
||||
`remote`, `database` or `completion` an error
|
||||
([issue #319](https://git.eeqj.de/sneak/vaultik/issues/319)); each had
|
||||
printed its help and exited 0.
|
||||
|
||||
- 2026-10-13: Made an exclude pattern starting with `**/` match
|
||||
directly under the snapshot path too, as in gitignore
|
||||
([issue #303](https://git.eeqj.de/sneak/vaultik/issues/303)).
|
||||
|
||||
- 2026-10-13: Refused a relative `file://` storage URL and decoded its
|
||||
percent-escapes
|
||||
([issue #299](https://git.eeqj.de/sneak/vaultik/issues/299)); a
|
||||
relative path had moved with the working directory.
|
||||
|
||||
- 2026-10-13: Removed the blob temp file when a backup stops part-way
|
||||
([issue #302](https://git.eeqj.de/sneak/vaultik/issues/302)); the
|
||||
half-written blob had stayed in `$TMPDIR`.
|
||||
|
||||
- 2026-10-13: Checked `--keep-newer-than` before a backup starts
|
||||
([issue #307](https://git.eeqj.de/sneak/vaultik/issues/307)); an
|
||||
invalid value had been read only by the prune after the backup, so
|
||||
retention never ran.
|
||||
|
||||
- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that
|
||||
is not a blob as an orphaned blob
|
||||
([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
|
||||
// under --skip-errors, a source directory that does not exist, which the
|
||||
// walk skips. The command exits non-zero and reports the backup as
|
||||
// completed, not as a failed snapshot creation. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
//
|
||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
|
||||
// writeBackupConfig would create the source directory.
|
||||
configPath := writeHermeticPruneConfig(t, false)
|
||||
|
||||
code, stdout, stderr := runEntry(t,
|
||||
flagConfig, configPath, "--skip-errors", cmdSnapshot, cmdCreate)
|
||||
|
||||
assert.Equal(t, 1, code)
|
||||
assert.Contains(t, stdout,
|
||||
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
|
||||
assert.NotContains(t, stdout, "Finished successfully.")
|
||||
assert.Contains(t, stderr,
|
||||
"Backup completed, but 1 file(s) could not be read")
|
||||
assert.NotContains(t, stderr, "Snapshot creation failed")
|
||||
}
|
||||
@@ -115,13 +115,6 @@ on Linux), then /etc/vaultik/config.yml.`,
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, vaultik.ErrFilesUnreadable) {
|
||||
log.Error("Backup completed with unreadable files", "error", err)
|
||||
ReportErrorf("Backup completed, but %v", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
log.Error("Snapshot creation failed", "error", err)
|
||||
ReportErrorf("Snapshot creation failed: %v", err)
|
||||
})
|
||||
@@ -132,8 +125,7 @@ on Linux), then /etc/vaultik/config.yml.`,
|
||||
"Run in cron mode (silent unless warning or error)")
|
||||
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
|
||||
"After backup, drop older snapshots of the same name and remove "+
|
||||
"orphaned blobs (not after a backup that skipped unreadable "+
|
||||
"files)")
|
||||
"orphaned blobs")
|
||||
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
||||
"With --prune: keep snapshots newer than this duration "+
|
||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
||||
|
||||
@@ -78,7 +78,8 @@ func TestProgressPercentWithSecondPathSmaller(t *testing.T) {
|
||||
|
||||
stats := progress.GetStats()
|
||||
|
||||
// With both paths counted, the percentage must not exceed 100%.
|
||||
// Directories count toward the total size but produce no chunks, so
|
||||
// the percentage ends just under 100%.
|
||||
percent := float64(stats.BytesProcessed.Load()) /
|
||||
float64(stats.TotalSize.Load()) * 100
|
||||
if percent > 100 {
|
||||
|
||||
+88
-167
@@ -141,12 +141,10 @@ type ScannerConfig struct {
|
||||
// ScanResult contains the results of a scan operation. Files and bytes
|
||||
// are counted per file: BytesScanned is the size of the new and changed
|
||||
// files, BytesSkipped that of the unchanged ones. FilesFailed counts the
|
||||
// new and changed files that phase 2 could not store, each entry removed
|
||||
// after the walk listed its directory and, under --skip-errors, each
|
||||
// entry the walk could not read, such as a directory it could not list;
|
||||
// FilesScanned includes them and BytesScanned does not. ChunksCreated
|
||||
// counts the chunks that were not already stored, and NewChunkSize is
|
||||
// their total size.
|
||||
// new and changed files that phase 2 could not store; FilesScanned
|
||||
// includes them and BytesScanned does not. ChunksCreated counts the
|
||||
// chunks that were not already stored, and NewChunkSize is their total
|
||||
// size.
|
||||
type ScanResult struct {
|
||||
FilesScanned int
|
||||
FilesSkipped int
|
||||
@@ -163,11 +161,6 @@ type ScanResult struct {
|
||||
UploadDuration time.Duration
|
||||
StartTime time.Time
|
||||
EndTime time.Time
|
||||
|
||||
// FilesUnreadable is the part of FilesFailed that --skip-errors
|
||||
// skipped because it could not be read or listed; the rest are files
|
||||
// removed after the walk listed them.
|
||||
FilesUnreadable int
|
||||
}
|
||||
|
||||
// NewScanner creates a new scanner instance
|
||||
@@ -291,11 +284,6 @@ func (s *Scanner) Scan(
|
||||
}
|
||||
}
|
||||
|
||||
err = s.recordDirectories(ctx, scanResult.Directories)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Summarize scan phase results and update progress
|
||||
s.summarizeScanPhase(result, filesToProcess)
|
||||
|
||||
@@ -867,15 +855,14 @@ func (s *Scanner) batchInsertFileData(
|
||||
// ScanPhaseResult contains the results of the scan phase
|
||||
type ScanPhaseResult struct {
|
||||
FilesToProcess []*FileToProcess
|
||||
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
|
||||
Directories []*FileToProcess // Every directory found, to record
|
||||
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
|
||||
}
|
||||
|
||||
// scanPhase performs the initial directory scan to identify files to process
|
||||
// It uses the pre-loaded knownFiles map for fast change detection without DB
|
||||
// queries. It also populates existingFiles map for deletion detection.
|
||||
// Returns files needing processing, IDs of unchanged files for snapshot
|
||||
// association, and the directories found.
|
||||
// Returns files needing processing and IDs of unchanged files for snapshot
|
||||
// association.
|
||||
func (s *Scanner) scanPhase(
|
||||
ctx context.Context,
|
||||
path string,
|
||||
@@ -883,10 +870,8 @@ func (s *Scanner) scanPhase(
|
||||
existingFiles map[string]struct{},
|
||||
knownFiles map[string]*database.File,
|
||||
) (*ScanPhaseResult, error) {
|
||||
// Use known file count as estimate for progress (accurate for
|
||||
// subsequent backups). The walk does not count directories, so the
|
||||
// estimate leaves them out.
|
||||
estimatedTotal := countNonDirectories(knownFiles)
|
||||
// Use known file count as estimate for progress (accurate for subsequent backups)
|
||||
estimatedTotal := int64(len(knownFiles))
|
||||
|
||||
collector := &scanCollector{}
|
||||
|
||||
@@ -897,28 +882,8 @@ func (s *Scanner) scanPhase(
|
||||
log.Debug("Starting directory walk", "path", path)
|
||||
|
||||
walkFn := func(filePath string, info os.FileInfo, err error) error {
|
||||
// Check exclude patterns first, so an excluded entry that is gone
|
||||
// or cannot be looked up is neither counted nor reported. For
|
||||
// directories, skip the entire subtree. info is nil when the
|
||||
// walk could not look the entry up.
|
||||
if s.shouldExclude(filePath, path) {
|
||||
if info != nil && info.IsDir() {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// The source path is in no directory listing, so its absence is
|
||||
// an error rather than a removal.
|
||||
if errors.Is(err, os.ErrNotExist) && filePath != path {
|
||||
countRemovedDuringWalk(filePath, existingFiles, result)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return s.handleWalkError(filePath, err, result)
|
||||
return s.handleWalkError(filePath, err)
|
||||
}
|
||||
|
||||
// Check context cancellation
|
||||
@@ -928,9 +893,18 @@ func (s *Scanner) scanPhase(
|
||||
default:
|
||||
}
|
||||
|
||||
// Check exclude patterns - for directories, skip the entire subtree
|
||||
if s.shouldExclude(filePath, path) {
|
||||
if info.IsDir() {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Handle symlinks and directories
|
||||
handled, err := s.recordSpecialEntry(
|
||||
filePath, info, existingFiles, knownFiles, collector, result)
|
||||
filePath, info, existingFiles, collector, result)
|
||||
if handled {
|
||||
return err
|
||||
}
|
||||
@@ -944,10 +918,9 @@ func (s *Scanner) scanPhase(
|
||||
existingFiles[filePath] = struct{}{}
|
||||
|
||||
// Check file against in-memory map (no DB query!)
|
||||
file := s.buildRegularFileEntry(filePath, info)
|
||||
needsProcessing := s.checkFileInMemory(file, knownFiles)
|
||||
file, needsProcessing := s.checkFileInMemory(filePath, info, knownFiles)
|
||||
|
||||
filesScanned, changedCount := collector.recordFile(
|
||||
filesScanned, changedCount := collector.recordRegularFile(
|
||||
filePath, info, file, needsProcessing)
|
||||
|
||||
// Update result stats
|
||||
@@ -971,45 +944,27 @@ func (s *Scanner) scanPhase(
|
||||
return &ScanPhaseResult{
|
||||
FilesToProcess: collector.filesToProcess,
|
||||
UnchangedFileIDs: collector.unchangedFileIDs,
|
||||
Directories: collector.directories,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// countNonDirectories returns how many of files are not directories.
|
||||
func countNonDirectories(files map[string]*database.File) int64 {
|
||||
var count int64
|
||||
|
||||
for _, f := range files {
|
||||
if f.Mode&uint32(os.ModeDir) == 0 {
|
||||
count++
|
||||
}
|
||||
}
|
||||
|
||||
return count
|
||||
}
|
||||
|
||||
// recordSpecialEntry records symlinks and directories (which have no
|
||||
// data to chunk) and reports whether it handled the entry. A symlink is
|
||||
// counted and checked for changes as a regular file is; a directory is
|
||||
// set aside to be recorded and is not counted. For a symlink whose
|
||||
// target cannot be read it returns handleWalkError's result.
|
||||
// data to chunk) and reports whether it handled the entry. For a symlink
|
||||
// whose target cannot be read it returns handleWalkError's result.
|
||||
func (s *Scanner) recordSpecialEntry(
|
||||
filePath string, info os.FileInfo,
|
||||
existingFiles map[string]struct{},
|
||||
knownFiles map[string]*database.File,
|
||||
collector *scanCollector, result *ScanResult,
|
||||
) (bool, error) {
|
||||
// Handle symlinks
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
file, err := s.buildSymlinkEntry(filePath, info)
|
||||
if err != nil {
|
||||
return true, s.handleWalkError(filePath, err, result)
|
||||
return true, s.handleWalkError(filePath, err)
|
||||
}
|
||||
|
||||
existingFiles[filePath] = struct{}{}
|
||||
needsProcessing := s.checkFileInMemory(file, knownFiles)
|
||||
collector.recordFile(filePath, info, file, needsProcessing)
|
||||
s.updateScanEntryStats(result, needsProcessing, info)
|
||||
collector.addToProcess(filePath, info, file)
|
||||
s.updateScanEntryStats(result, true, info)
|
||||
|
||||
return true, nil
|
||||
}
|
||||
@@ -1019,7 +974,7 @@ func (s *Scanner) recordSpecialEntry(
|
||||
if info.IsDir() {
|
||||
file := s.buildDirectoryEntry(filePath, info)
|
||||
existingFiles[filePath] = struct{}{}
|
||||
collector.addDirectory(filePath, info, file)
|
||||
collector.addToProcess(filePath, info, file)
|
||||
|
||||
return true, nil
|
||||
}
|
||||
@@ -1027,38 +982,15 @@ func (s *Scanner) recordSpecialEntry(
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// countRemovedDuringWalk counts an entry that was in its directory's
|
||||
// listing but was gone when the walk looked it up, like a file removed
|
||||
// before phase 2 reads it: as examined and failed, and not as an error,
|
||||
// with or without --skip-errors.
|
||||
func countRemovedDuringWalk(
|
||||
filePath string, existingFiles map[string]struct{}, result *ScanResult,
|
||||
) {
|
||||
log.Warn("File was deleted during the directory walk, skipping",
|
||||
"path", filePath)
|
||||
|
||||
// The walk found it, so it is not also counted as deleted.
|
||||
existingFiles[filePath] = struct{}{}
|
||||
result.FilesScanned++
|
||||
result.FilesFailed++
|
||||
}
|
||||
|
||||
// handleWalkError deals with a filesystem error surfaced by the walk:
|
||||
// under --skip-errors the entry is skipped with a UI notice and counted
|
||||
// as examined, failed and unreadable, otherwise the error is fatal.
|
||||
func (s *Scanner) handleWalkError(
|
||||
filePath string, err error, result *ScanResult,
|
||||
) error {
|
||||
// skipped (with a UI notice) under --skip-errors, fatal otherwise.
|
||||
func (s *Scanner) handleWalkError(filePath string, err error) error {
|
||||
if s.skipErrors {
|
||||
log.Error("Failed to access file (skipping due to --skip-errors)",
|
||||
"path", filePath, "error", err)
|
||||
s.ui.Errorf("Failed to access %s: %v. Skipping (--skip-errors).",
|
||||
s.ui.Path(filePath), err)
|
||||
|
||||
result.FilesScanned++
|
||||
result.FilesFailed++
|
||||
result.FilesUnreadable++
|
||||
|
||||
return nil // Continue scanning
|
||||
}
|
||||
|
||||
@@ -1072,29 +1004,28 @@ type scanCollector struct {
|
||||
mu sync.Mutex
|
||||
filesToProcess []*FileToProcess
|
||||
unchangedFileIDs []types.FileID // Just IDs - no new records needed
|
||||
directories []*FileToProcess
|
||||
filesScanned int64
|
||||
}
|
||||
|
||||
// addDirectory sets a directory aside to be recorded. It is not counted
|
||||
// as a file.
|
||||
func (c *scanCollector) addDirectory(
|
||||
// addToProcess queues a file entry for processing.
|
||||
func (c *scanCollector) addToProcess(
|
||||
path string, info os.FileInfo, file *database.File,
|
||||
) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
c.directories = append(c.directories, &FileToProcess{
|
||||
c.filesToProcess = append(c.filesToProcess, &FileToProcess{
|
||||
Path: path,
|
||||
FileInfo: info,
|
||||
File: file,
|
||||
})
|
||||
c.filesScanned++
|
||||
}
|
||||
|
||||
// recordFile records a regular file or symlink as changed (to process)
|
||||
// or unchanged (snapshot association only) and returns the running
|
||||
// scanned and changed counts.
|
||||
func (c *scanCollector) recordFile(
|
||||
// recordRegularFile records a regular file as changed (to process) or
|
||||
// unchanged (snapshot association only) and returns the running scanned
|
||||
// and changed counts.
|
||||
func (c *scanCollector) recordRegularFile(
|
||||
path string, info os.FileInfo, file *database.File, needsProcessing bool,
|
||||
) (int64, int) {
|
||||
c.mu.Lock()
|
||||
@@ -1197,28 +1128,7 @@ func (s *Scanner) printScanProgressLine(
|
||||
}
|
||||
}
|
||||
|
||||
// buildRegularFileEntry creates a File record for a regular file, with
|
||||
// no ID: checkFileInMemory sets it.
|
||||
func (s *Scanner) buildRegularFileEntry(path string, info os.FileInfo) *database.File {
|
||||
var uid, gid uint32
|
||||
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
|
||||
uid = stat.Uid
|
||||
gid = stat.Gid
|
||||
}
|
||||
|
||||
return &database.File{
|
||||
Path: types.FilePath(path),
|
||||
SourcePath: types.SourcePath(s.currentSourcePath),
|
||||
MTime: info.ModTime(),
|
||||
Size: info.Size(),
|
||||
Mode: uint32(info.Mode()),
|
||||
UID: uid,
|
||||
GID: gid,
|
||||
}
|
||||
}
|
||||
|
||||
// buildSymlinkEntry creates a File record for a symlink, with no ID:
|
||||
// checkFileInMemory sets it.
|
||||
// buildSymlinkEntry creates a File record for a symlink.
|
||||
func (s *Scanner) buildSymlinkEntry(
|
||||
path string, info os.FileInfo,
|
||||
) (*database.File, error) {
|
||||
@@ -1234,6 +1144,7 @@ func (s *Scanner) buildSymlinkEntry(
|
||||
}
|
||||
|
||||
return &database.File{
|
||||
ID: types.NewFileID(),
|
||||
Path: types.FilePath(path),
|
||||
SourcePath: types.SourcePath(s.currentSourcePath),
|
||||
MTime: info.ModTime(),
|
||||
@@ -1265,22 +1176,6 @@ func (s *Scanner) buildDirectoryEntry(path string, info os.FileInfo) *database.F
|
||||
}
|
||||
}
|
||||
|
||||
// recordDirectories records every directory the walk found. A directory
|
||||
// has no data to back up, so it is not among the files counted and
|
||||
// backed up.
|
||||
func (s *Scanner) recordDirectories(
|
||||
ctx context.Context, directories []*FileToProcess,
|
||||
) error {
|
||||
for _, dir := range directories {
|
||||
err := s.recordNonRegularFile(ctx, dir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("recording directory %s: %w", dir.Path, err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// recordNonRegularFile writes a symlink or directory entry to the database
|
||||
// and associates it with the current snapshot. No chunking is performed.
|
||||
func (s *Scanner) recordNonRegularFile(ctx context.Context, ftp *FileToProcess) error {
|
||||
@@ -1294,36 +1189,62 @@ func (s *Scanner) recordNonRegularFile(ctx context.Context, ftp *FileToProcess)
|
||||
})
|
||||
}
|
||||
|
||||
// checkFileInMemory reports whether a regular file or symlink needs
|
||||
// processing, and gives file the ID of its known record, or a new ID for
|
||||
// a new path.
|
||||
// checkFileInMemory checks if a file needs processing using the in-memory map
|
||||
// No database access is performed - this is purely CPU/memory work
|
||||
func (s *Scanner) checkFileInMemory(
|
||||
file *database.File, knownFiles map[string]*database.File,
|
||||
) bool {
|
||||
existingFile, exists := knownFiles[file.Path.String()]
|
||||
path string, info os.FileInfo, knownFiles map[string]*database.File,
|
||||
) (*database.File, bool) {
|
||||
var uid, gid uint32
|
||||
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
|
||||
uid = stat.Uid
|
||||
gid = stat.Gid
|
||||
}
|
||||
|
||||
// Check against in-memory map first to get existing ID if available
|
||||
existingFile, exists := knownFiles[path]
|
||||
|
||||
// Create file record with ID set upfront
|
||||
// For new files, generate UUID immediately so it's available for chunk associations
|
||||
// For existing files, reuse the existing ID
|
||||
var fileID types.FileID
|
||||
if exists {
|
||||
fileID = existingFile.ID
|
||||
} else {
|
||||
fileID = types.NewFileID()
|
||||
}
|
||||
|
||||
file := &database.File{
|
||||
ID: fileID,
|
||||
Path: types.FilePath(path),
|
||||
SourcePath: types.SourcePath(s.currentSourcePath),
|
||||
MTime: info.ModTime(),
|
||||
Size: info.Size(),
|
||||
Mode: uint32(info.Mode()),
|
||||
UID: uid,
|
||||
GID: gid,
|
||||
}
|
||||
|
||||
// New file - needs processing
|
||||
if !exists {
|
||||
file.ID = types.NewFileID()
|
||||
|
||||
return true
|
||||
return file, true
|
||||
}
|
||||
|
||||
file.ID = existingFile.ID
|
||||
|
||||
// No uploaded blob holds one of its chunks (see loadFilesToRechunk)
|
||||
if _, rechunk := s.filesToRechunk[file.ID]; rechunk {
|
||||
return true
|
||||
if _, rechunk := s.filesToRechunk[fileID]; rechunk {
|
||||
return file, true
|
||||
}
|
||||
|
||||
// Check if file has changed
|
||||
return existingFile.Size != file.Size ||
|
||||
if existingFile.Size != file.Size ||
|
||||
!existingFile.MTime.Equal(file.MTime) ||
|
||||
existingFile.Mode != file.Mode ||
|
||||
existingFile.UID != file.UID ||
|
||||
existingFile.GID != file.GID ||
|
||||
existingFile.LinkTarget != file.LinkTarget
|
||||
existingFile.GID != file.GID {
|
||||
return file, true
|
||||
}
|
||||
|
||||
// File unchanged
|
||||
return file, false
|
||||
}
|
||||
|
||||
// batchAddFilesToSnapshot adds existing file IDs to the snapshot association table
|
||||
@@ -1450,16 +1371,17 @@ func (s *Scanner) processPhase(
|
||||
return s.finalizeProcessPhase(ctx, result)
|
||||
}
|
||||
|
||||
// processFileWithErrorHandling records a symlink, or wraps
|
||||
// processFileWithErrorHandling records a directory or symlink, or wraps
|
||||
// processFileStreaming for a regular file with error recovery for deleted
|
||||
// files and skip-errors mode. Returns (skipped, error).
|
||||
func (s *Scanner) processFileWithErrorHandling(
|
||||
ctx context.Context, fileToProcess *FileToProcess, result *ScanResult,
|
||||
) (bool, error) {
|
||||
// A symlink has no data to open or read; it is only recorded in the
|
||||
// local index. An error recording it stops the run even under
|
||||
// --skip-errors, or the snapshot would complete without it.
|
||||
if os.FileMode(fileToProcess.File.Mode)&os.ModeSymlink != 0 {
|
||||
// A directory or symlink has no data to open or read; it is only
|
||||
// recorded in the local index. An error recording it stops the run
|
||||
// even under --skip-errors, or the snapshot would complete without it.
|
||||
mode := os.FileMode(fileToProcess.File.Mode)
|
||||
if mode&os.ModeSymlink != 0 || mode.IsDir() {
|
||||
err := s.recordNonRegularFile(ctx, fileToProcess)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("processing file %s: %w", fileToProcess.Path, err)
|
||||
@@ -1500,7 +1422,6 @@ func (s *Scanner) processFileWithErrorHandling(
|
||||
s.ui.Path(fileToProcess.Path), err)
|
||||
|
||||
countFailedFile(fileToProcess, result)
|
||||
result.FilesUnreadable++
|
||||
|
||||
return true, nil
|
||||
}
|
||||
|
||||
@@ -1,28 +0,0 @@
|
||||
//nolint:testpackage // exercises the unexported countNonDirectories helper
|
||||
package snapshot
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
)
|
||||
|
||||
// The walk counts regular files and symlinks, not directories, so the
|
||||
// estimate of its count made from the local index's records leaves the
|
||||
// directories out. See https://git.eeqj.de/sneak/vaultik/issues/305.
|
||||
func TestCountNonDirectories(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
known := map[string]*database.File{
|
||||
"/src": {Mode: uint32(os.ModeDir | 0o755)},
|
||||
"/src/sub": {Mode: uint32(os.ModeDir | 0o755)},
|
||||
"/src/a.txt": {Mode: 0o644},
|
||||
"/src/link": {Mode: uint32(os.ModeSymlink | 0o777), LinkTarget: "a.txt"},
|
||||
}
|
||||
|
||||
got := countNonDirectories(known)
|
||||
if got != 2 {
|
||||
t.Errorf("countNonDirectories = %d, want 2", got)
|
||||
}
|
||||
}
|
||||
@@ -39,11 +39,6 @@ var (
|
||||
// caller can report it apart from a failed backup.
|
||||
var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
|
||||
|
||||
// ErrFilesUnreadable is returned when --skip-errors skipped files or
|
||||
// directories it could not read or list. The snapshot is complete and
|
||||
// kept, so the caller can report it apart from a failed backup.
|
||||
var ErrFilesUnreadable = errors.New("file(s) could not be read")
|
||||
|
||||
// listRecentLimit caps how many snapshot rows are fetched from the
|
||||
// local index for listing and purge operations.
|
||||
const listRecentLimit = 10000
|
||||
@@ -135,16 +130,12 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
||||
}
|
||||
|
||||
// Process each named snapshot
|
||||
filesUnreadable := 0
|
||||
|
||||
for snapIdx, snapName := range snapshotNames {
|
||||
unreadable, err := v.createNamedSnapshot(
|
||||
err = v.createNamedSnapshot(
|
||||
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
filesUnreadable += unreadable
|
||||
}
|
||||
|
||||
// Print overall summary if multiple snapshots
|
||||
@@ -153,34 +144,13 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
||||
len(snapshotNames), v.UI.Duration(time.Since(overallStartTime)))
|
||||
}
|
||||
|
||||
switch {
|
||||
case opts.Prune && filesUnreadable > 0:
|
||||
// An older snapshot may be the only one that still holds a file
|
||||
// this run could not read.
|
||||
log.Warn("Skipping post-backup prune",
|
||||
"files_unreadable", filesUnreadable)
|
||||
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
|
||||
"could not be read; older snapshots are kept.", filesUnreadable)
|
||||
case opts.Prune:
|
||||
if opts.Prune {
|
||||
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: %w", ErrPostBackupPruneFailed, err)
|
||||
}
|
||||
}
|
||||
|
||||
return v.finishCreateSnapshot(filesUnreadable)
|
||||
}
|
||||
|
||||
// finishCreateSnapshot ends a run whose snapshots are all complete and
|
||||
// kept. Like finishRestore, it returns an error when --skip-errors
|
||||
// skipped anything it could not read or list; otherwise it prints the
|
||||
// closing line. A file removed during the run is counted as failed in
|
||||
// the summary but is not an error, with or without --skip-errors.
|
||||
func (v *Vaultik) finishCreateSnapshot(filesUnreadable int) error {
|
||||
if filesUnreadable > 0 {
|
||||
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
|
||||
}
|
||||
|
||||
// Terminus must obey the --cron invariant: silent on total
|
||||
// success only. UI.Complete is dropped in cron/quiet mode (that's
|
||||
// the success path), but if any warnings fired during the run we
|
||||
@@ -270,22 +240,16 @@ type snapshotStats struct {
|
||||
totalBlobsUploaded int
|
||||
uploadDuration time.Duration
|
||||
|
||||
// The part of totalFilesFailed that --skip-errors skipped because it
|
||||
// could not be read or listed.
|
||||
totalFilesUnreadable int
|
||||
|
||||
// The sizes of all blobs the snapshot references, set by
|
||||
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
||||
blobSize int64
|
||||
blobUncompressedSize int64
|
||||
}
|
||||
|
||||
// createNamedSnapshot creates a single named snapshot and returns the
|
||||
// number of entries --skip-errors skipped because it could not read or
|
||||
// list them.
|
||||
// createNamedSnapshot creates a single named snapshot
|
||||
func (v *Vaultik) createNamedSnapshot(
|
||||
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
|
||||
) (int, error) {
|
||||
) error {
|
||||
snapshotStartTime := time.Now()
|
||||
|
||||
if total > 1 {
|
||||
@@ -294,12 +258,12 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
previousSnapshotID, err := v.previousSnapshotID(snapName)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
scanner := v.ScannerFactory(snapshot.ScannerParams{
|
||||
@@ -314,7 +278,7 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
|
||||
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("creating snapshot: %w", err)
|
||||
return fmt.Errorf("creating snapshot: %w", err)
|
||||
}
|
||||
|
||||
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
|
||||
@@ -322,12 +286,12 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
err = v.finalizeSnapshotMetadata(snapshotID, stats)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
log.Info("Snapshot complete",
|
||||
@@ -340,7 +304,7 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
||||
|
||||
return stats.totalFilesUnreadable, nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// previousSnapshotID returns the ID of the most recent completed snapshot
|
||||
@@ -426,7 +390,6 @@ func (v *Vaultik) scanAllDirectories(
|
||||
stats.totalBlobs += result.BlobsCreated
|
||||
stats.totalFilesSkipped += result.FilesSkipped
|
||||
stats.totalFilesFailed += result.FilesFailed
|
||||
stats.totalFilesUnreadable += result.FilesUnreadable
|
||||
stats.totalBytesSkipped += result.BytesSkipped
|
||||
stats.totalFilesDeleted += result.FilesDeleted
|
||||
stats.totalBytesDeleted += result.BytesDeleted
|
||||
|
||||
@@ -178,13 +178,11 @@ func TestBackupDoesNotCountFileInUnopenableDirectoryAsDeleted(t *testing.T) {
|
||||
}
|
||||
|
||||
env.out.Reset()
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"work"},
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
||||
}))
|
||||
assert.Contains(t, env.out.String(), filesLine(1, 0, 1))
|
||||
}
|
||||
|
||||
// A backup counts only files that the previous snapshot of its own name
|
||||
|
||||
@@ -49,19 +49,13 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
||||
name string
|
||||
openErr error
|
||||
skipErrors bool
|
||||
wantErr error // nil for none
|
||||
}{
|
||||
// What a normal user gets opening a file with mode 000.
|
||||
{name: "unopenable under skip-errors",
|
||||
openErr: os.ErrPermission, skipErrors: true,
|
||||
wantErr: vaultik.ErrFilesUnreadable},
|
||||
openErr: os.ErrPermission, skipErrors: true},
|
||||
// What opening a file removed after phase 1 gives.
|
||||
{name: "removed between the phases",
|
||||
openErr: os.ErrNotExist, skipErrors: false},
|
||||
// A removal is not an error, so --skip-errors does not make the
|
||||
// run fail. See https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
{name: "removed between the phases under skip-errors",
|
||||
openErr: os.ErrNotExist, skipErrors: true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -105,11 +99,10 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
||||
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
|
||||
v.UI = ui.NewWithColor(out, false)
|
||||
|
||||
err = v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
require.NoError(t, v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: tt.skipErrors,
|
||||
Snapshots: []string{"src"},
|
||||
})
|
||||
require.ErrorIs(t, err, tt.wantErr)
|
||||
}))
|
||||
|
||||
summary := out.String()
|
||||
assert.Contains(t, summary,
|
||||
|
||||
@@ -3,7 +3,6 @@ package vaultik_test
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
@@ -74,210 +73,3 @@ func TestSkipErrorsBackupStopsOnIndexErrorRecordingDirectory(t *testing.T) {
|
||||
require.Len(t, snapshots, 1)
|
||||
assert.Nil(t, snapshots[0].CompletedAt)
|
||||
}
|
||||
|
||||
// A backup under --skip-errors that skips a file it cannot open, or a
|
||||
// directory it cannot list, keeps the snapshot, counts what it skipped
|
||||
// as failed and returns an error instead of finishing successfully. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestSkipErrorsBackupKeepsSnapshotAndFailsWhenItSkips(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
skipped string // relative to the source directory
|
||||
}{
|
||||
{name: "file it cannot open", skipped: "skipdir/file.txt"},
|
||||
{name: "directory it cannot list", skipped: "skipdir"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stored.txt": []byte("stored"),
|
||||
"skipdir/file.txt": []byte("in skipdir"),
|
||||
})
|
||||
env.setSnapshot("work", "")
|
||||
|
||||
// A mode does not lock out root, who runs the tests, so
|
||||
// opening the skipped path is made to fail instead.
|
||||
env.v.Fs = &openFailFs{
|
||||
path: filepath.Join(env.src, tt.skipped),
|
||||
err: os.ErrPermission,
|
||||
}
|
||||
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"work"},
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 1 backed up, 0 unchanged, 1 failed.")
|
||||
assert.NotContains(t, env.out.String(), "Finished successfully.")
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, snapshots, 1)
|
||||
assert.NotNil(t, snapshots[0].CompletedAt, "the snapshot is kept")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// removeOnLookupFs is the real filesystem, except that path is removed
|
||||
// just before it is looked up. The walk looks an entry up after listing
|
||||
// its directory, so path is listed but gone when the walk reaches it.
|
||||
type removeOnLookupFs struct {
|
||||
afero.OsFs
|
||||
|
||||
path string
|
||||
}
|
||||
|
||||
func (f *removeOnLookupFs) LstatIfPossible(
|
||||
name string,
|
||||
) (os.FileInfo, bool, error) {
|
||||
if name == f.path {
|
||||
_ = os.Remove(name)
|
||||
}
|
||||
|
||||
return f.OsFs.LstatIfPossible(name)
|
||||
}
|
||||
|
||||
// A file removed before the walk reaches it is counted once, as failed,
|
||||
// like a file removed before its data is read, and the backup finishes
|
||||
// successfully, with or without --skip-errors. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestBackupCountsFileRemovedBeforeWalkReachesItAsFailed(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
skipErrors bool
|
||||
}{
|
||||
{name: "under skip-errors", skipErrors: true},
|
||||
{name: "without skip-errors", skipErrors: false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stays.txt": []byte("stays"),
|
||||
"removed.txt": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("walk", "")
|
||||
|
||||
// The previous snapshot has the file, so a removal the walk
|
||||
// also counted as deleted would show.
|
||||
env.backUp(t, "walk")
|
||||
|
||||
env.v.Fs = &removeOnLookupFs{
|
||||
path: filepath.Join(env.src, "removed.txt"),
|
||||
}
|
||||
|
||||
env.out.Reset()
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: tt.skipErrors,
|
||||
Snapshots: []string{"walk"},
|
||||
}))
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A file an exclude pattern matches is not counted when it is removed
|
||||
// before the walk reaches it. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestBackupIgnoresExcludedFileRemovedBeforeWalkReachesIt(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stays.txt": []byte("stays"),
|
||||
"removed.tmp": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("walk", "", "*.tmp")
|
||||
env.v.Fs = &removeOnLookupFs{
|
||||
path: filepath.Join(env.src, "removed.tmp"),
|
||||
}
|
||||
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"walk"},
|
||||
}))
|
||||
assert.Contains(t, env.out.String(), filesLine(1, 1, 0))
|
||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
||||
}
|
||||
|
||||
// A backup under --skip-errors --prune that skipped a file it could not
|
||||
// read does not prune: the previous snapshot, the only one that holds
|
||||
// the file, is kept, and the output says the prune was skipped. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
||||
func TestSkipErrorsBackupDoesNotPruneWhenItSkips(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"other.txt": []byte("other"),
|
||||
"skipped.txt": []byte("skipped"),
|
||||
})
|
||||
env.setSnapshot("photos", "")
|
||||
env.backUp(t, "photos")
|
||||
|
||||
// An unchanged file is not read again, so the file is changed before
|
||||
// it becomes unreadable.
|
||||
skipped := filepath.Join(env.src, "skipped.txt")
|
||||
require.NoError(t, os.WriteFile(skipped, []byte("changed"), 0o600))
|
||||
env.v.Fs = &openFailFs{path: skipped, err: os.ErrPermission}
|
||||
|
||||
env.out.Reset()
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
Prune: true,
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"photos"},
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Skipped the post-backup prune because 1 file(s) could not be read")
|
||||
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 2, "the previous snapshot is kept")
|
||||
}
|
||||
|
||||
// A file removed during a backup under --skip-errors --prune is not a
|
||||
// skipped file, so the backup prunes as usual. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
||||
func TestSkipErrorsBackupPrunesWhenAFileIsRemovedDuringIt(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"other.txt": []byte("other"),
|
||||
"removed.txt": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("photos", "")
|
||||
env.backUp(t, "photos")
|
||||
|
||||
env.v.Fs = &removeOnLookupFs{path: filepath.Join(env.src, "removed.txt")}
|
||||
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
Prune: true,
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"photos"},
|
||||
}))
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 1, "the previous snapshot is pruned")
|
||||
}
|
||||
|
||||
@@ -13,7 +13,6 @@ import (
|
||||
"github.com/spf13/afero"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"golang.org/x/sys/unix"
|
||||
"sneak.berlin/go/vaultik/internal/config"
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
@@ -189,42 +188,6 @@ func (e *summaryEnv) dataLine(total, backedUp int64) string {
|
||||
e.v.UI.Size(total), e.v.UI.Size(backedUp))
|
||||
}
|
||||
|
||||
// enumeratedLine returns the line printed when the walk of a source path
|
||||
// ends.
|
||||
func (e *summaryEnv) enumeratedLine(
|
||||
files int, total int64, toBackUp int, toBackUpSize int64,
|
||||
) string {
|
||||
return fmt.Sprintf(
|
||||
"Enumerated %d snapshot source files (%s total), %d to back up (%s).",
|
||||
files, e.v.UI.Size(total), toBackUp, e.v.UI.Size(toBackUpSize))
|
||||
}
|
||||
|
||||
// snapshotPaths returns the paths the snapshot lists, sorted.
|
||||
func (e *summaryEnv) snapshotPaths(t *testing.T, snapshotID string) []string {
|
||||
t.Helper()
|
||||
|
||||
rows, err := e.db.Conn().QueryContext(context.Background(), `
|
||||
SELECT f.path FROM snapshot_files sf JOIN files f ON f.id = sf.file_id
|
||||
WHERE sf.snapshot_id = ? ORDER BY f.path`, snapshotID)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var paths []string
|
||||
|
||||
for rows.Next() {
|
||||
var path string
|
||||
|
||||
require.NoError(t, rows.Scan(&path))
|
||||
|
||||
paths = append(paths, path)
|
||||
}
|
||||
|
||||
require.NoError(t, rows.Err())
|
||||
|
||||
return paths
|
||||
}
|
||||
|
||||
// A first backup stores copy.bin's chunks while backing up a.bin, so
|
||||
// copy.bin's chunks are deduplicated within the run. Each file and byte
|
||||
// is still counted once.
|
||||
@@ -285,85 +248,6 @@ func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
|
||||
assert.Equal(t, uncompressed, snap.BlobUncompressedSize)
|
||||
}
|
||||
|
||||
// A backup of a tree with directories and a symlink, none of which
|
||||
// changed since the last backup, backs up nothing, and the snapshot
|
||||
// still lists every one of them. Both sides of the line that ends the
|
||||
// walk count the same entries, the files and the symlink, and no
|
||||
// directory's size. See https://git.eeqj.de/sneak/vaultik/issues/305.
|
||||
func TestSnapshotSummaryUnchangedRunWithDirectoriesAndSymlink(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newSummaryEnv(t)
|
||||
|
||||
const linkTarget = "a.bin"
|
||||
|
||||
require.NoError(t, os.Symlink(linkTarget,
|
||||
filepath.Join(filepath.Dir(env.aPath), "link")))
|
||||
|
||||
// A symlink's size is the length of its target.
|
||||
total := env.totalSize + int64(len(linkTarget))
|
||||
|
||||
first := env.backUp(t, "first", false)
|
||||
assert.Contains(t, first, env.enumeratedLine(4, total, 4, total))
|
||||
assert.Contains(t, first, filesLine(4, 4, 0))
|
||||
|
||||
second := env.backUp(t, "second", false)
|
||||
assert.Contains(t, second, env.enumeratedLine(4, total, 0, 0))
|
||||
assert.Contains(t, second, filesLine(4, 0, 4))
|
||||
assert.Contains(t, second, env.dataLine(total, 0))
|
||||
|
||||
assert.Equal(t,
|
||||
env.snapshotPaths(t, env.snapshot(t, "first").ID.String()),
|
||||
env.snapshotPaths(t, env.snapshot(t, "second").ID.String()))
|
||||
}
|
||||
|
||||
// A directory whose mtime changed and a symlink whose target changed are
|
||||
// recorded again by the next backup, which counts only the symlink as
|
||||
// backed up.
|
||||
func TestSnapshotSummaryChangedDirectoryAndSymlink(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newSummaryEnv(t)
|
||||
|
||||
// The local index keeps paths with symlinks resolved.
|
||||
dir, err := filepath.EvalSymlinks(filepath.Dir(env.aPath))
|
||||
require.NoError(t, err)
|
||||
|
||||
// The new link gets the old one's mtime, and a target, which need not
|
||||
// exist, of the old target's length, a symlink's size: only the
|
||||
// target differs.
|
||||
linkTime := unix.NsecToTimeval(
|
||||
time.Date(2029, time.March, 4, 5, 6, 7, 0, time.UTC).UnixNano())
|
||||
link := filepath.Join(dir, "link")
|
||||
require.NoError(t, os.Symlink("a.bin", link))
|
||||
require.NoError(t, unix.Lutimes(link, []unix.Timeval{linkTime, linkTime}))
|
||||
|
||||
env.backUp(t, "first", false)
|
||||
|
||||
require.NoError(t, os.Remove(link))
|
||||
require.NoError(t, os.Symlink("b.bin", link))
|
||||
require.NoError(t, unix.Lutimes(link, []unix.Timeval{linkTime, linkTime}))
|
||||
|
||||
mtime := time.Date(2030, time.January, 2, 3, 4, 5, 0, time.UTC)
|
||||
require.NoError(t, os.Chtimes(dir, mtime, mtime))
|
||||
|
||||
assert.Contains(t, env.backUp(t, "second", false), filesLine(4, 1, 3))
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
dirRow, err := env.repos.Files.GetByPath(ctx, dir)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, dirRow)
|
||||
assert.Equal(t, mtime.Unix(), dirRow.MTime.Unix())
|
||||
|
||||
linkRow, err := env.repos.Files.GetByPath(ctx, link)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, linkRow)
|
||||
assert.Equal(t, "b.bin", linkRow.LinkTarget.String())
|
||||
}
|
||||
|
||||
// Under --cron the progress reporter is off; the upload figures must
|
||||
// still reach the summary and the snapshots row. The snapshot has two
|
||||
// paths, each backed up by its own scan.
|
||||
|
||||
Reference in New Issue
Block a user