1 Commits
Author SHA1 Message Date
sneak 1ead24d193 Refuse a backup destination an unplugged volume left empty (closes #239)
check / check (push) Successful in 10m59s
Every backup writes vaultik-destination.txt at the destination root.
While the local index holds snapshots, a destination without it is
not present: snapshot create refuses before writing, prune and
snapshot purge fail, and snapshot list and snapshot remove act as for
a destination that cannot be listed. snapshot create checks before it
drops incomplete snapshots, snapshot remove before it looks on the
destination. remote nuke deletes the marker last. The file:// backend
never creates its destination directory; that protects a directory on
a removable volume, not a mount point.

Tests back up into directories that exist; fixtures for a written
destination carry the marker.

Judgement call: the marker is rewritten on every backup.
Deviation: for the complexity limit, two blocks moved, unchanged, into
snapshotNamesToCreate and confirmRemoveSnapshot.

Model: opus-5-5
2026-10-14 02:37:52 +00:00
21 changed files with 172 additions and 1234 deletions
+9 -10
View File
@@ -359,7 +359,7 @@ CreateSnapshot(opts)
## Deduplication Strategy ## Deduplication Strategy
1. **File-level**: Files and symlinks unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked 1. **File-level**: Files unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded. 2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded.
@@ -382,15 +382,14 @@ bucket/
└── manifest.json.zst # Blob list (for pruning/verification) └── manifest.json.zst # Blob list (for pruning/verification)
``` ```
A destination without `vaultik-destination.txt` is treated as not present While the local index holds snapshots, a destination without
while the local index holds snapshots stored there (`checkDestinationMarker` `vaultik-destination.txt` is treated as not present
in `internal/vaultik/destination_marker.go`): `snapshot create` refuses to (`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
write to it, `prune`, `snapshot purge` and `remote nuke` fail, and `snapshot create` refuses to write to it, and listing it fails as for a
`snapshot list` and `snapshot remove` behave as for a destination that cannot destination that cannot be listed. An unplugged volume's empty mount point
be listed. This keeps those commands from taking an unplugged volume's empty is therefore never taken for a destination whose snapshots are all gone.
mount point for a destination whose snapshots are all gone. The `file://` The `file://` backend never creates its destination directory, so a write
backend never creates its destination directory, so a write to a missing one to a missing one fails.
fails.
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
+14 -24
View File
@@ -182,7 +182,7 @@ vaultik version
* `--verbose`, `-v`: Enable verbose output (on stderr — see below) * `--verbose`, `-v`: Enable verbose output (on stderr — see below)
* `--debug`: Enable debug output (on stderr — see below) * `--debug`: Enable debug output (on stderr — see below)
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner) * `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run. A run that skipped anything still finishes but exits non-zero: `snapshot create` keeps the snapshot and counts each file or directory it skipped as failed. A file removed during the backup is also counted as failed but does not change the exit status. * `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run.
### locking ### locking
@@ -297,11 +297,7 @@ value where a list belongs or a word where a number belongs), is refused.
(for crontab) (for crontab)
* `--prune`: After backup, drop older snapshots of each backed-up name and * `--prune`: After backup, drop older snapshots of each backed-up name and
remove orphaned blobs from remote storage. By default keeps only the latest remove orphaned blobs from remote storage. By default keeps only the latest
snapshot per name; use `--keep-newer-than` for a rolling window. When snapshot per name; use `--keep-newer-than` for a rolling window.
`--skip-errors` skipped a file or directory it could not read or list, the
prune does not run and every older snapshot is kept, since one of them may
be the only one that holds what was skipped; the command says so and exits
non-zero. A file removed during the backup does not stop the prune.
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than * `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
Without `--prune`, or with a value that is not a valid duration, the Without `--prune`, or with a value that is not a valid duration, the
@@ -344,9 +340,9 @@ gives: revert the configuration, or run `vaultik database delete`.
If the destination store cannot be listed (unmounted volume, If the destination store cannot be listed (unmounted volume,
permission denied, network down), the command warns, falls back to the permission denied, network down), the command warns, falls back to the
local index alone, and still exits zero. It does the same when the local index alone, and still exits zero. It does the same when the
destination has no `vaultik-destination.txt` while the local index local index holds snapshots and the destination has no
holds snapshots stored there, as when a volume mounted at the `vaultik-destination.txt`, as when a volume mounted at the destination
destination is unplugged and leaves its empty mount point behind. is unplugged and leaves its empty mount point behind.
* `--json`: Output in JSON format. Each entry carries `locally_tracked` * `--json`: Output in JSON format. Each entry carries `locally_tracked`
(whether the snapshot is in the local index), `remote_key` (the full (whether the snapshot is in the local index), `remote_key` (the full
64-character storage key), and `remote_present` (whether it was seen 64-character storage key), and `remote_present` (whether it was seen
@@ -458,14 +454,9 @@ since `prune` never deletes it.
the backup destination store, whatever its name, and then the backup destination store, whatever its name, and then
`vaultik-destination.txt`, leaving the bucket prefix empty. Destructive `vaultik-destination.txt`, leaving the bucket prefix empty. Destructive
and irreversible. This is the single supported way to wipe the entire and irreversible. This is the single supported way to wipe the entire
destination store. A snapshot is removed from the local index only once destination store. If a listing afterwards still finds anything under
its metadata is deleted from the destination store. If the destination `metadata/` or `blobs/`, the command names each object left and exits
store cannot be listed, or the local index holds snapshots and the non-zero; running it again deletes what is left.
destination has no `vaultik-destination.txt` (an unplugged volume's
empty mount point), the command fails before it removes anything. If a
listing afterwards still finds anything under `metadata/` or `blobs/`,
the command names each object left and exits non-zero; running it again
deletes what is left.
* `--force`: Required to confirm destruction. * `--force`: Required to confirm destruction.
**`database delete`**: Delete the local SQLite state database file **`database delete`**: Delete the local SQLite state database file
@@ -536,13 +527,12 @@ For an implementation-level view of the internals — the data model, the
``` ```
* `vaultik-destination.txt` holds one fixed line of text, with no host * `vaultik-destination.txt` holds one fixed line of text, with no host
name, path or time. A destination without it is treated as not name, path or time. While the local index holds snapshots, a
present while the local index holds snapshots stored there: destination without it is treated as not present: `snapshot create`
`snapshot create` refuses to write to it, `prune`, `snapshot purge` refuses to write to it, `prune` and `snapshot purge` fail, and
and `remote nuke` fail, and `snapshot list` and `snapshot remove` `snapshot list` and `snapshot remove` behave as for a destination that
behave as for a destination that cannot be listed. This keeps an cannot be listed. This keeps an unplugged volume's empty mount point
unplugged volume's empty mount point from being taken for a from being taken for a destination whose snapshots are all gone. Mount
destination whose snapshots are all gone. Mount
the volume if it is removable; if the destination was emptied on the volume if it is removable; if the destination was emptied on
purpose, run `vaultik database delete` and back up again. purpose, run `vaultik database delete` and back up again.
* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash * Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
+4 -220
View File
@@ -15,229 +15,13 @@ pre-1.0
# Next Step # Next Step
The 1.0 work is complete on `next`: the scope settled on The 1.0 work is complete on `next`: the scope settled on
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) has landed, [issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) was the work
and so has every fix from the defect hunt before the release candidate already planned for 1.0, and all of it has landed. The mechanism to cut
([issue #288](https://git.eeqj.de/sneak/vaultik/issues/288)). What is the tag exists and is exercised; what is left is merging `next` to
left is merging `next` to `main` and tagging the release, both the `main` and tagging, both the owner's.
owner's; a `v*` tag runs the release build in
`.gitea/workflows/release.yml`.
# Completed Steps # Completed Steps
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
that skipped files it could not read
([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a
prune could delete the only snapshot still holding those files.
- 2026-10-14: Made `snapshot create --skip-errors` exit non-zero,
instead of 0, when it skips anything it cannot read
([issue #310](https://git.eeqj.de/sneak/vaultik/issues/310)).
- 2026-10-14: Kept the local index when `remote nuke` cannot list or
find the destination store
([issue #292](https://git.eeqj.de/sneak/vaultik/issues/292)). Nuke
had cleared the local snapshot records anyway; it now fails before
removing anything.
- 2026-10-14: Stopped counting directories and unchanged symlinks as
files to back up
([issue #305](https://git.eeqj.de/sneak/vaultik/issues/305)).
- 2026-10-14: Refused a backup destination that an unplugged volume
left empty, as the owner ruled
([issue #239](https://git.eeqj.de/sneak/vaultik/issues/239)). Every
backup now writes `vaultik-destination.txt` at the destination root,
and while the local index holds snapshots stored there, a destination
without it is treated as not present.
- 2026-10-14: Warned about restored ownership only when an entry's
owner or group differs from the recorded one
([issue #316](https://git.eeqj.de/sneak/vaultik/issues/316)); a
restore by a non-root user had warned even when none did.
- 2026-10-14: Made `snapshot list` report a changed destination instead
of advising `prune`, which refuses in that case
([issue #317](https://git.eeqj.de/sneak/vaultik/issues/317)).
- 2026-10-14: Counted a file as deleted by a backup only if the
previous snapshot of the same name listed it
([issue #304](https://git.eeqj.de/sneak/vaultik/issues/304)); files
that snapshot did not list had been counted too.
- 2026-10-14: Named each prefix once in restore, verify and purge
errors, some of which repeated it
([issue #327](https://git.eeqj.de/sneak/vaultik/issues/327)). A deep
verify log line that named S3 whatever the destination now names
storage.
- 2026-10-14: Refused credentials in a storage URL's endpoint or rclone
remote name, which were accepted and could then be printed
([issue #322](https://git.eeqj.de/sneak/vaultik/issues/322)).
- 2026-10-14: Ran `$EDITOR` through the shell in `config edit`, so a
value with arguments, such as `code --wait`, no longer fails
([issue #315](https://git.eeqj.de/sneak/vaultik/issues/315)).
- 2026-10-14: Made `snapshot remove` fail, instead of exiting 0, when
it cannot delete the metadata of a snapshot that is only on the
destination store
([issue #340](https://git.eeqj.de/sneak/vaultik/issues/340)). The
README and the help now say when it prints the `vaultik prune`
follow-up.
- 2026-10-14: Labelled the totals in the local index cleanup before
each backup as totals
([issue #306](https://git.eeqj.de/sneak/vaultik/issues/306)). Each
was shown as `(N remain)`, which read as rows left to clean up.
- 2026-10-13: Freed a cached blob once no file still to restore needs
it ([issue #324](https://git.eeqj.de/sneak/vaultik/issues/324)); one
also used by a file outside the selected paths had stayed in the
temporary cache until the restore ended.
- 2026-10-13: Kept the whole startup error when it contains `): `
([issue #320](https://git.eeqj.de/sneak/vaultik/issues/320)). Such
an error was cut at its last `): `, so the `blob_size_limit` one lost
the setting's name.
- 2026-10-13: Silenced the status lines of `prune` and snapshot removal
under `--cron` and `--quiet`
([issue #294](https://git.eeqj.de/sneak/vaultik/issues/294)).
- 2026-10-13: Read the rclone config file that `RCLONE_CONFIG` names
([issue #321](https://git.eeqj.de/sneak/vaultik/issues/321)); a
remote defined only there had been reported as not found.
- 2026-10-13: Made `/**/` in an exclude pattern match zero directories,
as in gitignore
([issue #338](https://git.eeqj.de/sneak/vaultik/issues/338));
`a/**/*.log` had missed `a/x.log`.
- 2026-10-13: Made the `chunk_size` and `blob_size_limit` defaults and
the `chunk_size` minimum 1000-based, like the size parser and the
docs ([issue #313](https://git.eeqj.de/sneak/vaultik/issues/313)).
- 2026-10-13: Kept a snapshot's remote metadata when `snapshot purge`
cannot remove it from the local index
([issue #337](https://git.eeqj.de/sneak/vaultik/issues/337)); the
purge had deleted the metadata anyway and exited 0.
- 2026-10-13: Made `snapshot restore` fail for a requested path that
matches nothing, which it had ignored
([issue #309](https://git.eeqj.de/sneak/vaultik/issues/309)).
- 2026-10-13: Made `remote nuke` delete every object under `metadata/`
and `blobs/`
([issue #291](https://git.eeqj.de/sneak/vaultik/issues/291)); it had
left some, such as a stray `.DS_Store`, while reporting the store
empty.
- 2026-10-13: Made `config set` create a list for an index under a
missing key
([issue #308](https://git.eeqj.de/sneak/vaultik/issues/308)); it had
written a map, after which no command could load the config.
- 2026-10-13: Listed a `file://` destination through a symlinked
`metadata/` or `blobs/`, which had listed as empty
([issue #296](https://git.eeqj.de/sneak/vaultik/issues/296)). A link
to a missing target now fails the listing
([issue #332](https://git.eeqj.de/sneak/vaultik/issues/332)).
- 2026-10-13: Made `snapshot verify --deep --json` report its blob
counts ([issue #314](https://git.eeqj.de/sneak/vaultik/issues/314));
a failed run had reported 0 missing and 0 verified.
- 2026-10-13: Limited and retried the module download in the Docker
build ([issue #351](https://git.eeqj.de/sneak/vaultik/issues/351)),
which had hung for as long as a stalled connection to the module
proxy lasted.
- 2026-10-13: Recorded each snapshot's uncompressed size when its
backup runs
([issue #360](https://git.eeqj.de/sneak/vaultik/issues/360)). A later
backup could change an older snapshot's UNCOMPRESSED SIZE in
`snapshot list`.
- 2026-10-13: Stopped `snapshot restore --skip-errors --verify`
reporting a file that failed to restore again as a checksum failure
([issue #323](https://git.eeqj.de/sneak/vaultik/issues/323)).
- 2026-10-13: Recorded the stored size of a blob a backup finds already
stored, not the size of its local copy, which can differ, so
`snapshot verify` could fail on a good snapshot
([issue #300](https://git.eeqj.de/sneak/vaultik/issues/300)).
- 2026-10-13: Stopped the local index and the blob upload step ending
the process on a failed close of query results or a failed temp file
cleanup ([issue #345](https://git.eeqj.de/sneak/vaultik/issues/345));
both are now returned as errors.
- 2026-10-13: Closed the snapshot database's storage reader too when
restore or deep verify is interrupted
([issue #356](https://git.eeqj.de/sneak/vaultik/issues/356)).
- 2026-10-13: Made `prune` and `remote nuke` fail when a delete on the
backup destination store fails
([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)).
- 2026-10-13: Made an S3 listing that the server cuts short an error
([issue #297](https://git.eeqj.de/sneak/vaultik/issues/297)); `prune`
had taken it as complete and could delete blobs still in use.
- 2026-10-13: Made shallow `snapshot verify` report a blob it could not
check apart from a missing one
([issue #318](https://git.eeqj.de/sneak/vaultik/issues/318)); any
error checking a blob had counted as missing, its cause dropped.
- 2026-10-13: Restored the other files when a file's chunk is in no
blob under `--skip-errors`
([issue #326](https://git.eeqj.de/sneak/vaultik/issues/326)); that
file had stopped the restore before anything was restored.
- 2026-10-13: Recorded each snapshot's new chunk size when its backup
runs ([issue #311](https://git.eeqj.de/sneak/vaultik/issues/311)).
`snapshot list` could show a NEW CHUNK SIZE of 0 B for a snapshot
whose change was a modified file.
- 2026-10-13: Made a backup interrupted during a local index write exit
as interrupted
([issue #301](https://git.eeqj.de/sneak/vaultik/issues/301)); it could
end with `FATAL: failed to rollback transaction` and exit 1 instead of
130.
- 2026-10-13: Printed `snapshot list` and `snapshot purge` sizes like
every other command
([issue #312](https://git.eeqj.de/sneak/vaultik/issues/312)). They,
and the `bytes` value in terminal log lines, were 1024-based but
labelled KB or MB, and now use 1000-based units.
- 2026-10-13: Closed each blob's storage reader when restore or deep
verify is interrupted, which ends a stalled read unless it is stuck in
the kernel ([issue #325](https://git.eeqj.de/sneak/vaultik/issues/325)).
- 2026-10-13: Made a mistyped subcommand under `snapshot`, `config`,
`remote`, `database` or `completion` an error
([issue #319](https://git.eeqj.de/sneak/vaultik/issues/319)); each had
printed its help and exited 0.
- 2026-10-13: Made an exclude pattern starting with `**/` match
directly under the snapshot path too, as in gitignore
([issue #303](https://git.eeqj.de/sneak/vaultik/issues/303)).
- 2026-10-13: Refused a relative `file://` storage URL and decoded its
percent-escapes
([issue #299](https://git.eeqj.de/sneak/vaultik/issues/299)); a
relative path had moved with the working directory.
- 2026-10-13: Removed the blob temp file when a backup stops part-way
([issue #302](https://git.eeqj.de/sneak/vaultik/issues/302)); the
half-written blob had stayed in `$TMPDIR`.
- 2026-10-13: Checked `--keep-newer-than` before a backup starts
([issue #307](https://git.eeqj.de/sneak/vaultik/issues/307)); an
invalid value had been read only by the prune after the backup, so
retention never ran.
- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that - 2026-10-13: Stopped `remote info` counting a file under `blobs/` that
is not a blob as an orphaned blob is not a blob as an orphaned blob
([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray ([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray
+1 -1
View File
@@ -27,7 +27,7 @@ The metadata subdirectory is named with the **remote key**, a one-way hash of th
## Marker File (`vaultik-destination.txt`) ## Marker File (`vaultik-destination.txt`)
Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. A destination without it is treated as not present while the local index holds snapshots stored there: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last. Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. While the local index holds snapshots, a destination without it is treated as not present: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove` and `snapshot list` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
## Blobs Directory (`blobs/`) ## Blobs Directory (`blobs/`)
@@ -1,30 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"testing"
"github.com/stretchr/testify/assert"
)
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
// under --skip-errors, a source directory that does not exist, which the
// walk skips. The command exits non-zero and reports the backup as
// completed, not as a failed snapshot creation. See
// https://git.eeqj.de/sneak/vaultik/issues/310.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
// writeBackupConfig would create the source directory.
configPath := writeHermeticPruneConfig(t, false)
code, stdout, stderr := runEntry(t,
flagConfig, configPath, "--skip-errors", cmdSnapshot, cmdCreate)
assert.Equal(t, 1, code)
assert.Contains(t, stdout,
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
assert.NotContains(t, stdout, "Finished successfully.")
assert.Contains(t, stderr,
"Backup completed, but 1 file(s) could not be read")
assert.NotContains(t, stderr, "Snapshot creation failed")
}
+1 -9
View File
@@ -115,13 +115,6 @@ on Linux), then /etc/vaultik/config.yml.`,
return return
} }
if errors.Is(err, vaultik.ErrFilesUnreadable) {
log.Error("Backup completed with unreadable files", "error", err)
ReportErrorf("Backup completed, but %v", err)
return
}
log.Error("Snapshot creation failed", "error", err) log.Error("Snapshot creation failed", "error", err)
ReportErrorf("Snapshot creation failed: %v", err) ReportErrorf("Snapshot creation failed: %v", err)
}) })
@@ -132,8 +125,7 @@ on Linux), then /etc/vaultik/config.yml.`,
"Run in cron mode (silent unless warning or error)") "Run in cron mode (silent unless warning or error)")
cmd.Flags().BoolVar(&opts.Prune, "prune", false, cmd.Flags().BoolVar(&opts.Prune, "prune", false,
"After backup, drop older snapshots of the same name and remove "+ "After backup, drop older snapshots of the same name and remove "+
"orphaned blobs (not after a backup that skipped unreadable "+ "orphaned blobs")
"files)")
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "", cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
"With --prune: keep snapshots newer than this duration "+ "With --prune: keep snapshots newer than this duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+ "(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
+2 -1
View File
@@ -78,7 +78,8 @@ func TestProgressPercentWithSecondPathSmaller(t *testing.T) {
stats := progress.GetStats() stats := progress.GetStats()
// With both paths counted, the percentage must not exceed 100%. // Directories count toward the total size but produce no chunks, so
// the percentage ends just under 100%.
percent := float64(stats.BytesProcessed.Load()) / percent := float64(stats.BytesProcessed.Load()) /
float64(stats.TotalSize.Load()) * 100 float64(stats.TotalSize.Load()) * 100
if percent > 100 { if percent > 100 {
+88 -167
View File
@@ -141,12 +141,10 @@ type ScannerConfig struct {
// ScanResult contains the results of a scan operation. Files and bytes // ScanResult contains the results of a scan operation. Files and bytes
// are counted per file: BytesScanned is the size of the new and changed // are counted per file: BytesScanned is the size of the new and changed
// files, BytesSkipped that of the unchanged ones. FilesFailed counts the // files, BytesSkipped that of the unchanged ones. FilesFailed counts the
// new and changed files that phase 2 could not store, each entry removed // new and changed files that phase 2 could not store; FilesScanned
// after the walk listed its directory and, under --skip-errors, each // includes them and BytesScanned does not. ChunksCreated counts the
// entry the walk could not read, such as a directory it could not list; // chunks that were not already stored, and NewChunkSize is their total
// FilesScanned includes them and BytesScanned does not. ChunksCreated // size.
// counts the chunks that were not already stored, and NewChunkSize is
// their total size.
type ScanResult struct { type ScanResult struct {
FilesScanned int FilesScanned int
FilesSkipped int FilesSkipped int
@@ -163,11 +161,6 @@ type ScanResult struct {
UploadDuration time.Duration UploadDuration time.Duration
StartTime time.Time StartTime time.Time
EndTime time.Time EndTime time.Time
// FilesUnreadable is the part of FilesFailed that --skip-errors
// skipped because it could not be read or listed; the rest are files
// removed after the walk listed them.
FilesUnreadable int
} }
// NewScanner creates a new scanner instance // NewScanner creates a new scanner instance
@@ -291,11 +284,6 @@ func (s *Scanner) Scan(
} }
} }
err = s.recordDirectories(ctx, scanResult.Directories)
if err != nil {
return nil, err
}
// Summarize scan phase results and update progress // Summarize scan phase results and update progress
s.summarizeScanPhase(result, filesToProcess) s.summarizeScanPhase(result, filesToProcess)
@@ -867,15 +855,14 @@ func (s *Scanner) batchInsertFileData(
// ScanPhaseResult contains the results of the scan phase // ScanPhaseResult contains the results of the scan phase
type ScanPhaseResult struct { type ScanPhaseResult struct {
FilesToProcess []*FileToProcess FilesToProcess []*FileToProcess
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
Directories []*FileToProcess // Every directory found, to record
} }
// scanPhase performs the initial directory scan to identify files to process // scanPhase performs the initial directory scan to identify files to process
// It uses the pre-loaded knownFiles map for fast change detection without DB // It uses the pre-loaded knownFiles map for fast change detection without DB
// queries. It also populates existingFiles map for deletion detection. // queries. It also populates existingFiles map for deletion detection.
// Returns files needing processing, IDs of unchanged files for snapshot // Returns files needing processing and IDs of unchanged files for snapshot
// association, and the directories found. // association.
func (s *Scanner) scanPhase( func (s *Scanner) scanPhase(
ctx context.Context, ctx context.Context,
path string, path string,
@@ -883,10 +870,8 @@ func (s *Scanner) scanPhase(
existingFiles map[string]struct{}, existingFiles map[string]struct{},
knownFiles map[string]*database.File, knownFiles map[string]*database.File,
) (*ScanPhaseResult, error) { ) (*ScanPhaseResult, error) {
// Use known file count as estimate for progress (accurate for // Use known file count as estimate for progress (accurate for subsequent backups)
// subsequent backups). The walk does not count directories, so the estimatedTotal := int64(len(knownFiles))
// estimate leaves them out.
estimatedTotal := countNonDirectories(knownFiles)
collector := &scanCollector{} collector := &scanCollector{}
@@ -897,28 +882,8 @@ func (s *Scanner) scanPhase(
log.Debug("Starting directory walk", "path", path) log.Debug("Starting directory walk", "path", path)
walkFn := func(filePath string, info os.FileInfo, err error) error { walkFn := func(filePath string, info os.FileInfo, err error) error {
// Check exclude patterns first, so an excluded entry that is gone
// or cannot be looked up is neither counted nor reported. For
// directories, skip the entire subtree. info is nil when the
// walk could not look the entry up.
if s.shouldExclude(filePath, path) {
if info != nil && info.IsDir() {
return filepath.SkipDir
}
return nil
}
// The source path is in no directory listing, so its absence is
// an error rather than a removal.
if errors.Is(err, os.ErrNotExist) && filePath != path {
countRemovedDuringWalk(filePath, existingFiles, result)
return nil
}
if err != nil { if err != nil {
return s.handleWalkError(filePath, err, result) return s.handleWalkError(filePath, err)
} }
// Check context cancellation // Check context cancellation
@@ -928,9 +893,18 @@ func (s *Scanner) scanPhase(
default: default:
} }
// Check exclude patterns - for directories, skip the entire subtree
if s.shouldExclude(filePath, path) {
if info.IsDir() {
return filepath.SkipDir
}
return nil
}
// Handle symlinks and directories // Handle symlinks and directories
handled, err := s.recordSpecialEntry( handled, err := s.recordSpecialEntry(
filePath, info, existingFiles, knownFiles, collector, result) filePath, info, existingFiles, collector, result)
if handled { if handled {
return err return err
} }
@@ -944,10 +918,9 @@ func (s *Scanner) scanPhase(
existingFiles[filePath] = struct{}{} existingFiles[filePath] = struct{}{}
// Check file against in-memory map (no DB query!) // Check file against in-memory map (no DB query!)
file := s.buildRegularFileEntry(filePath, info) file, needsProcessing := s.checkFileInMemory(filePath, info, knownFiles)
needsProcessing := s.checkFileInMemory(file, knownFiles)
filesScanned, changedCount := collector.recordFile( filesScanned, changedCount := collector.recordRegularFile(
filePath, info, file, needsProcessing) filePath, info, file, needsProcessing)
// Update result stats // Update result stats
@@ -971,45 +944,27 @@ func (s *Scanner) scanPhase(
return &ScanPhaseResult{ return &ScanPhaseResult{
FilesToProcess: collector.filesToProcess, FilesToProcess: collector.filesToProcess,
UnchangedFileIDs: collector.unchangedFileIDs, UnchangedFileIDs: collector.unchangedFileIDs,
Directories: collector.directories,
}, nil }, nil
} }
// countNonDirectories returns how many of files are not directories.
func countNonDirectories(files map[string]*database.File) int64 {
var count int64
for _, f := range files {
if f.Mode&uint32(os.ModeDir) == 0 {
count++
}
}
return count
}
// recordSpecialEntry records symlinks and directories (which have no // recordSpecialEntry records symlinks and directories (which have no
// data to chunk) and reports whether it handled the entry. A symlink is // data to chunk) and reports whether it handled the entry. For a symlink
// counted and checked for changes as a regular file is; a directory is // whose target cannot be read it returns handleWalkError's result.
// set aside to be recorded and is not counted. For a symlink whose
// target cannot be read it returns handleWalkError's result.
func (s *Scanner) recordSpecialEntry( func (s *Scanner) recordSpecialEntry(
filePath string, info os.FileInfo, filePath string, info os.FileInfo,
existingFiles map[string]struct{}, existingFiles map[string]struct{},
knownFiles map[string]*database.File,
collector *scanCollector, result *ScanResult, collector *scanCollector, result *ScanResult,
) (bool, error) { ) (bool, error) {
// Handle symlinks // Handle symlinks
if info.Mode()&os.ModeSymlink != 0 { if info.Mode()&os.ModeSymlink != 0 {
file, err := s.buildSymlinkEntry(filePath, info) file, err := s.buildSymlinkEntry(filePath, info)
if err != nil { if err != nil {
return true, s.handleWalkError(filePath, err, result) return true, s.handleWalkError(filePath, err)
} }
existingFiles[filePath] = struct{}{} existingFiles[filePath] = struct{}{}
needsProcessing := s.checkFileInMemory(file, knownFiles) collector.addToProcess(filePath, info, file)
collector.recordFile(filePath, info, file, needsProcessing) s.updateScanEntryStats(result, true, info)
s.updateScanEntryStats(result, needsProcessing, info)
return true, nil return true, nil
} }
@@ -1019,7 +974,7 @@ func (s *Scanner) recordSpecialEntry(
if info.IsDir() { if info.IsDir() {
file := s.buildDirectoryEntry(filePath, info) file := s.buildDirectoryEntry(filePath, info)
existingFiles[filePath] = struct{}{} existingFiles[filePath] = struct{}{}
collector.addDirectory(filePath, info, file) collector.addToProcess(filePath, info, file)
return true, nil return true, nil
} }
@@ -1027,38 +982,15 @@ func (s *Scanner) recordSpecialEntry(
return false, nil return false, nil
} }
// countRemovedDuringWalk counts an entry that was in its directory's
// listing but was gone when the walk looked it up, like a file removed
// before phase 2 reads it: as examined and failed, and not as an error,
// with or without --skip-errors.
func countRemovedDuringWalk(
filePath string, existingFiles map[string]struct{}, result *ScanResult,
) {
log.Warn("File was deleted during the directory walk, skipping",
"path", filePath)
// The walk found it, so it is not also counted as deleted.
existingFiles[filePath] = struct{}{}
result.FilesScanned++
result.FilesFailed++
}
// handleWalkError deals with a filesystem error surfaced by the walk: // handleWalkError deals with a filesystem error surfaced by the walk:
// under --skip-errors the entry is skipped with a UI notice and counted // skipped (with a UI notice) under --skip-errors, fatal otherwise.
// as examined, failed and unreadable, otherwise the error is fatal. func (s *Scanner) handleWalkError(filePath string, err error) error {
func (s *Scanner) handleWalkError(
filePath string, err error, result *ScanResult,
) error {
if s.skipErrors { if s.skipErrors {
log.Error("Failed to access file (skipping due to --skip-errors)", log.Error("Failed to access file (skipping due to --skip-errors)",
"path", filePath, "error", err) "path", filePath, "error", err)
s.ui.Errorf("Failed to access %s: %v. Skipping (--skip-errors).", s.ui.Errorf("Failed to access %s: %v. Skipping (--skip-errors).",
s.ui.Path(filePath), err) s.ui.Path(filePath), err)
result.FilesScanned++
result.FilesFailed++
result.FilesUnreadable++
return nil // Continue scanning return nil // Continue scanning
} }
@@ -1072,29 +1004,28 @@ type scanCollector struct {
mu sync.Mutex mu sync.Mutex
filesToProcess []*FileToProcess filesToProcess []*FileToProcess
unchangedFileIDs []types.FileID // Just IDs - no new records needed unchangedFileIDs []types.FileID // Just IDs - no new records needed
directories []*FileToProcess
filesScanned int64 filesScanned int64
} }
// addDirectory sets a directory aside to be recorded. It is not counted // addToProcess queues a file entry for processing.
// as a file. func (c *scanCollector) addToProcess(
func (c *scanCollector) addDirectory(
path string, info os.FileInfo, file *database.File, path string, info os.FileInfo, file *database.File,
) { ) {
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()
c.directories = append(c.directories, &FileToProcess{ c.filesToProcess = append(c.filesToProcess, &FileToProcess{
Path: path, Path: path,
FileInfo: info, FileInfo: info,
File: file, File: file,
}) })
c.filesScanned++
} }
// recordFile records a regular file or symlink as changed (to process) // recordRegularFile records a regular file as changed (to process) or
// or unchanged (snapshot association only) and returns the running // unchanged (snapshot association only) and returns the running scanned
// scanned and changed counts. // and changed counts.
func (c *scanCollector) recordFile( func (c *scanCollector) recordRegularFile(
path string, info os.FileInfo, file *database.File, needsProcessing bool, path string, info os.FileInfo, file *database.File, needsProcessing bool,
) (int64, int) { ) (int64, int) {
c.mu.Lock() c.mu.Lock()
@@ -1197,28 +1128,7 @@ func (s *Scanner) printScanProgressLine(
} }
} }
// buildRegularFileEntry creates a File record for a regular file, with // buildSymlinkEntry creates a File record for a symlink.
// no ID: checkFileInMemory sets it.
func (s *Scanner) buildRegularFileEntry(path string, info os.FileInfo) *database.File {
var uid, gid uint32
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
uid = stat.Uid
gid = stat.Gid
}
return &database.File{
Path: types.FilePath(path),
SourcePath: types.SourcePath(s.currentSourcePath),
MTime: info.ModTime(),
Size: info.Size(),
Mode: uint32(info.Mode()),
UID: uid,
GID: gid,
}
}
// buildSymlinkEntry creates a File record for a symlink, with no ID:
// checkFileInMemory sets it.
func (s *Scanner) buildSymlinkEntry( func (s *Scanner) buildSymlinkEntry(
path string, info os.FileInfo, path string, info os.FileInfo,
) (*database.File, error) { ) (*database.File, error) {
@@ -1234,6 +1144,7 @@ func (s *Scanner) buildSymlinkEntry(
} }
return &database.File{ return &database.File{
ID: types.NewFileID(),
Path: types.FilePath(path), Path: types.FilePath(path),
SourcePath: types.SourcePath(s.currentSourcePath), SourcePath: types.SourcePath(s.currentSourcePath),
MTime: info.ModTime(), MTime: info.ModTime(),
@@ -1265,22 +1176,6 @@ func (s *Scanner) buildDirectoryEntry(path string, info os.FileInfo) *database.F
} }
} }
// recordDirectories records every directory the walk found. A directory
// has no data to back up, so it is not among the files counted and
// backed up.
func (s *Scanner) recordDirectories(
ctx context.Context, directories []*FileToProcess,
) error {
for _, dir := range directories {
err := s.recordNonRegularFile(ctx, dir)
if err != nil {
return fmt.Errorf("recording directory %s: %w", dir.Path, err)
}
}
return nil
}
// recordNonRegularFile writes a symlink or directory entry to the database // recordNonRegularFile writes a symlink or directory entry to the database
// and associates it with the current snapshot. No chunking is performed. // and associates it with the current snapshot. No chunking is performed.
func (s *Scanner) recordNonRegularFile(ctx context.Context, ftp *FileToProcess) error { func (s *Scanner) recordNonRegularFile(ctx context.Context, ftp *FileToProcess) error {
@@ -1294,36 +1189,62 @@ func (s *Scanner) recordNonRegularFile(ctx context.Context, ftp *FileToProcess)
}) })
} }
// checkFileInMemory reports whether a regular file or symlink needs // checkFileInMemory checks if a file needs processing using the in-memory map
// processing, and gives file the ID of its known record, or a new ID for
// a new path.
// No database access is performed - this is purely CPU/memory work // No database access is performed - this is purely CPU/memory work
func (s *Scanner) checkFileInMemory( func (s *Scanner) checkFileInMemory(
file *database.File, knownFiles map[string]*database.File, path string, info os.FileInfo, knownFiles map[string]*database.File,
) bool { ) (*database.File, bool) {
existingFile, exists := knownFiles[file.Path.String()] var uid, gid uint32
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
uid = stat.Uid
gid = stat.Gid
}
// Check against in-memory map first to get existing ID if available
existingFile, exists := knownFiles[path]
// Create file record with ID set upfront
// For new files, generate UUID immediately so it's available for chunk associations
// For existing files, reuse the existing ID
var fileID types.FileID
if exists {
fileID = existingFile.ID
} else {
fileID = types.NewFileID()
}
file := &database.File{
ID: fileID,
Path: types.FilePath(path),
SourcePath: types.SourcePath(s.currentSourcePath),
MTime: info.ModTime(),
Size: info.Size(),
Mode: uint32(info.Mode()),
UID: uid,
GID: gid,
}
// New file - needs processing // New file - needs processing
if !exists { if !exists {
file.ID = types.NewFileID() return file, true
return true
} }
file.ID = existingFile.ID
// No uploaded blob holds one of its chunks (see loadFilesToRechunk) // No uploaded blob holds one of its chunks (see loadFilesToRechunk)
if _, rechunk := s.filesToRechunk[file.ID]; rechunk { if _, rechunk := s.filesToRechunk[fileID]; rechunk {
return true return file, true
} }
// Check if file has changed // Check if file has changed
return existingFile.Size != file.Size || if existingFile.Size != file.Size ||
!existingFile.MTime.Equal(file.MTime) || !existingFile.MTime.Equal(file.MTime) ||
existingFile.Mode != file.Mode || existingFile.Mode != file.Mode ||
existingFile.UID != file.UID || existingFile.UID != file.UID ||
existingFile.GID != file.GID || existingFile.GID != file.GID {
existingFile.LinkTarget != file.LinkTarget return file, true
}
// File unchanged
return file, false
} }
// batchAddFilesToSnapshot adds existing file IDs to the snapshot association table // batchAddFilesToSnapshot adds existing file IDs to the snapshot association table
@@ -1450,16 +1371,17 @@ func (s *Scanner) processPhase(
return s.finalizeProcessPhase(ctx, result) return s.finalizeProcessPhase(ctx, result)
} }
// processFileWithErrorHandling records a symlink, or wraps // processFileWithErrorHandling records a directory or symlink, or wraps
// processFileStreaming for a regular file with error recovery for deleted // processFileStreaming for a regular file with error recovery for deleted
// files and skip-errors mode. Returns (skipped, error). // files and skip-errors mode. Returns (skipped, error).
func (s *Scanner) processFileWithErrorHandling( func (s *Scanner) processFileWithErrorHandling(
ctx context.Context, fileToProcess *FileToProcess, result *ScanResult, ctx context.Context, fileToProcess *FileToProcess, result *ScanResult,
) (bool, error) { ) (bool, error) {
// A symlink has no data to open or read; it is only recorded in the // A directory or symlink has no data to open or read; it is only
// local index. An error recording it stops the run even under // recorded in the local index. An error recording it stops the run
// --skip-errors, or the snapshot would complete without it. // even under --skip-errors, or the snapshot would complete without it.
if os.FileMode(fileToProcess.File.Mode)&os.ModeSymlink != 0 { mode := os.FileMode(fileToProcess.File.Mode)
if mode&os.ModeSymlink != 0 || mode.IsDir() {
err := s.recordNonRegularFile(ctx, fileToProcess) err := s.recordNonRegularFile(ctx, fileToProcess)
if err != nil { if err != nil {
return false, fmt.Errorf("processing file %s: %w", fileToProcess.Path, err) return false, fmt.Errorf("processing file %s: %w", fileToProcess.Path, err)
@@ -1500,7 +1422,6 @@ func (s *Scanner) processFileWithErrorHandling(
s.ui.Path(fileToProcess.Path), err) s.ui.Path(fileToProcess.Path), err)
countFailedFile(fileToProcess, result) countFailedFile(fileToProcess, result)
result.FilesUnreadable++
return true, nil return true, nil
} }
@@ -1,28 +0,0 @@
//nolint:testpackage // exercises the unexported countNonDirectories helper
package snapshot
import (
"os"
"testing"
"sneak.berlin/go/vaultik/internal/database"
)
// The walk counts regular files and symlinks, not directories, so the
// estimate of its count made from the local index's records leaves the
// directories out. See https://git.eeqj.de/sneak/vaultik/issues/305.
func TestCountNonDirectories(t *testing.T) {
t.Parallel()
known := map[string]*database.File{
"/src": {Mode: uint32(os.ModeDir | 0o755)},
"/src/sub": {Mode: uint32(os.ModeDir | 0o755)},
"/src/a.txt": {Mode: 0o644},
"/src/link": {Mode: uint32(os.ModeSymlink | 0o777), LinkTarget: "a.txt"},
}
got := countNonDirectories(known)
if got != 2 {
t.Errorf("countNonDirectories = %d, want 2", got)
}
}
+8 -24
View File
@@ -9,10 +9,10 @@ import (
) )
// destinationMarkerKey is the object every backup writes at the root of // destinationMarkerKey is the object every backup writes at the root of
// the backup destination store. While the local index holds a snapshot // the backup destination store. While the local index holds a snapshot,
// stored there, a destination without it is treated as not present: an // a destination without it is treated as not present: an unplugged
// unplugged volume leaves an empty directory at its mount point, which // volume leaves an empty directory at its mount point, which would
// would otherwise read as a store whose snapshots are all gone. // otherwise read as a store whose snapshots are all gone.
const destinationMarkerKey = "vaultik-destination.txt" const destinationMarkerKey = "vaultik-destination.txt"
// destinationMarkerText is the whole content of the marker. It names no // destinationMarkerText is the whole content of the marker. It names no
@@ -20,16 +20,13 @@ const destinationMarkerKey = "vaultik-destination.txt"
const destinationMarkerText = "This is a vaultik backup destination store.\n" const destinationMarkerText = "This is a vaultik backup destination store.\n"
// errDestinationNotPresent is returned when the local index holds a // errDestinationNotPresent is returned when the local index holds a
// snapshot stored on the backup destination store and the store has no // snapshot and the backup destination store has no marker.
// marker.
var errDestinationNotPresent = errors.New("backup destination store is not present") var errDestinationNotPresent = errors.New("backup destination store is not present")
// checkDestinationMarker returns errDestinationNotPresent when the local // checkDestinationMarker returns errDestinationNotPresent when the local
// index holds a snapshot stored on the backup destination store and the // index holds a snapshot and the backup destination store has no marker.
// store has no marker. With no snapshot in the local index, as on a // With no snapshot in the local index, as on a first backup or on
// first backup or on another machine, there is nothing to check; nor // another machine, there is nothing to check.
// when the local index is bound to a different destination than the
// configured one, since its snapshots are stored there.
func (v *Vaultik) checkDestinationMarker() error { func (v *Vaultik) checkDestinationMarker() error {
ids, err := v.localSnapshotIDs() ids, err := v.localSnapshotIDs()
if err != nil { if err != nil {
@@ -40,19 +37,6 @@ func (v *Vaultik) checkDestinationMarker() error {
return nil return nil
} }
// Config is nil only in tests, which then record no binding (see
// EnsureStorageBinding).
if v.Config != nil {
stored, configured, err := v.storageBinding()
if err != nil {
return err
}
if stored != "" && stored != configured {
return nil
}
}
_, err = v.Storage.Stat(v.ctx, destinationMarkerKey) _, err = v.Storage.Stat(v.ctx, destinationMarkerKey)
if errors.Is(err, storage.ErrNotFound) { if errors.Is(err, storage.ErrNotFound) {
return fmt.Errorf("%w: %s has no %s, but the local index holds "+ return fmt.Errorf("%w: %s has no %s, but the local index holds "+
@@ -12,7 +12,6 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
@@ -152,25 +151,6 @@ func TestPurgeRefusesDestinationWithoutMarker(t *testing.T) {
assertUntouched(ctx, t, repos, storeDir) assertUntouched(ctx, t, repos, storeDir)
} }
// TestNukeRemoteRefusesDestinationWithoutMarker checks that remote nuke
// fails, deletes no local snapshot record, and neither reports the
// destination empty nor advises running nuke again when the destination
// has no marker.
func TestNukeRemoteRefusesDestinationWithoutMarker(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
v, repos, out, storeDir := backUpThenUnplugMountPoint(ctx, t)
err := v.NukeRemote(true)
require.ErrorContains(t, err, markerFile)
assert.NotContains(t, out.String(), "now empty")
assert.NotContains(t, out.String(), "vaultik remote nuke --force")
assertUntouched(ctx, t, repos, storeDir)
}
// TestRemoveSnapshotWarnsWhenDestinationHasNoMarker checks that snapshot // TestRemoveSnapshotWarnsWhenDestinationHasNoMarker checks that snapshot
// remove warns that the metadata could not be removed from a destination // remove warns that the metadata could not be removed from a destination
// without a marker, instead of reporting that it was. // without a marker, instead of reporting that it was.
@@ -241,44 +221,6 @@ func TestListSnapshotsWarnsWhenDestinationHasNoMarker(t *testing.T) {
assert.NotContains(t, out.String(), "not found in backup destination store") assert.NotContains(t, out.String(), "not found in backup destination store")
} }
// TestListSnapshotsAfterStorageURLChangeToEmptyDirectory checks that
// snapshot list, after storage_url is changed to an empty directory,
// warns only that the local index is bound to a different destination:
// the local snapshots are stored at the old one, so the new one's lack
// of a marker says nothing. With --json the local row is not on the new
// destination, so remote_present is false, not unknown.
func TestListSnapshotsAfterStorageURLChangeToEmptyDirectory(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
v, repos, out := backUpToFileDestination(ctx, t, t.TempDir())
id := localSnapshotID(ctx, t, repos, "first")
newDir := t.TempDir()
store, err := storage.NewFileStorer(newDir)
require.NoError(t, err)
v.Storage = store
v.Config.StorageURL = "file://" + newDir
require.NoError(t, v.ListSnapshots(false))
assert.Contains(t, out.String(), id)
assert.Contains(t, out.String(),
"local index is bound to a different backup destination")
assert.NotContains(t, out.String(), "Could not list backup destination store")
assert.NotContains(t, out.String(), markerFile)
out.Reset()
require.NoError(t, v.ListSnapshots(true))
rows := decodeListJSON(t, out.String())
require.Len(t, rows, 1)
require.NotNil(t, rows[0].RemotePresent)
assert.False(t, *rows[0].RemotePresent)
}
// TestBackupAfterNukeRemoteSucceeds checks that remote nuke leaves the // TestBackupAfterNukeRemoteSucceeds checks that remote nuke leaves the
// destination ready for a new first backup, which writes the marker // destination ready for a new first backup, which writes the marker
// again. // again.
@@ -188,41 +188,3 @@ func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"), assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
err.Error()) err.Error())
} }
// TestNukeRemoteKeepsLocalIndexWhenDestinationMissing checks that remote
// nuke fails on a destination it cannot list, leaves the local index
// unchanged, and neither reports the destination empty nor advises
// running nuke again.
func TestNukeRemoteKeepsLocalIndexWhenDestinationMissing(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
v, repos, out := backUpThenUnplug(ctx, t)
filesBefore, err := repos.Files.ListAll(ctx)
require.NoError(t, err)
require.NotEmpty(t, filesBefore)
blobsBefore, err := repos.Blobs.GetAll(ctx)
require.NoError(t, err)
require.NotEmpty(t, blobsBefore)
err = v.NukeRemote(true)
require.ErrorIs(t, err, fs.ErrNotExist)
require.ErrorContains(t, err, "listing remote snapshots")
assert.NotContains(t, out.String(), "now empty")
assert.NotContains(t, out.String(), "vaultik remote nuke --force")
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err)
assert.Len(t, snapshots, 1, "nuke must delete no local snapshot record")
files, err := repos.Files.ListAll(ctx)
require.NoError(t, err)
assert.Len(t, files, len(filesBefore), "nuke must delete no local file record")
blobs, err := repos.Blobs.GetAll(ctx)
require.NoError(t, err)
assert.Len(t, blobs, len(blobsBefore), "nuke must delete no local blob record")
}
+2 -27
View File
@@ -104,32 +104,6 @@ func TestNukeRemoteFailsWhenADeleteFails(t *testing.T) {
} }
} }
// TestNukeRemoteKeepsLocalSnapshotWhoseMetadataIsNotDeleted checks that
// remote nuke keeps a snapshot in the local index while its metadata
// cannot be deleted from the destination store: dropped from the local
// index, it would be left there as a remote-only snapshot without its
// name.
func TestNukeRemoteKeepsLocalSnapshotWhoseMetadataIsNotDeleted(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
storeDir := filepath.Join(t.TempDir(), "store")
v, repos, _ := backUpToFileDestination(ctx, t, storeDir)
store := faultstore.New(v.Storage)
store.OnDelete = func(key string) bool {
return strings.HasPrefix(key, "metadata/")
}
v.Storage = store
require.Error(t, v.NukeRemote(true))
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err)
assert.Len(t, snapshots, 1)
}
// TestNukeRemoteNamesWhatIsLeft checks that remote nuke, when an object // TestNukeRemoteNamesWhatIsLeft checks that remote nuke, when an object
// it does not recognise cannot be deleted, does not report the store // it does not recognise cannot be deleted, does not report the store
// empty but names that object and fails. // empty but names that object and fails.
@@ -180,7 +154,8 @@ func TestNukeRemoteRefusesAnotherDestinationWithoutRetryAdvice(t *testing.T) {
// TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved checks that a // TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved checks that a
// snapshot remote nuke cannot remove from the local index fails the // snapshot remote nuke cannot remove from the local index fails the
// run, and that nuke then does not report the destination store empty. // run: nuke then leaves that snapshot's metadata on the destination
// store.
func TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved(t *testing.T) { func TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
+1 -8
View File
@@ -55,19 +55,12 @@ func (v *Vaultik) NukeRemote(force bool) error {
} }
// Checked before anything is deleted: running nuke again would fail // Checked before anything is deleted: running nuke again would fail
// the same way, so the advice below must not follow these errors. // the same way, so the advice below must not follow this error.
// The listing also fails on a destination without the marker while
// the local index holds snapshots.
err := v.EnsureStorageBinding() err := v.EnsureStorageBinding()
if err != nil { if err != nil {
return err return err
} }
_, err = v.listAllRemoteSnapshotKeys()
if err != nil {
return err
}
err = v.emptyDestinationStore() err = v.emptyDestinationStore()
if err != nil { if err != nil {
v.UI.Warningf("Run '%s' again to delete what is left on the "+ v.UI.Warningf("Run '%s' again to delete what is left on the "+
+5 -42
View File
@@ -12,7 +12,6 @@ import (
"path/filepath" "path/filepath"
"slices" "slices"
"strings" "strings"
"syscall"
"time" "time"
"filippo.io/age" "filippo.io/age"
@@ -220,7 +219,9 @@ func (v *Vaultik) finishRestore(
) )
if os.Geteuid() != 0 { if os.Geteuid() != 0 {
v.warnIfOwnershipNotPreserved(files, opts.TargetDir) v.UI.Warningf("Restore did not preserve file ownership: chown(2) " +
"requires root. Re-run as root (e.g. with sudo) if you need " +
"original UID/GID preserved.")
} }
if result.FilesFailed > 0 { if result.FilesFailed > 0 {
@@ -246,43 +247,6 @@ func (v *Vaultik) finishRestore(
return unmatchedErr return unmatchedErr
} }
// warnIfOwnershipNotPreserved warns once if some entry restored under
// targetDir has another owner or group on disk than the snapshot
// records. It is meant for a restore by a normal user, which changes no
// owner or group: a new entry is owned by that user, and its group is
// that user's, or the group of the directory it is created in on macOS
// and in a setgid directory on Linux. An entry that is not on disk, or
// whose filesystem reports no owner, is skipped.
func (v *Vaultik) warnIfOwnershipNotPreserved(
files []*database.File, targetDir string,
) {
for _, file := range files {
targetPath, err := containedRestorePath(
v.Fs, targetDir, file.Path.String())
if err != nil {
continue
}
info, err := lstatIfPossible(v.Fs, targetPath)
if err != nil {
continue
}
owner, ok := info.Sys().(*syscall.Stat_t)
if !ok {
continue
}
if owner.Uid != file.UID || owner.Gid != file.GID {
v.UI.Warningf("Restore did not preserve file ownership: chown(2) " +
"requires root. Re-run as root (e.g. with sudo) if you need " +
"original UID/GID preserved.")
return
}
}
}
// reportUnmatchedPaths warns, by name, about each requested path that // reportUnmatchedPaths warns, by name, about each requested path that
// matched nothing in the snapshot, and returns an error counting them, // matched nothing in the snapshot, and returns an error counting them,
// or nil when there are none. // or nil when there are none.
@@ -1001,9 +965,8 @@ type restoreSession struct {
// root can chown a file to an arbitrary UID/GID — non-root chown // root can chown a file to an arbitrary UID/GID — non-root chown
// always fails with EPERM. Attempting it anyway produces N // always fails with EPERM. Attempting it anyway produces N
// guaranteed-failed syscalls + N noisy debug lines, so we skip // guaranteed-failed syscalls + N noisy debug lines, so we skip
// the call entirely as non-root; warnIfOwnershipNotPreserved warns // the call entirely as non-root and emit one warning at the end
// once at the end of the restore if that left any owner or group // of the restore explaining that ownership was not preserved.
// other than recorded.
runningAsRoot bool runningAsRoot bool
// directories holds every restored directory, for // directories holds every restored directory, for
// applyDirectoryMetadata to finish after the restore loop. // applyDirectoryMetadata to finish after the restore loop.
@@ -1,131 +0,0 @@
package vaultik //nolint:testpackage // calls the unexported ownership check
import (
"bytes"
"context"
"os"
"strings"
"syscall"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/ui"
)
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/316: a
// restore by a normal user warns that ownership was not preserved only
// when some restored entry has another owner or group on disk than the
// snapshot records.
const ownershipWarning = "did not preserve file ownership"
// TestOwnershipWarningSilentWhenEntriesMatch checks that a restore whose
// entries all end up with their recorded owner and group prints no
// warning.
func TestOwnershipWarningSilentWhenEntriesMatch(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
targetDir := t.TempDir()
owner := ownerOf(t, targetDir)
out := restoreAsNormalUser(t, targetDir, []*database.File{
{Path: "/a", Mode: plainFileMode, UID: owner.Uid, GID: owner.Gid},
{Path: "/b", Mode: plainFileMode, UID: owner.Uid, GID: owner.Gid},
})
assert.Empty(t, out)
}
// TestOwnershipWarningWhenOwnerDiffers checks that the warning is printed
// when an entry recorded with another owner is restored owned by the
// restoring user, even though its group matches the recorded one.
func TestOwnershipWarningWhenOwnerDiffers(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
targetDir := t.TempDir()
owner := ownerOf(t, targetDir)
out := restoreAsNormalUser(t, targetDir, []*database.File{
{Path: "/a", Mode: plainFileMode, UID: otherOwnerID, GID: owner.Gid},
})
assert.Equal(t, 1, strings.Count(out, ownershipWarning), out)
}
// TestOwnershipWarningWhenDirectoryGivesItsGroup checks that the warning
// is printed, once, when entries recorded with the restoring user's
// owner and group take another group from the setgid directory they are
// restored into, as every new entry takes its directory's group on
// macOS.
func TestOwnershipWarningWhenDirectoryGivesItsGroup(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
if os.Geteuid() != 0 {
t.Skip("giving a directory to another group needs root")
}
targetDir := t.TempDir()
user := ownerOf(t, targetDir)
require.NoError(t, os.Chown(targetDir, -1, int(otherOwnerID)))
require.NoError(t, os.Chmod(targetDir, os.ModeSetgid|writableTestMode))
out := restoreAsNormalUser(t, targetDir, []*database.File{
{Path: "/a", Mode: plainFileMode, UID: user.Uid, GID: user.Gid},
{Path: "/b", Mode: plainFileMode, UID: user.Uid, GID: user.Gid},
})
assert.Equal(t, 1, strings.Count(out, ownershipWarning), out)
}
// ownerOf returns the owner and group path has on disk.
func ownerOf(t *testing.T, path string) *syscall.Stat_t {
t.Helper()
info, err := os.Lstat(path)
require.NoError(t, err)
owner, ok := info.Sys().(*syscall.Stat_t)
require.True(t, ok)
return owner
}
// restoreAsNormalUser restores rows, all empty regular files, into
// targetDir without changing any owner or group, as restore does for a
// normal user, and returns what warnIfOwnershipNotPreserved prints.
func restoreAsNormalUser(
t *testing.T, targetDir string, rows []*database.File,
) string {
t.Helper()
ctx := context.Background()
rows, repos := makeFiles(ctx, t, rows)
var out bytes.Buffer
v := newContainmentVaultik(ctx, afero.NewOsFs())
v.UI = ui.NewWithColor(&out, false)
session := &restoreSession{
v: v,
ctx: ctx,
repos: repos,
opts: &RestoreOptions{TargetDir: targetDir},
result: &RestoreResult{},
}
for _, row := range rows {
require.NoError(t, session.restoreFile(row))
}
v.warnIfOwnershipNotPreserved(rows, targetDir)
return out.String()
}
+31 -77
View File
@@ -39,11 +39,6 @@ var (
// caller can report it apart from a failed backup. // caller can report it apart from a failed backup.
var ErrPostBackupPruneFailed = errors.New("post-backup prune failed") var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
// ErrFilesUnreadable is returned when --skip-errors skipped files or
// directories it could not read or list. The snapshot is complete and
// kept, so the caller can report it apart from a failed backup.
var ErrFilesUnreadable = errors.New("file(s) could not be read")
// listRecentLimit caps how many snapshot rows are fetched from the // listRecentLimit caps how many snapshot rows are fetched from the
// local index for listing and purge operations. // local index for listing and purge operations.
const listRecentLimit = 10000 const listRecentLimit = 10000
@@ -135,16 +130,12 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
} }
// Process each named snapshot // Process each named snapshot
filesUnreadable := 0
for snapIdx, snapName := range snapshotNames { for snapIdx, snapName := range snapshotNames {
unreadable, err := v.createNamedSnapshot( err = v.createNamedSnapshot(
opts, hostname, snapName, snapIdx+1, len(snapshotNames)) opts, hostname, snapName, snapIdx+1, len(snapshotNames))
if err != nil { if err != nil {
return err return err
} }
filesUnreadable += unreadable
} }
// Print overall summary if multiple snapshots // Print overall summary if multiple snapshots
@@ -153,34 +144,13 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
len(snapshotNames), v.UI.Duration(time.Since(overallStartTime))) len(snapshotNames), v.UI.Duration(time.Since(overallStartTime)))
} }
switch { if opts.Prune {
case opts.Prune && filesUnreadable > 0:
// An older snapshot may be the only one that still holds a file
// this run could not read.
log.Warn("Skipping post-backup prune",
"files_unreadable", filesUnreadable)
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
"could not be read; older snapshots are kept.", filesUnreadable)
case opts.Prune:
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan) err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
if err != nil { if err != nil {
return fmt.Errorf("%w: %w", ErrPostBackupPruneFailed, err) return fmt.Errorf("%w: %w", ErrPostBackupPruneFailed, err)
} }
} }
return v.finishCreateSnapshot(filesUnreadable)
}
// finishCreateSnapshot ends a run whose snapshots are all complete and
// kept. Like finishRestore, it returns an error when --skip-errors
// skipped anything it could not read or list; otherwise it prints the
// closing line. A file removed during the run is counted as failed in
// the summary but is not an error, with or without --skip-errors.
func (v *Vaultik) finishCreateSnapshot(filesUnreadable int) error {
if filesUnreadable > 0 {
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
}
// Terminus must obey the --cron invariant: silent on total // Terminus must obey the --cron invariant: silent on total
// success only. UI.Complete is dropped in cron/quiet mode (that's // success only. UI.Complete is dropped in cron/quiet mode (that's
// the success path), but if any warnings fired during the run we // the success path), but if any warnings fired during the run we
@@ -270,22 +240,16 @@ type snapshotStats struct {
totalBlobsUploaded int totalBlobsUploaded int
uploadDuration time.Duration uploadDuration time.Duration
// The part of totalFilesFailed that --skip-errors skipped because it
// could not be read or listed.
totalFilesUnreadable int
// The sizes of all blobs the snapshot references, set by // The sizes of all blobs the snapshot references, set by
// finalizeSnapshotMetadata once snapshot_blobs is populated. // finalizeSnapshotMetadata once snapshot_blobs is populated.
blobSize int64 blobSize int64
blobUncompressedSize int64 blobUncompressedSize int64
} }
// createNamedSnapshot creates a single named snapshot and returns the // createNamedSnapshot creates a single named snapshot
// number of entries --skip-errors skipped because it could not read or
// list them.
func (v *Vaultik) createNamedSnapshot( func (v *Vaultik) createNamedSnapshot(
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int, opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
) (int, error) { ) error {
snapshotStartTime := time.Now() snapshotStartTime := time.Now()
if total > 1 { if total > 1 {
@@ -294,12 +258,12 @@ func (v *Vaultik) createNamedSnapshot(
resolvedDirs, err := v.resolveSnapshotPaths(snapName) resolvedDirs, err := v.resolveSnapshotPaths(snapName)
if err != nil { if err != nil {
return 0, err return err
} }
previousSnapshotID, err := v.previousSnapshotID(snapName) previousSnapshotID, err := v.previousSnapshotID(snapName)
if err != nil { if err != nil {
return 0, err return err
} }
scanner := v.ScannerFactory(snapshot.ScannerParams{ scanner := v.ScannerFactory(snapshot.ScannerParams{
@@ -314,7 +278,7 @@ func (v *Vaultik) createNamedSnapshot(
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName( snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit) v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
if err != nil { if err != nil {
return 0, fmt.Errorf("creating snapshot: %w", err) return fmt.Errorf("creating snapshot: %w", err)
} }
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName) log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
@@ -322,12 +286,12 @@ func (v *Vaultik) createNamedSnapshot(
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID) stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
if err != nil { if err != nil {
return 0, err return err
} }
err = v.finalizeSnapshotMetadata(snapshotID, stats) err = v.finalizeSnapshotMetadata(snapshotID, stats)
if err != nil { if err != nil {
return 0, err return err
} }
log.Info("Snapshot complete", log.Info("Snapshot complete",
@@ -340,7 +304,7 @@ func (v *Vaultik) createNamedSnapshot(
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats) v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
return stats.totalFilesUnreadable, nil return nil
} }
// previousSnapshotID returns the ID of the most recent completed snapshot // previousSnapshotID returns the ID of the most recent completed snapshot
@@ -426,7 +390,6 @@ func (v *Vaultik) scanAllDirectories(
stats.totalBlobs += result.BlobsCreated stats.totalBlobs += result.BlobsCreated
stats.totalFilesSkipped += result.FilesSkipped stats.totalFilesSkipped += result.FilesSkipped
stats.totalFilesFailed += result.FilesFailed stats.totalFilesFailed += result.FilesFailed
stats.totalFilesUnreadable += result.FilesUnreadable
stats.totalBytesSkipped += result.BytesSkipped stats.totalBytesSkipped += result.BytesSkipped
stats.totalFilesDeleted += result.FilesDeleted stats.totalFilesDeleted += result.FilesDeleted
stats.totalBytesDeleted += result.BytesDeleted stats.totalBytesDeleted += result.BytesDeleted
@@ -1553,9 +1516,7 @@ func (v *Vaultik) removeSnapshotRemote(
// database from the local index, and (with --remote) every snapshot // database from the local index, and (with --remote) every snapshot
// metadata directory in remote storage. Both sides are processed so a // metadata directory in remote storage. Both sides are processed so a
// "remove --all" leaves nothing behind, even when the local DB and // "remove --all" leaves nothing behind, even when the local DB and
// remote storage have diverged. Unless LocalOnly is set, it fails // remote storage have diverged.
// before it removes anything when the destination store cannot be
// listed or is not present by checkDestinationMarker.
func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) { func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) {
err := v.EnsureStorageBinding() err := v.EnsureStorageBinding()
if err != nil { if err != nil {
@@ -1568,15 +1529,11 @@ func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error)
} }
// remoteKeys is the set of metadata/<key>/ subdirectories on the // remoteKeys is the set of metadata/<key>/ subdirectories on the
// destination store. Only for a local-only remove is a failure // destination store; failures are downgraded to a warning so a
// downgraded to a warning, so a permission-denied or unreachable // permission-denied or unreachable remote can't block a local-only
// remote can't block it. // remove.
remoteKeys, remoteErr := v.listAllRemoteSnapshotKeys() remoteKeys, remoteErr := v.listAllRemoteSnapshotKeys()
if remoteErr != nil { if remoteErr != nil {
if !opts.LocalOnly {
return nil, remoteErr
}
log.Warn("Could not list remote snapshots", "error", remoteErr) log.Warn("Could not list remote snapshots", "error", remoteErr)
v.UI.Warningf("Could not list remote snapshots: %v.", remoteErr) v.UI.Warningf("Could not list remote snapshots: %v.", remoteErr)
} }
@@ -1638,7 +1595,8 @@ func (v *Vaultik) localSnapshotIDs() ([]string, error) {
// listAllRemoteSnapshotKeys collects the hashed remote keys // listAllRemoteSnapshotKeys collects the hashed remote keys
// (subdirectories under metadata/) currently present in the // (subdirectories under metadata/) currently present in the
// destination store. Returns (nil, err) when the store cannot be // destination store. Returns (nil, err) when the store cannot be
// listed, or is not present by checkDestinationMarker. // listed, or is not present by checkDestinationMarker; callers must
// treat that as "no remote info available," not fatal.
func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) { func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) {
log.Info("Listing all remote snapshots") log.Info("Listing all remote snapshots")
@@ -1822,12 +1780,10 @@ func (v *Vaultik) deleteIncompleteSnapshot(snapshotIDStr string, id any) bool {
return true return true
} }
// removeAllLocalSnapshots removes each known local snapshot's remote // removeAllLocalSnapshots removes each known local snapshot (and, unless
// metadata, unless --local-only, and then its local record, returning // --local-only, its remote metadata), returning the number of snapshots
// the number of snapshots it could not remove. The local record goes // it could not remove. A snapshot left in the local index keeps its
// only once the metadata is deleted: dropped first, it would leave that // remote metadata too, since it is not an orphan remote key.
// metadata on the destination store as a remote-only snapshot whose name
// is lost.
func (v *Vaultik) removeAllLocalSnapshots( func (v *Vaultik) removeAllLocalSnapshots(
localSnaps []string, opts *RemoveOptions, result *RemoveResult, localSnaps []string, opts *RemoveOptions, result *RemoveResult,
) int { ) int {
@@ -1836,19 +1792,6 @@ func (v *Vaultik) removeAllLocalSnapshots(
for _, snapshotID := range localSnaps { for _, snapshotID := range localSnaps {
log.Info("Removing snapshot", "snapshot_id", snapshotID) log.Info("Removing snapshot", "snapshot_id", snapshotID)
if !opts.LocalOnly {
_, err := v.deleteRemoteSnapshotByKey(
snapshot.RemoteSnapshotKey(snapshotID))
if err != nil {
log.Warn("Failed to remove snapshot metadata from remote",
"snapshot_id", snapshotID, "error", err)
failures++
continue
}
}
err := v.deleteSnapshotFromLocalDB(snapshotID) err := v.deleteSnapshotFromLocalDB(snapshotID)
if err != nil { if err != nil {
log.Error("Failed to remove from local database", log.Error("Failed to remove from local database",
@@ -1859,6 +1802,17 @@ func (v *Vaultik) removeAllLocalSnapshots(
continue continue
} }
if !opts.LocalOnly {
_, err = v.deleteRemoteSnapshotByKey(
snapshot.RemoteSnapshotKey(snapshotID))
if err != nil {
log.Warn("Failed to remove snapshot metadata from remote",
"snapshot_id", snapshotID, "error", err)
failures++
}
}
result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID) result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID)
} }
@@ -178,13 +178,11 @@ func TestBackupDoesNotCountFileInUnopenableDirectoryAsDeleted(t *testing.T) {
} }
env.out.Reset() env.out.Reset()
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{ require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
SkipErrors: true, SkipErrors: true,
Snapshots: []string{"work"}, Snapshots: []string{"work"},
}) }))
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable) assert.Contains(t, env.out.String(), filesLine(1, 0, 1))
assert.Contains(t, env.out.String(),
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
} }
// A backup counts only files that the previous snapshot of its own name // A backup counts only files that the previous snapshot of its own name
+3 -10
View File
@@ -49,19 +49,13 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
name string name string
openErr error openErr error
skipErrors bool skipErrors bool
wantErr error // nil for none
}{ }{
// What a normal user gets opening a file with mode 000. // What a normal user gets opening a file with mode 000.
{name: "unopenable under skip-errors", {name: "unopenable under skip-errors",
openErr: os.ErrPermission, skipErrors: true, openErr: os.ErrPermission, skipErrors: true},
wantErr: vaultik.ErrFilesUnreadable},
// What opening a file removed after phase 1 gives. // What opening a file removed after phase 1 gives.
{name: "removed between the phases", {name: "removed between the phases",
openErr: os.ErrNotExist, skipErrors: false}, openErr: os.ErrNotExist, skipErrors: false},
// A removal is not an error, so --skip-errors does not make the
// run fail. See https://git.eeqj.de/sneak/vaultik/issues/310.
{name: "removed between the phases under skip-errors",
openErr: os.ErrNotExist, skipErrors: true},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -105,11 +99,10 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
v := newBackupVaultik(ctx, cfg, store, repos, db, fs) v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
v.UI = ui.NewWithColor(out, false) v.UI = ui.NewWithColor(out, false)
err = v.CreateSnapshot(&vaultik.SnapshotCreateOptions{ require.NoError(t, v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
SkipErrors: tt.skipErrors, SkipErrors: tt.skipErrors,
Snapshots: []string{"src"}, Snapshots: []string{"src"},
}) }))
require.ErrorIs(t, err, tt.wantErr)
summary := out.String() summary := out.String()
assert.Contains(t, summary, assert.Contains(t, summary,
@@ -3,7 +3,6 @@ package vaultik_test
import ( import (
"context" "context"
"fmt" "fmt"
"os"
"path/filepath" "path/filepath"
"testing" "testing"
@@ -74,210 +73,3 @@ func TestSkipErrorsBackupStopsOnIndexErrorRecordingDirectory(t *testing.T) {
require.Len(t, snapshots, 1) require.Len(t, snapshots, 1)
assert.Nil(t, snapshots[0].CompletedAt) assert.Nil(t, snapshots[0].CompletedAt)
} }
// A backup under --skip-errors that skips a file it cannot open, or a
// directory it cannot list, keeps the snapshot, counts what it skipped
// as failed and returns an error instead of finishing successfully. See
// https://git.eeqj.de/sneak/vaultik/issues/310.
func TestSkipErrorsBackupKeepsSnapshotAndFailsWhenItSkips(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
tests := []struct {
name string
skipped string // relative to the source directory
}{
{name: "file it cannot open", skipped: "skipdir/file.txt"},
{name: "directory it cannot list", skipped: "skipdir"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
env := newDeletedCountEnv(t, map[string][]byte{
"stored.txt": []byte("stored"),
"skipdir/file.txt": []byte("in skipdir"),
})
env.setSnapshot("work", "")
// A mode does not lock out root, who runs the tests, so
// opening the skipped path is made to fail instead.
env.v.Fs = &openFailFs{
path: filepath.Join(env.src, tt.skipped),
err: os.ErrPermission,
}
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
SkipErrors: true,
Snapshots: []string{"work"},
})
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
assert.Contains(t, env.out.String(),
"Files: 2 examined, 1 backed up, 0 unchanged, 1 failed.")
assert.NotContains(t, env.out.String(), "Finished successfully.")
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
context.Background(), listRecentTestLimit)
require.NoError(t, err)
require.Len(t, snapshots, 1)
assert.NotNil(t, snapshots[0].CompletedAt, "the snapshot is kept")
})
}
}
// removeOnLookupFs is the real filesystem, except that path is removed
// just before it is looked up. The walk looks an entry up after listing
// its directory, so path is listed but gone when the walk reaches it.
type removeOnLookupFs struct {
afero.OsFs
path string
}
func (f *removeOnLookupFs) LstatIfPossible(
name string,
) (os.FileInfo, bool, error) {
if name == f.path {
_ = os.Remove(name)
}
return f.OsFs.LstatIfPossible(name)
}
// A file removed before the walk reaches it is counted once, as failed,
// like a file removed before its data is read, and the backup finishes
// successfully, with or without --skip-errors. See
// https://git.eeqj.de/sneak/vaultik/issues/310.
func TestBackupCountsFileRemovedBeforeWalkReachesItAsFailed(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
tests := []struct {
name string
skipErrors bool
}{
{name: "under skip-errors", skipErrors: true},
{name: "without skip-errors", skipErrors: false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
env := newDeletedCountEnv(t, map[string][]byte{
"stays.txt": []byte("stays"),
"removed.txt": []byte("removed"),
})
env.setSnapshot("walk", "")
// The previous snapshot has the file, so a removal the walk
// also counted as deleted would show.
env.backUp(t, "walk")
env.v.Fs = &removeOnLookupFs{
path: filepath.Join(env.src, "removed.txt"),
}
env.out.Reset()
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
SkipErrors: tt.skipErrors,
Snapshots: []string{"walk"},
}))
assert.Contains(t, env.out.String(),
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
assert.Contains(t, env.out.String(), "Finished successfully.")
})
}
}
// A file an exclude pattern matches is not counted when it is removed
// before the walk reaches it. See
// https://git.eeqj.de/sneak/vaultik/issues/310.
func TestBackupIgnoresExcludedFileRemovedBeforeWalkReachesIt(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newDeletedCountEnv(t, map[string][]byte{
"stays.txt": []byte("stays"),
"removed.tmp": []byte("removed"),
})
env.setSnapshot("walk", "", "*.tmp")
env.v.Fs = &removeOnLookupFs{
path: filepath.Join(env.src, "removed.tmp"),
}
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
SkipErrors: true,
Snapshots: []string{"walk"},
}))
assert.Contains(t, env.out.String(), filesLine(1, 1, 0))
assert.Contains(t, env.out.String(), "Finished successfully.")
}
// A backup under --skip-errors --prune that skipped a file it could not
// read does not prune: the previous snapshot, the only one that holds
// the file, is kept, and the output says the prune was skipped. See
// https://git.eeqj.de/sneak/vaultik/issues/382.
func TestSkipErrorsBackupDoesNotPruneWhenItSkips(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newDeletedCountEnv(t, map[string][]byte{
"other.txt": []byte("other"),
"skipped.txt": []byte("skipped"),
})
env.setSnapshot("photos", "")
env.backUp(t, "photos")
// An unchanged file is not read again, so the file is changed before
// it becomes unreadable.
skipped := filepath.Join(env.src, "skipped.txt")
require.NoError(t, os.WriteFile(skipped, []byte("changed"), 0o600))
env.v.Fs = &openFailFs{path: skipped, err: os.ErrPermission}
env.out.Reset()
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Prune: true,
SkipErrors: true,
Snapshots: []string{"photos"},
})
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
assert.Contains(t, env.out.String(),
"Skipped the post-backup prune because 1 file(s) could not be read")
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
context.Background(), listRecentTestLimit)
require.NoError(t, err)
assert.Len(t, snapshots, 2, "the previous snapshot is kept")
}
// A file removed during a backup under --skip-errors --prune is not a
// skipped file, so the backup prunes as usual. See
// https://git.eeqj.de/sneak/vaultik/issues/382.
func TestSkipErrorsBackupPrunesWhenAFileIsRemovedDuringIt(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newDeletedCountEnv(t, map[string][]byte{
"other.txt": []byte("other"),
"removed.txt": []byte("removed"),
})
env.setSnapshot("photos", "")
env.backUp(t, "photos")
env.v.Fs = &removeOnLookupFs{path: filepath.Join(env.src, "removed.txt")}
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Prune: true,
SkipErrors: true,
Snapshots: []string{"photos"},
}))
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
context.Background(), listRecentTestLimit)
require.NoError(t, err)
assert.Len(t, snapshots, 1, "the previous snapshot is pruned")
}
-116
View File
@@ -13,7 +13,6 @@ import (
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"golang.org/x/sys/unix"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
@@ -189,42 +188,6 @@ func (e *summaryEnv) dataLine(total, backedUp int64) string {
e.v.UI.Size(total), e.v.UI.Size(backedUp)) e.v.UI.Size(total), e.v.UI.Size(backedUp))
} }
// enumeratedLine returns the line printed when the walk of a source path
// ends.
func (e *summaryEnv) enumeratedLine(
files int, total int64, toBackUp int, toBackUpSize int64,
) string {
return fmt.Sprintf(
"Enumerated %d snapshot source files (%s total), %d to back up (%s).",
files, e.v.UI.Size(total), toBackUp, e.v.UI.Size(toBackUpSize))
}
// snapshotPaths returns the paths the snapshot lists, sorted.
func (e *summaryEnv) snapshotPaths(t *testing.T, snapshotID string) []string {
t.Helper()
rows, err := e.db.Conn().QueryContext(context.Background(), `
SELECT f.path FROM snapshot_files sf JOIN files f ON f.id = sf.file_id
WHERE sf.snapshot_id = ? ORDER BY f.path`, snapshotID)
require.NoError(t, err)
defer func() { _ = rows.Close() }()
var paths []string
for rows.Next() {
var path string
require.NoError(t, rows.Scan(&path))
paths = append(paths, path)
}
require.NoError(t, rows.Err())
return paths
}
// A first backup stores copy.bin's chunks while backing up a.bin, so // A first backup stores copy.bin's chunks while backing up a.bin, so
// copy.bin's chunks are deduplicated within the run. Each file and byte // copy.bin's chunks are deduplicated within the run. Each file and byte
// is still counted once. // is still counted once.
@@ -285,85 +248,6 @@ func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
assert.Equal(t, uncompressed, snap.BlobUncompressedSize) assert.Equal(t, uncompressed, snap.BlobUncompressedSize)
} }
// A backup of a tree with directories and a symlink, none of which
// changed since the last backup, backs up nothing, and the snapshot
// still lists every one of them. Both sides of the line that ends the
// walk count the same entries, the files and the symlink, and no
// directory's size. See https://git.eeqj.de/sneak/vaultik/issues/305.
func TestSnapshotSummaryUnchangedRunWithDirectoriesAndSymlink(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newSummaryEnv(t)
const linkTarget = "a.bin"
require.NoError(t, os.Symlink(linkTarget,
filepath.Join(filepath.Dir(env.aPath), "link")))
// A symlink's size is the length of its target.
total := env.totalSize + int64(len(linkTarget))
first := env.backUp(t, "first", false)
assert.Contains(t, first, env.enumeratedLine(4, total, 4, total))
assert.Contains(t, first, filesLine(4, 4, 0))
second := env.backUp(t, "second", false)
assert.Contains(t, second, env.enumeratedLine(4, total, 0, 0))
assert.Contains(t, second, filesLine(4, 0, 4))
assert.Contains(t, second, env.dataLine(total, 0))
assert.Equal(t,
env.snapshotPaths(t, env.snapshot(t, "first").ID.String()),
env.snapshotPaths(t, env.snapshot(t, "second").ID.String()))
}
// A directory whose mtime changed and a symlink whose target changed are
// recorded again by the next backup, which counts only the symlink as
// backed up.
func TestSnapshotSummaryChangedDirectoryAndSymlink(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newSummaryEnv(t)
// The local index keeps paths with symlinks resolved.
dir, err := filepath.EvalSymlinks(filepath.Dir(env.aPath))
require.NoError(t, err)
// The new link gets the old one's mtime, and a target, which need not
// exist, of the old target's length, a symlink's size: only the
// target differs.
linkTime := unix.NsecToTimeval(
time.Date(2029, time.March, 4, 5, 6, 7, 0, time.UTC).UnixNano())
link := filepath.Join(dir, "link")
require.NoError(t, os.Symlink("a.bin", link))
require.NoError(t, unix.Lutimes(link, []unix.Timeval{linkTime, linkTime}))
env.backUp(t, "first", false)
require.NoError(t, os.Remove(link))
require.NoError(t, os.Symlink("b.bin", link))
require.NoError(t, unix.Lutimes(link, []unix.Timeval{linkTime, linkTime}))
mtime := time.Date(2030, time.January, 2, 3, 4, 5, 0, time.UTC)
require.NoError(t, os.Chtimes(dir, mtime, mtime))
assert.Contains(t, env.backUp(t, "second", false), filesLine(4, 1, 3))
ctx := context.Background()
dirRow, err := env.repos.Files.GetByPath(ctx, dir)
require.NoError(t, err)
require.NotNil(t, dirRow)
assert.Equal(t, mtime.Unix(), dirRow.MTime.Unix())
linkRow, err := env.repos.Files.GetByPath(ctx, link)
require.NoError(t, err)
require.NotNil(t, linkRow)
assert.Equal(t, "b.bin", linkRow.LinkTarget.String())
}
// Under --cron the progress reporter is off; the upload figures must // Under --cron the progress reporter is off; the upload figures must
// still reach the summary and the snapshots row. The snapshot has two // still reach the summary and the snapshots row. The snapshot has two
// paths, each backed up by its own scan. // paths, each backed up by its own scan.