Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b3cf02c817 |
+5
-6
@@ -385,12 +385,11 @@ bucket/
|
|||||||
A destination without `vaultik-destination.txt` is treated as not present
|
A destination without `vaultik-destination.txt` is treated as not present
|
||||||
while the local index holds snapshots stored there (`checkDestinationMarker`
|
while the local index holds snapshots stored there (`checkDestinationMarker`
|
||||||
in `internal/vaultik/destination_marker.go`): `snapshot create` refuses to
|
in `internal/vaultik/destination_marker.go`): `snapshot create` refuses to
|
||||||
write to it, `prune`, `snapshot purge` and `remote nuke` fail, and
|
write to it, `prune` and `snapshot purge` fail, and `snapshot list` and
|
||||||
`snapshot list` and `snapshot remove` behave as for a destination that cannot
|
`snapshot remove` behave as for a destination that cannot be listed. This
|
||||||
be listed. This keeps those commands from taking an unplugged volume's empty
|
keeps those commands from taking an unplugged volume's empty mount point for
|
||||||
mount point for a destination whose snapshots are all gone. The `file://`
|
a destination whose snapshots are all gone. The `file://` backend never
|
||||||
backend never creates its destination directory, so a write to a missing one
|
creates its destination directory, so a write to a missing one fails.
|
||||||
fails.
|
|
||||||
|
|
||||||
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
|
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
|
||||||
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
|
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
|
||||||
|
|||||||
@@ -182,7 +182,7 @@ vaultik version
|
|||||||
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
|
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
|
||||||
* `--debug`: Enable debug output (on stderr — see below)
|
* `--debug`: Enable debug output (on stderr — see below)
|
||||||
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
|
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
|
||||||
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run. A run that skipped anything still finishes but exits non-zero: `snapshot create` keeps the snapshot and counts each file or directory it skipped as failed. A file removed during the backup is also counted as failed but does not change the exit status.
|
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run.
|
||||||
|
|
||||||
### locking
|
### locking
|
||||||
|
|
||||||
@@ -297,11 +297,7 @@ value where a list belongs or a word where a number belongs), is refused.
|
|||||||
(for crontab)
|
(for crontab)
|
||||||
* `--prune`: After backup, drop older snapshots of each backed-up name and
|
* `--prune`: After backup, drop older snapshots of each backed-up name and
|
||||||
remove orphaned blobs from remote storage. By default keeps only the latest
|
remove orphaned blobs from remote storage. By default keeps only the latest
|
||||||
snapshot per name; use `--keep-newer-than` for a rolling window. When
|
snapshot per name; use `--keep-newer-than` for a rolling window.
|
||||||
`--skip-errors` skipped a file or directory it could not read or list, the
|
|
||||||
prune does not run and every older snapshot is kept, since one of them may
|
|
||||||
be the only one that holds what was skipped; the command says so and exits
|
|
||||||
non-zero. A file removed during the backup does not stop the prune.
|
|
||||||
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
||||||
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
||||||
Without `--prune`, or with a value that is not a valid duration, the
|
Without `--prune`, or with a value that is not a valid duration, the
|
||||||
@@ -458,14 +454,9 @@ since `prune` never deletes it.
|
|||||||
the backup destination store, whatever its name, and then
|
the backup destination store, whatever its name, and then
|
||||||
`vaultik-destination.txt`, leaving the bucket prefix empty. Destructive
|
`vaultik-destination.txt`, leaving the bucket prefix empty. Destructive
|
||||||
and irreversible. This is the single supported way to wipe the entire
|
and irreversible. This is the single supported way to wipe the entire
|
||||||
destination store. A snapshot is removed from the local index only once
|
destination store. If a listing afterwards still finds anything under
|
||||||
its metadata is deleted from the destination store. If the destination
|
`metadata/` or `blobs/`, the command names each object left and exits
|
||||||
store cannot be listed, or the local index holds snapshots and the
|
non-zero; running it again deletes what is left.
|
||||||
destination has no `vaultik-destination.txt` (an unplugged volume's
|
|
||||||
empty mount point), the command fails before it removes anything. If a
|
|
||||||
listing afterwards still finds anything under `metadata/` or `blobs/`,
|
|
||||||
the command names each object left and exits non-zero; running it again
|
|
||||||
deletes what is left.
|
|
||||||
* `--force`: Required to confirm destruction.
|
* `--force`: Required to confirm destruction.
|
||||||
|
|
||||||
**`database delete`**: Delete the local SQLite state database file
|
**`database delete`**: Delete the local SQLite state database file
|
||||||
@@ -538,11 +529,10 @@ For an implementation-level view of the internals — the data model, the
|
|||||||
* `vaultik-destination.txt` holds one fixed line of text, with no host
|
* `vaultik-destination.txt` holds one fixed line of text, with no host
|
||||||
name, path or time. A destination without it is treated as not
|
name, path or time. A destination without it is treated as not
|
||||||
present while the local index holds snapshots stored there:
|
present while the local index holds snapshots stored there:
|
||||||
`snapshot create` refuses to write to it, `prune`, `snapshot purge`
|
`snapshot create` refuses to write to it, `prune` and `snapshot purge`
|
||||||
and `remote nuke` fail, and `snapshot list` and `snapshot remove`
|
fail, and `snapshot list` and `snapshot remove` behave as for a
|
||||||
behave as for a destination that cannot be listed. This keeps an
|
destination that cannot be listed. This keeps an unplugged volume's empty mount point
|
||||||
unplugged volume's empty mount point from being taken for a
|
from being taken for a destination whose snapshots are all gone. Mount
|
||||||
destination whose snapshots are all gone. Mount
|
|
||||||
the volume if it is removable; if the destination was emptied on
|
the volume if it is removable; if the destination was emptied on
|
||||||
purpose, run `vaultik database delete` and back up again.
|
purpose, run `vaultik database delete` and back up again.
|
||||||
* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
|
* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
|
||||||
|
|||||||
@@ -15,229 +15,13 @@ pre-1.0
|
|||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
The 1.0 work is complete on `next`: the scope settled on
|
The 1.0 work is complete on `next`: the scope settled on
|
||||||
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) has landed,
|
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) was the work
|
||||||
and so has every fix from the defect hunt before the release candidate
|
already planned for 1.0, and all of it has landed. The mechanism to cut
|
||||||
([issue #288](https://git.eeqj.de/sneak/vaultik/issues/288)). What is
|
the tag exists and is exercised; what is left is merging `next` to
|
||||||
left is merging `next` to `main` and tagging the release, both the
|
`main` and tagging, both the owner's.
|
||||||
owner's; a `v*` tag runs the release build in
|
|
||||||
`.gitea/workflows/release.yml`.
|
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
|
||||||
that skipped files it could not read
|
|
||||||
([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a
|
|
||||||
prune could delete the only snapshot still holding those files.
|
|
||||||
|
|
||||||
- 2026-10-14: Made `snapshot create --skip-errors` exit non-zero,
|
|
||||||
instead of 0, when it skips anything it cannot read
|
|
||||||
([issue #310](https://git.eeqj.de/sneak/vaultik/issues/310)).
|
|
||||||
|
|
||||||
- 2026-10-14: Kept the local index when `remote nuke` cannot list or
|
|
||||||
find the destination store
|
|
||||||
([issue #292](https://git.eeqj.de/sneak/vaultik/issues/292)). Nuke
|
|
||||||
had cleared the local snapshot records anyway; it now fails before
|
|
||||||
removing anything.
|
|
||||||
|
|
||||||
- 2026-10-14: Stopped counting directories and unchanged symlinks as
|
|
||||||
files to back up
|
|
||||||
([issue #305](https://git.eeqj.de/sneak/vaultik/issues/305)).
|
|
||||||
|
|
||||||
- 2026-10-14: Refused a backup destination that an unplugged volume
|
|
||||||
left empty, as the owner ruled
|
|
||||||
([issue #239](https://git.eeqj.de/sneak/vaultik/issues/239)). Every
|
|
||||||
backup now writes `vaultik-destination.txt` at the destination root,
|
|
||||||
and while the local index holds snapshots stored there, a destination
|
|
||||||
without it is treated as not present.
|
|
||||||
|
|
||||||
- 2026-10-14: Warned about restored ownership only when an entry's
|
|
||||||
owner or group differs from the recorded one
|
|
||||||
([issue #316](https://git.eeqj.de/sneak/vaultik/issues/316)); a
|
|
||||||
restore by a non-root user had warned even when none did.
|
|
||||||
|
|
||||||
- 2026-10-14: Made `snapshot list` report a changed destination instead
|
|
||||||
of advising `prune`, which refuses in that case
|
|
||||||
([issue #317](https://git.eeqj.de/sneak/vaultik/issues/317)).
|
|
||||||
|
|
||||||
- 2026-10-14: Counted a file as deleted by a backup only if the
|
|
||||||
previous snapshot of the same name listed it
|
|
||||||
([issue #304](https://git.eeqj.de/sneak/vaultik/issues/304)); files
|
|
||||||
that snapshot did not list had been counted too.
|
|
||||||
|
|
||||||
- 2026-10-14: Named each prefix once in restore, verify and purge
|
|
||||||
errors, some of which repeated it
|
|
||||||
([issue #327](https://git.eeqj.de/sneak/vaultik/issues/327)). A deep
|
|
||||||
verify log line that named S3 whatever the destination now names
|
|
||||||
storage.
|
|
||||||
|
|
||||||
- 2026-10-14: Refused credentials in a storage URL's endpoint or rclone
|
|
||||||
remote name, which were accepted and could then be printed
|
|
||||||
([issue #322](https://git.eeqj.de/sneak/vaultik/issues/322)).
|
|
||||||
|
|
||||||
- 2026-10-14: Ran `$EDITOR` through the shell in `config edit`, so a
|
|
||||||
value with arguments, such as `code --wait`, no longer fails
|
|
||||||
([issue #315](https://git.eeqj.de/sneak/vaultik/issues/315)).
|
|
||||||
|
|
||||||
- 2026-10-14: Made `snapshot remove` fail, instead of exiting 0, when
|
|
||||||
it cannot delete the metadata of a snapshot that is only on the
|
|
||||||
destination store
|
|
||||||
([issue #340](https://git.eeqj.de/sneak/vaultik/issues/340)). The
|
|
||||||
README and the help now say when it prints the `vaultik prune`
|
|
||||||
follow-up.
|
|
||||||
|
|
||||||
- 2026-10-14: Labelled the totals in the local index cleanup before
|
|
||||||
each backup as totals
|
|
||||||
([issue #306](https://git.eeqj.de/sneak/vaultik/issues/306)). Each
|
|
||||||
was shown as `(N remain)`, which read as rows left to clean up.
|
|
||||||
|
|
||||||
- 2026-10-13: Freed a cached blob once no file still to restore needs
|
|
||||||
it ([issue #324](https://git.eeqj.de/sneak/vaultik/issues/324)); one
|
|
||||||
also used by a file outside the selected paths had stayed in the
|
|
||||||
temporary cache until the restore ended.
|
|
||||||
|
|
||||||
- 2026-10-13: Kept the whole startup error when it contains `): `
|
|
||||||
([issue #320](https://git.eeqj.de/sneak/vaultik/issues/320)). Such
|
|
||||||
an error was cut at its last `): `, so the `blob_size_limit` one lost
|
|
||||||
the setting's name.
|
|
||||||
|
|
||||||
- 2026-10-13: Silenced the status lines of `prune` and snapshot removal
|
|
||||||
under `--cron` and `--quiet`
|
|
||||||
([issue #294](https://git.eeqj.de/sneak/vaultik/issues/294)).
|
|
||||||
|
|
||||||
- 2026-10-13: Read the rclone config file that `RCLONE_CONFIG` names
|
|
||||||
([issue #321](https://git.eeqj.de/sneak/vaultik/issues/321)); a
|
|
||||||
remote defined only there had been reported as not found.
|
|
||||||
|
|
||||||
- 2026-10-13: Made `/**/` in an exclude pattern match zero directories,
|
|
||||||
as in gitignore
|
|
||||||
([issue #338](https://git.eeqj.de/sneak/vaultik/issues/338));
|
|
||||||
`a/**/*.log` had missed `a/x.log`.
|
|
||||||
|
|
||||||
- 2026-10-13: Made the `chunk_size` and `blob_size_limit` defaults and
|
|
||||||
the `chunk_size` minimum 1000-based, like the size parser and the
|
|
||||||
docs ([issue #313](https://git.eeqj.de/sneak/vaultik/issues/313)).
|
|
||||||
|
|
||||||
- 2026-10-13: Kept a snapshot's remote metadata when `snapshot purge`
|
|
||||||
cannot remove it from the local index
|
|
||||||
([issue #337](https://git.eeqj.de/sneak/vaultik/issues/337)); the
|
|
||||||
purge had deleted the metadata anyway and exited 0.
|
|
||||||
|
|
||||||
- 2026-10-13: Made `snapshot restore` fail for a requested path that
|
|
||||||
matches nothing, which it had ignored
|
|
||||||
([issue #309](https://git.eeqj.de/sneak/vaultik/issues/309)).
|
|
||||||
|
|
||||||
- 2026-10-13: Made `remote nuke` delete every object under `metadata/`
|
|
||||||
and `blobs/`
|
|
||||||
([issue #291](https://git.eeqj.de/sneak/vaultik/issues/291)); it had
|
|
||||||
left some, such as a stray `.DS_Store`, while reporting the store
|
|
||||||
empty.
|
|
||||||
|
|
||||||
- 2026-10-13: Made `config set` create a list for an index under a
|
|
||||||
missing key
|
|
||||||
([issue #308](https://git.eeqj.de/sneak/vaultik/issues/308)); it had
|
|
||||||
written a map, after which no command could load the config.
|
|
||||||
|
|
||||||
- 2026-10-13: Listed a `file://` destination through a symlinked
|
|
||||||
`metadata/` or `blobs/`, which had listed as empty
|
|
||||||
([issue #296](https://git.eeqj.de/sneak/vaultik/issues/296)). A link
|
|
||||||
to a missing target now fails the listing
|
|
||||||
([issue #332](https://git.eeqj.de/sneak/vaultik/issues/332)).
|
|
||||||
|
|
||||||
- 2026-10-13: Made `snapshot verify --deep --json` report its blob
|
|
||||||
counts ([issue #314](https://git.eeqj.de/sneak/vaultik/issues/314));
|
|
||||||
a failed run had reported 0 missing and 0 verified.
|
|
||||||
|
|
||||||
- 2026-10-13: Limited and retried the module download in the Docker
|
|
||||||
build ([issue #351](https://git.eeqj.de/sneak/vaultik/issues/351)),
|
|
||||||
which had hung for as long as a stalled connection to the module
|
|
||||||
proxy lasted.
|
|
||||||
|
|
||||||
- 2026-10-13: Recorded each snapshot's uncompressed size when its
|
|
||||||
backup runs
|
|
||||||
([issue #360](https://git.eeqj.de/sneak/vaultik/issues/360)). A later
|
|
||||||
backup could change an older snapshot's UNCOMPRESSED SIZE in
|
|
||||||
`snapshot list`.
|
|
||||||
|
|
||||||
- 2026-10-13: Stopped `snapshot restore --skip-errors --verify`
|
|
||||||
reporting a file that failed to restore again as a checksum failure
|
|
||||||
([issue #323](https://git.eeqj.de/sneak/vaultik/issues/323)).
|
|
||||||
|
|
||||||
- 2026-10-13: Recorded the stored size of a blob a backup finds already
|
|
||||||
stored, not the size of its local copy, which can differ, so
|
|
||||||
`snapshot verify` could fail on a good snapshot
|
|
||||||
([issue #300](https://git.eeqj.de/sneak/vaultik/issues/300)).
|
|
||||||
|
|
||||||
- 2026-10-13: Stopped the local index and the blob upload step ending
|
|
||||||
the process on a failed close of query results or a failed temp file
|
|
||||||
cleanup ([issue #345](https://git.eeqj.de/sneak/vaultik/issues/345));
|
|
||||||
both are now returned as errors.
|
|
||||||
|
|
||||||
- 2026-10-13: Closed the snapshot database's storage reader too when
|
|
||||||
restore or deep verify is interrupted
|
|
||||||
([issue #356](https://git.eeqj.de/sneak/vaultik/issues/356)).
|
|
||||||
|
|
||||||
- 2026-10-13: Made `prune` and `remote nuke` fail when a delete on the
|
|
||||||
backup destination store fails
|
|
||||||
([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)).
|
|
||||||
|
|
||||||
- 2026-10-13: Made an S3 listing that the server cuts short an error
|
|
||||||
([issue #297](https://git.eeqj.de/sneak/vaultik/issues/297)); `prune`
|
|
||||||
had taken it as complete and could delete blobs still in use.
|
|
||||||
|
|
||||||
- 2026-10-13: Made shallow `snapshot verify` report a blob it could not
|
|
||||||
check apart from a missing one
|
|
||||||
([issue #318](https://git.eeqj.de/sneak/vaultik/issues/318)); any
|
|
||||||
error checking a blob had counted as missing, its cause dropped.
|
|
||||||
|
|
||||||
- 2026-10-13: Restored the other files when a file's chunk is in no
|
|
||||||
blob under `--skip-errors`
|
|
||||||
([issue #326](https://git.eeqj.de/sneak/vaultik/issues/326)); that
|
|
||||||
file had stopped the restore before anything was restored.
|
|
||||||
|
|
||||||
- 2026-10-13: Recorded each snapshot's new chunk size when its backup
|
|
||||||
runs ([issue #311](https://git.eeqj.de/sneak/vaultik/issues/311)).
|
|
||||||
`snapshot list` could show a NEW CHUNK SIZE of 0 B for a snapshot
|
|
||||||
whose change was a modified file.
|
|
||||||
|
|
||||||
- 2026-10-13: Made a backup interrupted during a local index write exit
|
|
||||||
as interrupted
|
|
||||||
([issue #301](https://git.eeqj.de/sneak/vaultik/issues/301)); it could
|
|
||||||
end with `FATAL: failed to rollback transaction` and exit 1 instead of
|
|
||||||
130.
|
|
||||||
|
|
||||||
- 2026-10-13: Printed `snapshot list` and `snapshot purge` sizes like
|
|
||||||
every other command
|
|
||||||
([issue #312](https://git.eeqj.de/sneak/vaultik/issues/312)). They,
|
|
||||||
and the `bytes` value in terminal log lines, were 1024-based but
|
|
||||||
labelled KB or MB, and now use 1000-based units.
|
|
||||||
|
|
||||||
- 2026-10-13: Closed each blob's storage reader when restore or deep
|
|
||||||
verify is interrupted, which ends a stalled read unless it is stuck in
|
|
||||||
the kernel ([issue #325](https://git.eeqj.de/sneak/vaultik/issues/325)).
|
|
||||||
|
|
||||||
- 2026-10-13: Made a mistyped subcommand under `snapshot`, `config`,
|
|
||||||
`remote`, `database` or `completion` an error
|
|
||||||
([issue #319](https://git.eeqj.de/sneak/vaultik/issues/319)); each had
|
|
||||||
printed its help and exited 0.
|
|
||||||
|
|
||||||
- 2026-10-13: Made an exclude pattern starting with `**/` match
|
|
||||||
directly under the snapshot path too, as in gitignore
|
|
||||||
([issue #303](https://git.eeqj.de/sneak/vaultik/issues/303)).
|
|
||||||
|
|
||||||
- 2026-10-13: Refused a relative `file://` storage URL and decoded its
|
|
||||||
percent-escapes
|
|
||||||
([issue #299](https://git.eeqj.de/sneak/vaultik/issues/299)); a
|
|
||||||
relative path had moved with the working directory.
|
|
||||||
|
|
||||||
- 2026-10-13: Removed the blob temp file when a backup stops part-way
|
|
||||||
([issue #302](https://git.eeqj.de/sneak/vaultik/issues/302)); the
|
|
||||||
half-written blob had stayed in `$TMPDIR`.
|
|
||||||
|
|
||||||
- 2026-10-13: Checked `--keep-newer-than` before a backup starts
|
|
||||||
([issue #307](https://git.eeqj.de/sneak/vaultik/issues/307)); an
|
|
||||||
invalid value had been read only by the prune after the backup, so
|
|
||||||
retention never ran.
|
|
||||||
|
|
||||||
- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that
|
- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that
|
||||||
is not a blob as an orphaned blob
|
is not a blob as an orphaned blob
|
||||||
([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray
|
([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ The metadata subdirectory is named with the **remote key**, a one-way hash of th
|
|||||||
|
|
||||||
## Marker File (`vaultik-destination.txt`)
|
## Marker File (`vaultik-destination.txt`)
|
||||||
|
|
||||||
Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. A destination without it is treated as not present while the local index holds snapshots stored there: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
|
Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. A destination without it is treated as not present while the local index holds snapshots stored there: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove` and `snapshot list` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
|
||||||
|
|
||||||
## Blobs Directory (`blobs/`)
|
## Blobs Directory (`blobs/`)
|
||||||
|
|
||||||
|
|||||||
@@ -1,30 +0,0 @@
|
|||||||
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
|
|
||||||
// under --skip-errors, a source directory that does not exist, which the
|
|
||||||
// walk skips. The command exits non-zero and reports the backup as
|
|
||||||
// completed, not as a failed snapshot creation. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
|
||||||
//
|
|
||||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
|
||||||
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
|
|
||||||
// writeBackupConfig would create the source directory.
|
|
||||||
configPath := writeHermeticPruneConfig(t, false)
|
|
||||||
|
|
||||||
code, stdout, stderr := runEntry(t,
|
|
||||||
flagConfig, configPath, "--skip-errors", cmdSnapshot, cmdCreate)
|
|
||||||
|
|
||||||
assert.Equal(t, 1, code)
|
|
||||||
assert.Contains(t, stdout,
|
|
||||||
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
|
|
||||||
assert.NotContains(t, stdout, "Finished successfully.")
|
|
||||||
assert.Contains(t, stderr,
|
|
||||||
"Backup completed, but 1 file(s) could not be read")
|
|
||||||
assert.NotContains(t, stderr, "Snapshot creation failed")
|
|
||||||
}
|
|
||||||
@@ -115,13 +115,6 @@ on Linux), then /etc/vaultik/config.yml.`,
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
if errors.Is(err, vaultik.ErrFilesUnreadable) {
|
|
||||||
log.Error("Backup completed with unreadable files", "error", err)
|
|
||||||
ReportErrorf("Backup completed, but %v", err)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
log.Error("Snapshot creation failed", "error", err)
|
log.Error("Snapshot creation failed", "error", err)
|
||||||
ReportErrorf("Snapshot creation failed: %v", err)
|
ReportErrorf("Snapshot creation failed: %v", err)
|
||||||
})
|
})
|
||||||
@@ -132,8 +125,7 @@ on Linux), then /etc/vaultik/config.yml.`,
|
|||||||
"Run in cron mode (silent unless warning or error)")
|
"Run in cron mode (silent unless warning or error)")
|
||||||
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
|
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
|
||||||
"After backup, drop older snapshots of the same name and remove "+
|
"After backup, drop older snapshots of the same name and remove "+
|
||||||
"orphaned blobs (not after a backup that skipped unreadable "+
|
"orphaned blobs")
|
||||||
"files)")
|
|
||||||
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
||||||
"With --prune: keep snapshots newer than this duration "+
|
"With --prune: keep snapshots newer than this duration "+
|
||||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
||||||
|
|||||||
@@ -141,12 +141,10 @@ type ScannerConfig struct {
|
|||||||
// ScanResult contains the results of a scan operation. Files and bytes
|
// ScanResult contains the results of a scan operation. Files and bytes
|
||||||
// are counted per file: BytesScanned is the size of the new and changed
|
// are counted per file: BytesScanned is the size of the new and changed
|
||||||
// files, BytesSkipped that of the unchanged ones. FilesFailed counts the
|
// files, BytesSkipped that of the unchanged ones. FilesFailed counts the
|
||||||
// new and changed files that phase 2 could not store, each entry removed
|
// new and changed files that phase 2 could not store; FilesScanned
|
||||||
// after the walk listed its directory and, under --skip-errors, each
|
// includes them and BytesScanned does not. ChunksCreated counts the
|
||||||
// entry the walk could not read, such as a directory it could not list;
|
// chunks that were not already stored, and NewChunkSize is their total
|
||||||
// FilesScanned includes them and BytesScanned does not. ChunksCreated
|
// size.
|
||||||
// counts the chunks that were not already stored, and NewChunkSize is
|
|
||||||
// their total size.
|
|
||||||
type ScanResult struct {
|
type ScanResult struct {
|
||||||
FilesScanned int
|
FilesScanned int
|
||||||
FilesSkipped int
|
FilesSkipped int
|
||||||
@@ -163,11 +161,6 @@ type ScanResult struct {
|
|||||||
UploadDuration time.Duration
|
UploadDuration time.Duration
|
||||||
StartTime time.Time
|
StartTime time.Time
|
||||||
EndTime time.Time
|
EndTime time.Time
|
||||||
|
|
||||||
// FilesUnreadable is the part of FilesFailed that --skip-errors
|
|
||||||
// skipped because it could not be read or listed; the rest are files
|
|
||||||
// removed after the walk listed them.
|
|
||||||
FilesUnreadable int
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewScanner creates a new scanner instance
|
// NewScanner creates a new scanner instance
|
||||||
@@ -897,28 +890,8 @@ func (s *Scanner) scanPhase(
|
|||||||
log.Debug("Starting directory walk", "path", path)
|
log.Debug("Starting directory walk", "path", path)
|
||||||
|
|
||||||
walkFn := func(filePath string, info os.FileInfo, err error) error {
|
walkFn := func(filePath string, info os.FileInfo, err error) error {
|
||||||
// Check exclude patterns first, so an excluded entry that is gone
|
|
||||||
// or cannot be looked up is neither counted nor reported. For
|
|
||||||
// directories, skip the entire subtree. info is nil when the
|
|
||||||
// walk could not look the entry up.
|
|
||||||
if s.shouldExclude(filePath, path) {
|
|
||||||
if info != nil && info.IsDir() {
|
|
||||||
return filepath.SkipDir
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// The source path is in no directory listing, so its absence is
|
|
||||||
// an error rather than a removal.
|
|
||||||
if errors.Is(err, os.ErrNotExist) && filePath != path {
|
|
||||||
countRemovedDuringWalk(filePath, existingFiles, result)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return s.handleWalkError(filePath, err, result)
|
return s.handleWalkError(filePath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check context cancellation
|
// Check context cancellation
|
||||||
@@ -928,6 +901,15 @@ func (s *Scanner) scanPhase(
|
|||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Check exclude patterns - for directories, skip the entire subtree
|
||||||
|
if s.shouldExclude(filePath, path) {
|
||||||
|
if info.IsDir() {
|
||||||
|
return filepath.SkipDir
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// Handle symlinks and directories
|
// Handle symlinks and directories
|
||||||
handled, err := s.recordSpecialEntry(
|
handled, err := s.recordSpecialEntry(
|
||||||
filePath, info, existingFiles, knownFiles, collector, result)
|
filePath, info, existingFiles, knownFiles, collector, result)
|
||||||
@@ -1003,7 +985,7 @@ func (s *Scanner) recordSpecialEntry(
|
|||||||
if info.Mode()&os.ModeSymlink != 0 {
|
if info.Mode()&os.ModeSymlink != 0 {
|
||||||
file, err := s.buildSymlinkEntry(filePath, info)
|
file, err := s.buildSymlinkEntry(filePath, info)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return true, s.handleWalkError(filePath, err, result)
|
return true, s.handleWalkError(filePath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
existingFiles[filePath] = struct{}{}
|
existingFiles[filePath] = struct{}{}
|
||||||
@@ -1027,38 +1009,15 @@ func (s *Scanner) recordSpecialEntry(
|
|||||||
return false, nil
|
return false, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// countRemovedDuringWalk counts an entry that was in its directory's
|
|
||||||
// listing but was gone when the walk looked it up, like a file removed
|
|
||||||
// before phase 2 reads it: as examined and failed, and not as an error,
|
|
||||||
// with or without --skip-errors.
|
|
||||||
func countRemovedDuringWalk(
|
|
||||||
filePath string, existingFiles map[string]struct{}, result *ScanResult,
|
|
||||||
) {
|
|
||||||
log.Warn("File was deleted during the directory walk, skipping",
|
|
||||||
"path", filePath)
|
|
||||||
|
|
||||||
// The walk found it, so it is not also counted as deleted.
|
|
||||||
existingFiles[filePath] = struct{}{}
|
|
||||||
result.FilesScanned++
|
|
||||||
result.FilesFailed++
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleWalkError deals with a filesystem error surfaced by the walk:
|
// handleWalkError deals with a filesystem error surfaced by the walk:
|
||||||
// under --skip-errors the entry is skipped with a UI notice and counted
|
// skipped (with a UI notice) under --skip-errors, fatal otherwise.
|
||||||
// as examined, failed and unreadable, otherwise the error is fatal.
|
func (s *Scanner) handleWalkError(filePath string, err error) error {
|
||||||
func (s *Scanner) handleWalkError(
|
|
||||||
filePath string, err error, result *ScanResult,
|
|
||||||
) error {
|
|
||||||
if s.skipErrors {
|
if s.skipErrors {
|
||||||
log.Error("Failed to access file (skipping due to --skip-errors)",
|
log.Error("Failed to access file (skipping due to --skip-errors)",
|
||||||
"path", filePath, "error", err)
|
"path", filePath, "error", err)
|
||||||
s.ui.Errorf("Failed to access %s: %v. Skipping (--skip-errors).",
|
s.ui.Errorf("Failed to access %s: %v. Skipping (--skip-errors).",
|
||||||
s.ui.Path(filePath), err)
|
s.ui.Path(filePath), err)
|
||||||
|
|
||||||
result.FilesScanned++
|
|
||||||
result.FilesFailed++
|
|
||||||
result.FilesUnreadable++
|
|
||||||
|
|
||||||
return nil // Continue scanning
|
return nil // Continue scanning
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1500,7 +1459,6 @@ func (s *Scanner) processFileWithErrorHandling(
|
|||||||
s.ui.Path(fileToProcess.Path), err)
|
s.ui.Path(fileToProcess.Path), err)
|
||||||
|
|
||||||
countFailedFile(fileToProcess, result)
|
countFailedFile(fileToProcess, result)
|
||||||
result.FilesUnreadable++
|
|
||||||
|
|
||||||
return true, nil
|
return true, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -152,25 +152,6 @@ func TestPurgeRefusesDestinationWithoutMarker(t *testing.T) {
|
|||||||
assertUntouched(ctx, t, repos, storeDir)
|
assertUntouched(ctx, t, repos, storeDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestNukeRemoteRefusesDestinationWithoutMarker checks that remote nuke
|
|
||||||
// fails, deletes no local snapshot record, and neither reports the
|
|
||||||
// destination empty nor advises running nuke again when the destination
|
|
||||||
// has no marker.
|
|
||||||
func TestNukeRemoteRefusesDestinationWithoutMarker(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
v, repos, out, storeDir := backUpThenUnplugMountPoint(ctx, t)
|
|
||||||
|
|
||||||
err := v.NukeRemote(true)
|
|
||||||
require.ErrorContains(t, err, markerFile)
|
|
||||||
assert.NotContains(t, out.String(), "now empty")
|
|
||||||
assert.NotContains(t, out.String(), "vaultik remote nuke --force")
|
|
||||||
|
|
||||||
assertUntouched(ctx, t, repos, storeDir)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRemoveSnapshotWarnsWhenDestinationHasNoMarker checks that snapshot
|
// TestRemoveSnapshotWarnsWhenDestinationHasNoMarker checks that snapshot
|
||||||
// remove warns that the metadata could not be removed from a destination
|
// remove warns that the metadata could not be removed from a destination
|
||||||
// without a marker, instead of reporting that it was.
|
// without a marker, instead of reporting that it was.
|
||||||
|
|||||||
@@ -188,41 +188,3 @@ func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
|
|||||||
assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
|
assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
|
||||||
err.Error())
|
err.Error())
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestNukeRemoteKeepsLocalIndexWhenDestinationMissing checks that remote
|
|
||||||
// nuke fails on a destination it cannot list, leaves the local index
|
|
||||||
// unchanged, and neither reports the destination empty nor advises
|
|
||||||
// running nuke again.
|
|
||||||
func TestNukeRemoteKeepsLocalIndexWhenDestinationMissing(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
v, repos, out := backUpThenUnplug(ctx, t)
|
|
||||||
|
|
||||||
filesBefore, err := repos.Files.ListAll(ctx)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NotEmpty(t, filesBefore)
|
|
||||||
|
|
||||||
blobsBefore, err := repos.Blobs.GetAll(ctx)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NotEmpty(t, blobsBefore)
|
|
||||||
|
|
||||||
err = v.NukeRemote(true)
|
|
||||||
require.ErrorIs(t, err, fs.ErrNotExist)
|
|
||||||
require.ErrorContains(t, err, "listing remote snapshots")
|
|
||||||
assert.NotContains(t, out.String(), "now empty")
|
|
||||||
assert.NotContains(t, out.String(), "vaultik remote nuke --force")
|
|
||||||
|
|
||||||
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, snapshots, 1, "nuke must delete no local snapshot record")
|
|
||||||
|
|
||||||
files, err := repos.Files.ListAll(ctx)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, files, len(filesBefore), "nuke must delete no local file record")
|
|
||||||
|
|
||||||
blobs, err := repos.Blobs.GetAll(ctx)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, blobs, len(blobsBefore), "nuke must delete no local blob record")
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -104,32 +104,6 @@ func TestNukeRemoteFailsWhenADeleteFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestNukeRemoteKeepsLocalSnapshotWhoseMetadataIsNotDeleted checks that
|
|
||||||
// remote nuke keeps a snapshot in the local index while its metadata
|
|
||||||
// cannot be deleted from the destination store: dropped from the local
|
|
||||||
// index, it would be left there as a remote-only snapshot without its
|
|
||||||
// name.
|
|
||||||
func TestNukeRemoteKeepsLocalSnapshotWhoseMetadataIsNotDeleted(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
storeDir := filepath.Join(t.TempDir(), "store")
|
|
||||||
v, repos, _ := backUpToFileDestination(ctx, t, storeDir)
|
|
||||||
|
|
||||||
store := faultstore.New(v.Storage)
|
|
||||||
store.OnDelete = func(key string) bool {
|
|
||||||
return strings.HasPrefix(key, "metadata/")
|
|
||||||
}
|
|
||||||
v.Storage = store
|
|
||||||
|
|
||||||
require.Error(t, v.NukeRemote(true))
|
|
||||||
|
|
||||||
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, snapshots, 1)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestNukeRemoteNamesWhatIsLeft checks that remote nuke, when an object
|
// TestNukeRemoteNamesWhatIsLeft checks that remote nuke, when an object
|
||||||
// it does not recognise cannot be deleted, does not report the store
|
// it does not recognise cannot be deleted, does not report the store
|
||||||
// empty but names that object and fails.
|
// empty but names that object and fails.
|
||||||
@@ -180,7 +154,8 @@ func TestNukeRemoteRefusesAnotherDestinationWithoutRetryAdvice(t *testing.T) {
|
|||||||
|
|
||||||
// TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved checks that a
|
// TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved checks that a
|
||||||
// snapshot remote nuke cannot remove from the local index fails the
|
// snapshot remote nuke cannot remove from the local index fails the
|
||||||
// run, and that nuke then does not report the destination store empty.
|
// run: nuke then leaves that snapshot's metadata on the destination
|
||||||
|
// store.
|
||||||
func TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved(t *testing.T) {
|
func TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|||||||
@@ -55,19 +55,12 @@ func (v *Vaultik) NukeRemote(force bool) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Checked before anything is deleted: running nuke again would fail
|
// Checked before anything is deleted: running nuke again would fail
|
||||||
// the same way, so the advice below must not follow these errors.
|
// the same way, so the advice below must not follow this error.
|
||||||
// The listing also fails on a destination without the marker while
|
|
||||||
// the local index holds snapshots.
|
|
||||||
err := v.EnsureStorageBinding()
|
err := v.EnsureStorageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err = v.listAllRemoteSnapshotKeys()
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
err = v.emptyDestinationStore()
|
err = v.emptyDestinationStore()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
v.UI.Warningf("Run '%s' again to delete what is left on the "+
|
v.UI.Warningf("Run '%s' again to delete what is left on the "+
|
||||||
|
|||||||
@@ -39,11 +39,6 @@ var (
|
|||||||
// caller can report it apart from a failed backup.
|
// caller can report it apart from a failed backup.
|
||||||
var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
|
var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
|
||||||
|
|
||||||
// ErrFilesUnreadable is returned when --skip-errors skipped files or
|
|
||||||
// directories it could not read or list. The snapshot is complete and
|
|
||||||
// kept, so the caller can report it apart from a failed backup.
|
|
||||||
var ErrFilesUnreadable = errors.New("file(s) could not be read")
|
|
||||||
|
|
||||||
// listRecentLimit caps how many snapshot rows are fetched from the
|
// listRecentLimit caps how many snapshot rows are fetched from the
|
||||||
// local index for listing and purge operations.
|
// local index for listing and purge operations.
|
||||||
const listRecentLimit = 10000
|
const listRecentLimit = 10000
|
||||||
@@ -135,16 +130,12 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Process each named snapshot
|
// Process each named snapshot
|
||||||
filesUnreadable := 0
|
|
||||||
|
|
||||||
for snapIdx, snapName := range snapshotNames {
|
for snapIdx, snapName := range snapshotNames {
|
||||||
unreadable, err := v.createNamedSnapshot(
|
err = v.createNamedSnapshot(
|
||||||
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
filesUnreadable += unreadable
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Print overall summary if multiple snapshots
|
// Print overall summary if multiple snapshots
|
||||||
@@ -153,34 +144,13 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
|||||||
len(snapshotNames), v.UI.Duration(time.Since(overallStartTime)))
|
len(snapshotNames), v.UI.Duration(time.Since(overallStartTime)))
|
||||||
}
|
}
|
||||||
|
|
||||||
switch {
|
if opts.Prune {
|
||||||
case opts.Prune && filesUnreadable > 0:
|
|
||||||
// An older snapshot may be the only one that still holds a file
|
|
||||||
// this run could not read.
|
|
||||||
log.Warn("Skipping post-backup prune",
|
|
||||||
"files_unreadable", filesUnreadable)
|
|
||||||
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
|
|
||||||
"could not be read; older snapshots are kept.", filesUnreadable)
|
|
||||||
case opts.Prune:
|
|
||||||
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("%w: %w", ErrPostBackupPruneFailed, err)
|
return fmt.Errorf("%w: %w", ErrPostBackupPruneFailed, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return v.finishCreateSnapshot(filesUnreadable)
|
|
||||||
}
|
|
||||||
|
|
||||||
// finishCreateSnapshot ends a run whose snapshots are all complete and
|
|
||||||
// kept. Like finishRestore, it returns an error when --skip-errors
|
|
||||||
// skipped anything it could not read or list; otherwise it prints the
|
|
||||||
// closing line. A file removed during the run is counted as failed in
|
|
||||||
// the summary but is not an error, with or without --skip-errors.
|
|
||||||
func (v *Vaultik) finishCreateSnapshot(filesUnreadable int) error {
|
|
||||||
if filesUnreadable > 0 {
|
|
||||||
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Terminus must obey the --cron invariant: silent on total
|
// Terminus must obey the --cron invariant: silent on total
|
||||||
// success only. UI.Complete is dropped in cron/quiet mode (that's
|
// success only. UI.Complete is dropped in cron/quiet mode (that's
|
||||||
// the success path), but if any warnings fired during the run we
|
// the success path), but if any warnings fired during the run we
|
||||||
@@ -270,22 +240,16 @@ type snapshotStats struct {
|
|||||||
totalBlobsUploaded int
|
totalBlobsUploaded int
|
||||||
uploadDuration time.Duration
|
uploadDuration time.Duration
|
||||||
|
|
||||||
// The part of totalFilesFailed that --skip-errors skipped because it
|
|
||||||
// could not be read or listed.
|
|
||||||
totalFilesUnreadable int
|
|
||||||
|
|
||||||
// The sizes of all blobs the snapshot references, set by
|
// The sizes of all blobs the snapshot references, set by
|
||||||
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
||||||
blobSize int64
|
blobSize int64
|
||||||
blobUncompressedSize int64
|
blobUncompressedSize int64
|
||||||
}
|
}
|
||||||
|
|
||||||
// createNamedSnapshot creates a single named snapshot and returns the
|
// createNamedSnapshot creates a single named snapshot
|
||||||
// number of entries --skip-errors skipped because it could not read or
|
|
||||||
// list them.
|
|
||||||
func (v *Vaultik) createNamedSnapshot(
|
func (v *Vaultik) createNamedSnapshot(
|
||||||
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
|
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
|
||||||
) (int, error) {
|
) error {
|
||||||
snapshotStartTime := time.Now()
|
snapshotStartTime := time.Now()
|
||||||
|
|
||||||
if total > 1 {
|
if total > 1 {
|
||||||
@@ -294,12 +258,12 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
|
|
||||||
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
|
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
previousSnapshotID, err := v.previousSnapshotID(snapName)
|
previousSnapshotID, err := v.previousSnapshotID(snapName)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
scanner := v.ScannerFactory(snapshot.ScannerParams{
|
scanner := v.ScannerFactory(snapshot.ScannerParams{
|
||||||
@@ -314,7 +278,7 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
|
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
|
||||||
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
|
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, fmt.Errorf("creating snapshot: %w", err)
|
return fmt.Errorf("creating snapshot: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
|
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
|
||||||
@@ -322,12 +286,12 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
|
|
||||||
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
|
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
err = v.finalizeSnapshotMetadata(snapshotID, stats)
|
err = v.finalizeSnapshotMetadata(snapshotID, stats)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
log.Info("Snapshot complete",
|
log.Info("Snapshot complete",
|
||||||
@@ -340,7 +304,7 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
|
|
||||||
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
||||||
|
|
||||||
return stats.totalFilesUnreadable, nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// previousSnapshotID returns the ID of the most recent completed snapshot
|
// previousSnapshotID returns the ID of the most recent completed snapshot
|
||||||
@@ -426,7 +390,6 @@ func (v *Vaultik) scanAllDirectories(
|
|||||||
stats.totalBlobs += result.BlobsCreated
|
stats.totalBlobs += result.BlobsCreated
|
||||||
stats.totalFilesSkipped += result.FilesSkipped
|
stats.totalFilesSkipped += result.FilesSkipped
|
||||||
stats.totalFilesFailed += result.FilesFailed
|
stats.totalFilesFailed += result.FilesFailed
|
||||||
stats.totalFilesUnreadable += result.FilesUnreadable
|
|
||||||
stats.totalBytesSkipped += result.BytesSkipped
|
stats.totalBytesSkipped += result.BytesSkipped
|
||||||
stats.totalFilesDeleted += result.FilesDeleted
|
stats.totalFilesDeleted += result.FilesDeleted
|
||||||
stats.totalBytesDeleted += result.BytesDeleted
|
stats.totalBytesDeleted += result.BytesDeleted
|
||||||
@@ -1553,9 +1516,7 @@ func (v *Vaultik) removeSnapshotRemote(
|
|||||||
// database from the local index, and (with --remote) every snapshot
|
// database from the local index, and (with --remote) every snapshot
|
||||||
// metadata directory in remote storage. Both sides are processed so a
|
// metadata directory in remote storage. Both sides are processed so a
|
||||||
// "remove --all" leaves nothing behind, even when the local DB and
|
// "remove --all" leaves nothing behind, even when the local DB and
|
||||||
// remote storage have diverged. Unless LocalOnly is set, it fails
|
// remote storage have diverged.
|
||||||
// before it removes anything when the destination store cannot be
|
|
||||||
// listed or is not present by checkDestinationMarker.
|
|
||||||
func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) {
|
func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) {
|
||||||
err := v.EnsureStorageBinding()
|
err := v.EnsureStorageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1568,15 +1529,11 @@ func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error)
|
|||||||
}
|
}
|
||||||
|
|
||||||
// remoteKeys is the set of metadata/<key>/ subdirectories on the
|
// remoteKeys is the set of metadata/<key>/ subdirectories on the
|
||||||
// destination store. Only for a local-only remove is a failure
|
// destination store; failures are downgraded to a warning so a
|
||||||
// downgraded to a warning, so a permission-denied or unreachable
|
// permission-denied or unreachable remote can't block a local-only
|
||||||
// remote can't block it.
|
// remove.
|
||||||
remoteKeys, remoteErr := v.listAllRemoteSnapshotKeys()
|
remoteKeys, remoteErr := v.listAllRemoteSnapshotKeys()
|
||||||
if remoteErr != nil {
|
if remoteErr != nil {
|
||||||
if !opts.LocalOnly {
|
|
||||||
return nil, remoteErr
|
|
||||||
}
|
|
||||||
|
|
||||||
log.Warn("Could not list remote snapshots", "error", remoteErr)
|
log.Warn("Could not list remote snapshots", "error", remoteErr)
|
||||||
v.UI.Warningf("Could not list remote snapshots: %v.", remoteErr)
|
v.UI.Warningf("Could not list remote snapshots: %v.", remoteErr)
|
||||||
}
|
}
|
||||||
@@ -1638,7 +1595,8 @@ func (v *Vaultik) localSnapshotIDs() ([]string, error) {
|
|||||||
// listAllRemoteSnapshotKeys collects the hashed remote keys
|
// listAllRemoteSnapshotKeys collects the hashed remote keys
|
||||||
// (subdirectories under metadata/) currently present in the
|
// (subdirectories under metadata/) currently present in the
|
||||||
// destination store. Returns (nil, err) when the store cannot be
|
// destination store. Returns (nil, err) when the store cannot be
|
||||||
// listed, or is not present by checkDestinationMarker.
|
// listed, or is not present by checkDestinationMarker; callers must
|
||||||
|
// treat that as "no remote info available," not fatal.
|
||||||
func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) {
|
func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) {
|
||||||
log.Info("Listing all remote snapshots")
|
log.Info("Listing all remote snapshots")
|
||||||
|
|
||||||
@@ -1822,12 +1780,10 @@ func (v *Vaultik) deleteIncompleteSnapshot(snapshotIDStr string, id any) bool {
|
|||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// removeAllLocalSnapshots removes each known local snapshot's remote
|
// removeAllLocalSnapshots removes each known local snapshot (and, unless
|
||||||
// metadata, unless --local-only, and then its local record, returning
|
// --local-only, its remote metadata), returning the number of snapshots
|
||||||
// the number of snapshots it could not remove. The local record goes
|
// it could not remove. A snapshot left in the local index keeps its
|
||||||
// only once the metadata is deleted: dropped first, it would leave that
|
// remote metadata too, since it is not an orphan remote key.
|
||||||
// metadata on the destination store as a remote-only snapshot whose name
|
|
||||||
// is lost.
|
|
||||||
func (v *Vaultik) removeAllLocalSnapshots(
|
func (v *Vaultik) removeAllLocalSnapshots(
|
||||||
localSnaps []string, opts *RemoveOptions, result *RemoveResult,
|
localSnaps []string, opts *RemoveOptions, result *RemoveResult,
|
||||||
) int {
|
) int {
|
||||||
@@ -1836,19 +1792,6 @@ func (v *Vaultik) removeAllLocalSnapshots(
|
|||||||
for _, snapshotID := range localSnaps {
|
for _, snapshotID := range localSnaps {
|
||||||
log.Info("Removing snapshot", "snapshot_id", snapshotID)
|
log.Info("Removing snapshot", "snapshot_id", snapshotID)
|
||||||
|
|
||||||
if !opts.LocalOnly {
|
|
||||||
_, err := v.deleteRemoteSnapshotByKey(
|
|
||||||
snapshot.RemoteSnapshotKey(snapshotID))
|
|
||||||
if err != nil {
|
|
||||||
log.Warn("Failed to remove snapshot metadata from remote",
|
|
||||||
"snapshot_id", snapshotID, "error", err)
|
|
||||||
|
|
||||||
failures++
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err := v.deleteSnapshotFromLocalDB(snapshotID)
|
err := v.deleteSnapshotFromLocalDB(snapshotID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Error("Failed to remove from local database",
|
log.Error("Failed to remove from local database",
|
||||||
@@ -1859,6 +1802,17 @@ func (v *Vaultik) removeAllLocalSnapshots(
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if !opts.LocalOnly {
|
||||||
|
_, err = v.deleteRemoteSnapshotByKey(
|
||||||
|
snapshot.RemoteSnapshotKey(snapshotID))
|
||||||
|
if err != nil {
|
||||||
|
log.Warn("Failed to remove snapshot metadata from remote",
|
||||||
|
"snapshot_id", snapshotID, "error", err)
|
||||||
|
|
||||||
|
failures++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID)
|
result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -178,13 +178,11 @@ func TestBackupDoesNotCountFileInUnopenableDirectoryAsDeleted(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
env.out.Reset()
|
env.out.Reset()
|
||||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||||
SkipErrors: true,
|
SkipErrors: true,
|
||||||
Snapshots: []string{"work"},
|
Snapshots: []string{"work"},
|
||||||
})
|
}))
|
||||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
assert.Contains(t, env.out.String(), filesLine(1, 0, 1))
|
||||||
assert.Contains(t, env.out.String(),
|
|
||||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// A backup counts only files that the previous snapshot of its own name
|
// A backup counts only files that the previous snapshot of its own name
|
||||||
|
|||||||
@@ -49,19 +49,13 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
|||||||
name string
|
name string
|
||||||
openErr error
|
openErr error
|
||||||
skipErrors bool
|
skipErrors bool
|
||||||
wantErr error // nil for none
|
|
||||||
}{
|
}{
|
||||||
// What a normal user gets opening a file with mode 000.
|
// What a normal user gets opening a file with mode 000.
|
||||||
{name: "unopenable under skip-errors",
|
{name: "unopenable under skip-errors",
|
||||||
openErr: os.ErrPermission, skipErrors: true,
|
openErr: os.ErrPermission, skipErrors: true},
|
||||||
wantErr: vaultik.ErrFilesUnreadable},
|
|
||||||
// What opening a file removed after phase 1 gives.
|
// What opening a file removed after phase 1 gives.
|
||||||
{name: "removed between the phases",
|
{name: "removed between the phases",
|
||||||
openErr: os.ErrNotExist, skipErrors: false},
|
openErr: os.ErrNotExist, skipErrors: false},
|
||||||
// A removal is not an error, so --skip-errors does not make the
|
|
||||||
// run fail. See https://git.eeqj.de/sneak/vaultik/issues/310.
|
|
||||||
{name: "removed between the phases under skip-errors",
|
|
||||||
openErr: os.ErrNotExist, skipErrors: true},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
@@ -105,11 +99,10 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
|||||||
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
|
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
|
||||||
v.UI = ui.NewWithColor(out, false)
|
v.UI = ui.NewWithColor(out, false)
|
||||||
|
|
||||||
err = v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
require.NoError(t, v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||||
SkipErrors: tt.skipErrors,
|
SkipErrors: tt.skipErrors,
|
||||||
Snapshots: []string{"src"},
|
Snapshots: []string{"src"},
|
||||||
})
|
}))
|
||||||
require.ErrorIs(t, err, tt.wantErr)
|
|
||||||
|
|
||||||
summary := out.String()
|
summary := out.String()
|
||||||
assert.Contains(t, summary,
|
assert.Contains(t, summary,
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package vaultik_test
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
@@ -74,210 +73,3 @@ func TestSkipErrorsBackupStopsOnIndexErrorRecordingDirectory(t *testing.T) {
|
|||||||
require.Len(t, snapshots, 1)
|
require.Len(t, snapshots, 1)
|
||||||
assert.Nil(t, snapshots[0].CompletedAt)
|
assert.Nil(t, snapshots[0].CompletedAt)
|
||||||
}
|
}
|
||||||
|
|
||||||
// A backup under --skip-errors that skips a file it cannot open, or a
|
|
||||||
// directory it cannot list, keeps the snapshot, counts what it skipped
|
|
||||||
// as failed and returns an error instead of finishing successfully. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
|
||||||
func TestSkipErrorsBackupKeepsSnapshotAndFailsWhenItSkips(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
skipped string // relative to the source directory
|
|
||||||
}{
|
|
||||||
{name: "file it cannot open", skipped: "skipdir/file.txt"},
|
|
||||||
{name: "directory it cannot list", skipped: "skipdir"},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newDeletedCountEnv(t, map[string][]byte{
|
|
||||||
"stored.txt": []byte("stored"),
|
|
||||||
"skipdir/file.txt": []byte("in skipdir"),
|
|
||||||
})
|
|
||||||
env.setSnapshot("work", "")
|
|
||||||
|
|
||||||
// A mode does not lock out root, who runs the tests, so
|
|
||||||
// opening the skipped path is made to fail instead.
|
|
||||||
env.v.Fs = &openFailFs{
|
|
||||||
path: filepath.Join(env.src, tt.skipped),
|
|
||||||
err: os.ErrPermission,
|
|
||||||
}
|
|
||||||
|
|
||||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
|
||||||
SkipErrors: true,
|
|
||||||
Snapshots: []string{"work"},
|
|
||||||
})
|
|
||||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
|
||||||
|
|
||||||
assert.Contains(t, env.out.String(),
|
|
||||||
"Files: 2 examined, 1 backed up, 0 unchanged, 1 failed.")
|
|
||||||
assert.NotContains(t, env.out.String(), "Finished successfully.")
|
|
||||||
|
|
||||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
|
||||||
context.Background(), listRecentTestLimit)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.Len(t, snapshots, 1)
|
|
||||||
assert.NotNil(t, snapshots[0].CompletedAt, "the snapshot is kept")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// removeOnLookupFs is the real filesystem, except that path is removed
|
|
||||||
// just before it is looked up. The walk looks an entry up after listing
|
|
||||||
// its directory, so path is listed but gone when the walk reaches it.
|
|
||||||
type removeOnLookupFs struct {
|
|
||||||
afero.OsFs
|
|
||||||
|
|
||||||
path string
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *removeOnLookupFs) LstatIfPossible(
|
|
||||||
name string,
|
|
||||||
) (os.FileInfo, bool, error) {
|
|
||||||
if name == f.path {
|
|
||||||
_ = os.Remove(name)
|
|
||||||
}
|
|
||||||
|
|
||||||
return f.OsFs.LstatIfPossible(name)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A file removed before the walk reaches it is counted once, as failed,
|
|
||||||
// like a file removed before its data is read, and the backup finishes
|
|
||||||
// successfully, with or without --skip-errors. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
|
||||||
func TestBackupCountsFileRemovedBeforeWalkReachesItAsFailed(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
skipErrors bool
|
|
||||||
}{
|
|
||||||
{name: "under skip-errors", skipErrors: true},
|
|
||||||
{name: "without skip-errors", skipErrors: false},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newDeletedCountEnv(t, map[string][]byte{
|
|
||||||
"stays.txt": []byte("stays"),
|
|
||||||
"removed.txt": []byte("removed"),
|
|
||||||
})
|
|
||||||
env.setSnapshot("walk", "")
|
|
||||||
|
|
||||||
// The previous snapshot has the file, so a removal the walk
|
|
||||||
// also counted as deleted would show.
|
|
||||||
env.backUp(t, "walk")
|
|
||||||
|
|
||||||
env.v.Fs = &removeOnLookupFs{
|
|
||||||
path: filepath.Join(env.src, "removed.txt"),
|
|
||||||
}
|
|
||||||
|
|
||||||
env.out.Reset()
|
|
||||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
|
||||||
SkipErrors: tt.skipErrors,
|
|
||||||
Snapshots: []string{"walk"},
|
|
||||||
}))
|
|
||||||
assert.Contains(t, env.out.String(),
|
|
||||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
|
||||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// A file an exclude pattern matches is not counted when it is removed
|
|
||||||
// before the walk reaches it. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
|
||||||
func TestBackupIgnoresExcludedFileRemovedBeforeWalkReachesIt(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newDeletedCountEnv(t, map[string][]byte{
|
|
||||||
"stays.txt": []byte("stays"),
|
|
||||||
"removed.tmp": []byte("removed"),
|
|
||||||
})
|
|
||||||
env.setSnapshot("walk", "", "*.tmp")
|
|
||||||
env.v.Fs = &removeOnLookupFs{
|
|
||||||
path: filepath.Join(env.src, "removed.tmp"),
|
|
||||||
}
|
|
||||||
|
|
||||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
|
||||||
SkipErrors: true,
|
|
||||||
Snapshots: []string{"walk"},
|
|
||||||
}))
|
|
||||||
assert.Contains(t, env.out.String(), filesLine(1, 1, 0))
|
|
||||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
|
||||||
}
|
|
||||||
|
|
||||||
// A backup under --skip-errors --prune that skipped a file it could not
|
|
||||||
// read does not prune: the previous snapshot, the only one that holds
|
|
||||||
// the file, is kept, and the output says the prune was skipped. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
|
||||||
func TestSkipErrorsBackupDoesNotPruneWhenItSkips(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newDeletedCountEnv(t, map[string][]byte{
|
|
||||||
"other.txt": []byte("other"),
|
|
||||||
"skipped.txt": []byte("skipped"),
|
|
||||||
})
|
|
||||||
env.setSnapshot("photos", "")
|
|
||||||
env.backUp(t, "photos")
|
|
||||||
|
|
||||||
// An unchanged file is not read again, so the file is changed before
|
|
||||||
// it becomes unreadable.
|
|
||||||
skipped := filepath.Join(env.src, "skipped.txt")
|
|
||||||
require.NoError(t, os.WriteFile(skipped, []byte("changed"), 0o600))
|
|
||||||
env.v.Fs = &openFailFs{path: skipped, err: os.ErrPermission}
|
|
||||||
|
|
||||||
env.out.Reset()
|
|
||||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
|
||||||
Prune: true,
|
|
||||||
SkipErrors: true,
|
|
||||||
Snapshots: []string{"photos"},
|
|
||||||
})
|
|
||||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
|
||||||
assert.Contains(t, env.out.String(),
|
|
||||||
"Skipped the post-backup prune because 1 file(s) could not be read")
|
|
||||||
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
|
|
||||||
|
|
||||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
|
||||||
context.Background(), listRecentTestLimit)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, snapshots, 2, "the previous snapshot is kept")
|
|
||||||
}
|
|
||||||
|
|
||||||
// A file removed during a backup under --skip-errors --prune is not a
|
|
||||||
// skipped file, so the backup prunes as usual. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
|
||||||
func TestSkipErrorsBackupPrunesWhenAFileIsRemovedDuringIt(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newDeletedCountEnv(t, map[string][]byte{
|
|
||||||
"other.txt": []byte("other"),
|
|
||||||
"removed.txt": []byte("removed"),
|
|
||||||
})
|
|
||||||
env.setSnapshot("photos", "")
|
|
||||||
env.backUp(t, "photos")
|
|
||||||
|
|
||||||
env.v.Fs = &removeOnLookupFs{path: filepath.Join(env.src, "removed.txt")}
|
|
||||||
|
|
||||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
|
||||||
Prune: true,
|
|
||||||
SkipErrors: true,
|
|
||||||
Snapshots: []string{"photos"},
|
|
||||||
}))
|
|
||||||
|
|
||||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
|
||||||
context.Background(), listRecentTestLimit)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, snapshots, 1, "the previous snapshot is pruned")
|
|
||||||
}
|
|
||||||
|
|||||||
Reference in New Issue
Block a user