Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1ead24d193 |
+9
-10
@@ -359,7 +359,7 @@ CreateSnapshot(opts)
|
|||||||
|
|
||||||
## Deduplication Strategy
|
## Deduplication Strategy
|
||||||
|
|
||||||
1. **File-level**: Files and symlinks unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked
|
1. **File-level**: Files unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked
|
||||||
|
|
||||||
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded.
|
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded.
|
||||||
|
|
||||||
@@ -382,15 +382,14 @@ bucket/
|
|||||||
└── manifest.json.zst # Blob list (for pruning/verification)
|
└── manifest.json.zst # Blob list (for pruning/verification)
|
||||||
```
|
```
|
||||||
|
|
||||||
A destination without `vaultik-destination.txt` is treated as not present
|
While the local index holds snapshots, a destination without
|
||||||
while the local index holds snapshots stored there (`checkDestinationMarker`
|
`vaultik-destination.txt` is treated as not present
|
||||||
in `internal/vaultik/destination_marker.go`): `snapshot create` refuses to
|
(`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
|
||||||
write to it, `prune`, `snapshot purge` and `remote nuke` fail, and
|
`snapshot create` refuses to write to it, and listing it fails as for a
|
||||||
`snapshot list` and `snapshot remove` behave as for a destination that cannot
|
destination that cannot be listed. An unplugged volume's empty mount point
|
||||||
be listed. This keeps those commands from taking an unplugged volume's empty
|
is therefore never taken for a destination whose snapshots are all gone.
|
||||||
mount point for a destination whose snapshots are all gone. The `file://`
|
The `file://` backend never creates its destination directory, so a write
|
||||||
backend never creates its destination directory, so a write to a missing one
|
to a missing one fails.
|
||||||
fails.
|
|
||||||
|
|
||||||
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
|
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
|
||||||
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
|
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
|
||||||
|
|||||||
@@ -182,7 +182,7 @@ vaultik version
|
|||||||
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
|
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
|
||||||
* `--debug`: Enable debug output (on stderr — see below)
|
* `--debug`: Enable debug output (on stderr — see below)
|
||||||
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
|
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
|
||||||
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run. A run that skipped anything still finishes but exits non-zero: `snapshot create` keeps the snapshot and counts each file or directory it skipped as failed. A file removed during the backup is also counted as failed but does not change the exit status.
|
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run.
|
||||||
|
|
||||||
### locking
|
### locking
|
||||||
|
|
||||||
@@ -297,11 +297,7 @@ value where a list belongs or a word where a number belongs), is refused.
|
|||||||
(for crontab)
|
(for crontab)
|
||||||
* `--prune`: After backup, drop older snapshots of each backed-up name and
|
* `--prune`: After backup, drop older snapshots of each backed-up name and
|
||||||
remove orphaned blobs from remote storage. By default keeps only the latest
|
remove orphaned blobs from remote storage. By default keeps only the latest
|
||||||
snapshot per name; use `--keep-newer-than` for a rolling window. When
|
snapshot per name; use `--keep-newer-than` for a rolling window.
|
||||||
`--skip-errors` skipped a file or directory it could not read or list, the
|
|
||||||
prune does not run and every older snapshot is kept, since one of them may
|
|
||||||
be the only one that holds what was skipped; the command says so and exits
|
|
||||||
non-zero. A file removed during the backup does not stop the prune.
|
|
||||||
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
||||||
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
||||||
Without `--prune`, or with a value that is not a valid duration, the
|
Without `--prune`, or with a value that is not a valid duration, the
|
||||||
@@ -344,9 +340,9 @@ gives: revert the configuration, or run `vaultik database delete`.
|
|||||||
If the destination store cannot be listed (unmounted volume,
|
If the destination store cannot be listed (unmounted volume,
|
||||||
permission denied, network down), the command warns, falls back to the
|
permission denied, network down), the command warns, falls back to the
|
||||||
local index alone, and still exits zero. It does the same when the
|
local index alone, and still exits zero. It does the same when the
|
||||||
destination has no `vaultik-destination.txt` while the local index
|
local index holds snapshots and the destination has no
|
||||||
holds snapshots stored there, as when a volume mounted at the
|
`vaultik-destination.txt`, as when a volume mounted at the destination
|
||||||
destination is unplugged and leaves its empty mount point behind.
|
is unplugged and leaves its empty mount point behind.
|
||||||
* `--json`: Output in JSON format. Each entry carries `locally_tracked`
|
* `--json`: Output in JSON format. Each entry carries `locally_tracked`
|
||||||
(whether the snapshot is in the local index), `remote_key` (the full
|
(whether the snapshot is in the local index), `remote_key` (the full
|
||||||
64-character storage key), and `remote_present` (whether it was seen
|
64-character storage key), and `remote_present` (whether it was seen
|
||||||
@@ -458,14 +454,9 @@ since `prune` never deletes it.
|
|||||||
the backup destination store, whatever its name, and then
|
the backup destination store, whatever its name, and then
|
||||||
`vaultik-destination.txt`, leaving the bucket prefix empty. Destructive
|
`vaultik-destination.txt`, leaving the bucket prefix empty. Destructive
|
||||||
and irreversible. This is the single supported way to wipe the entire
|
and irreversible. This is the single supported way to wipe the entire
|
||||||
destination store. A snapshot is removed from the local index only once
|
destination store. If a listing afterwards still finds anything under
|
||||||
its metadata is deleted from the destination store. If the destination
|
`metadata/` or `blobs/`, the command names each object left and exits
|
||||||
store cannot be listed, or the local index holds snapshots and the
|
non-zero; running it again deletes what is left.
|
||||||
destination has no `vaultik-destination.txt` (an unplugged volume's
|
|
||||||
empty mount point), the command fails before it removes anything. If a
|
|
||||||
listing afterwards still finds anything under `metadata/` or `blobs/`,
|
|
||||||
the command names each object left and exits non-zero; running it again
|
|
||||||
deletes what is left.
|
|
||||||
* `--force`: Required to confirm destruction.
|
* `--force`: Required to confirm destruction.
|
||||||
|
|
||||||
**`database delete`**: Delete the local SQLite state database file
|
**`database delete`**: Delete the local SQLite state database file
|
||||||
@@ -536,13 +527,12 @@ For an implementation-level view of the internals — the data model, the
|
|||||||
```
|
```
|
||||||
|
|
||||||
* `vaultik-destination.txt` holds one fixed line of text, with no host
|
* `vaultik-destination.txt` holds one fixed line of text, with no host
|
||||||
name, path or time. A destination without it is treated as not
|
name, path or time. While the local index holds snapshots, a
|
||||||
present while the local index holds snapshots stored there:
|
destination without it is treated as not present: `snapshot create`
|
||||||
`snapshot create` refuses to write to it, `prune`, `snapshot purge`
|
refuses to write to it, `prune` and `snapshot purge` fail, and
|
||||||
and `remote nuke` fail, and `snapshot list` and `snapshot remove`
|
`snapshot list` and `snapshot remove` behave as for a destination that
|
||||||
behave as for a destination that cannot be listed. This keeps an
|
cannot be listed. This keeps an unplugged volume's empty mount point
|
||||||
unplugged volume's empty mount point from being taken for a
|
from being taken for a destination whose snapshots are all gone. Mount
|
||||||
destination whose snapshots are all gone. Mount
|
|
||||||
the volume if it is removable; if the destination was emptied on
|
the volume if it is removable; if the destination was emptied on
|
||||||
purpose, run `vaultik database delete` and back up again.
|
purpose, run `vaultik database delete` and back up again.
|
||||||
* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
|
* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
|
||||||
|
|||||||
@@ -15,229 +15,13 @@ pre-1.0
|
|||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
The 1.0 work is complete on `next`: the scope settled on
|
The 1.0 work is complete on `next`: the scope settled on
|
||||||
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) has landed,
|
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) was the work
|
||||||
and so has every fix from the defect hunt before the release candidate
|
already planned for 1.0, and all of it has landed. The mechanism to cut
|
||||||
([issue #288](https://git.eeqj.de/sneak/vaultik/issues/288)). What is
|
the tag exists and is exercised; what is left is merging `next` to
|
||||||
left is merging `next` to `main` and tagging the release, both the
|
`main` and tagging, both the owner's.
|
||||||
owner's; a `v*` tag runs the release build in
|
|
||||||
`.gitea/workflows/release.yml`.
|
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
|
||||||
that skipped files it could not read
|
|
||||||
([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a
|
|
||||||
prune could delete the only snapshot still holding those files.
|
|
||||||
|
|
||||||
- 2026-10-14: Made `snapshot create --skip-errors` exit non-zero,
|
|
||||||
instead of 0, when it skips anything it cannot read
|
|
||||||
([issue #310](https://git.eeqj.de/sneak/vaultik/issues/310)).
|
|
||||||
|
|
||||||
- 2026-10-14: Kept the local index when `remote nuke` cannot list or
|
|
||||||
find the destination store
|
|
||||||
([issue #292](https://git.eeqj.de/sneak/vaultik/issues/292)). Nuke
|
|
||||||
had cleared the local snapshot records anyway; it now fails before
|
|
||||||
removing anything.
|
|
||||||
|
|
||||||
- 2026-10-14: Stopped counting directories and unchanged symlinks as
|
|
||||||
files to back up
|
|
||||||
([issue #305](https://git.eeqj.de/sneak/vaultik/issues/305)).
|
|
||||||
|
|
||||||
- 2026-10-14: Refused a backup destination that an unplugged volume
|
|
||||||
left empty, as the owner ruled
|
|
||||||
([issue #239](https://git.eeqj.de/sneak/vaultik/issues/239)). Every
|
|
||||||
backup now writes `vaultik-destination.txt` at the destination root,
|
|
||||||
and while the local index holds snapshots stored there, a destination
|
|
||||||
without it is treated as not present.
|
|
||||||
|
|
||||||
- 2026-10-14: Warned about restored ownership only when an entry's
|
|
||||||
owner or group differs from the recorded one
|
|
||||||
([issue #316](https://git.eeqj.de/sneak/vaultik/issues/316)); a
|
|
||||||
restore by a non-root user had warned even when none did.
|
|
||||||
|
|
||||||
- 2026-10-14: Made `snapshot list` report a changed destination instead
|
|
||||||
of advising `prune`, which refuses in that case
|
|
||||||
([issue #317](https://git.eeqj.de/sneak/vaultik/issues/317)).
|
|
||||||
|
|
||||||
- 2026-10-14: Counted a file as deleted by a backup only if the
|
|
||||||
previous snapshot of the same name listed it
|
|
||||||
([issue #304](https://git.eeqj.de/sneak/vaultik/issues/304)); files
|
|
||||||
that snapshot did not list had been counted too.
|
|
||||||
|
|
||||||
- 2026-10-14: Named each prefix once in restore, verify and purge
|
|
||||||
errors, some of which repeated it
|
|
||||||
([issue #327](https://git.eeqj.de/sneak/vaultik/issues/327)). A deep
|
|
||||||
verify log line that named S3 whatever the destination now names
|
|
||||||
storage.
|
|
||||||
|
|
||||||
- 2026-10-14: Refused credentials in a storage URL's endpoint or rclone
|
|
||||||
remote name, which were accepted and could then be printed
|
|
||||||
([issue #322](https://git.eeqj.de/sneak/vaultik/issues/322)).
|
|
||||||
|
|
||||||
- 2026-10-14: Ran `$EDITOR` through the shell in `config edit`, so a
|
|
||||||
value with arguments, such as `code --wait`, no longer fails
|
|
||||||
([issue #315](https://git.eeqj.de/sneak/vaultik/issues/315)).
|
|
||||||
|
|
||||||
- 2026-10-14: Made `snapshot remove` fail, instead of exiting 0, when
|
|
||||||
it cannot delete the metadata of a snapshot that is only on the
|
|
||||||
destination store
|
|
||||||
([issue #340](https://git.eeqj.de/sneak/vaultik/issues/340)). The
|
|
||||||
README and the help now say when it prints the `vaultik prune`
|
|
||||||
follow-up.
|
|
||||||
|
|
||||||
- 2026-10-14: Labelled the totals in the local index cleanup before
|
|
||||||
each backup as totals
|
|
||||||
([issue #306](https://git.eeqj.de/sneak/vaultik/issues/306)). Each
|
|
||||||
was shown as `(N remain)`, which read as rows left to clean up.
|
|
||||||
|
|
||||||
- 2026-10-13: Freed a cached blob once no file still to restore needs
|
|
||||||
it ([issue #324](https://git.eeqj.de/sneak/vaultik/issues/324)); one
|
|
||||||
also used by a file outside the selected paths had stayed in the
|
|
||||||
temporary cache until the restore ended.
|
|
||||||
|
|
||||||
- 2026-10-13: Kept the whole startup error when it contains `): `
|
|
||||||
([issue #320](https://git.eeqj.de/sneak/vaultik/issues/320)). Such
|
|
||||||
an error was cut at its last `): `, so the `blob_size_limit` one lost
|
|
||||||
the setting's name.
|
|
||||||
|
|
||||||
- 2026-10-13: Silenced the status lines of `prune` and snapshot removal
|
|
||||||
under `--cron` and `--quiet`
|
|
||||||
([issue #294](https://git.eeqj.de/sneak/vaultik/issues/294)).
|
|
||||||
|
|
||||||
- 2026-10-13: Read the rclone config file that `RCLONE_CONFIG` names
|
|
||||||
([issue #321](https://git.eeqj.de/sneak/vaultik/issues/321)); a
|
|
||||||
remote defined only there had been reported as not found.
|
|
||||||
|
|
||||||
- 2026-10-13: Made `/**/` in an exclude pattern match zero directories,
|
|
||||||
as in gitignore
|
|
||||||
([issue #338](https://git.eeqj.de/sneak/vaultik/issues/338));
|
|
||||||
`a/**/*.log` had missed `a/x.log`.
|
|
||||||
|
|
||||||
- 2026-10-13: Made the `chunk_size` and `blob_size_limit` defaults and
|
|
||||||
the `chunk_size` minimum 1000-based, like the size parser and the
|
|
||||||
docs ([issue #313](https://git.eeqj.de/sneak/vaultik/issues/313)).
|
|
||||||
|
|
||||||
- 2026-10-13: Kept a snapshot's remote metadata when `snapshot purge`
|
|
||||||
cannot remove it from the local index
|
|
||||||
([issue #337](https://git.eeqj.de/sneak/vaultik/issues/337)); the
|
|
||||||
purge had deleted the metadata anyway and exited 0.
|
|
||||||
|
|
||||||
- 2026-10-13: Made `snapshot restore` fail for a requested path that
|
|
||||||
matches nothing, which it had ignored
|
|
||||||
([issue #309](https://git.eeqj.de/sneak/vaultik/issues/309)).
|
|
||||||
|
|
||||||
- 2026-10-13: Made `remote nuke` delete every object under `metadata/`
|
|
||||||
and `blobs/`
|
|
||||||
([issue #291](https://git.eeqj.de/sneak/vaultik/issues/291)); it had
|
|
||||||
left some, such as a stray `.DS_Store`, while reporting the store
|
|
||||||
empty.
|
|
||||||
|
|
||||||
- 2026-10-13: Made `config set` create a list for an index under a
|
|
||||||
missing key
|
|
||||||
([issue #308](https://git.eeqj.de/sneak/vaultik/issues/308)); it had
|
|
||||||
written a map, after which no command could load the config.
|
|
||||||
|
|
||||||
- 2026-10-13: Listed a `file://` destination through a symlinked
|
|
||||||
`metadata/` or `blobs/`, which had listed as empty
|
|
||||||
([issue #296](https://git.eeqj.de/sneak/vaultik/issues/296)). A link
|
|
||||||
to a missing target now fails the listing
|
|
||||||
([issue #332](https://git.eeqj.de/sneak/vaultik/issues/332)).
|
|
||||||
|
|
||||||
- 2026-10-13: Made `snapshot verify --deep --json` report its blob
|
|
||||||
counts ([issue #314](https://git.eeqj.de/sneak/vaultik/issues/314));
|
|
||||||
a failed run had reported 0 missing and 0 verified.
|
|
||||||
|
|
||||||
- 2026-10-13: Limited and retried the module download in the Docker
|
|
||||||
build ([issue #351](https://git.eeqj.de/sneak/vaultik/issues/351)),
|
|
||||||
which had hung for as long as a stalled connection to the module
|
|
||||||
proxy lasted.
|
|
||||||
|
|
||||||
- 2026-10-13: Recorded each snapshot's uncompressed size when its
|
|
||||||
backup runs
|
|
||||||
([issue #360](https://git.eeqj.de/sneak/vaultik/issues/360)). A later
|
|
||||||
backup could change an older snapshot's UNCOMPRESSED SIZE in
|
|
||||||
`snapshot list`.
|
|
||||||
|
|
||||||
- 2026-10-13: Stopped `snapshot restore --skip-errors --verify`
|
|
||||||
reporting a file that failed to restore again as a checksum failure
|
|
||||||
([issue #323](https://git.eeqj.de/sneak/vaultik/issues/323)).
|
|
||||||
|
|
||||||
- 2026-10-13: Recorded the stored size of a blob a backup finds already
|
|
||||||
stored, not the size of its local copy, which can differ, so
|
|
||||||
`snapshot verify` could fail on a good snapshot
|
|
||||||
([issue #300](https://git.eeqj.de/sneak/vaultik/issues/300)).
|
|
||||||
|
|
||||||
- 2026-10-13: Stopped the local index and the blob upload step ending
|
|
||||||
the process on a failed close of query results or a failed temp file
|
|
||||||
cleanup ([issue #345](https://git.eeqj.de/sneak/vaultik/issues/345));
|
|
||||||
both are now returned as errors.
|
|
||||||
|
|
||||||
- 2026-10-13: Closed the snapshot database's storage reader too when
|
|
||||||
restore or deep verify is interrupted
|
|
||||||
([issue #356](https://git.eeqj.de/sneak/vaultik/issues/356)).
|
|
||||||
|
|
||||||
- 2026-10-13: Made `prune` and `remote nuke` fail when a delete on the
|
|
||||||
backup destination store fails
|
|
||||||
([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)).
|
|
||||||
|
|
||||||
- 2026-10-13: Made an S3 listing that the server cuts short an error
|
|
||||||
([issue #297](https://git.eeqj.de/sneak/vaultik/issues/297)); `prune`
|
|
||||||
had taken it as complete and could delete blobs still in use.
|
|
||||||
|
|
||||||
- 2026-10-13: Made shallow `snapshot verify` report a blob it could not
|
|
||||||
check apart from a missing one
|
|
||||||
([issue #318](https://git.eeqj.de/sneak/vaultik/issues/318)); any
|
|
||||||
error checking a blob had counted as missing, its cause dropped.
|
|
||||||
|
|
||||||
- 2026-10-13: Restored the other files when a file's chunk is in no
|
|
||||||
blob under `--skip-errors`
|
|
||||||
([issue #326](https://git.eeqj.de/sneak/vaultik/issues/326)); that
|
|
||||||
file had stopped the restore before anything was restored.
|
|
||||||
|
|
||||||
- 2026-10-13: Recorded each snapshot's new chunk size when its backup
|
|
||||||
runs ([issue #311](https://git.eeqj.de/sneak/vaultik/issues/311)).
|
|
||||||
`snapshot list` could show a NEW CHUNK SIZE of 0 B for a snapshot
|
|
||||||
whose change was a modified file.
|
|
||||||
|
|
||||||
- 2026-10-13: Made a backup interrupted during a local index write exit
|
|
||||||
as interrupted
|
|
||||||
([issue #301](https://git.eeqj.de/sneak/vaultik/issues/301)); it could
|
|
||||||
end with `FATAL: failed to rollback transaction` and exit 1 instead of
|
|
||||||
130.
|
|
||||||
|
|
||||||
- 2026-10-13: Printed `snapshot list` and `snapshot purge` sizes like
|
|
||||||
every other command
|
|
||||||
([issue #312](https://git.eeqj.de/sneak/vaultik/issues/312)). They,
|
|
||||||
and the `bytes` value in terminal log lines, were 1024-based but
|
|
||||||
labelled KB or MB, and now use 1000-based units.
|
|
||||||
|
|
||||||
- 2026-10-13: Closed each blob's storage reader when restore or deep
|
|
||||||
verify is interrupted, which ends a stalled read unless it is stuck in
|
|
||||||
the kernel ([issue #325](https://git.eeqj.de/sneak/vaultik/issues/325)).
|
|
||||||
|
|
||||||
- 2026-10-13: Made a mistyped subcommand under `snapshot`, `config`,
|
|
||||||
`remote`, `database` or `completion` an error
|
|
||||||
([issue #319](https://git.eeqj.de/sneak/vaultik/issues/319)); each had
|
|
||||||
printed its help and exited 0.
|
|
||||||
|
|
||||||
- 2026-10-13: Made an exclude pattern starting with `**/` match
|
|
||||||
directly under the snapshot path too, as in gitignore
|
|
||||||
([issue #303](https://git.eeqj.de/sneak/vaultik/issues/303)).
|
|
||||||
|
|
||||||
- 2026-10-13: Refused a relative `file://` storage URL and decoded its
|
|
||||||
percent-escapes
|
|
||||||
([issue #299](https://git.eeqj.de/sneak/vaultik/issues/299)); a
|
|
||||||
relative path had moved with the working directory.
|
|
||||||
|
|
||||||
- 2026-10-13: Removed the blob temp file when a backup stops part-way
|
|
||||||
([issue #302](https://git.eeqj.de/sneak/vaultik/issues/302)); the
|
|
||||||
half-written blob had stayed in `$TMPDIR`.
|
|
||||||
|
|
||||||
- 2026-10-13: Checked `--keep-newer-than` before a backup starts
|
|
||||||
([issue #307](https://git.eeqj.de/sneak/vaultik/issues/307)); an
|
|
||||||
invalid value had been read only by the prune after the backup, so
|
|
||||||
retention never ran.
|
|
||||||
|
|
||||||
- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that
|
- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that
|
||||||
is not a blob as an orphaned blob
|
is not a blob as an orphaned blob
|
||||||
([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray
|
([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ The metadata subdirectory is named with the **remote key**, a one-way hash of th
|
|||||||
|
|
||||||
## Marker File (`vaultik-destination.txt`)
|
## Marker File (`vaultik-destination.txt`)
|
||||||
|
|
||||||
Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. A destination without it is treated as not present while the local index holds snapshots stored there: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
|
Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. While the local index holds snapshots, a destination without it is treated as not present: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove` and `snapshot list` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
|
||||||
|
|
||||||
## Blobs Directory (`blobs/`)
|
## Blobs Directory (`blobs/`)
|
||||||
|
|
||||||
|
|||||||
@@ -1,30 +0,0 @@
|
|||||||
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
|
|
||||||
// under --skip-errors, a source directory that does not exist, which the
|
|
||||||
// walk skips. The command exits non-zero and reports the backup as
|
|
||||||
// completed, not as a failed snapshot creation. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
|
||||||
//
|
|
||||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
|
||||||
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
|
|
||||||
// writeBackupConfig would create the source directory.
|
|
||||||
configPath := writeHermeticPruneConfig(t, false)
|
|
||||||
|
|
||||||
code, stdout, stderr := runEntry(t,
|
|
||||||
flagConfig, configPath, "--skip-errors", cmdSnapshot, cmdCreate)
|
|
||||||
|
|
||||||
assert.Equal(t, 1, code)
|
|
||||||
assert.Contains(t, stdout,
|
|
||||||
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
|
|
||||||
assert.NotContains(t, stdout, "Finished successfully.")
|
|
||||||
assert.Contains(t, stderr,
|
|
||||||
"Backup completed, but 1 file(s) could not be read")
|
|
||||||
assert.NotContains(t, stderr, "Snapshot creation failed")
|
|
||||||
}
|
|
||||||
@@ -115,13 +115,6 @@ on Linux), then /etc/vaultik/config.yml.`,
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
if errors.Is(err, vaultik.ErrFilesUnreadable) {
|
|
||||||
log.Error("Backup completed with unreadable files", "error", err)
|
|
||||||
ReportErrorf("Backup completed, but %v", err)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
log.Error("Snapshot creation failed", "error", err)
|
log.Error("Snapshot creation failed", "error", err)
|
||||||
ReportErrorf("Snapshot creation failed: %v", err)
|
ReportErrorf("Snapshot creation failed: %v", err)
|
||||||
})
|
})
|
||||||
@@ -132,8 +125,7 @@ on Linux), then /etc/vaultik/config.yml.`,
|
|||||||
"Run in cron mode (silent unless warning or error)")
|
"Run in cron mode (silent unless warning or error)")
|
||||||
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
|
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
|
||||||
"After backup, drop older snapshots of the same name and remove "+
|
"After backup, drop older snapshots of the same name and remove "+
|
||||||
"orphaned blobs (not after a backup that skipped unreadable "+
|
"orphaned blobs")
|
||||||
"files)")
|
|
||||||
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
||||||
"With --prune: keep snapshots newer than this duration "+
|
"With --prune: keep snapshots newer than this duration "+
|
||||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
||||||
|
|||||||
@@ -78,7 +78,8 @@ func TestProgressPercentWithSecondPathSmaller(t *testing.T) {
|
|||||||
|
|
||||||
stats := progress.GetStats()
|
stats := progress.GetStats()
|
||||||
|
|
||||||
// With both paths counted, the percentage must not exceed 100%.
|
// Directories count toward the total size but produce no chunks, so
|
||||||
|
// the percentage ends just under 100%.
|
||||||
percent := float64(stats.BytesProcessed.Load()) /
|
percent := float64(stats.BytesProcessed.Load()) /
|
||||||
float64(stats.TotalSize.Load()) * 100
|
float64(stats.TotalSize.Load()) * 100
|
||||||
if percent > 100 {
|
if percent > 100 {
|
||||||
|
|||||||
+88
-167
@@ -141,12 +141,10 @@ type ScannerConfig struct {
|
|||||||
// ScanResult contains the results of a scan operation. Files and bytes
|
// ScanResult contains the results of a scan operation. Files and bytes
|
||||||
// are counted per file: BytesScanned is the size of the new and changed
|
// are counted per file: BytesScanned is the size of the new and changed
|
||||||
// files, BytesSkipped that of the unchanged ones. FilesFailed counts the
|
// files, BytesSkipped that of the unchanged ones. FilesFailed counts the
|
||||||
// new and changed files that phase 2 could not store, each entry removed
|
// new and changed files that phase 2 could not store; FilesScanned
|
||||||
// after the walk listed its directory and, under --skip-errors, each
|
// includes them and BytesScanned does not. ChunksCreated counts the
|
||||||
// entry the walk could not read, such as a directory it could not list;
|
// chunks that were not already stored, and NewChunkSize is their total
|
||||||
// FilesScanned includes them and BytesScanned does not. ChunksCreated
|
// size.
|
||||||
// counts the chunks that were not already stored, and NewChunkSize is
|
|
||||||
// their total size.
|
|
||||||
type ScanResult struct {
|
type ScanResult struct {
|
||||||
FilesScanned int
|
FilesScanned int
|
||||||
FilesSkipped int
|
FilesSkipped int
|
||||||
@@ -163,11 +161,6 @@ type ScanResult struct {
|
|||||||
UploadDuration time.Duration
|
UploadDuration time.Duration
|
||||||
StartTime time.Time
|
StartTime time.Time
|
||||||
EndTime time.Time
|
EndTime time.Time
|
||||||
|
|
||||||
// FilesUnreadable is the part of FilesFailed that --skip-errors
|
|
||||||
// skipped because it could not be read or listed; the rest are files
|
|
||||||
// removed after the walk listed them.
|
|
||||||
FilesUnreadable int
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewScanner creates a new scanner instance
|
// NewScanner creates a new scanner instance
|
||||||
@@ -291,11 +284,6 @@ func (s *Scanner) Scan(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
err = s.recordDirectories(ctx, scanResult.Directories)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
// Summarize scan phase results and update progress
|
// Summarize scan phase results and update progress
|
||||||
s.summarizeScanPhase(result, filesToProcess)
|
s.summarizeScanPhase(result, filesToProcess)
|
||||||
|
|
||||||
@@ -867,15 +855,14 @@ func (s *Scanner) batchInsertFileData(
|
|||||||
// ScanPhaseResult contains the results of the scan phase
|
// ScanPhaseResult contains the results of the scan phase
|
||||||
type ScanPhaseResult struct {
|
type ScanPhaseResult struct {
|
||||||
FilesToProcess []*FileToProcess
|
FilesToProcess []*FileToProcess
|
||||||
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
|
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
|
||||||
Directories []*FileToProcess // Every directory found, to record
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// scanPhase performs the initial directory scan to identify files to process
|
// scanPhase performs the initial directory scan to identify files to process
|
||||||
// It uses the pre-loaded knownFiles map for fast change detection without DB
|
// It uses the pre-loaded knownFiles map for fast change detection without DB
|
||||||
// queries. It also populates existingFiles map for deletion detection.
|
// queries. It also populates existingFiles map for deletion detection.
|
||||||
// Returns files needing processing, IDs of unchanged files for snapshot
|
// Returns files needing processing and IDs of unchanged files for snapshot
|
||||||
// association, and the directories found.
|
// association.
|
||||||
func (s *Scanner) scanPhase(
|
func (s *Scanner) scanPhase(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
path string,
|
path string,
|
||||||
@@ -883,10 +870,8 @@ func (s *Scanner) scanPhase(
|
|||||||
existingFiles map[string]struct{},
|
existingFiles map[string]struct{},
|
||||||
knownFiles map[string]*database.File,
|
knownFiles map[string]*database.File,
|
||||||
) (*ScanPhaseResult, error) {
|
) (*ScanPhaseResult, error) {
|
||||||
// Use known file count as estimate for progress (accurate for
|
// Use known file count as estimate for progress (accurate for subsequent backups)
|
||||||
// subsequent backups). The walk does not count directories, so the
|
estimatedTotal := int64(len(knownFiles))
|
||||||
// estimate leaves them out.
|
|
||||||
estimatedTotal := countNonDirectories(knownFiles)
|
|
||||||
|
|
||||||
collector := &scanCollector{}
|
collector := &scanCollector{}
|
||||||
|
|
||||||
@@ -897,28 +882,8 @@ func (s *Scanner) scanPhase(
|
|||||||
log.Debug("Starting directory walk", "path", path)
|
log.Debug("Starting directory walk", "path", path)
|
||||||
|
|
||||||
walkFn := func(filePath string, info os.FileInfo, err error) error {
|
walkFn := func(filePath string, info os.FileInfo, err error) error {
|
||||||
// Check exclude patterns first, so an excluded entry that is gone
|
|
||||||
// or cannot be looked up is neither counted nor reported. For
|
|
||||||
// directories, skip the entire subtree. info is nil when the
|
|
||||||
// walk could not look the entry up.
|
|
||||||
if s.shouldExclude(filePath, path) {
|
|
||||||
if info != nil && info.IsDir() {
|
|
||||||
return filepath.SkipDir
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// The source path is in no directory listing, so its absence is
|
|
||||||
// an error rather than a removal.
|
|
||||||
if errors.Is(err, os.ErrNotExist) && filePath != path {
|
|
||||||
countRemovedDuringWalk(filePath, existingFiles, result)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return s.handleWalkError(filePath, err, result)
|
return s.handleWalkError(filePath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check context cancellation
|
// Check context cancellation
|
||||||
@@ -928,9 +893,18 @@ func (s *Scanner) scanPhase(
|
|||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Check exclude patterns - for directories, skip the entire subtree
|
||||||
|
if s.shouldExclude(filePath, path) {
|
||||||
|
if info.IsDir() {
|
||||||
|
return filepath.SkipDir
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// Handle symlinks and directories
|
// Handle symlinks and directories
|
||||||
handled, err := s.recordSpecialEntry(
|
handled, err := s.recordSpecialEntry(
|
||||||
filePath, info, existingFiles, knownFiles, collector, result)
|
filePath, info, existingFiles, collector, result)
|
||||||
if handled {
|
if handled {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -944,10 +918,9 @@ func (s *Scanner) scanPhase(
|
|||||||
existingFiles[filePath] = struct{}{}
|
existingFiles[filePath] = struct{}{}
|
||||||
|
|
||||||
// Check file against in-memory map (no DB query!)
|
// Check file against in-memory map (no DB query!)
|
||||||
file := s.buildRegularFileEntry(filePath, info)
|
file, needsProcessing := s.checkFileInMemory(filePath, info, knownFiles)
|
||||||
needsProcessing := s.checkFileInMemory(file, knownFiles)
|
|
||||||
|
|
||||||
filesScanned, changedCount := collector.recordFile(
|
filesScanned, changedCount := collector.recordRegularFile(
|
||||||
filePath, info, file, needsProcessing)
|
filePath, info, file, needsProcessing)
|
||||||
|
|
||||||
// Update result stats
|
// Update result stats
|
||||||
@@ -971,45 +944,27 @@ func (s *Scanner) scanPhase(
|
|||||||
return &ScanPhaseResult{
|
return &ScanPhaseResult{
|
||||||
FilesToProcess: collector.filesToProcess,
|
FilesToProcess: collector.filesToProcess,
|
||||||
UnchangedFileIDs: collector.unchangedFileIDs,
|
UnchangedFileIDs: collector.unchangedFileIDs,
|
||||||
Directories: collector.directories,
|
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// countNonDirectories returns how many of files are not directories.
|
|
||||||
func countNonDirectories(files map[string]*database.File) int64 {
|
|
||||||
var count int64
|
|
||||||
|
|
||||||
for _, f := range files {
|
|
||||||
if f.Mode&uint32(os.ModeDir) == 0 {
|
|
||||||
count++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return count
|
|
||||||
}
|
|
||||||
|
|
||||||
// recordSpecialEntry records symlinks and directories (which have no
|
// recordSpecialEntry records symlinks and directories (which have no
|
||||||
// data to chunk) and reports whether it handled the entry. A symlink is
|
// data to chunk) and reports whether it handled the entry. For a symlink
|
||||||
// counted and checked for changes as a regular file is; a directory is
|
// whose target cannot be read it returns handleWalkError's result.
|
||||||
// set aside to be recorded and is not counted. For a symlink whose
|
|
||||||
// target cannot be read it returns handleWalkError's result.
|
|
||||||
func (s *Scanner) recordSpecialEntry(
|
func (s *Scanner) recordSpecialEntry(
|
||||||
filePath string, info os.FileInfo,
|
filePath string, info os.FileInfo,
|
||||||
existingFiles map[string]struct{},
|
existingFiles map[string]struct{},
|
||||||
knownFiles map[string]*database.File,
|
|
||||||
collector *scanCollector, result *ScanResult,
|
collector *scanCollector, result *ScanResult,
|
||||||
) (bool, error) {
|
) (bool, error) {
|
||||||
// Handle symlinks
|
// Handle symlinks
|
||||||
if info.Mode()&os.ModeSymlink != 0 {
|
if info.Mode()&os.ModeSymlink != 0 {
|
||||||
file, err := s.buildSymlinkEntry(filePath, info)
|
file, err := s.buildSymlinkEntry(filePath, info)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return true, s.handleWalkError(filePath, err, result)
|
return true, s.handleWalkError(filePath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
existingFiles[filePath] = struct{}{}
|
existingFiles[filePath] = struct{}{}
|
||||||
needsProcessing := s.checkFileInMemory(file, knownFiles)
|
collector.addToProcess(filePath, info, file)
|
||||||
collector.recordFile(filePath, info, file, needsProcessing)
|
s.updateScanEntryStats(result, true, info)
|
||||||
s.updateScanEntryStats(result, needsProcessing, info)
|
|
||||||
|
|
||||||
return true, nil
|
return true, nil
|
||||||
}
|
}
|
||||||
@@ -1019,7 +974,7 @@ func (s *Scanner) recordSpecialEntry(
|
|||||||
if info.IsDir() {
|
if info.IsDir() {
|
||||||
file := s.buildDirectoryEntry(filePath, info)
|
file := s.buildDirectoryEntry(filePath, info)
|
||||||
existingFiles[filePath] = struct{}{}
|
existingFiles[filePath] = struct{}{}
|
||||||
collector.addDirectory(filePath, info, file)
|
collector.addToProcess(filePath, info, file)
|
||||||
|
|
||||||
return true, nil
|
return true, nil
|
||||||
}
|
}
|
||||||
@@ -1027,38 +982,15 @@ func (s *Scanner) recordSpecialEntry(
|
|||||||
return false, nil
|
return false, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// countRemovedDuringWalk counts an entry that was in its directory's
|
|
||||||
// listing but was gone when the walk looked it up, like a file removed
|
|
||||||
// before phase 2 reads it: as examined and failed, and not as an error,
|
|
||||||
// with or without --skip-errors.
|
|
||||||
func countRemovedDuringWalk(
|
|
||||||
filePath string, existingFiles map[string]struct{}, result *ScanResult,
|
|
||||||
) {
|
|
||||||
log.Warn("File was deleted during the directory walk, skipping",
|
|
||||||
"path", filePath)
|
|
||||||
|
|
||||||
// The walk found it, so it is not also counted as deleted.
|
|
||||||
existingFiles[filePath] = struct{}{}
|
|
||||||
result.FilesScanned++
|
|
||||||
result.FilesFailed++
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleWalkError deals with a filesystem error surfaced by the walk:
|
// handleWalkError deals with a filesystem error surfaced by the walk:
|
||||||
// under --skip-errors the entry is skipped with a UI notice and counted
|
// skipped (with a UI notice) under --skip-errors, fatal otherwise.
|
||||||
// as examined, failed and unreadable, otherwise the error is fatal.
|
func (s *Scanner) handleWalkError(filePath string, err error) error {
|
||||||
func (s *Scanner) handleWalkError(
|
|
||||||
filePath string, err error, result *ScanResult,
|
|
||||||
) error {
|
|
||||||
if s.skipErrors {
|
if s.skipErrors {
|
||||||
log.Error("Failed to access file (skipping due to --skip-errors)",
|
log.Error("Failed to access file (skipping due to --skip-errors)",
|
||||||
"path", filePath, "error", err)
|
"path", filePath, "error", err)
|
||||||
s.ui.Errorf("Failed to access %s: %v. Skipping (--skip-errors).",
|
s.ui.Errorf("Failed to access %s: %v. Skipping (--skip-errors).",
|
||||||
s.ui.Path(filePath), err)
|
s.ui.Path(filePath), err)
|
||||||
|
|
||||||
result.FilesScanned++
|
|
||||||
result.FilesFailed++
|
|
||||||
result.FilesUnreadable++
|
|
||||||
|
|
||||||
return nil // Continue scanning
|
return nil // Continue scanning
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1072,29 +1004,28 @@ type scanCollector struct {
|
|||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
filesToProcess []*FileToProcess
|
filesToProcess []*FileToProcess
|
||||||
unchangedFileIDs []types.FileID // Just IDs - no new records needed
|
unchangedFileIDs []types.FileID // Just IDs - no new records needed
|
||||||
directories []*FileToProcess
|
|
||||||
filesScanned int64
|
filesScanned int64
|
||||||
}
|
}
|
||||||
|
|
||||||
// addDirectory sets a directory aside to be recorded. It is not counted
|
// addToProcess queues a file entry for processing.
|
||||||
// as a file.
|
func (c *scanCollector) addToProcess(
|
||||||
func (c *scanCollector) addDirectory(
|
|
||||||
path string, info os.FileInfo, file *database.File,
|
path string, info os.FileInfo, file *database.File,
|
||||||
) {
|
) {
|
||||||
c.mu.Lock()
|
c.mu.Lock()
|
||||||
defer c.mu.Unlock()
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
c.directories = append(c.directories, &FileToProcess{
|
c.filesToProcess = append(c.filesToProcess, &FileToProcess{
|
||||||
Path: path,
|
Path: path,
|
||||||
FileInfo: info,
|
FileInfo: info,
|
||||||
File: file,
|
File: file,
|
||||||
})
|
})
|
||||||
|
c.filesScanned++
|
||||||
}
|
}
|
||||||
|
|
||||||
// recordFile records a regular file or symlink as changed (to process)
|
// recordRegularFile records a regular file as changed (to process) or
|
||||||
// or unchanged (snapshot association only) and returns the running
|
// unchanged (snapshot association only) and returns the running scanned
|
||||||
// scanned and changed counts.
|
// and changed counts.
|
||||||
func (c *scanCollector) recordFile(
|
func (c *scanCollector) recordRegularFile(
|
||||||
path string, info os.FileInfo, file *database.File, needsProcessing bool,
|
path string, info os.FileInfo, file *database.File, needsProcessing bool,
|
||||||
) (int64, int) {
|
) (int64, int) {
|
||||||
c.mu.Lock()
|
c.mu.Lock()
|
||||||
@@ -1197,28 +1128,7 @@ func (s *Scanner) printScanProgressLine(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// buildRegularFileEntry creates a File record for a regular file, with
|
// buildSymlinkEntry creates a File record for a symlink.
|
||||||
// no ID: checkFileInMemory sets it.
|
|
||||||
func (s *Scanner) buildRegularFileEntry(path string, info os.FileInfo) *database.File {
|
|
||||||
var uid, gid uint32
|
|
||||||
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
|
|
||||||
uid = stat.Uid
|
|
||||||
gid = stat.Gid
|
|
||||||
}
|
|
||||||
|
|
||||||
return &database.File{
|
|
||||||
Path: types.FilePath(path),
|
|
||||||
SourcePath: types.SourcePath(s.currentSourcePath),
|
|
||||||
MTime: info.ModTime(),
|
|
||||||
Size: info.Size(),
|
|
||||||
Mode: uint32(info.Mode()),
|
|
||||||
UID: uid,
|
|
||||||
GID: gid,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// buildSymlinkEntry creates a File record for a symlink, with no ID:
|
|
||||||
// checkFileInMemory sets it.
|
|
||||||
func (s *Scanner) buildSymlinkEntry(
|
func (s *Scanner) buildSymlinkEntry(
|
||||||
path string, info os.FileInfo,
|
path string, info os.FileInfo,
|
||||||
) (*database.File, error) {
|
) (*database.File, error) {
|
||||||
@@ -1234,6 +1144,7 @@ func (s *Scanner) buildSymlinkEntry(
|
|||||||
}
|
}
|
||||||
|
|
||||||
return &database.File{
|
return &database.File{
|
||||||
|
ID: types.NewFileID(),
|
||||||
Path: types.FilePath(path),
|
Path: types.FilePath(path),
|
||||||
SourcePath: types.SourcePath(s.currentSourcePath),
|
SourcePath: types.SourcePath(s.currentSourcePath),
|
||||||
MTime: info.ModTime(),
|
MTime: info.ModTime(),
|
||||||
@@ -1265,22 +1176,6 @@ func (s *Scanner) buildDirectoryEntry(path string, info os.FileInfo) *database.F
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// recordDirectories records every directory the walk found. A directory
|
|
||||||
// has no data to back up, so it is not among the files counted and
|
|
||||||
// backed up.
|
|
||||||
func (s *Scanner) recordDirectories(
|
|
||||||
ctx context.Context, directories []*FileToProcess,
|
|
||||||
) error {
|
|
||||||
for _, dir := range directories {
|
|
||||||
err := s.recordNonRegularFile(ctx, dir)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("recording directory %s: %w", dir.Path, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// recordNonRegularFile writes a symlink or directory entry to the database
|
// recordNonRegularFile writes a symlink or directory entry to the database
|
||||||
// and associates it with the current snapshot. No chunking is performed.
|
// and associates it with the current snapshot. No chunking is performed.
|
||||||
func (s *Scanner) recordNonRegularFile(ctx context.Context, ftp *FileToProcess) error {
|
func (s *Scanner) recordNonRegularFile(ctx context.Context, ftp *FileToProcess) error {
|
||||||
@@ -1294,36 +1189,62 @@ func (s *Scanner) recordNonRegularFile(ctx context.Context, ftp *FileToProcess)
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkFileInMemory reports whether a regular file or symlink needs
|
// checkFileInMemory checks if a file needs processing using the in-memory map
|
||||||
// processing, and gives file the ID of its known record, or a new ID for
|
|
||||||
// a new path.
|
|
||||||
// No database access is performed - this is purely CPU/memory work
|
// No database access is performed - this is purely CPU/memory work
|
||||||
func (s *Scanner) checkFileInMemory(
|
func (s *Scanner) checkFileInMemory(
|
||||||
file *database.File, knownFiles map[string]*database.File,
|
path string, info os.FileInfo, knownFiles map[string]*database.File,
|
||||||
) bool {
|
) (*database.File, bool) {
|
||||||
existingFile, exists := knownFiles[file.Path.String()]
|
var uid, gid uint32
|
||||||
|
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
|
||||||
|
uid = stat.Uid
|
||||||
|
gid = stat.Gid
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check against in-memory map first to get existing ID if available
|
||||||
|
existingFile, exists := knownFiles[path]
|
||||||
|
|
||||||
|
// Create file record with ID set upfront
|
||||||
|
// For new files, generate UUID immediately so it's available for chunk associations
|
||||||
|
// For existing files, reuse the existing ID
|
||||||
|
var fileID types.FileID
|
||||||
|
if exists {
|
||||||
|
fileID = existingFile.ID
|
||||||
|
} else {
|
||||||
|
fileID = types.NewFileID()
|
||||||
|
}
|
||||||
|
|
||||||
|
file := &database.File{
|
||||||
|
ID: fileID,
|
||||||
|
Path: types.FilePath(path),
|
||||||
|
SourcePath: types.SourcePath(s.currentSourcePath),
|
||||||
|
MTime: info.ModTime(),
|
||||||
|
Size: info.Size(),
|
||||||
|
Mode: uint32(info.Mode()),
|
||||||
|
UID: uid,
|
||||||
|
GID: gid,
|
||||||
|
}
|
||||||
|
|
||||||
// New file - needs processing
|
// New file - needs processing
|
||||||
if !exists {
|
if !exists {
|
||||||
file.ID = types.NewFileID()
|
return file, true
|
||||||
|
|
||||||
return true
|
|
||||||
}
|
}
|
||||||
|
|
||||||
file.ID = existingFile.ID
|
|
||||||
|
|
||||||
// No uploaded blob holds one of its chunks (see loadFilesToRechunk)
|
// No uploaded blob holds one of its chunks (see loadFilesToRechunk)
|
||||||
if _, rechunk := s.filesToRechunk[file.ID]; rechunk {
|
if _, rechunk := s.filesToRechunk[fileID]; rechunk {
|
||||||
return true
|
return file, true
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check if file has changed
|
// Check if file has changed
|
||||||
return existingFile.Size != file.Size ||
|
if existingFile.Size != file.Size ||
|
||||||
!existingFile.MTime.Equal(file.MTime) ||
|
!existingFile.MTime.Equal(file.MTime) ||
|
||||||
existingFile.Mode != file.Mode ||
|
existingFile.Mode != file.Mode ||
|
||||||
existingFile.UID != file.UID ||
|
existingFile.UID != file.UID ||
|
||||||
existingFile.GID != file.GID ||
|
existingFile.GID != file.GID {
|
||||||
existingFile.LinkTarget != file.LinkTarget
|
return file, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// File unchanged
|
||||||
|
return file, false
|
||||||
}
|
}
|
||||||
|
|
||||||
// batchAddFilesToSnapshot adds existing file IDs to the snapshot association table
|
// batchAddFilesToSnapshot adds existing file IDs to the snapshot association table
|
||||||
@@ -1450,16 +1371,17 @@ func (s *Scanner) processPhase(
|
|||||||
return s.finalizeProcessPhase(ctx, result)
|
return s.finalizeProcessPhase(ctx, result)
|
||||||
}
|
}
|
||||||
|
|
||||||
// processFileWithErrorHandling records a symlink, or wraps
|
// processFileWithErrorHandling records a directory or symlink, or wraps
|
||||||
// processFileStreaming for a regular file with error recovery for deleted
|
// processFileStreaming for a regular file with error recovery for deleted
|
||||||
// files and skip-errors mode. Returns (skipped, error).
|
// files and skip-errors mode. Returns (skipped, error).
|
||||||
func (s *Scanner) processFileWithErrorHandling(
|
func (s *Scanner) processFileWithErrorHandling(
|
||||||
ctx context.Context, fileToProcess *FileToProcess, result *ScanResult,
|
ctx context.Context, fileToProcess *FileToProcess, result *ScanResult,
|
||||||
) (bool, error) {
|
) (bool, error) {
|
||||||
// A symlink has no data to open or read; it is only recorded in the
|
// A directory or symlink has no data to open or read; it is only
|
||||||
// local index. An error recording it stops the run even under
|
// recorded in the local index. An error recording it stops the run
|
||||||
// --skip-errors, or the snapshot would complete without it.
|
// even under --skip-errors, or the snapshot would complete without it.
|
||||||
if os.FileMode(fileToProcess.File.Mode)&os.ModeSymlink != 0 {
|
mode := os.FileMode(fileToProcess.File.Mode)
|
||||||
|
if mode&os.ModeSymlink != 0 || mode.IsDir() {
|
||||||
err := s.recordNonRegularFile(ctx, fileToProcess)
|
err := s.recordNonRegularFile(ctx, fileToProcess)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, fmt.Errorf("processing file %s: %w", fileToProcess.Path, err)
|
return false, fmt.Errorf("processing file %s: %w", fileToProcess.Path, err)
|
||||||
@@ -1500,7 +1422,6 @@ func (s *Scanner) processFileWithErrorHandling(
|
|||||||
s.ui.Path(fileToProcess.Path), err)
|
s.ui.Path(fileToProcess.Path), err)
|
||||||
|
|
||||||
countFailedFile(fileToProcess, result)
|
countFailedFile(fileToProcess, result)
|
||||||
result.FilesUnreadable++
|
|
||||||
|
|
||||||
return true, nil
|
return true, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,28 +0,0 @@
|
|||||||
//nolint:testpackage // exercises the unexported countNonDirectories helper
|
|
||||||
package snapshot
|
|
||||||
|
|
||||||
import (
|
|
||||||
"os"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"sneak.berlin/go/vaultik/internal/database"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The walk counts regular files and symlinks, not directories, so the
|
|
||||||
// estimate of its count made from the local index's records leaves the
|
|
||||||
// directories out. See https://git.eeqj.de/sneak/vaultik/issues/305.
|
|
||||||
func TestCountNonDirectories(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
known := map[string]*database.File{
|
|
||||||
"/src": {Mode: uint32(os.ModeDir | 0o755)},
|
|
||||||
"/src/sub": {Mode: uint32(os.ModeDir | 0o755)},
|
|
||||||
"/src/a.txt": {Mode: 0o644},
|
|
||||||
"/src/link": {Mode: uint32(os.ModeSymlink | 0o777), LinkTarget: "a.txt"},
|
|
||||||
}
|
|
||||||
|
|
||||||
got := countNonDirectories(known)
|
|
||||||
if got != 2 {
|
|
||||||
t.Errorf("countNonDirectories = %d, want 2", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -9,10 +9,10 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// destinationMarkerKey is the object every backup writes at the root of
|
// destinationMarkerKey is the object every backup writes at the root of
|
||||||
// the backup destination store. While the local index holds a snapshot
|
// the backup destination store. While the local index holds a snapshot,
|
||||||
// stored there, a destination without it is treated as not present: an
|
// a destination without it is treated as not present: an unplugged
|
||||||
// unplugged volume leaves an empty directory at its mount point, which
|
// volume leaves an empty directory at its mount point, which would
|
||||||
// would otherwise read as a store whose snapshots are all gone.
|
// otherwise read as a store whose snapshots are all gone.
|
||||||
const destinationMarkerKey = "vaultik-destination.txt"
|
const destinationMarkerKey = "vaultik-destination.txt"
|
||||||
|
|
||||||
// destinationMarkerText is the whole content of the marker. It names no
|
// destinationMarkerText is the whole content of the marker. It names no
|
||||||
@@ -20,16 +20,13 @@ const destinationMarkerKey = "vaultik-destination.txt"
|
|||||||
const destinationMarkerText = "This is a vaultik backup destination store.\n"
|
const destinationMarkerText = "This is a vaultik backup destination store.\n"
|
||||||
|
|
||||||
// errDestinationNotPresent is returned when the local index holds a
|
// errDestinationNotPresent is returned when the local index holds a
|
||||||
// snapshot stored on the backup destination store and the store has no
|
// snapshot and the backup destination store has no marker.
|
||||||
// marker.
|
|
||||||
var errDestinationNotPresent = errors.New("backup destination store is not present")
|
var errDestinationNotPresent = errors.New("backup destination store is not present")
|
||||||
|
|
||||||
// checkDestinationMarker returns errDestinationNotPresent when the local
|
// checkDestinationMarker returns errDestinationNotPresent when the local
|
||||||
// index holds a snapshot stored on the backup destination store and the
|
// index holds a snapshot and the backup destination store has no marker.
|
||||||
// store has no marker. With no snapshot in the local index, as on a
|
// With no snapshot in the local index, as on a first backup or on
|
||||||
// first backup or on another machine, there is nothing to check; nor
|
// another machine, there is nothing to check.
|
||||||
// when the local index is bound to a different destination than the
|
|
||||||
// configured one, since its snapshots are stored there.
|
|
||||||
func (v *Vaultik) checkDestinationMarker() error {
|
func (v *Vaultik) checkDestinationMarker() error {
|
||||||
ids, err := v.localSnapshotIDs()
|
ids, err := v.localSnapshotIDs()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -40,19 +37,6 @@ func (v *Vaultik) checkDestinationMarker() error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Config is nil only in tests, which then record no binding (see
|
|
||||||
// EnsureStorageBinding).
|
|
||||||
if v.Config != nil {
|
|
||||||
stored, configured, err := v.storageBinding()
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
if stored != "" && stored != configured {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = v.Storage.Stat(v.ctx, destinationMarkerKey)
|
_, err = v.Storage.Stat(v.ctx, destinationMarkerKey)
|
||||||
if errors.Is(err, storage.ErrNotFound) {
|
if errors.Is(err, storage.ErrNotFound) {
|
||||||
return fmt.Errorf("%w: %s has no %s, but the local index holds "+
|
return fmt.Errorf("%w: %s has no %s, but the local index holds "+
|
||||||
|
|||||||
@@ -12,7 +12,6 @@ import (
|
|||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/vaultik/internal/database"
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
|
||||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -152,25 +151,6 @@ func TestPurgeRefusesDestinationWithoutMarker(t *testing.T) {
|
|||||||
assertUntouched(ctx, t, repos, storeDir)
|
assertUntouched(ctx, t, repos, storeDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestNukeRemoteRefusesDestinationWithoutMarker checks that remote nuke
|
|
||||||
// fails, deletes no local snapshot record, and neither reports the
|
|
||||||
// destination empty nor advises running nuke again when the destination
|
|
||||||
// has no marker.
|
|
||||||
func TestNukeRemoteRefusesDestinationWithoutMarker(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
v, repos, out, storeDir := backUpThenUnplugMountPoint(ctx, t)
|
|
||||||
|
|
||||||
err := v.NukeRemote(true)
|
|
||||||
require.ErrorContains(t, err, markerFile)
|
|
||||||
assert.NotContains(t, out.String(), "now empty")
|
|
||||||
assert.NotContains(t, out.String(), "vaultik remote nuke --force")
|
|
||||||
|
|
||||||
assertUntouched(ctx, t, repos, storeDir)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRemoveSnapshotWarnsWhenDestinationHasNoMarker checks that snapshot
|
// TestRemoveSnapshotWarnsWhenDestinationHasNoMarker checks that snapshot
|
||||||
// remove warns that the metadata could not be removed from a destination
|
// remove warns that the metadata could not be removed from a destination
|
||||||
// without a marker, instead of reporting that it was.
|
// without a marker, instead of reporting that it was.
|
||||||
@@ -241,44 +221,6 @@ func TestListSnapshotsWarnsWhenDestinationHasNoMarker(t *testing.T) {
|
|||||||
assert.NotContains(t, out.String(), "not found in backup destination store")
|
assert.NotContains(t, out.String(), "not found in backup destination store")
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestListSnapshotsAfterStorageURLChangeToEmptyDirectory checks that
|
|
||||||
// snapshot list, after storage_url is changed to an empty directory,
|
|
||||||
// warns only that the local index is bound to a different destination:
|
|
||||||
// the local snapshots are stored at the old one, so the new one's lack
|
|
||||||
// of a marker says nothing. With --json the local row is not on the new
|
|
||||||
// destination, so remote_present is false, not unknown.
|
|
||||||
func TestListSnapshotsAfterStorageURLChangeToEmptyDirectory(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
v, repos, out := backUpToFileDestination(ctx, t, t.TempDir())
|
|
||||||
id := localSnapshotID(ctx, t, repos, "first")
|
|
||||||
|
|
||||||
newDir := t.TempDir()
|
|
||||||
store, err := storage.NewFileStorer(newDir)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
v.Storage = store
|
|
||||||
v.Config.StorageURL = "file://" + newDir
|
|
||||||
|
|
||||||
require.NoError(t, v.ListSnapshots(false))
|
|
||||||
|
|
||||||
assert.Contains(t, out.String(), id)
|
|
||||||
assert.Contains(t, out.String(),
|
|
||||||
"local index is bound to a different backup destination")
|
|
||||||
assert.NotContains(t, out.String(), "Could not list backup destination store")
|
|
||||||
assert.NotContains(t, out.String(), markerFile)
|
|
||||||
|
|
||||||
out.Reset()
|
|
||||||
require.NoError(t, v.ListSnapshots(true))
|
|
||||||
|
|
||||||
rows := decodeListJSON(t, out.String())
|
|
||||||
require.Len(t, rows, 1)
|
|
||||||
require.NotNil(t, rows[0].RemotePresent)
|
|
||||||
assert.False(t, *rows[0].RemotePresent)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestBackupAfterNukeRemoteSucceeds checks that remote nuke leaves the
|
// TestBackupAfterNukeRemoteSucceeds checks that remote nuke leaves the
|
||||||
// destination ready for a new first backup, which writes the marker
|
// destination ready for a new first backup, which writes the marker
|
||||||
// again.
|
// again.
|
||||||
|
|||||||
@@ -188,41 +188,3 @@ func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
|
|||||||
assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
|
assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
|
||||||
err.Error())
|
err.Error())
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestNukeRemoteKeepsLocalIndexWhenDestinationMissing checks that remote
|
|
||||||
// nuke fails on a destination it cannot list, leaves the local index
|
|
||||||
// unchanged, and neither reports the destination empty nor advises
|
|
||||||
// running nuke again.
|
|
||||||
func TestNukeRemoteKeepsLocalIndexWhenDestinationMissing(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
v, repos, out := backUpThenUnplug(ctx, t)
|
|
||||||
|
|
||||||
filesBefore, err := repos.Files.ListAll(ctx)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NotEmpty(t, filesBefore)
|
|
||||||
|
|
||||||
blobsBefore, err := repos.Blobs.GetAll(ctx)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NotEmpty(t, blobsBefore)
|
|
||||||
|
|
||||||
err = v.NukeRemote(true)
|
|
||||||
require.ErrorIs(t, err, fs.ErrNotExist)
|
|
||||||
require.ErrorContains(t, err, "listing remote snapshots")
|
|
||||||
assert.NotContains(t, out.String(), "now empty")
|
|
||||||
assert.NotContains(t, out.String(), "vaultik remote nuke --force")
|
|
||||||
|
|
||||||
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, snapshots, 1, "nuke must delete no local snapshot record")
|
|
||||||
|
|
||||||
files, err := repos.Files.ListAll(ctx)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, files, len(filesBefore), "nuke must delete no local file record")
|
|
||||||
|
|
||||||
blobs, err := repos.Blobs.GetAll(ctx)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, blobs, len(blobsBefore), "nuke must delete no local blob record")
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -104,32 +104,6 @@ func TestNukeRemoteFailsWhenADeleteFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestNukeRemoteKeepsLocalSnapshotWhoseMetadataIsNotDeleted checks that
|
|
||||||
// remote nuke keeps a snapshot in the local index while its metadata
|
|
||||||
// cannot be deleted from the destination store: dropped from the local
|
|
||||||
// index, it would be left there as a remote-only snapshot without its
|
|
||||||
// name.
|
|
||||||
func TestNukeRemoteKeepsLocalSnapshotWhoseMetadataIsNotDeleted(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
storeDir := filepath.Join(t.TempDir(), "store")
|
|
||||||
v, repos, _ := backUpToFileDestination(ctx, t, storeDir)
|
|
||||||
|
|
||||||
store := faultstore.New(v.Storage)
|
|
||||||
store.OnDelete = func(key string) bool {
|
|
||||||
return strings.HasPrefix(key, "metadata/")
|
|
||||||
}
|
|
||||||
v.Storage = store
|
|
||||||
|
|
||||||
require.Error(t, v.NukeRemote(true))
|
|
||||||
|
|
||||||
snapshots, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, snapshots, 1)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestNukeRemoteNamesWhatIsLeft checks that remote nuke, when an object
|
// TestNukeRemoteNamesWhatIsLeft checks that remote nuke, when an object
|
||||||
// it does not recognise cannot be deleted, does not report the store
|
// it does not recognise cannot be deleted, does not report the store
|
||||||
// empty but names that object and fails.
|
// empty but names that object and fails.
|
||||||
@@ -180,7 +154,8 @@ func TestNukeRemoteRefusesAnotherDestinationWithoutRetryAdvice(t *testing.T) {
|
|||||||
|
|
||||||
// TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved checks that a
|
// TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved checks that a
|
||||||
// snapshot remote nuke cannot remove from the local index fails the
|
// snapshot remote nuke cannot remove from the local index fails the
|
||||||
// run, and that nuke then does not report the destination store empty.
|
// run: nuke then leaves that snapshot's metadata on the destination
|
||||||
|
// store.
|
||||||
func TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved(t *testing.T) {
|
func TestNukeRemoteFailsWhenALocalSnapshotCannotBeRemoved(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|||||||
@@ -55,19 +55,12 @@ func (v *Vaultik) NukeRemote(force bool) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Checked before anything is deleted: running nuke again would fail
|
// Checked before anything is deleted: running nuke again would fail
|
||||||
// the same way, so the advice below must not follow these errors.
|
// the same way, so the advice below must not follow this error.
|
||||||
// The listing also fails on a destination without the marker while
|
|
||||||
// the local index holds snapshots.
|
|
||||||
err := v.EnsureStorageBinding()
|
err := v.EnsureStorageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err = v.listAllRemoteSnapshotKeys()
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
err = v.emptyDestinationStore()
|
err = v.emptyDestinationStore()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
v.UI.Warningf("Run '%s' again to delete what is left on the "+
|
v.UI.Warningf("Run '%s' again to delete what is left on the "+
|
||||||
|
|||||||
@@ -12,7 +12,6 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"syscall"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"filippo.io/age"
|
"filippo.io/age"
|
||||||
@@ -220,7 +219,9 @@ func (v *Vaultik) finishRestore(
|
|||||||
)
|
)
|
||||||
|
|
||||||
if os.Geteuid() != 0 {
|
if os.Geteuid() != 0 {
|
||||||
v.warnIfOwnershipNotPreserved(files, opts.TargetDir)
|
v.UI.Warningf("Restore did not preserve file ownership: chown(2) " +
|
||||||
|
"requires root. Re-run as root (e.g. with sudo) if you need " +
|
||||||
|
"original UID/GID preserved.")
|
||||||
}
|
}
|
||||||
|
|
||||||
if result.FilesFailed > 0 {
|
if result.FilesFailed > 0 {
|
||||||
@@ -246,43 +247,6 @@ func (v *Vaultik) finishRestore(
|
|||||||
return unmatchedErr
|
return unmatchedErr
|
||||||
}
|
}
|
||||||
|
|
||||||
// warnIfOwnershipNotPreserved warns once if some entry restored under
|
|
||||||
// targetDir has another owner or group on disk than the snapshot
|
|
||||||
// records. It is meant for a restore by a normal user, which changes no
|
|
||||||
// owner or group: a new entry is owned by that user, and its group is
|
|
||||||
// that user's, or the group of the directory it is created in on macOS
|
|
||||||
// and in a setgid directory on Linux. An entry that is not on disk, or
|
|
||||||
// whose filesystem reports no owner, is skipped.
|
|
||||||
func (v *Vaultik) warnIfOwnershipNotPreserved(
|
|
||||||
files []*database.File, targetDir string,
|
|
||||||
) {
|
|
||||||
for _, file := range files {
|
|
||||||
targetPath, err := containedRestorePath(
|
|
||||||
v.Fs, targetDir, file.Path.String())
|
|
||||||
if err != nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
info, err := lstatIfPossible(v.Fs, targetPath)
|
|
||||||
if err != nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
owner, ok := info.Sys().(*syscall.Stat_t)
|
|
||||||
if !ok {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if owner.Uid != file.UID || owner.Gid != file.GID {
|
|
||||||
v.UI.Warningf("Restore did not preserve file ownership: chown(2) " +
|
|
||||||
"requires root. Re-run as root (e.g. with sudo) if you need " +
|
|
||||||
"original UID/GID preserved.")
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// reportUnmatchedPaths warns, by name, about each requested path that
|
// reportUnmatchedPaths warns, by name, about each requested path that
|
||||||
// matched nothing in the snapshot, and returns an error counting them,
|
// matched nothing in the snapshot, and returns an error counting them,
|
||||||
// or nil when there are none.
|
// or nil when there are none.
|
||||||
@@ -1001,9 +965,8 @@ type restoreSession struct {
|
|||||||
// root can chown a file to an arbitrary UID/GID — non-root chown
|
// root can chown a file to an arbitrary UID/GID — non-root chown
|
||||||
// always fails with EPERM. Attempting it anyway produces N
|
// always fails with EPERM. Attempting it anyway produces N
|
||||||
// guaranteed-failed syscalls + N noisy debug lines, so we skip
|
// guaranteed-failed syscalls + N noisy debug lines, so we skip
|
||||||
// the call entirely as non-root; warnIfOwnershipNotPreserved warns
|
// the call entirely as non-root and emit one warning at the end
|
||||||
// once at the end of the restore if that left any owner or group
|
// of the restore explaining that ownership was not preserved.
|
||||||
// other than recorded.
|
|
||||||
runningAsRoot bool
|
runningAsRoot bool
|
||||||
// directories holds every restored directory, for
|
// directories holds every restored directory, for
|
||||||
// applyDirectoryMetadata to finish after the restore loop.
|
// applyDirectoryMetadata to finish after the restore loop.
|
||||||
|
|||||||
@@ -1,131 +0,0 @@
|
|||||||
package vaultik //nolint:testpackage // calls the unexported ownership check
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"os"
|
|
||||||
"strings"
|
|
||||||
"syscall"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/spf13/afero"
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
"sneak.berlin/go/vaultik/internal/database"
|
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
|
||||||
"sneak.berlin/go/vaultik/internal/ui"
|
|
||||||
)
|
|
||||||
|
|
||||||
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/316: a
|
|
||||||
// restore by a normal user warns that ownership was not preserved only
|
|
||||||
// when some restored entry has another owner or group on disk than the
|
|
||||||
// snapshot records.
|
|
||||||
|
|
||||||
const ownershipWarning = "did not preserve file ownership"
|
|
||||||
|
|
||||||
// TestOwnershipWarningSilentWhenEntriesMatch checks that a restore whose
|
|
||||||
// entries all end up with their recorded owner and group prints no
|
|
||||||
// warning.
|
|
||||||
func TestOwnershipWarningSilentWhenEntriesMatch(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
targetDir := t.TempDir()
|
|
||||||
owner := ownerOf(t, targetDir)
|
|
||||||
|
|
||||||
out := restoreAsNormalUser(t, targetDir, []*database.File{
|
|
||||||
{Path: "/a", Mode: plainFileMode, UID: owner.Uid, GID: owner.Gid},
|
|
||||||
{Path: "/b", Mode: plainFileMode, UID: owner.Uid, GID: owner.Gid},
|
|
||||||
})
|
|
||||||
|
|
||||||
assert.Empty(t, out)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOwnershipWarningWhenOwnerDiffers checks that the warning is printed
|
|
||||||
// when an entry recorded with another owner is restored owned by the
|
|
||||||
// restoring user, even though its group matches the recorded one.
|
|
||||||
func TestOwnershipWarningWhenOwnerDiffers(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
targetDir := t.TempDir()
|
|
||||||
owner := ownerOf(t, targetDir)
|
|
||||||
|
|
||||||
out := restoreAsNormalUser(t, targetDir, []*database.File{
|
|
||||||
{Path: "/a", Mode: plainFileMode, UID: otherOwnerID, GID: owner.Gid},
|
|
||||||
})
|
|
||||||
|
|
||||||
assert.Equal(t, 1, strings.Count(out, ownershipWarning), out)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOwnershipWarningWhenDirectoryGivesItsGroup checks that the warning
|
|
||||||
// is printed, once, when entries recorded with the restoring user's
|
|
||||||
// owner and group take another group from the setgid directory they are
|
|
||||||
// restored into, as every new entry takes its directory's group on
|
|
||||||
// macOS.
|
|
||||||
func TestOwnershipWarningWhenDirectoryGivesItsGroup(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
if os.Geteuid() != 0 {
|
|
||||||
t.Skip("giving a directory to another group needs root")
|
|
||||||
}
|
|
||||||
|
|
||||||
targetDir := t.TempDir()
|
|
||||||
user := ownerOf(t, targetDir)
|
|
||||||
|
|
||||||
require.NoError(t, os.Chown(targetDir, -1, int(otherOwnerID)))
|
|
||||||
require.NoError(t, os.Chmod(targetDir, os.ModeSetgid|writableTestMode))
|
|
||||||
|
|
||||||
out := restoreAsNormalUser(t, targetDir, []*database.File{
|
|
||||||
{Path: "/a", Mode: plainFileMode, UID: user.Uid, GID: user.Gid},
|
|
||||||
{Path: "/b", Mode: plainFileMode, UID: user.Uid, GID: user.Gid},
|
|
||||||
})
|
|
||||||
|
|
||||||
assert.Equal(t, 1, strings.Count(out, ownershipWarning), out)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ownerOf returns the owner and group path has on disk.
|
|
||||||
func ownerOf(t *testing.T, path string) *syscall.Stat_t {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
info, err := os.Lstat(path)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
owner, ok := info.Sys().(*syscall.Stat_t)
|
|
||||||
require.True(t, ok)
|
|
||||||
|
|
||||||
return owner
|
|
||||||
}
|
|
||||||
|
|
||||||
// restoreAsNormalUser restores rows, all empty regular files, into
|
|
||||||
// targetDir without changing any owner or group, as restore does for a
|
|
||||||
// normal user, and returns what warnIfOwnershipNotPreserved prints.
|
|
||||||
func restoreAsNormalUser(
|
|
||||||
t *testing.T, targetDir string, rows []*database.File,
|
|
||||||
) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
rows, repos := makeFiles(ctx, t, rows)
|
|
||||||
|
|
||||||
var out bytes.Buffer
|
|
||||||
|
|
||||||
v := newContainmentVaultik(ctx, afero.NewOsFs())
|
|
||||||
v.UI = ui.NewWithColor(&out, false)
|
|
||||||
|
|
||||||
session := &restoreSession{
|
|
||||||
v: v,
|
|
||||||
ctx: ctx,
|
|
||||||
repos: repos,
|
|
||||||
opts: &RestoreOptions{TargetDir: targetDir},
|
|
||||||
result: &RestoreResult{},
|
|
||||||
}
|
|
||||||
for _, row := range rows {
|
|
||||||
require.NoError(t, session.restoreFile(row))
|
|
||||||
}
|
|
||||||
|
|
||||||
v.warnIfOwnershipNotPreserved(rows, targetDir)
|
|
||||||
|
|
||||||
return out.String()
|
|
||||||
}
|
|
||||||
@@ -39,11 +39,6 @@ var (
|
|||||||
// caller can report it apart from a failed backup.
|
// caller can report it apart from a failed backup.
|
||||||
var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
|
var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
|
||||||
|
|
||||||
// ErrFilesUnreadable is returned when --skip-errors skipped files or
|
|
||||||
// directories it could not read or list. The snapshot is complete and
|
|
||||||
// kept, so the caller can report it apart from a failed backup.
|
|
||||||
var ErrFilesUnreadable = errors.New("file(s) could not be read")
|
|
||||||
|
|
||||||
// listRecentLimit caps how many snapshot rows are fetched from the
|
// listRecentLimit caps how many snapshot rows are fetched from the
|
||||||
// local index for listing and purge operations.
|
// local index for listing and purge operations.
|
||||||
const listRecentLimit = 10000
|
const listRecentLimit = 10000
|
||||||
@@ -135,16 +130,12 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Process each named snapshot
|
// Process each named snapshot
|
||||||
filesUnreadable := 0
|
|
||||||
|
|
||||||
for snapIdx, snapName := range snapshotNames {
|
for snapIdx, snapName := range snapshotNames {
|
||||||
unreadable, err := v.createNamedSnapshot(
|
err = v.createNamedSnapshot(
|
||||||
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
filesUnreadable += unreadable
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Print overall summary if multiple snapshots
|
// Print overall summary if multiple snapshots
|
||||||
@@ -153,34 +144,13 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
|||||||
len(snapshotNames), v.UI.Duration(time.Since(overallStartTime)))
|
len(snapshotNames), v.UI.Duration(time.Since(overallStartTime)))
|
||||||
}
|
}
|
||||||
|
|
||||||
switch {
|
if opts.Prune {
|
||||||
case opts.Prune && filesUnreadable > 0:
|
|
||||||
// An older snapshot may be the only one that still holds a file
|
|
||||||
// this run could not read.
|
|
||||||
log.Warn("Skipping post-backup prune",
|
|
||||||
"files_unreadable", filesUnreadable)
|
|
||||||
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
|
|
||||||
"could not be read; older snapshots are kept.", filesUnreadable)
|
|
||||||
case opts.Prune:
|
|
||||||
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("%w: %w", ErrPostBackupPruneFailed, err)
|
return fmt.Errorf("%w: %w", ErrPostBackupPruneFailed, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return v.finishCreateSnapshot(filesUnreadable)
|
|
||||||
}
|
|
||||||
|
|
||||||
// finishCreateSnapshot ends a run whose snapshots are all complete and
|
|
||||||
// kept. Like finishRestore, it returns an error when --skip-errors
|
|
||||||
// skipped anything it could not read or list; otherwise it prints the
|
|
||||||
// closing line. A file removed during the run is counted as failed in
|
|
||||||
// the summary but is not an error, with or without --skip-errors.
|
|
||||||
func (v *Vaultik) finishCreateSnapshot(filesUnreadable int) error {
|
|
||||||
if filesUnreadable > 0 {
|
|
||||||
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Terminus must obey the --cron invariant: silent on total
|
// Terminus must obey the --cron invariant: silent on total
|
||||||
// success only. UI.Complete is dropped in cron/quiet mode (that's
|
// success only. UI.Complete is dropped in cron/quiet mode (that's
|
||||||
// the success path), but if any warnings fired during the run we
|
// the success path), but if any warnings fired during the run we
|
||||||
@@ -270,22 +240,16 @@ type snapshotStats struct {
|
|||||||
totalBlobsUploaded int
|
totalBlobsUploaded int
|
||||||
uploadDuration time.Duration
|
uploadDuration time.Duration
|
||||||
|
|
||||||
// The part of totalFilesFailed that --skip-errors skipped because it
|
|
||||||
// could not be read or listed.
|
|
||||||
totalFilesUnreadable int
|
|
||||||
|
|
||||||
// The sizes of all blobs the snapshot references, set by
|
// The sizes of all blobs the snapshot references, set by
|
||||||
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
||||||
blobSize int64
|
blobSize int64
|
||||||
blobUncompressedSize int64
|
blobUncompressedSize int64
|
||||||
}
|
}
|
||||||
|
|
||||||
// createNamedSnapshot creates a single named snapshot and returns the
|
// createNamedSnapshot creates a single named snapshot
|
||||||
// number of entries --skip-errors skipped because it could not read or
|
|
||||||
// list them.
|
|
||||||
func (v *Vaultik) createNamedSnapshot(
|
func (v *Vaultik) createNamedSnapshot(
|
||||||
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
|
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
|
||||||
) (int, error) {
|
) error {
|
||||||
snapshotStartTime := time.Now()
|
snapshotStartTime := time.Now()
|
||||||
|
|
||||||
if total > 1 {
|
if total > 1 {
|
||||||
@@ -294,12 +258,12 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
|
|
||||||
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
|
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
previousSnapshotID, err := v.previousSnapshotID(snapName)
|
previousSnapshotID, err := v.previousSnapshotID(snapName)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
scanner := v.ScannerFactory(snapshot.ScannerParams{
|
scanner := v.ScannerFactory(snapshot.ScannerParams{
|
||||||
@@ -314,7 +278,7 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
|
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
|
||||||
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
|
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, fmt.Errorf("creating snapshot: %w", err)
|
return fmt.Errorf("creating snapshot: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
|
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
|
||||||
@@ -322,12 +286,12 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
|
|
||||||
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
|
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
err = v.finalizeSnapshotMetadata(snapshotID, stats)
|
err = v.finalizeSnapshotMetadata(snapshotID, stats)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
log.Info("Snapshot complete",
|
log.Info("Snapshot complete",
|
||||||
@@ -340,7 +304,7 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
|
|
||||||
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
||||||
|
|
||||||
return stats.totalFilesUnreadable, nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// previousSnapshotID returns the ID of the most recent completed snapshot
|
// previousSnapshotID returns the ID of the most recent completed snapshot
|
||||||
@@ -426,7 +390,6 @@ func (v *Vaultik) scanAllDirectories(
|
|||||||
stats.totalBlobs += result.BlobsCreated
|
stats.totalBlobs += result.BlobsCreated
|
||||||
stats.totalFilesSkipped += result.FilesSkipped
|
stats.totalFilesSkipped += result.FilesSkipped
|
||||||
stats.totalFilesFailed += result.FilesFailed
|
stats.totalFilesFailed += result.FilesFailed
|
||||||
stats.totalFilesUnreadable += result.FilesUnreadable
|
|
||||||
stats.totalBytesSkipped += result.BytesSkipped
|
stats.totalBytesSkipped += result.BytesSkipped
|
||||||
stats.totalFilesDeleted += result.FilesDeleted
|
stats.totalFilesDeleted += result.FilesDeleted
|
||||||
stats.totalBytesDeleted += result.BytesDeleted
|
stats.totalBytesDeleted += result.BytesDeleted
|
||||||
@@ -1553,9 +1516,7 @@ func (v *Vaultik) removeSnapshotRemote(
|
|||||||
// database from the local index, and (with --remote) every snapshot
|
// database from the local index, and (with --remote) every snapshot
|
||||||
// metadata directory in remote storage. Both sides are processed so a
|
// metadata directory in remote storage. Both sides are processed so a
|
||||||
// "remove --all" leaves nothing behind, even when the local DB and
|
// "remove --all" leaves nothing behind, even when the local DB and
|
||||||
// remote storage have diverged. Unless LocalOnly is set, it fails
|
// remote storage have diverged.
|
||||||
// before it removes anything when the destination store cannot be
|
|
||||||
// listed or is not present by checkDestinationMarker.
|
|
||||||
func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) {
|
func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) {
|
||||||
err := v.EnsureStorageBinding()
|
err := v.EnsureStorageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1568,15 +1529,11 @@ func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error)
|
|||||||
}
|
}
|
||||||
|
|
||||||
// remoteKeys is the set of metadata/<key>/ subdirectories on the
|
// remoteKeys is the set of metadata/<key>/ subdirectories on the
|
||||||
// destination store. Only for a local-only remove is a failure
|
// destination store; failures are downgraded to a warning so a
|
||||||
// downgraded to a warning, so a permission-denied or unreachable
|
// permission-denied or unreachable remote can't block a local-only
|
||||||
// remote can't block it.
|
// remove.
|
||||||
remoteKeys, remoteErr := v.listAllRemoteSnapshotKeys()
|
remoteKeys, remoteErr := v.listAllRemoteSnapshotKeys()
|
||||||
if remoteErr != nil {
|
if remoteErr != nil {
|
||||||
if !opts.LocalOnly {
|
|
||||||
return nil, remoteErr
|
|
||||||
}
|
|
||||||
|
|
||||||
log.Warn("Could not list remote snapshots", "error", remoteErr)
|
log.Warn("Could not list remote snapshots", "error", remoteErr)
|
||||||
v.UI.Warningf("Could not list remote snapshots: %v.", remoteErr)
|
v.UI.Warningf("Could not list remote snapshots: %v.", remoteErr)
|
||||||
}
|
}
|
||||||
@@ -1638,7 +1595,8 @@ func (v *Vaultik) localSnapshotIDs() ([]string, error) {
|
|||||||
// listAllRemoteSnapshotKeys collects the hashed remote keys
|
// listAllRemoteSnapshotKeys collects the hashed remote keys
|
||||||
// (subdirectories under metadata/) currently present in the
|
// (subdirectories under metadata/) currently present in the
|
||||||
// destination store. Returns (nil, err) when the store cannot be
|
// destination store. Returns (nil, err) when the store cannot be
|
||||||
// listed, or is not present by checkDestinationMarker.
|
// listed, or is not present by checkDestinationMarker; callers must
|
||||||
|
// treat that as "no remote info available," not fatal.
|
||||||
func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) {
|
func (v *Vaultik) listAllRemoteSnapshotKeys() ([]string, error) {
|
||||||
log.Info("Listing all remote snapshots")
|
log.Info("Listing all remote snapshots")
|
||||||
|
|
||||||
@@ -1822,12 +1780,10 @@ func (v *Vaultik) deleteIncompleteSnapshot(snapshotIDStr string, id any) bool {
|
|||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// removeAllLocalSnapshots removes each known local snapshot's remote
|
// removeAllLocalSnapshots removes each known local snapshot (and, unless
|
||||||
// metadata, unless --local-only, and then its local record, returning
|
// --local-only, its remote metadata), returning the number of snapshots
|
||||||
// the number of snapshots it could not remove. The local record goes
|
// it could not remove. A snapshot left in the local index keeps its
|
||||||
// only once the metadata is deleted: dropped first, it would leave that
|
// remote metadata too, since it is not an orphan remote key.
|
||||||
// metadata on the destination store as a remote-only snapshot whose name
|
|
||||||
// is lost.
|
|
||||||
func (v *Vaultik) removeAllLocalSnapshots(
|
func (v *Vaultik) removeAllLocalSnapshots(
|
||||||
localSnaps []string, opts *RemoveOptions, result *RemoveResult,
|
localSnaps []string, opts *RemoveOptions, result *RemoveResult,
|
||||||
) int {
|
) int {
|
||||||
@@ -1836,19 +1792,6 @@ func (v *Vaultik) removeAllLocalSnapshots(
|
|||||||
for _, snapshotID := range localSnaps {
|
for _, snapshotID := range localSnaps {
|
||||||
log.Info("Removing snapshot", "snapshot_id", snapshotID)
|
log.Info("Removing snapshot", "snapshot_id", snapshotID)
|
||||||
|
|
||||||
if !opts.LocalOnly {
|
|
||||||
_, err := v.deleteRemoteSnapshotByKey(
|
|
||||||
snapshot.RemoteSnapshotKey(snapshotID))
|
|
||||||
if err != nil {
|
|
||||||
log.Warn("Failed to remove snapshot metadata from remote",
|
|
||||||
"snapshot_id", snapshotID, "error", err)
|
|
||||||
|
|
||||||
failures++
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err := v.deleteSnapshotFromLocalDB(snapshotID)
|
err := v.deleteSnapshotFromLocalDB(snapshotID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Error("Failed to remove from local database",
|
log.Error("Failed to remove from local database",
|
||||||
@@ -1859,6 +1802,17 @@ func (v *Vaultik) removeAllLocalSnapshots(
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if !opts.LocalOnly {
|
||||||
|
_, err = v.deleteRemoteSnapshotByKey(
|
||||||
|
snapshot.RemoteSnapshotKey(snapshotID))
|
||||||
|
if err != nil {
|
||||||
|
log.Warn("Failed to remove snapshot metadata from remote",
|
||||||
|
"snapshot_id", snapshotID, "error", err)
|
||||||
|
|
||||||
|
failures++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID)
|
result.SnapshotsRemoved = append(result.SnapshotsRemoved, snapshotID)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -178,13 +178,11 @@ func TestBackupDoesNotCountFileInUnopenableDirectoryAsDeleted(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
env.out.Reset()
|
env.out.Reset()
|
||||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||||
SkipErrors: true,
|
SkipErrors: true,
|
||||||
Snapshots: []string{"work"},
|
Snapshots: []string{"work"},
|
||||||
})
|
}))
|
||||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
assert.Contains(t, env.out.String(), filesLine(1, 0, 1))
|
||||||
assert.Contains(t, env.out.String(),
|
|
||||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// A backup counts only files that the previous snapshot of its own name
|
// A backup counts only files that the previous snapshot of its own name
|
||||||
|
|||||||
@@ -49,19 +49,13 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
|||||||
name string
|
name string
|
||||||
openErr error
|
openErr error
|
||||||
skipErrors bool
|
skipErrors bool
|
||||||
wantErr error // nil for none
|
|
||||||
}{
|
}{
|
||||||
// What a normal user gets opening a file with mode 000.
|
// What a normal user gets opening a file with mode 000.
|
||||||
{name: "unopenable under skip-errors",
|
{name: "unopenable under skip-errors",
|
||||||
openErr: os.ErrPermission, skipErrors: true,
|
openErr: os.ErrPermission, skipErrors: true},
|
||||||
wantErr: vaultik.ErrFilesUnreadable},
|
|
||||||
// What opening a file removed after phase 1 gives.
|
// What opening a file removed after phase 1 gives.
|
||||||
{name: "removed between the phases",
|
{name: "removed between the phases",
|
||||||
openErr: os.ErrNotExist, skipErrors: false},
|
openErr: os.ErrNotExist, skipErrors: false},
|
||||||
// A removal is not an error, so --skip-errors does not make the
|
|
||||||
// run fail. See https://git.eeqj.de/sneak/vaultik/issues/310.
|
|
||||||
{name: "removed between the phases under skip-errors",
|
|
||||||
openErr: os.ErrNotExist, skipErrors: true},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
@@ -105,11 +99,10 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
|||||||
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
|
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
|
||||||
v.UI = ui.NewWithColor(out, false)
|
v.UI = ui.NewWithColor(out, false)
|
||||||
|
|
||||||
err = v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
require.NoError(t, v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||||
SkipErrors: tt.skipErrors,
|
SkipErrors: tt.skipErrors,
|
||||||
Snapshots: []string{"src"},
|
Snapshots: []string{"src"},
|
||||||
})
|
}))
|
||||||
require.ErrorIs(t, err, tt.wantErr)
|
|
||||||
|
|
||||||
summary := out.String()
|
summary := out.String()
|
||||||
assert.Contains(t, summary,
|
assert.Contains(t, summary,
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package vaultik_test
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
@@ -74,210 +73,3 @@ func TestSkipErrorsBackupStopsOnIndexErrorRecordingDirectory(t *testing.T) {
|
|||||||
require.Len(t, snapshots, 1)
|
require.Len(t, snapshots, 1)
|
||||||
assert.Nil(t, snapshots[0].CompletedAt)
|
assert.Nil(t, snapshots[0].CompletedAt)
|
||||||
}
|
}
|
||||||
|
|
||||||
// A backup under --skip-errors that skips a file it cannot open, or a
|
|
||||||
// directory it cannot list, keeps the snapshot, counts what it skipped
|
|
||||||
// as failed and returns an error instead of finishing successfully. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
|
||||||
func TestSkipErrorsBackupKeepsSnapshotAndFailsWhenItSkips(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
skipped string // relative to the source directory
|
|
||||||
}{
|
|
||||||
{name: "file it cannot open", skipped: "skipdir/file.txt"},
|
|
||||||
{name: "directory it cannot list", skipped: "skipdir"},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newDeletedCountEnv(t, map[string][]byte{
|
|
||||||
"stored.txt": []byte("stored"),
|
|
||||||
"skipdir/file.txt": []byte("in skipdir"),
|
|
||||||
})
|
|
||||||
env.setSnapshot("work", "")
|
|
||||||
|
|
||||||
// A mode does not lock out root, who runs the tests, so
|
|
||||||
// opening the skipped path is made to fail instead.
|
|
||||||
env.v.Fs = &openFailFs{
|
|
||||||
path: filepath.Join(env.src, tt.skipped),
|
|
||||||
err: os.ErrPermission,
|
|
||||||
}
|
|
||||||
|
|
||||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
|
||||||
SkipErrors: true,
|
|
||||||
Snapshots: []string{"work"},
|
|
||||||
})
|
|
||||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
|
||||||
|
|
||||||
assert.Contains(t, env.out.String(),
|
|
||||||
"Files: 2 examined, 1 backed up, 0 unchanged, 1 failed.")
|
|
||||||
assert.NotContains(t, env.out.String(), "Finished successfully.")
|
|
||||||
|
|
||||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
|
||||||
context.Background(), listRecentTestLimit)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.Len(t, snapshots, 1)
|
|
||||||
assert.NotNil(t, snapshots[0].CompletedAt, "the snapshot is kept")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// removeOnLookupFs is the real filesystem, except that path is removed
|
|
||||||
// just before it is looked up. The walk looks an entry up after listing
|
|
||||||
// its directory, so path is listed but gone when the walk reaches it.
|
|
||||||
type removeOnLookupFs struct {
|
|
||||||
afero.OsFs
|
|
||||||
|
|
||||||
path string
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *removeOnLookupFs) LstatIfPossible(
|
|
||||||
name string,
|
|
||||||
) (os.FileInfo, bool, error) {
|
|
||||||
if name == f.path {
|
|
||||||
_ = os.Remove(name)
|
|
||||||
}
|
|
||||||
|
|
||||||
return f.OsFs.LstatIfPossible(name)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A file removed before the walk reaches it is counted once, as failed,
|
|
||||||
// like a file removed before its data is read, and the backup finishes
|
|
||||||
// successfully, with or without --skip-errors. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
|
||||||
func TestBackupCountsFileRemovedBeforeWalkReachesItAsFailed(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
skipErrors bool
|
|
||||||
}{
|
|
||||||
{name: "under skip-errors", skipErrors: true},
|
|
||||||
{name: "without skip-errors", skipErrors: false},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newDeletedCountEnv(t, map[string][]byte{
|
|
||||||
"stays.txt": []byte("stays"),
|
|
||||||
"removed.txt": []byte("removed"),
|
|
||||||
})
|
|
||||||
env.setSnapshot("walk", "")
|
|
||||||
|
|
||||||
// The previous snapshot has the file, so a removal the walk
|
|
||||||
// also counted as deleted would show.
|
|
||||||
env.backUp(t, "walk")
|
|
||||||
|
|
||||||
env.v.Fs = &removeOnLookupFs{
|
|
||||||
path: filepath.Join(env.src, "removed.txt"),
|
|
||||||
}
|
|
||||||
|
|
||||||
env.out.Reset()
|
|
||||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
|
||||||
SkipErrors: tt.skipErrors,
|
|
||||||
Snapshots: []string{"walk"},
|
|
||||||
}))
|
|
||||||
assert.Contains(t, env.out.String(),
|
|
||||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
|
||||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// A file an exclude pattern matches is not counted when it is removed
|
|
||||||
// before the walk reaches it. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
|
||||||
func TestBackupIgnoresExcludedFileRemovedBeforeWalkReachesIt(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newDeletedCountEnv(t, map[string][]byte{
|
|
||||||
"stays.txt": []byte("stays"),
|
|
||||||
"removed.tmp": []byte("removed"),
|
|
||||||
})
|
|
||||||
env.setSnapshot("walk", "", "*.tmp")
|
|
||||||
env.v.Fs = &removeOnLookupFs{
|
|
||||||
path: filepath.Join(env.src, "removed.tmp"),
|
|
||||||
}
|
|
||||||
|
|
||||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
|
||||||
SkipErrors: true,
|
|
||||||
Snapshots: []string{"walk"},
|
|
||||||
}))
|
|
||||||
assert.Contains(t, env.out.String(), filesLine(1, 1, 0))
|
|
||||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
|
||||||
}
|
|
||||||
|
|
||||||
// A backup under --skip-errors --prune that skipped a file it could not
|
|
||||||
// read does not prune: the previous snapshot, the only one that holds
|
|
||||||
// the file, is kept, and the output says the prune was skipped. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
|
||||||
func TestSkipErrorsBackupDoesNotPruneWhenItSkips(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newDeletedCountEnv(t, map[string][]byte{
|
|
||||||
"other.txt": []byte("other"),
|
|
||||||
"skipped.txt": []byte("skipped"),
|
|
||||||
})
|
|
||||||
env.setSnapshot("photos", "")
|
|
||||||
env.backUp(t, "photos")
|
|
||||||
|
|
||||||
// An unchanged file is not read again, so the file is changed before
|
|
||||||
// it becomes unreadable.
|
|
||||||
skipped := filepath.Join(env.src, "skipped.txt")
|
|
||||||
require.NoError(t, os.WriteFile(skipped, []byte("changed"), 0o600))
|
|
||||||
env.v.Fs = &openFailFs{path: skipped, err: os.ErrPermission}
|
|
||||||
|
|
||||||
env.out.Reset()
|
|
||||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
|
||||||
Prune: true,
|
|
||||||
SkipErrors: true,
|
|
||||||
Snapshots: []string{"photos"},
|
|
||||||
})
|
|
||||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
|
||||||
assert.Contains(t, env.out.String(),
|
|
||||||
"Skipped the post-backup prune because 1 file(s) could not be read")
|
|
||||||
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
|
|
||||||
|
|
||||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
|
||||||
context.Background(), listRecentTestLimit)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, snapshots, 2, "the previous snapshot is kept")
|
|
||||||
}
|
|
||||||
|
|
||||||
// A file removed during a backup under --skip-errors --prune is not a
|
|
||||||
// skipped file, so the backup prunes as usual. See
|
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
|
||||||
func TestSkipErrorsBackupPrunesWhenAFileIsRemovedDuringIt(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newDeletedCountEnv(t, map[string][]byte{
|
|
||||||
"other.txt": []byte("other"),
|
|
||||||
"removed.txt": []byte("removed"),
|
|
||||||
})
|
|
||||||
env.setSnapshot("photos", "")
|
|
||||||
env.backUp(t, "photos")
|
|
||||||
|
|
||||||
env.v.Fs = &removeOnLookupFs{path: filepath.Join(env.src, "removed.txt")}
|
|
||||||
|
|
||||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
|
||||||
Prune: true,
|
|
||||||
SkipErrors: true,
|
|
||||||
Snapshots: []string{"photos"},
|
|
||||||
}))
|
|
||||||
|
|
||||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
|
||||||
context.Background(), listRecentTestLimit)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Len(t, snapshots, 1, "the previous snapshot is pruned")
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -13,7 +13,6 @@ import (
|
|||||||
"github.com/spf13/afero"
|
"github.com/spf13/afero"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"golang.org/x/sys/unix"
|
|
||||||
"sneak.berlin/go/vaultik/internal/config"
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
"sneak.berlin/go/vaultik/internal/database"
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
@@ -189,42 +188,6 @@ func (e *summaryEnv) dataLine(total, backedUp int64) string {
|
|||||||
e.v.UI.Size(total), e.v.UI.Size(backedUp))
|
e.v.UI.Size(total), e.v.UI.Size(backedUp))
|
||||||
}
|
}
|
||||||
|
|
||||||
// enumeratedLine returns the line printed when the walk of a source path
|
|
||||||
// ends.
|
|
||||||
func (e *summaryEnv) enumeratedLine(
|
|
||||||
files int, total int64, toBackUp int, toBackUpSize int64,
|
|
||||||
) string {
|
|
||||||
return fmt.Sprintf(
|
|
||||||
"Enumerated %d snapshot source files (%s total), %d to back up (%s).",
|
|
||||||
files, e.v.UI.Size(total), toBackUp, e.v.UI.Size(toBackUpSize))
|
|
||||||
}
|
|
||||||
|
|
||||||
// snapshotPaths returns the paths the snapshot lists, sorted.
|
|
||||||
func (e *summaryEnv) snapshotPaths(t *testing.T, snapshotID string) []string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
rows, err := e.db.Conn().QueryContext(context.Background(), `
|
|
||||||
SELECT f.path FROM snapshot_files sf JOIN files f ON f.id = sf.file_id
|
|
||||||
WHERE sf.snapshot_id = ? ORDER BY f.path`, snapshotID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
defer func() { _ = rows.Close() }()
|
|
||||||
|
|
||||||
var paths []string
|
|
||||||
|
|
||||||
for rows.Next() {
|
|
||||||
var path string
|
|
||||||
|
|
||||||
require.NoError(t, rows.Scan(&path))
|
|
||||||
|
|
||||||
paths = append(paths, path)
|
|
||||||
}
|
|
||||||
|
|
||||||
require.NoError(t, rows.Err())
|
|
||||||
|
|
||||||
return paths
|
|
||||||
}
|
|
||||||
|
|
||||||
// A first backup stores copy.bin's chunks while backing up a.bin, so
|
// A first backup stores copy.bin's chunks while backing up a.bin, so
|
||||||
// copy.bin's chunks are deduplicated within the run. Each file and byte
|
// copy.bin's chunks are deduplicated within the run. Each file and byte
|
||||||
// is still counted once.
|
// is still counted once.
|
||||||
@@ -285,85 +248,6 @@ func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
|
|||||||
assert.Equal(t, uncompressed, snap.BlobUncompressedSize)
|
assert.Equal(t, uncompressed, snap.BlobUncompressedSize)
|
||||||
}
|
}
|
||||||
|
|
||||||
// A backup of a tree with directories and a symlink, none of which
|
|
||||||
// changed since the last backup, backs up nothing, and the snapshot
|
|
||||||
// still lists every one of them. Both sides of the line that ends the
|
|
||||||
// walk count the same entries, the files and the symlink, and no
|
|
||||||
// directory's size. See https://git.eeqj.de/sneak/vaultik/issues/305.
|
|
||||||
func TestSnapshotSummaryUnchangedRunWithDirectoriesAndSymlink(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newSummaryEnv(t)
|
|
||||||
|
|
||||||
const linkTarget = "a.bin"
|
|
||||||
|
|
||||||
require.NoError(t, os.Symlink(linkTarget,
|
|
||||||
filepath.Join(filepath.Dir(env.aPath), "link")))
|
|
||||||
|
|
||||||
// A symlink's size is the length of its target.
|
|
||||||
total := env.totalSize + int64(len(linkTarget))
|
|
||||||
|
|
||||||
first := env.backUp(t, "first", false)
|
|
||||||
assert.Contains(t, first, env.enumeratedLine(4, total, 4, total))
|
|
||||||
assert.Contains(t, first, filesLine(4, 4, 0))
|
|
||||||
|
|
||||||
second := env.backUp(t, "second", false)
|
|
||||||
assert.Contains(t, second, env.enumeratedLine(4, total, 0, 0))
|
|
||||||
assert.Contains(t, second, filesLine(4, 0, 4))
|
|
||||||
assert.Contains(t, second, env.dataLine(total, 0))
|
|
||||||
|
|
||||||
assert.Equal(t,
|
|
||||||
env.snapshotPaths(t, env.snapshot(t, "first").ID.String()),
|
|
||||||
env.snapshotPaths(t, env.snapshot(t, "second").ID.String()))
|
|
||||||
}
|
|
||||||
|
|
||||||
// A directory whose mtime changed and a symlink whose target changed are
|
|
||||||
// recorded again by the next backup, which counts only the symlink as
|
|
||||||
// backed up.
|
|
||||||
func TestSnapshotSummaryChangedDirectoryAndSymlink(t *testing.T) {
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
env := newSummaryEnv(t)
|
|
||||||
|
|
||||||
// The local index keeps paths with symlinks resolved.
|
|
||||||
dir, err := filepath.EvalSymlinks(filepath.Dir(env.aPath))
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
// The new link gets the old one's mtime, and a target, which need not
|
|
||||||
// exist, of the old target's length, a symlink's size: only the
|
|
||||||
// target differs.
|
|
||||||
linkTime := unix.NsecToTimeval(
|
|
||||||
time.Date(2029, time.March, 4, 5, 6, 7, 0, time.UTC).UnixNano())
|
|
||||||
link := filepath.Join(dir, "link")
|
|
||||||
require.NoError(t, os.Symlink("a.bin", link))
|
|
||||||
require.NoError(t, unix.Lutimes(link, []unix.Timeval{linkTime, linkTime}))
|
|
||||||
|
|
||||||
env.backUp(t, "first", false)
|
|
||||||
|
|
||||||
require.NoError(t, os.Remove(link))
|
|
||||||
require.NoError(t, os.Symlink("b.bin", link))
|
|
||||||
require.NoError(t, unix.Lutimes(link, []unix.Timeval{linkTime, linkTime}))
|
|
||||||
|
|
||||||
mtime := time.Date(2030, time.January, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
require.NoError(t, os.Chtimes(dir, mtime, mtime))
|
|
||||||
|
|
||||||
assert.Contains(t, env.backUp(t, "second", false), filesLine(4, 1, 3))
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
|
|
||||||
dirRow, err := env.repos.Files.GetByPath(ctx, dir)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NotNil(t, dirRow)
|
|
||||||
assert.Equal(t, mtime.Unix(), dirRow.MTime.Unix())
|
|
||||||
|
|
||||||
linkRow, err := env.repos.Files.GetByPath(ctx, link)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NotNil(t, linkRow)
|
|
||||||
assert.Equal(t, "b.bin", linkRow.LinkTarget.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
// Under --cron the progress reporter is off; the upload figures must
|
// Under --cron the progress reporter is off; the upload figures must
|
||||||
// still reach the summary and the snapshots row. The snapshot has two
|
// still reach the summary and the snapshots row. The snapshot has two
|
||||||
// paths, each backed up by its own scan.
|
// paths, each backed up by its own scan.
|
||||||
|
|||||||
Reference in New Issue
Block a user