Compare commits
1
Commits
next
..
523629f1a8
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
523629f1a8 |
+1
-1
@@ -359,7 +359,7 @@ CreateSnapshot(opts)
|
||||
|
||||
## Deduplication Strategy
|
||||
|
||||
1. **File-level**: Files and symlinks unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid, and a symlink's target), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked
|
||||
1. **File-level**: Files unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid), unless the file lists a chunk that no uploaded blob holds; such a file is re-chunked
|
||||
|
||||
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded.
|
||||
|
||||
|
||||
@@ -96,14 +96,6 @@ vaultik snapshot restore <snapshot-id> ~/vaultik-restore
|
||||
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
|
||||
```
|
||||
|
||||
On a destination shared between hosts or users, the prune in that cron line
|
||||
relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup
|
||||
another one is still running, so it must be longer than the longest backup
|
||||
run. It does not protect a blob that backup reuses from an earlier run that
|
||||
stopped before its manifest: the prune can delete such a blob, and the new
|
||||
snapshot then cannot be restored. To not depend on the grace period, give each
|
||||
host its own prefix or directory. See [locking](#locking).
|
||||
|
||||
## restoring on another machine
|
||||
|
||||
Restoring on a host that never ran the backup — a replacement machine
|
||||
@@ -190,7 +182,7 @@ vaultik version
|
||||
* `--verbose`, `-v`: Enable verbose output (on stderr — see below)
|
||||
* `--debug`: Enable debug output (on stderr — see below)
|
||||
* `--quiet`, `-q`: Suppress non-error output (also suppresses startup banner)
|
||||
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run. A run that skipped anything still finishes but exits non-zero: `snapshot create` keeps the snapshot and counts each file or directory it skipped as failed. A file removed during the backup is also counted as failed but does not change the exit status.
|
||||
* `--skip-errors`: Skip files that cannot be read when creating a snapshot, or that cannot be restored when restoring, instead of aborting. Packing and storage errors while creating a snapshot (which would leave a chunk recorded but not stored) still abort the run.
|
||||
|
||||
### locking
|
||||
|
||||
@@ -200,22 +192,7 @@ process-wide lock at `$XDG_DATA_HOME/vaultik/vaultik.pid`
|
||||
(`~/.local/share/vaultik/vaultik.pid` on Linux) for the whole run. Only
|
||||
one of them runs at a time: a second one exits immediately with an
|
||||
"already running" error rather than waiting, so two writers can never
|
||||
corrupt the local index.
|
||||
|
||||
The lock covers one user on one machine. It does not stop another user,
|
||||
or another host, from writing to the same destination store at the same
|
||||
time. A backup uploads its blobs before the manifest that lists them, so a
|
||||
prune from elsewhere at that moment finds blobs no snapshot lists. It
|
||||
deletes such a blob only once it is older than `prune_grace_period`
|
||||
(default `24h`), which therefore must be longer than the longest backup run
|
||||
of any host or user sharing the destination. The grace period does not
|
||||
protect a blob a backup reuses from an earlier run that stopped before its
|
||||
manifest: a backup skips the upload of a blob already on the destination, so
|
||||
that blob keeps the time of its first upload, and a prune from elsewhere
|
||||
during the backup can delete it; the new snapshot then cannot be restored.
|
||||
`remote nuke` deletes everything regardless. To not depend on the grace
|
||||
period, give each host or user its own prefix (or directory) on the
|
||||
destination.
|
||||
corrupt the local index or the destination store.
|
||||
|
||||
Read-only commands — `info`, `snapshot list`, `snapshot verify`, and
|
||||
`remote info` — do not take the lock and are never blocked, so they run
|
||||
@@ -319,13 +296,8 @@ value where a list belongs or a word where a number belongs), is refused.
|
||||
* `--cron`: Silent on total success; warnings and errors are still printed
|
||||
(for crontab)
|
||||
* `--prune`: After backup, drop older snapshots of each backed-up name and
|
||||
remove orphaned blobs older than `prune_grace_period` from remote storage,
|
||||
as `prune` does. By default keeps only the latest
|
||||
snapshot per name; use `--keep-newer-than` for a rolling window. When
|
||||
`--skip-errors` skipped a file or directory it could not read or list, the
|
||||
prune does not run and every older snapshot is kept, since one of them may
|
||||
be the only one that holds what was skipped; the command says so and exits
|
||||
non-zero. A file removed during the backup does not stop the prune.
|
||||
remove orphaned blobs from remote storage. By default keeps only the latest
|
||||
snapshot per name; use `--keep-newer-than` for a rolling window.
|
||||
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
||||
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
||||
Without `--prune`, or with a value that is not a valid duration, the
|
||||
@@ -447,15 +419,9 @@ local snapshot whose remote metadata is missing is dropped from the
|
||||
local index; (2) delete orphaned local rows (files, chunks, blobs no
|
||||
longer referenced by any completed snapshot); (3) list every remote
|
||||
manifest on the destination store to compute the still-referenced blob
|
||||
set and delete any blob not in that set that is older than
|
||||
`prune_grace_period` (default `24h`). A newer one is kept and counted: it
|
||||
may belong to a backup still running, on this host or another sharing the
|
||||
destination store, that has not yet uploaded its manifest. The grace period
|
||||
does not protect a blob such a backup reuses from an earlier run that stopped
|
||||
before its manifest: that blob keeps the time of its first upload, so it can
|
||||
be deleted. Step (3) reads all remote manifests — network cost scales with
|
||||
the number of snapshots.
|
||||
`snapshot create --prune` runs the same cleanup automatically; this is the
|
||||
set and delete any blob not in that set. Step (3) reads all remote
|
||||
manifests — network cost scales with the number of snapshots. `snapshot
|
||||
create --prune` runs the same cleanup automatically; this is the
|
||||
manual entry point for the same work. If the destination store cannot
|
||||
be listed, or the local index holds snapshots and the destination has
|
||||
no `vaultik-destination.txt` (an unplugged volume's empty mount point),
|
||||
@@ -466,9 +432,6 @@ exits non-zero.
|
||||
* `--json`: Output stats as JSON. Also skips the confirmation prompt, as
|
||||
`--force` does. A blob that cannot be deleted is counted in
|
||||
`blobs_failed`, and the document is printed before the non-zero exit.
|
||||
A blob kept for being newer than `prune_grace_period`, or because the
|
||||
listing of the destination gives no time for it, is counted in
|
||||
`blobs_too_new` and not in `blobs_found`.
|
||||
|
||||
**`info`**: Display system configuration, storage settings, encryption
|
||||
recipients, and local database statistics.
|
||||
@@ -543,14 +506,6 @@ being written, a killed upload can leave a truncated object under its name,
|
||||
which a later backup takes for complete. A leftover `.partial` file is ignored
|
||||
and can be deleted.
|
||||
|
||||
A destination shared between hosts or users, each with its own local index,
|
||||
relies on `prune_grace_period` to keep one's prune from deleting the blobs of
|
||||
another's backup still running, so it must be longer than the longest backup
|
||||
run. It does not protect a blob that backup reuses from an earlier run that
|
||||
stopped before its manifest, which the prune can delete. To not depend on the
|
||||
grace period, give each host its own prefix (or directory). See
|
||||
[locking](#locking).
|
||||
|
||||
Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is
|
||||
still supported for backward compatibility. `storage_url` takes precedence if
|
||||
both are set.
|
||||
@@ -637,8 +592,7 @@ format does and does not protect.
|
||||
1. List all snapshot manifests
|
||||
2. Build set of all referenced blob hashes
|
||||
3. List all blobs in storage
|
||||
4. Delete any blob not in the referenced set that is older than
|
||||
`prune_grace_period`
|
||||
4. Delete any blob not in the referenced set
|
||||
|
||||
### chunking and deduplication
|
||||
|
||||
@@ -683,7 +637,6 @@ complete annotated example also lives in
|
||||
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
||||
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
|
||||
| `compression_level` | `3` | zstd compression level (1-19) |
|
||||
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it. A backup uploads its blobs before the manifest that lists them, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs at once |
|
||||
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
||||
| `index_path` | platform data dir | Local SQLite index path |
|
||||
|
||||
|
||||
@@ -15,229 +15,13 @@ pre-1.0
|
||||
# Next Step
|
||||
|
||||
The 1.0 work is complete on `next`: the scope settled on
|
||||
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) has landed,
|
||||
and so has every fix from the defect hunt before the release candidate
|
||||
([issue #288](https://git.eeqj.de/sneak/vaultik/issues/288)). What is
|
||||
left is merging `next` to `main` and tagging the release, both the
|
||||
owner's; a `v*` tag runs the release build in
|
||||
`.gitea/workflows/release.yml`.
|
||||
[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) was the work
|
||||
already planned for 1.0, and all of it has landed. The mechanism to cut
|
||||
the tag exists and is exercised; what is left is merging `next` to
|
||||
`main` and tagging, both the owner's.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
||||
that skipped files it could not read
|
||||
([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a
|
||||
prune could delete the only snapshot still holding those files.
|
||||
|
||||
- 2026-10-14: Made `snapshot create --skip-errors` exit non-zero,
|
||||
instead of 0, when it skips anything it cannot read
|
||||
([issue #310](https://git.eeqj.de/sneak/vaultik/issues/310)).
|
||||
|
||||
- 2026-10-14: Kept the local index when `remote nuke` cannot list or
|
||||
find the destination store
|
||||
([issue #292](https://git.eeqj.de/sneak/vaultik/issues/292)). Nuke
|
||||
had cleared the local snapshot records anyway; it now fails before
|
||||
removing anything.
|
||||
|
||||
- 2026-10-14: Stopped counting directories and unchanged symlinks as
|
||||
files to back up
|
||||
([issue #305](https://git.eeqj.de/sneak/vaultik/issues/305)).
|
||||
|
||||
- 2026-10-14: Refused a backup destination that an unplugged volume
|
||||
left empty, as the owner ruled
|
||||
([issue #239](https://git.eeqj.de/sneak/vaultik/issues/239)). Every
|
||||
backup now writes `vaultik-destination.txt` at the destination root,
|
||||
and while the local index holds snapshots stored there, a destination
|
||||
without it is treated as not present.
|
||||
|
||||
- 2026-10-14: Warned about restored ownership only when an entry's
|
||||
owner or group differs from the recorded one
|
||||
([issue #316](https://git.eeqj.de/sneak/vaultik/issues/316)); a
|
||||
restore by a non-root user had warned even when none did.
|
||||
|
||||
- 2026-10-14: Made `snapshot list` report a changed destination instead
|
||||
of advising `prune`, which refuses in that case
|
||||
([issue #317](https://git.eeqj.de/sneak/vaultik/issues/317)).
|
||||
|
||||
- 2026-10-14: Counted a file as deleted by a backup only if the
|
||||
previous snapshot of the same name listed it
|
||||
([issue #304](https://git.eeqj.de/sneak/vaultik/issues/304)); files
|
||||
that snapshot did not list had been counted too.
|
||||
|
||||
- 2026-10-14: Named each prefix once in restore, verify and purge
|
||||
errors, some of which repeated it
|
||||
([issue #327](https://git.eeqj.de/sneak/vaultik/issues/327)). A deep
|
||||
verify log line that named S3 whatever the destination now names
|
||||
storage.
|
||||
|
||||
- 2026-10-14: Refused credentials in a storage URL's endpoint or rclone
|
||||
remote name, which were accepted and could then be printed
|
||||
([issue #322](https://git.eeqj.de/sneak/vaultik/issues/322)).
|
||||
|
||||
- 2026-10-14: Ran `$EDITOR` through the shell in `config edit`, so a
|
||||
value with arguments, such as `code --wait`, no longer fails
|
||||
([issue #315](https://git.eeqj.de/sneak/vaultik/issues/315)).
|
||||
|
||||
- 2026-10-14: Made `snapshot remove` fail, instead of exiting 0, when
|
||||
it cannot delete the metadata of a snapshot that is only on the
|
||||
destination store
|
||||
([issue #340](https://git.eeqj.de/sneak/vaultik/issues/340)). The
|
||||
README and the help now say when it prints the `vaultik prune`
|
||||
follow-up.
|
||||
|
||||
- 2026-10-14: Labelled the totals in the local index cleanup before
|
||||
each backup as totals
|
||||
([issue #306](https://git.eeqj.de/sneak/vaultik/issues/306)). Each
|
||||
was shown as `(N remain)`, which read as rows left to clean up.
|
||||
|
||||
- 2026-10-13: Freed a cached blob once no file still to restore needs
|
||||
it ([issue #324](https://git.eeqj.de/sneak/vaultik/issues/324)); one
|
||||
also used by a file outside the selected paths had stayed in the
|
||||
temporary cache until the restore ended.
|
||||
|
||||
- 2026-10-13: Kept the whole startup error when it contains `): `
|
||||
([issue #320](https://git.eeqj.de/sneak/vaultik/issues/320)). Such
|
||||
an error was cut at its last `): `, so the `blob_size_limit` one lost
|
||||
the setting's name.
|
||||
|
||||
- 2026-10-13: Silenced the status lines of `prune` and snapshot removal
|
||||
under `--cron` and `--quiet`
|
||||
([issue #294](https://git.eeqj.de/sneak/vaultik/issues/294)).
|
||||
|
||||
- 2026-10-13: Read the rclone config file that `RCLONE_CONFIG` names
|
||||
([issue #321](https://git.eeqj.de/sneak/vaultik/issues/321)); a
|
||||
remote defined only there had been reported as not found.
|
||||
|
||||
- 2026-10-13: Made `/**/` in an exclude pattern match zero directories,
|
||||
as in gitignore
|
||||
([issue #338](https://git.eeqj.de/sneak/vaultik/issues/338));
|
||||
`a/**/*.log` had missed `a/x.log`.
|
||||
|
||||
- 2026-10-13: Made the `chunk_size` and `blob_size_limit` defaults and
|
||||
the `chunk_size` minimum 1000-based, like the size parser and the
|
||||
docs ([issue #313](https://git.eeqj.de/sneak/vaultik/issues/313)).
|
||||
|
||||
- 2026-10-13: Kept a snapshot's remote metadata when `snapshot purge`
|
||||
cannot remove it from the local index
|
||||
([issue #337](https://git.eeqj.de/sneak/vaultik/issues/337)); the
|
||||
purge had deleted the metadata anyway and exited 0.
|
||||
|
||||
- 2026-10-13: Made `snapshot restore` fail for a requested path that
|
||||
matches nothing, which it had ignored
|
||||
([issue #309](https://git.eeqj.de/sneak/vaultik/issues/309)).
|
||||
|
||||
- 2026-10-13: Made `remote nuke` delete every object under `metadata/`
|
||||
and `blobs/`
|
||||
([issue #291](https://git.eeqj.de/sneak/vaultik/issues/291)); it had
|
||||
left some, such as a stray `.DS_Store`, while reporting the store
|
||||
empty.
|
||||
|
||||
- 2026-10-13: Made `config set` create a list for an index under a
|
||||
missing key
|
||||
([issue #308](https://git.eeqj.de/sneak/vaultik/issues/308)); it had
|
||||
written a map, after which no command could load the config.
|
||||
|
||||
- 2026-10-13: Listed a `file://` destination through a symlinked
|
||||
`metadata/` or `blobs/`, which had listed as empty
|
||||
([issue #296](https://git.eeqj.de/sneak/vaultik/issues/296)). A link
|
||||
to a missing target now fails the listing
|
||||
([issue #332](https://git.eeqj.de/sneak/vaultik/issues/332)).
|
||||
|
||||
- 2026-10-13: Made `snapshot verify --deep --json` report its blob
|
||||
counts ([issue #314](https://git.eeqj.de/sneak/vaultik/issues/314));
|
||||
a failed run had reported 0 missing and 0 verified.
|
||||
|
||||
- 2026-10-13: Limited and retried the module download in the Docker
|
||||
build ([issue #351](https://git.eeqj.de/sneak/vaultik/issues/351)),
|
||||
which had hung for as long as a stalled connection to the module
|
||||
proxy lasted.
|
||||
|
||||
- 2026-10-13: Recorded each snapshot's uncompressed size when its
|
||||
backup runs
|
||||
([issue #360](https://git.eeqj.de/sneak/vaultik/issues/360)). A later
|
||||
backup could change an older snapshot's UNCOMPRESSED SIZE in
|
||||
`snapshot list`.
|
||||
|
||||
- 2026-10-13: Stopped `snapshot restore --skip-errors --verify`
|
||||
reporting a file that failed to restore again as a checksum failure
|
||||
([issue #323](https://git.eeqj.de/sneak/vaultik/issues/323)).
|
||||
|
||||
- 2026-10-13: Recorded the stored size of a blob a backup finds already
|
||||
stored, not the size of its local copy, which can differ, so
|
||||
`snapshot verify` could fail on a good snapshot
|
||||
([issue #300](https://git.eeqj.de/sneak/vaultik/issues/300)).
|
||||
|
||||
- 2026-10-13: Stopped the local index and the blob upload step ending
|
||||
the process on a failed close of query results or a failed temp file
|
||||
cleanup ([issue #345](https://git.eeqj.de/sneak/vaultik/issues/345));
|
||||
both are now returned as errors.
|
||||
|
||||
- 2026-10-13: Closed the snapshot database's storage reader too when
|
||||
restore or deep verify is interrupted
|
||||
([issue #356](https://git.eeqj.de/sneak/vaultik/issues/356)).
|
||||
|
||||
- 2026-10-13: Made `prune` and `remote nuke` fail when a delete on the
|
||||
backup destination store fails
|
||||
([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)).
|
||||
|
||||
- 2026-10-13: Made an S3 listing that the server cuts short an error
|
||||
([issue #297](https://git.eeqj.de/sneak/vaultik/issues/297)); `prune`
|
||||
had taken it as complete and could delete blobs still in use.
|
||||
|
||||
- 2026-10-13: Made shallow `snapshot verify` report a blob it could not
|
||||
check apart from a missing one
|
||||
([issue #318](https://git.eeqj.de/sneak/vaultik/issues/318)); any
|
||||
error checking a blob had counted as missing, its cause dropped.
|
||||
|
||||
- 2026-10-13: Restored the other files when a file's chunk is in no
|
||||
blob under `--skip-errors`
|
||||
([issue #326](https://git.eeqj.de/sneak/vaultik/issues/326)); that
|
||||
file had stopped the restore before anything was restored.
|
||||
|
||||
- 2026-10-13: Recorded each snapshot's new chunk size when its backup
|
||||
runs ([issue #311](https://git.eeqj.de/sneak/vaultik/issues/311)).
|
||||
`snapshot list` could show a NEW CHUNK SIZE of 0 B for a snapshot
|
||||
whose change was a modified file.
|
||||
|
||||
- 2026-10-13: Made a backup interrupted during a local index write exit
|
||||
as interrupted
|
||||
([issue #301](https://git.eeqj.de/sneak/vaultik/issues/301)); it could
|
||||
end with `FATAL: failed to rollback transaction` and exit 1 instead of
|
||||
130.
|
||||
|
||||
- 2026-10-13: Printed `snapshot list` and `snapshot purge` sizes like
|
||||
every other command
|
||||
([issue #312](https://git.eeqj.de/sneak/vaultik/issues/312)). They,
|
||||
and the `bytes` value in terminal log lines, were 1024-based but
|
||||
labelled KB or MB, and now use 1000-based units.
|
||||
|
||||
- 2026-10-13: Closed each blob's storage reader when restore or deep
|
||||
verify is interrupted, which ends a stalled read unless it is stuck in
|
||||
the kernel ([issue #325](https://git.eeqj.de/sneak/vaultik/issues/325)).
|
||||
|
||||
- 2026-10-13: Made a mistyped subcommand under `snapshot`, `config`,
|
||||
`remote`, `database` or `completion` an error
|
||||
([issue #319](https://git.eeqj.de/sneak/vaultik/issues/319)); each had
|
||||
printed its help and exited 0.
|
||||
|
||||
- 2026-10-13: Made an exclude pattern starting with `**/` match
|
||||
directly under the snapshot path too, as in gitignore
|
||||
([issue #303](https://git.eeqj.de/sneak/vaultik/issues/303)).
|
||||
|
||||
- 2026-10-13: Refused a relative `file://` storage URL and decoded its
|
||||
percent-escapes
|
||||
([issue #299](https://git.eeqj.de/sneak/vaultik/issues/299)); a
|
||||
relative path had moved with the working directory.
|
||||
|
||||
- 2026-10-13: Removed the blob temp file when a backup stops part-way
|
||||
([issue #302](https://git.eeqj.de/sneak/vaultik/issues/302)); the
|
||||
half-written blob had stayed in `$TMPDIR`.
|
||||
|
||||
- 2026-10-13: Checked `--keep-newer-than` before a backup starts
|
||||
([issue #307](https://git.eeqj.de/sneak/vaultik/issues/307)); an
|
||||
invalid value had been read only by the prune after the backup, so
|
||||
retention never ran.
|
||||
|
||||
- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that
|
||||
is not a blob as an orphaned blob
|
||||
([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray
|
||||
|
||||
@@ -328,18 +328,6 @@ storage_url: "rclone://myremote/path/to/backups"
|
||||
# Higher = better compression but slower
|
||||
# Default: 3
|
||||
compression_level: 5
|
||||
|
||||
# How old a blob that no snapshot lists must be before prune deletes it
|
||||
# A backup uploads its blobs before the manifest that lists them, so on a
|
||||
# destination shared between hosts or users this must be longer than the
|
||||
# longest backup run
|
||||
# It does not protect a blob a backup reuses from an earlier run that stopped
|
||||
# before its manifest: that blob keeps the time of its first upload, so a
|
||||
# prune can delete it
|
||||
# Supports: 36h, 90m, etc.; 0s deletes them at once
|
||||
# Default: 24h
|
||||
#prune_grace_period: 24h
|
||||
|
||||
# Hostname to use in backup metadata
|
||||
# Default: system hostname
|
||||
#hostname: myserver
|
||||
|
||||
+1
-2
@@ -228,8 +228,7 @@ The restore process doesn't use the local database. Instead:
|
||||
|
||||
1. **Identify Unreferenced Blobs**
|
||||
- Query blobs not referenced by any remaining snapshot
|
||||
- Delete from S3, once older than `prune_grace_period`, and from the local
|
||||
database
|
||||
- Delete from S3 and local database
|
||||
|
||||
### 6. Incomplete Snapshot Cleanup
|
||||
|
||||
|
||||
@@ -166,7 +166,7 @@ The prune operation is safe because:
|
||||
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references
|
||||
2. Manifests are unencrypted and can be read without keys
|
||||
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced
|
||||
4. It deletes an unreferenced blob only once the blob is older than `prune_grace_period` (default `24h`). A backup uploads its blobs before the manifest that lists them, so a prune running meanwhile, on this host or another sharing the destination, leaves them as long as no backup run takes longer than the grace period. The grace period does not protect a blob a backup reuses from an earlier run that stopped before its manifest: the backup skips the upload of a blob already on the destination, so that blob keeps the time of its first upload, and a prune before the backup's manifest is written can delete it; the new snapshot then cannot be restored. `remote nuke` has no such check
|
||||
4. Prune requires exclusive access to the destination: running it during a concurrent backup can race a snapshot whose manifest is not yet written, so do not prune while a backup is in progress
|
||||
|
||||
## Restoration Requirements
|
||||
|
||||
|
||||
@@ -239,16 +239,6 @@ storage_url: ""
|
||||
# Default: 3
|
||||
# compression_level: 3
|
||||
|
||||
# How old a blob that no snapshot lists must be before prune deletes it.
|
||||
# A backup uploads its blobs before the manifest that lists them, so on a
|
||||
# destination shared between hosts or users this must be longer than the
|
||||
# longest backup run. It does not protect a blob a backup reuses from an
|
||||
# earlier run that stopped before its manifest: that blob keeps the time of
|
||||
# its first upload, so a prune can delete it.
|
||||
# Accepts: 36h, 90m, etc. 0s deletes them at once.
|
||||
# Default: 24h
|
||||
# prune_grace_period: 24h
|
||||
|
||||
# Hostname used in snapshot IDs. Default: system hostname.
|
||||
# hostname: myserver
|
||||
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -15,8 +14,8 @@ import (
|
||||
const flagCron = "--cron"
|
||||
|
||||
// plantUnreferencedBlob writes a blob that no snapshot references to the
|
||||
// destination store at storeDir, dated long before any prune grace period,
|
||||
// so that a prune deletes it. Returns its path.
|
||||
// destination store at storeDir, so that a prune deletes it. Returns its
|
||||
// path.
|
||||
func plantUnreferencedBlob(t *testing.T, storeDir string) string {
|
||||
t.Helper()
|
||||
|
||||
@@ -26,9 +25,6 @@ func plantUnreferencedBlob(t *testing.T, storeDir string) string {
|
||||
require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o750))
|
||||
require.NoError(t, os.WriteFile(path, []byte("unreferenced"), 0o600))
|
||||
|
||||
longAgo := time.Unix(0, 0)
|
||||
require.NoError(t, os.Chtimes(path, longAgo, longAgo))
|
||||
|
||||
return path
|
||||
}
|
||||
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
|
||||
// under --skip-errors, a source directory that does not exist, which the
|
||||
// walk skips. The command exits non-zero and reports the backup as
|
||||
// completed, not as a failed snapshot creation. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
//
|
||||
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
|
||||
// writeBackupConfig would create the source directory.
|
||||
configPath := writeHermeticPruneConfig(t, false)
|
||||
|
||||
code, stdout, stderr := runEntry(t,
|
||||
flagConfig, configPath, "--skip-errors", cmdSnapshot, cmdCreate)
|
||||
|
||||
assert.Equal(t, 1, code)
|
||||
assert.Contains(t, stdout,
|
||||
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
|
||||
assert.NotContains(t, stdout, "Finished successfully.")
|
||||
assert.Contains(t, stderr,
|
||||
"Backup completed, but 1 file(s) could not be read")
|
||||
assert.NotContains(t, stderr, "Snapshot creation failed")
|
||||
}
|
||||
@@ -20,12 +20,7 @@ Local cleanup drops incomplete snapshots and any files, chunks, or
|
||||
blobs no longer referenced by a completed snapshot. Remote cleanup
|
||||
scans every snapshot manifest in the destination store, builds the
|
||||
set of still-referenced blob hashes, and deletes any blob not in that
|
||||
set that is older than prune_grace_period (default 24h). A newer one
|
||||
is kept: it may belong to a backup still running, on this host or
|
||||
another sharing the destination store, that has not yet uploaded its
|
||||
manifest. The grace period does not protect a blob such a backup
|
||||
reuses from an earlier run that stopped before its manifest: that
|
||||
blob keeps the time of its first upload, so it can be deleted.
|
||||
set.
|
||||
|
||||
Snapshot create --prune runs the same cleanup automatically; this
|
||||
command is the manual entry point for the same work (e.g. after a
|
||||
|
||||
@@ -115,13 +115,6 @@ on Linux), then /etc/vaultik/config.yml.`,
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, vaultik.ErrFilesUnreadable) {
|
||||
log.Error("Backup completed with unreadable files", "error", err)
|
||||
ReportErrorf("Backup completed, but %v", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
log.Error("Snapshot creation failed", "error", err)
|
||||
ReportErrorf("Snapshot creation failed: %v", err)
|
||||
})
|
||||
@@ -132,8 +125,7 @@ on Linux), then /etc/vaultik/config.yml.`,
|
||||
"Run in cron mode (silent unless warning or error)")
|
||||
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
|
||||
"After backup, drop older snapshots of the same name and remove "+
|
||||
"orphaned blobs older than prune_grace_period (not after a "+
|
||||
"backup that skipped unreadable files)")
|
||||
"orphaned blobs")
|
||||
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
||||
"With --prune: keep snapshots newer than this duration "+
|
||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
||||
|
||||
@@ -11,7 +11,6 @@ import (
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"filippo.io/age"
|
||||
"git.eeqj.de/sneak/smartconfig"
|
||||
@@ -39,7 +38,6 @@ const (
|
||||
defaultChunkSize = Size(10 * 1000 * 1000) // 10MB
|
||||
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MiB
|
||||
defaultCompressionLevel = 3
|
||||
defaultPruneGracePeriod = 24 * time.Hour
|
||||
minChunkSize = 1000 * 1000 // 1MB
|
||||
minCompressionLevel = 1
|
||||
maxCompressionLevel = 19
|
||||
@@ -69,8 +67,6 @@ var (
|
||||
errFileURLNotAbsolute = errors.New(
|
||||
"a file:// storage_url needs an absolute path, " +
|
||||
"as in file:///path/to/backup")
|
||||
errNegativePruneGracePeriod = errors.New(
|
||||
"prune_grace_period must not be negative")
|
||||
errStorageNotConfigured = errors.New(
|
||||
"storage not configured; set storage_url or provide s3.endpoint + " +
|
||||
"s3.bucket + credentials")
|
||||
@@ -191,13 +187,6 @@ type Config struct {
|
||||
Snapshots map[string]SnapshotConfig `yaml:"snapshots"`
|
||||
CompressionLevel int `yaml:"compression_level"`
|
||||
|
||||
// PruneGracePeriod is how old a blob no manifest lists must be before
|
||||
// a prune deletes it. A backup lists its blobs in a manifest only
|
||||
// once it has uploaded them all, so a younger one may belong to a
|
||||
// backup still running, on this host or another sharing the
|
||||
// destination store.
|
||||
PruneGracePeriod time.Duration `yaml:"prune_grace_period"`
|
||||
|
||||
// StorageURL specifies the storage backend using a URL format.
|
||||
// Takes precedence over S3Config if set.
|
||||
// Supported formats:
|
||||
@@ -299,7 +288,6 @@ func Parse(data []byte) (*Config, error) {
|
||||
ChunkSize: defaultChunkSize,
|
||||
IndexPath: filepath.Join(xdg.DataHome, appName, "index.sqlite"),
|
||||
CompressionLevel: defaultCompressionLevel,
|
||||
PruneGracePeriod: defaultPruneGracePeriod,
|
||||
S3: S3Config{PartSize: defaultS3PartSize},
|
||||
}
|
||||
|
||||
@@ -368,7 +356,6 @@ func Parse(data []byte) (*Config, error) {
|
||||
// exceeds the configured limit
|
||||
// - Compression level must be between 1 and 19
|
||||
// - S3 part size must be between 5MiB and 5GiB, the part sizes S3 accepts
|
||||
// - Prune grace period must not be negative
|
||||
//
|
||||
// Returns an error describing the first validation failure encountered.
|
||||
func (c *Config) Validate() error {
|
||||
@@ -418,10 +405,6 @@ func (c *Config) Validate() error {
|
||||
return errBadS3PartSize
|
||||
}
|
||||
|
||||
if c.PruneGracePeriod < 0 {
|
||||
return errNegativePruneGracePeriod
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -6,7 +6,6 @@ import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/chunker"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
@@ -413,73 +412,6 @@ func TestDocumentedSizeExamples(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestValidatePruneGracePeriod checks that a negative prune_grace_period
|
||||
// is refused rather than acting like 0s, which deletes unreferenced blobs
|
||||
// at once, and that 0s and the documented examples are accepted.
|
||||
func TestValidatePruneGracePeriod(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const withoutGracePeriod = "snapshots:\n" +
|
||||
" test:\n" +
|
||||
" paths: [/tmp/vaultik-test-source]\n" +
|
||||
"storage_url: file:///tmp/vaultik-test-storage\n"
|
||||
|
||||
tests := []struct {
|
||||
setting string
|
||||
wantErr error
|
||||
}{
|
||||
{"prune_grace_period: 0s", nil},
|
||||
{"prune_grace_period: 90m", nil},
|
||||
{"prune_grace_period: 36h", nil},
|
||||
{"prune_grace_period: -1h", errNegativePruneGracePeriod},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.setting, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg, err := Parse([]byte(withoutGracePeriod + tt.setting + "\n"))
|
||||
if err != nil {
|
||||
t.Fatalf("Parse() unexpected error: %v", err)
|
||||
}
|
||||
|
||||
err = cfg.Validate()
|
||||
if !errors.Is(err, tt.wantErr) {
|
||||
t.Fatalf("Validate() error = %v, want %v", err, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoadPruneGracePeriod checks that a config file without
|
||||
// prune_grace_period loads with the documented 24h. The value is written out
|
||||
// here rather than taken from defaultPruneGracePeriod, so that a change of the
|
||||
// default fails this test.
|
||||
func TestLoadPruneGracePeriod(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const withoutGracePeriod = "snapshots:\n" +
|
||||
" test:\n" +
|
||||
" paths: [/tmp/vaultik-test-source]\n" +
|
||||
"storage_url: file:///tmp/vaultik-test-storage\n"
|
||||
|
||||
path := filepath.Join(t.TempDir(), "config.yml")
|
||||
|
||||
err := os.WriteFile(path, []byte(withoutGracePeriod), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
|
||||
cfg, err := Load(path)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() unexpected error: %v", err)
|
||||
}
|
||||
|
||||
if cfg.PruneGracePeriod != 24*time.Hour {
|
||||
t.Errorf("prune_grace_period = %s, want 24h", cfg.PruneGracePeriod)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoadRejectsRelativeFileURL checks that a file:// storage_url with a
|
||||
// relative path fails at load. Accepted, it would resolve against the
|
||||
// working directory, so a backup run from another directory would write
|
||||
|
||||
@@ -8,7 +8,6 @@ import (
|
||||
"io"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go-v2/aws"
|
||||
"github.com/aws/aws-sdk-go-v2/config"
|
||||
@@ -303,10 +302,7 @@ func IsNotFound(err error) bool {
|
||||
type ObjectInfo struct {
|
||||
Key string
|
||||
Size int64
|
||||
// ModTime is the object's LastModified time. Only ListObjectsStream
|
||||
// sets it.
|
||||
ModTime time.Time
|
||||
Err error
|
||||
Err error
|
||||
}
|
||||
|
||||
// ListObjectsStream lists objects with the given prefix and returns a channel.
|
||||
@@ -347,9 +343,8 @@ func (c *Client) ListObjectsStream(
|
||||
}
|
||||
|
||||
ch <- ObjectInfo{
|
||||
Key: key,
|
||||
Size: *obj.Size,
|
||||
ModTime: aws.ToTime(obj.LastModified),
|
||||
Key: key,
|
||||
Size: *obj.Size,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -78,7 +78,8 @@ func TestProgressPercentWithSecondPathSmaller(t *testing.T) {
|
||||
|
||||
stats := progress.GetStats()
|
||||
|
||||
// With both paths counted, the percentage must not exceed 100%.
|
||||
// Directories count toward the total size but produce no chunks, so
|
||||
// the percentage ends just under 100%.
|
||||
percent := float64(stats.BytesProcessed.Load()) /
|
||||
float64(stats.TotalSize.Load()) * 100
|
||||
if percent > 100 {
|
||||
|
||||
+88
-167
@@ -141,12 +141,10 @@ type ScannerConfig struct {
|
||||
// ScanResult contains the results of a scan operation. Files and bytes
|
||||
// are counted per file: BytesScanned is the size of the new and changed
|
||||
// files, BytesSkipped that of the unchanged ones. FilesFailed counts the
|
||||
// new and changed files that phase 2 could not store, each entry removed
|
||||
// after the walk listed its directory and, under --skip-errors, each
|
||||
// entry the walk could not read, such as a directory it could not list;
|
||||
// FilesScanned includes them and BytesScanned does not. ChunksCreated
|
||||
// counts the chunks that were not already stored, and NewChunkSize is
|
||||
// their total size.
|
||||
// new and changed files that phase 2 could not store; FilesScanned
|
||||
// includes them and BytesScanned does not. ChunksCreated counts the
|
||||
// chunks that were not already stored, and NewChunkSize is their total
|
||||
// size.
|
||||
type ScanResult struct {
|
||||
FilesScanned int
|
||||
FilesSkipped int
|
||||
@@ -163,11 +161,6 @@ type ScanResult struct {
|
||||
UploadDuration time.Duration
|
||||
StartTime time.Time
|
||||
EndTime time.Time
|
||||
|
||||
// FilesUnreadable is the part of FilesFailed that --skip-errors
|
||||
// skipped because it could not be read or listed; the rest are files
|
||||
// removed after the walk listed them.
|
||||
FilesUnreadable int
|
||||
}
|
||||
|
||||
// NewScanner creates a new scanner instance
|
||||
@@ -291,11 +284,6 @@ func (s *Scanner) Scan(
|
||||
}
|
||||
}
|
||||
|
||||
err = s.recordDirectories(ctx, scanResult.Directories)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Summarize scan phase results and update progress
|
||||
s.summarizeScanPhase(result, filesToProcess)
|
||||
|
||||
@@ -867,15 +855,14 @@ func (s *Scanner) batchInsertFileData(
|
||||
// ScanPhaseResult contains the results of the scan phase
|
||||
type ScanPhaseResult struct {
|
||||
FilesToProcess []*FileToProcess
|
||||
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
|
||||
Directories []*FileToProcess // Every directory found, to record
|
||||
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
|
||||
}
|
||||
|
||||
// scanPhase performs the initial directory scan to identify files to process
|
||||
// It uses the pre-loaded knownFiles map for fast change detection without DB
|
||||
// queries. It also populates existingFiles map for deletion detection.
|
||||
// Returns files needing processing, IDs of unchanged files for snapshot
|
||||
// association, and the directories found.
|
||||
// Returns files needing processing and IDs of unchanged files for snapshot
|
||||
// association.
|
||||
func (s *Scanner) scanPhase(
|
||||
ctx context.Context,
|
||||
path string,
|
||||
@@ -883,10 +870,8 @@ func (s *Scanner) scanPhase(
|
||||
existingFiles map[string]struct{},
|
||||
knownFiles map[string]*database.File,
|
||||
) (*ScanPhaseResult, error) {
|
||||
// Use known file count as estimate for progress (accurate for
|
||||
// subsequent backups). The walk does not count directories, so the
|
||||
// estimate leaves them out.
|
||||
estimatedTotal := countNonDirectories(knownFiles)
|
||||
// Use known file count as estimate for progress (accurate for subsequent backups)
|
||||
estimatedTotal := int64(len(knownFiles))
|
||||
|
||||
collector := &scanCollector{}
|
||||
|
||||
@@ -897,28 +882,8 @@ func (s *Scanner) scanPhase(
|
||||
log.Debug("Starting directory walk", "path", path)
|
||||
|
||||
walkFn := func(filePath string, info os.FileInfo, err error) error {
|
||||
// Check exclude patterns first, so an excluded entry that is gone
|
||||
// or cannot be looked up is neither counted nor reported. For
|
||||
// directories, skip the entire subtree. info is nil when the
|
||||
// walk could not look the entry up.
|
||||
if s.shouldExclude(filePath, path) {
|
||||
if info != nil && info.IsDir() {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// The source path is in no directory listing, so its absence is
|
||||
// an error rather than a removal.
|
||||
if errors.Is(err, os.ErrNotExist) && filePath != path {
|
||||
countRemovedDuringWalk(filePath, existingFiles, result)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return s.handleWalkError(filePath, err, result)
|
||||
return s.handleWalkError(filePath, err)
|
||||
}
|
||||
|
||||
// Check context cancellation
|
||||
@@ -928,9 +893,18 @@ func (s *Scanner) scanPhase(
|
||||
default:
|
||||
}
|
||||
|
||||
// Check exclude patterns - for directories, skip the entire subtree
|
||||
if s.shouldExclude(filePath, path) {
|
||||
if info.IsDir() {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Handle symlinks and directories
|
||||
handled, err := s.recordSpecialEntry(
|
||||
filePath, info, existingFiles, knownFiles, collector, result)
|
||||
filePath, info, existingFiles, collector, result)
|
||||
if handled {
|
||||
return err
|
||||
}
|
||||
@@ -944,10 +918,9 @@ func (s *Scanner) scanPhase(
|
||||
existingFiles[filePath] = struct{}{}
|
||||
|
||||
// Check file against in-memory map (no DB query!)
|
||||
file := s.buildRegularFileEntry(filePath, info)
|
||||
needsProcessing := s.checkFileInMemory(file, knownFiles)
|
||||
file, needsProcessing := s.checkFileInMemory(filePath, info, knownFiles)
|
||||
|
||||
filesScanned, changedCount := collector.recordFile(
|
||||
filesScanned, changedCount := collector.recordRegularFile(
|
||||
filePath, info, file, needsProcessing)
|
||||
|
||||
// Update result stats
|
||||
@@ -971,45 +944,27 @@ func (s *Scanner) scanPhase(
|
||||
return &ScanPhaseResult{
|
||||
FilesToProcess: collector.filesToProcess,
|
||||
UnchangedFileIDs: collector.unchangedFileIDs,
|
||||
Directories: collector.directories,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// countNonDirectories returns how many of files are not directories.
|
||||
func countNonDirectories(files map[string]*database.File) int64 {
|
||||
var count int64
|
||||
|
||||
for _, f := range files {
|
||||
if f.Mode&uint32(os.ModeDir) == 0 {
|
||||
count++
|
||||
}
|
||||
}
|
||||
|
||||
return count
|
||||
}
|
||||
|
||||
// recordSpecialEntry records symlinks and directories (which have no
|
||||
// data to chunk) and reports whether it handled the entry. A symlink is
|
||||
// counted and checked for changes as a regular file is; a directory is
|
||||
// set aside to be recorded and is not counted. For a symlink whose
|
||||
// target cannot be read it returns handleWalkError's result.
|
||||
// data to chunk) and reports whether it handled the entry. For a symlink
|
||||
// whose target cannot be read it returns handleWalkError's result.
|
||||
func (s *Scanner) recordSpecialEntry(
|
||||
filePath string, info os.FileInfo,
|
||||
existingFiles map[string]struct{},
|
||||
knownFiles map[string]*database.File,
|
||||
collector *scanCollector, result *ScanResult,
|
||||
) (bool, error) {
|
||||
// Handle symlinks
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
file, err := s.buildSymlinkEntry(filePath, info)
|
||||
if err != nil {
|
||||
return true, s.handleWalkError(filePath, err, result)
|
||||
return true, s.handleWalkError(filePath, err)
|
||||
}
|
||||
|
||||
existingFiles[filePath] = struct{}{}
|
||||
needsProcessing := s.checkFileInMemory(file, knownFiles)
|
||||
collector.recordFile(filePath, info, file, needsProcessing)
|
||||
s.updateScanEntryStats(result, needsProcessing, info)
|
||||
collector.addToProcess(filePath, info, file)
|
||||
s.updateScanEntryStats(result, true, info)
|
||||
|
||||
return true, nil
|
||||
}
|
||||
@@ -1019,7 +974,7 @@ func (s *Scanner) recordSpecialEntry(
|
||||
if info.IsDir() {
|
||||
file := s.buildDirectoryEntry(filePath, info)
|
||||
existingFiles[filePath] = struct{}{}
|
||||
collector.addDirectory(filePath, info, file)
|
||||
collector.addToProcess(filePath, info, file)
|
||||
|
||||
return true, nil
|
||||
}
|
||||
@@ -1027,38 +982,15 @@ func (s *Scanner) recordSpecialEntry(
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// countRemovedDuringWalk counts an entry that was in its directory's
|
||||
// listing but was gone when the walk looked it up, like a file removed
|
||||
// before phase 2 reads it: as examined and failed, and not as an error,
|
||||
// with or without --skip-errors.
|
||||
func countRemovedDuringWalk(
|
||||
filePath string, existingFiles map[string]struct{}, result *ScanResult,
|
||||
) {
|
||||
log.Warn("File was deleted during the directory walk, skipping",
|
||||
"path", filePath)
|
||||
|
||||
// The walk found it, so it is not also counted as deleted.
|
||||
existingFiles[filePath] = struct{}{}
|
||||
result.FilesScanned++
|
||||
result.FilesFailed++
|
||||
}
|
||||
|
||||
// handleWalkError deals with a filesystem error surfaced by the walk:
|
||||
// under --skip-errors the entry is skipped with a UI notice and counted
|
||||
// as examined, failed and unreadable, otherwise the error is fatal.
|
||||
func (s *Scanner) handleWalkError(
|
||||
filePath string, err error, result *ScanResult,
|
||||
) error {
|
||||
// skipped (with a UI notice) under --skip-errors, fatal otherwise.
|
||||
func (s *Scanner) handleWalkError(filePath string, err error) error {
|
||||
if s.skipErrors {
|
||||
log.Error("Failed to access file (skipping due to --skip-errors)",
|
||||
"path", filePath, "error", err)
|
||||
s.ui.Errorf("Failed to access %s: %v. Skipping (--skip-errors).",
|
||||
s.ui.Path(filePath), err)
|
||||
|
||||
result.FilesScanned++
|
||||
result.FilesFailed++
|
||||
result.FilesUnreadable++
|
||||
|
||||
return nil // Continue scanning
|
||||
}
|
||||
|
||||
@@ -1072,29 +1004,28 @@ type scanCollector struct {
|
||||
mu sync.Mutex
|
||||
filesToProcess []*FileToProcess
|
||||
unchangedFileIDs []types.FileID // Just IDs - no new records needed
|
||||
directories []*FileToProcess
|
||||
filesScanned int64
|
||||
}
|
||||
|
||||
// addDirectory sets a directory aside to be recorded. It is not counted
|
||||
// as a file.
|
||||
func (c *scanCollector) addDirectory(
|
||||
// addToProcess queues a file entry for processing.
|
||||
func (c *scanCollector) addToProcess(
|
||||
path string, info os.FileInfo, file *database.File,
|
||||
) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
c.directories = append(c.directories, &FileToProcess{
|
||||
c.filesToProcess = append(c.filesToProcess, &FileToProcess{
|
||||
Path: path,
|
||||
FileInfo: info,
|
||||
File: file,
|
||||
})
|
||||
c.filesScanned++
|
||||
}
|
||||
|
||||
// recordFile records a regular file or symlink as changed (to process)
|
||||
// or unchanged (snapshot association only) and returns the running
|
||||
// scanned and changed counts.
|
||||
func (c *scanCollector) recordFile(
|
||||
// recordRegularFile records a regular file as changed (to process) or
|
||||
// unchanged (snapshot association only) and returns the running scanned
|
||||
// and changed counts.
|
||||
func (c *scanCollector) recordRegularFile(
|
||||
path string, info os.FileInfo, file *database.File, needsProcessing bool,
|
||||
) (int64, int) {
|
||||
c.mu.Lock()
|
||||
@@ -1197,28 +1128,7 @@ func (s *Scanner) printScanProgressLine(
|
||||
}
|
||||
}
|
||||
|
||||
// buildRegularFileEntry creates a File record for a regular file, with
|
||||
// no ID: checkFileInMemory sets it.
|
||||
func (s *Scanner) buildRegularFileEntry(path string, info os.FileInfo) *database.File {
|
||||
var uid, gid uint32
|
||||
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
|
||||
uid = stat.Uid
|
||||
gid = stat.Gid
|
||||
}
|
||||
|
||||
return &database.File{
|
||||
Path: types.FilePath(path),
|
||||
SourcePath: types.SourcePath(s.currentSourcePath),
|
||||
MTime: info.ModTime(),
|
||||
Size: info.Size(),
|
||||
Mode: uint32(info.Mode()),
|
||||
UID: uid,
|
||||
GID: gid,
|
||||
}
|
||||
}
|
||||
|
||||
// buildSymlinkEntry creates a File record for a symlink, with no ID:
|
||||
// checkFileInMemory sets it.
|
||||
// buildSymlinkEntry creates a File record for a symlink.
|
||||
func (s *Scanner) buildSymlinkEntry(
|
||||
path string, info os.FileInfo,
|
||||
) (*database.File, error) {
|
||||
@@ -1234,6 +1144,7 @@ func (s *Scanner) buildSymlinkEntry(
|
||||
}
|
||||
|
||||
return &database.File{
|
||||
ID: types.NewFileID(),
|
||||
Path: types.FilePath(path),
|
||||
SourcePath: types.SourcePath(s.currentSourcePath),
|
||||
MTime: info.ModTime(),
|
||||
@@ -1265,22 +1176,6 @@ func (s *Scanner) buildDirectoryEntry(path string, info os.FileInfo) *database.F
|
||||
}
|
||||
}
|
||||
|
||||
// recordDirectories records every directory the walk found. A directory
|
||||
// has no data to back up, so it is not among the files counted and
|
||||
// backed up.
|
||||
func (s *Scanner) recordDirectories(
|
||||
ctx context.Context, directories []*FileToProcess,
|
||||
) error {
|
||||
for _, dir := range directories {
|
||||
err := s.recordNonRegularFile(ctx, dir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("recording directory %s: %w", dir.Path, err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// recordNonRegularFile writes a symlink or directory entry to the database
|
||||
// and associates it with the current snapshot. No chunking is performed.
|
||||
func (s *Scanner) recordNonRegularFile(ctx context.Context, ftp *FileToProcess) error {
|
||||
@@ -1294,36 +1189,62 @@ func (s *Scanner) recordNonRegularFile(ctx context.Context, ftp *FileToProcess)
|
||||
})
|
||||
}
|
||||
|
||||
// checkFileInMemory reports whether a regular file or symlink needs
|
||||
// processing, and gives file the ID of its known record, or a new ID for
|
||||
// a new path.
|
||||
// checkFileInMemory checks if a file needs processing using the in-memory map
|
||||
// No database access is performed - this is purely CPU/memory work
|
||||
func (s *Scanner) checkFileInMemory(
|
||||
file *database.File, knownFiles map[string]*database.File,
|
||||
) bool {
|
||||
existingFile, exists := knownFiles[file.Path.String()]
|
||||
path string, info os.FileInfo, knownFiles map[string]*database.File,
|
||||
) (*database.File, bool) {
|
||||
var uid, gid uint32
|
||||
if stat, ok := info.Sys().(*syscall.Stat_t); ok {
|
||||
uid = stat.Uid
|
||||
gid = stat.Gid
|
||||
}
|
||||
|
||||
// Check against in-memory map first to get existing ID if available
|
||||
existingFile, exists := knownFiles[path]
|
||||
|
||||
// Create file record with ID set upfront
|
||||
// For new files, generate UUID immediately so it's available for chunk associations
|
||||
// For existing files, reuse the existing ID
|
||||
var fileID types.FileID
|
||||
if exists {
|
||||
fileID = existingFile.ID
|
||||
} else {
|
||||
fileID = types.NewFileID()
|
||||
}
|
||||
|
||||
file := &database.File{
|
||||
ID: fileID,
|
||||
Path: types.FilePath(path),
|
||||
SourcePath: types.SourcePath(s.currentSourcePath),
|
||||
MTime: info.ModTime(),
|
||||
Size: info.Size(),
|
||||
Mode: uint32(info.Mode()),
|
||||
UID: uid,
|
||||
GID: gid,
|
||||
}
|
||||
|
||||
// New file - needs processing
|
||||
if !exists {
|
||||
file.ID = types.NewFileID()
|
||||
|
||||
return true
|
||||
return file, true
|
||||
}
|
||||
|
||||
file.ID = existingFile.ID
|
||||
|
||||
// No uploaded blob holds one of its chunks (see loadFilesToRechunk)
|
||||
if _, rechunk := s.filesToRechunk[file.ID]; rechunk {
|
||||
return true
|
||||
if _, rechunk := s.filesToRechunk[fileID]; rechunk {
|
||||
return file, true
|
||||
}
|
||||
|
||||
// Check if file has changed
|
||||
return existingFile.Size != file.Size ||
|
||||
if existingFile.Size != file.Size ||
|
||||
!existingFile.MTime.Equal(file.MTime) ||
|
||||
existingFile.Mode != file.Mode ||
|
||||
existingFile.UID != file.UID ||
|
||||
existingFile.GID != file.GID ||
|
||||
existingFile.LinkTarget != file.LinkTarget
|
||||
existingFile.GID != file.GID {
|
||||
return file, true
|
||||
}
|
||||
|
||||
// File unchanged
|
||||
return file, false
|
||||
}
|
||||
|
||||
// batchAddFilesToSnapshot adds existing file IDs to the snapshot association table
|
||||
@@ -1450,16 +1371,17 @@ func (s *Scanner) processPhase(
|
||||
return s.finalizeProcessPhase(ctx, result)
|
||||
}
|
||||
|
||||
// processFileWithErrorHandling records a symlink, or wraps
|
||||
// processFileWithErrorHandling records a directory or symlink, or wraps
|
||||
// processFileStreaming for a regular file with error recovery for deleted
|
||||
// files and skip-errors mode. Returns (skipped, error).
|
||||
func (s *Scanner) processFileWithErrorHandling(
|
||||
ctx context.Context, fileToProcess *FileToProcess, result *ScanResult,
|
||||
) (bool, error) {
|
||||
// A symlink has no data to open or read; it is only recorded in the
|
||||
// local index. An error recording it stops the run even under
|
||||
// --skip-errors, or the snapshot would complete without it.
|
||||
if os.FileMode(fileToProcess.File.Mode)&os.ModeSymlink != 0 {
|
||||
// A directory or symlink has no data to open or read; it is only
|
||||
// recorded in the local index. An error recording it stops the run
|
||||
// even under --skip-errors, or the snapshot would complete without it.
|
||||
mode := os.FileMode(fileToProcess.File.Mode)
|
||||
if mode&os.ModeSymlink != 0 || mode.IsDir() {
|
||||
err := s.recordNonRegularFile(ctx, fileToProcess)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("processing file %s: %w", fileToProcess.Path, err)
|
||||
@@ -1500,7 +1422,6 @@ func (s *Scanner) processFileWithErrorHandling(
|
||||
s.ui.Path(fileToProcess.Path), err)
|
||||
|
||||
countFailedFile(fileToProcess, result)
|
||||
result.FilesUnreadable++
|
||||
|
||||
return true, nil
|
||||
}
|
||||
|
||||
@@ -1,28 +0,0 @@
|
||||
//nolint:testpackage // exercises the unexported countNonDirectories helper
|
||||
package snapshot
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
)
|
||||
|
||||
// The walk counts regular files and symlinks, not directories, so the
|
||||
// estimate of its count made from the local index's records leaves the
|
||||
// directories out. See https://git.eeqj.de/sneak/vaultik/issues/305.
|
||||
func TestCountNonDirectories(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
known := map[string]*database.File{
|
||||
"/src": {Mode: uint32(os.ModeDir | 0o755)},
|
||||
"/src/sub": {Mode: uint32(os.ModeDir | 0o755)},
|
||||
"/src/a.txt": {Mode: 0o644},
|
||||
"/src/link": {Mode: uint32(os.ModeSymlink | 0o777), LinkTarget: "a.txt"},
|
||||
}
|
||||
|
||||
got := countNonDirectories(known)
|
||||
if got != 2 {
|
||||
t.Errorf("countNonDirectories = %d, want 2", got)
|
||||
}
|
||||
}
|
||||
@@ -8,7 +8,6 @@ import (
|
||||
"reflect"
|
||||
"sort"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
)
|
||||
@@ -20,17 +19,15 @@ import (
|
||||
// constructor, so the contract is defined once.
|
||||
//
|
||||
// It exercises the public Storer interface: round-trip, stat, list with
|
||||
// prefix filtering, the modification time ListStream reports, overwrite,
|
||||
// delete, delete-of-missing, and not-found on Get and Stat. Each section
|
||||
// takes its own fresh backend instance, so the order of sections never
|
||||
// matters and no section sees another's objects.
|
||||
// prefix filtering, overwrite, delete, delete-of-missing, and not-found on
|
||||
// Get and Stat. Each section takes its own fresh backend instance, so the
|
||||
// order of sections never matters and no section sees another's objects.
|
||||
func runStorerConformance(t *testing.T, newStorer func(*testing.T) storage.Storer) {
|
||||
t.Helper()
|
||||
|
||||
conformanceRoundTrip(t, newStorer(t))
|
||||
conformanceOverwrite(t, newStorer(t))
|
||||
conformanceList(t, newStorer(t))
|
||||
conformanceListStreamModTime(t, newStorer(t))
|
||||
conformanceDelete(t, newStorer(t))
|
||||
conformanceNotFound(t, newStorer(t))
|
||||
}
|
||||
@@ -121,35 +118,6 @@ func conformanceList(t *testing.T, s storage.Storer) {
|
||||
}
|
||||
}
|
||||
|
||||
// conformanceListStreamModTime checks that ListStream gives an object
|
||||
// just written a modification time of about now, which prune compares
|
||||
// with its grace period.
|
||||
func conformanceListStreamModTime(t *testing.T, s storage.Storer) {
|
||||
t.Helper()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
err := s.Put(ctx, "blobs/dd/written-now", bytes.NewReader([]byte("data")))
|
||||
if err != nil {
|
||||
t.Fatalf("Put: %v", err)
|
||||
}
|
||||
|
||||
var listed []storage.ObjectInfo
|
||||
|
||||
for info := range s.ListStream(ctx, "blobs/") {
|
||||
listed = append(listed, info)
|
||||
}
|
||||
|
||||
if len(listed) != 1 || listed[0].Err != nil {
|
||||
t.Fatalf("ListStream(\"blobs/\") = %v, want the one object", listed)
|
||||
}
|
||||
|
||||
if time.Since(listed[0].ModTime).Abs() > time.Minute {
|
||||
t.Errorf("ListStream ModTime = %v, want within a minute of now",
|
||||
listed[0].ModTime)
|
||||
}
|
||||
}
|
||||
|
||||
// conformanceDelete checks that Delete removes an object and that deleting
|
||||
// a missing key is not an error.
|
||||
func conformanceDelete(t *testing.T, s storage.Storer) {
|
||||
|
||||
@@ -235,9 +235,8 @@ func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan Objec
|
||||
// Normalize path separators
|
||||
relPath = strings.ReplaceAll(relPath, string(filepath.Separator), "/")
|
||||
ch <- ObjectInfo{
|
||||
Key: relPath,
|
||||
Size: info.Size(),
|
||||
ModTime: info.ModTime(),
|
||||
Key: relPath,
|
||||
Size: info.Size(),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -53,14 +53,6 @@ func NewRcloneStorer(ctx context.Context, remote, path string) (*RcloneStorer, e
|
||||
rclonePath += path
|
||||
}
|
||||
|
||||
// On S3, Swift and some other remotes rclone keeps a time of its own in
|
||||
// each object's metadata, which costs one request per object to read.
|
||||
// rclone's use_server_modtime setting makes ModTime the time the
|
||||
// listing gives instead: when the object was stored, which is what
|
||||
// prune compares with prune_grace_period.
|
||||
ctx, ci := fs.AddConfig(ctx)
|
||||
ci.UseServerModTime = true
|
||||
|
||||
// Create the rclone filesystem
|
||||
fsys, err := fs.NewFs(ctx, rclonePath)
|
||||
if err != nil {
|
||||
@@ -219,9 +211,8 @@ func (r *RcloneStorer) ListStream(
|
||||
|
||||
if prefix == "" || strings.HasPrefix(key, prefix) {
|
||||
ch <- ObjectInfo{
|
||||
Key: key,
|
||||
Size: obj.Size(),
|
||||
ModTime: obj.ModTime(ctx),
|
||||
Key: key,
|
||||
Size: obj.Size(),
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
@@ -5,17 +5,11 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/johannesboyne/gofakes3"
|
||||
"github.com/johannesboyne/gofakes3/backend/s3mem"
|
||||
"github.com/rclone/rclone/fs"
|
||||
"github.com/rclone/rclone/fs/config"
|
||||
"github.com/rclone/rclone/fs/config/configmap"
|
||||
@@ -435,72 +429,3 @@ func TestNewRcloneStorerReadsRcloneConfigEnv(t *testing.T) {
|
||||
t.Errorf("object is not in the remote's directory: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRcloneListStreamOnS3MakesNoRequestPerObject checks that listing an
|
||||
// rclone S3 remote takes each object's time from the listing itself. By
|
||||
// default rclone reads the time it keeps in each object's metadata, one
|
||||
// request per object, and every command that lists the destination store
|
||||
// would slow down as it grows.
|
||||
//
|
||||
//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
|
||||
func TestRcloneListStreamOnS3MakesNoRequestPerObject(t *testing.T) {
|
||||
backend := s3mem.New()
|
||||
|
||||
err := backend.CreateBucket(s3TestBucket)
|
||||
if err != nil {
|
||||
t.Fatalf("create bucket: %v", err)
|
||||
}
|
||||
|
||||
const body = "blob-bytes"
|
||||
|
||||
_, err = backend.PutObject(s3TestBucket, testBlobKey, nil,
|
||||
strings.NewReader(body), int64(len(body)))
|
||||
if err != nil {
|
||||
t.Fatalf("seed object: %v", err)
|
||||
}
|
||||
|
||||
var headRequests atomic.Int64
|
||||
|
||||
fakeS3 := gofakes3.New(backend).Server()
|
||||
srv := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method == http.MethodHead {
|
||||
headRequests.Add(1)
|
||||
}
|
||||
|
||||
fakeS3.ServeHTTP(w, r)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
remote := ":s3,provider=Other" +
|
||||
",access_key_id=" + s3TestAccessKeyID +
|
||||
",secret_access_key=" + s3TestSecretAccessKey +
|
||||
",endpoint='" + srv.URL + "'"
|
||||
ctx := context.Background()
|
||||
|
||||
s, err := storage.NewRcloneStorer(ctx, remote, s3TestBucket)
|
||||
if err != nil {
|
||||
t.Fatalf("NewRcloneStorer: %v", err)
|
||||
}
|
||||
|
||||
headRequests.Store(0)
|
||||
|
||||
var listed []storage.ObjectInfo
|
||||
|
||||
for info := range s.ListStream(ctx, "blobs/") {
|
||||
listed = append(listed, info)
|
||||
}
|
||||
|
||||
if len(listed) != 1 || listed[0].Err != nil {
|
||||
t.Fatalf("ListStream(\"blobs/\") = %v, want the one object", listed)
|
||||
}
|
||||
|
||||
if time.Since(listed[0].ModTime).Abs() > time.Minute {
|
||||
t.Errorf("ListStream ModTime = %v, want within a minute of now",
|
||||
listed[0].ModTime)
|
||||
}
|
||||
|
||||
if n := headRequests.Load(); n != 0 {
|
||||
t.Errorf("ListStream made %d HEAD request(s), want none", n)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -89,10 +89,9 @@ func (s *S3Storer) ListStream(ctx context.Context, prefix string) <-chan ObjectI
|
||||
|
||||
for info := range s.client.ListObjectsStream(ctx, prefix, false) {
|
||||
ch <- ObjectInfo{
|
||||
Key: info.Key,
|
||||
Size: info.Size,
|
||||
ModTime: info.ModTime,
|
||||
Err: info.Err,
|
||||
Key: info.Key,
|
||||
Size: info.Size,
|
||||
Err: info.Err,
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
@@ -14,7 +14,6 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrNotFound is returned when an object does not exist.
|
||||
@@ -28,10 +27,7 @@ type ProgressCallback func(bytesTransferred int64) error
|
||||
type ObjectInfo struct {
|
||||
Key string // Object key/path
|
||||
Size int64 // Size in bytes
|
||||
// ModTime is when the object was written, as the store reports it.
|
||||
// Only ListStream sets it; Stat leaves it zero.
|
||||
ModTime time.Time
|
||||
Err error // Error for streaming results (nil on success)
|
||||
Err error // Error for streaming results (nil on success)
|
||||
}
|
||||
|
||||
// Info provides human-readable storage configuration.
|
||||
@@ -71,8 +67,7 @@ type Storer interface {
|
||||
// For large result sets, prefer ListStream.
|
||||
List(ctx context.Context, prefix string) ([]string, error)
|
||||
|
||||
// ListStream returns a channel of ObjectInfo, with each object's key,
|
||||
// size and modification time, for large result sets.
|
||||
// ListStream returns a channel of ObjectInfo for large result sets.
|
||||
// The channel is closed when listing completes.
|
||||
// If an error occurs during listing, the final item will have Err set.
|
||||
ListStream(ctx context.Context, prefix string) <-chan ObjectInfo
|
||||
|
||||
@@ -457,9 +457,9 @@ func (v *Vaultik) scanRemoteBlobStorage(
|
||||
return err
|
||||
}
|
||||
|
||||
for _, blob := range allBlobs {
|
||||
for _, size := range allBlobs {
|
||||
result.TotalBlobCount++
|
||||
result.TotalBlobSize += blob.Size
|
||||
result.TotalBlobSize += size
|
||||
}
|
||||
|
||||
// A blob named only by a manifest that could not be read, or by one
|
||||
@@ -474,10 +474,10 @@ func (v *Vaultik) scanRemoteBlobStorage(
|
||||
orphanedSize int64
|
||||
)
|
||||
|
||||
for hash, blob := range allBlobs {
|
||||
for hash, size := range allBlobs {
|
||||
if _, referenced := referencedBlobs[hash]; !referenced {
|
||||
orphanedCount++
|
||||
orphanedSize += blob.Size
|
||||
orphanedSize += size
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+17
-59
@@ -5,10 +5,8 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
)
|
||||
|
||||
// PruneOptions contains options for the prune command
|
||||
@@ -165,10 +163,6 @@ type PruneBlobsResult struct {
|
||||
BlobsDeleted int `json:"blobs_deleted"`
|
||||
BlobsFailed int `json:"blobs_failed,omitempty"`
|
||||
BytesFreed int64 `json:"bytes_freed"`
|
||||
// BlobsTooNew counts the unreferenced blobs kept because they are
|
||||
// newer than the prune grace period, or of unknown age. BlobsFound
|
||||
// does not include them.
|
||||
BlobsTooNew int `json:"blobs_too_new"`
|
||||
}
|
||||
|
||||
// Prune removes orphaned data from the local index database AND
|
||||
@@ -214,15 +208,9 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
|
||||
return err
|
||||
}
|
||||
|
||||
unreferencedBlobs, totalSize, tooNew := v.findUnreferencedBlobs(
|
||||
allBlobs, allBlobsReferenced)
|
||||
unreferencedBlobs, totalSize := v.findUnreferencedBlobs(allBlobs, allBlobsReferenced)
|
||||
|
||||
result := &PruneBlobsResult{
|
||||
BlobsFound: len(unreferencedBlobs),
|
||||
BlobsTooNew: tooNew,
|
||||
}
|
||||
|
||||
v.reportBlobsTooNew(tooNew, opts)
|
||||
result := &PruneBlobsResult{BlobsFound: len(unreferencedBlobs)}
|
||||
|
||||
if len(unreferencedBlobs) == 0 {
|
||||
log.Info("No unreferenced blobs found")
|
||||
@@ -356,12 +344,11 @@ func (v *Vaultik) listUniqueSnapshotIDs() ([]string, error) {
|
||||
return snapshotIDs, nil
|
||||
}
|
||||
|
||||
// listAllRemoteBlobs returns a map of all blob hashes in remote storage
|
||||
// to their listing entries.
|
||||
func (v *Vaultik) listAllRemoteBlobs() (map[string]storage.ObjectInfo, error) {
|
||||
// listAllRemoteBlobs returns a map of all blob hashes to their sizes in remote storage
|
||||
func (v *Vaultik) listAllRemoteBlobs() (map[string]int64, error) {
|
||||
log.Info("Listing all blobs in storage")
|
||||
|
||||
allBlobs := make(map[string]storage.ObjectInfo)
|
||||
allBlobs := make(map[string]int64)
|
||||
blobObjectCh := v.Storage.ListStream(v.ctx, "blobs/")
|
||||
|
||||
for object := range blobObjectCh {
|
||||
@@ -385,7 +372,7 @@ func (v *Vaultik) listAllRemoteBlobs() (map[string]storage.ObjectInfo, error) {
|
||||
continue
|
||||
}
|
||||
|
||||
allBlobs[parts[3]] = object
|
||||
allBlobs[parts[3]] = object.Size
|
||||
}
|
||||
|
||||
log.Info("Found blobs in storage", "count", len(allBlobs))
|
||||
@@ -393,59 +380,30 @@ func (v *Vaultik) listAllRemoteBlobs() (map[string]storage.ObjectInfo, error) {
|
||||
return allBlobs, nil
|
||||
}
|
||||
|
||||
// findUnreferencedBlobs returns the hashes of the blobs no manifest
|
||||
// references that are older than the prune grace period, and their total
|
||||
// size. It also returns how many unreferenced blobs it left out for
|
||||
// being newer than that, or of unknown age because the listing gave no
|
||||
// time for them.
|
||||
// findUnreferencedBlobs returns blob hashes not referenced by any
|
||||
// manifest and their total size.
|
||||
func (v *Vaultik) findUnreferencedBlobs(
|
||||
allBlobs map[string]storage.ObjectInfo, referenced map[string]bool,
|
||||
) ([]string, int64, int) {
|
||||
allBlobs map[string]int64, referenced map[string]bool,
|
||||
) ([]string, int64) {
|
||||
var (
|
||||
unreferenced []string
|
||||
totalSize int64
|
||||
tooNew int
|
||||
)
|
||||
|
||||
cutoff := time.Now().UTC().Add(-v.Config.PruneGracePeriod)
|
||||
|
||||
for hash, blob := range allBlobs {
|
||||
if referenced[hash] {
|
||||
continue
|
||||
for hash, size := range allBlobs {
|
||||
if !referenced[hash] {
|
||||
unreferenced = append(unreferenced, hash)
|
||||
totalSize += size
|
||||
}
|
||||
|
||||
if blob.ModTime.IsZero() || blob.ModTime.After(cutoff) {
|
||||
tooNew++
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
unreferenced = append(unreferenced, hash)
|
||||
totalSize += blob.Size
|
||||
}
|
||||
|
||||
return unreferenced, totalSize, tooNew
|
||||
}
|
||||
|
||||
// reportBlobsTooNew says how many unreferenced blobs prune kept for being
|
||||
// newer than the grace period or of unknown age. Under --json the
|
||||
// document carries the count instead.
|
||||
func (v *Vaultik) reportBlobsTooNew(tooNew int, opts *PruneOptions) {
|
||||
if tooNew == 0 || opts.JSON {
|
||||
return
|
||||
}
|
||||
|
||||
v.UI.Infof("Kept %d unreferenced blob(s) newer than "+
|
||||
"prune_grace_period (%s), or of unknown age: a backup still "+
|
||||
"running may need them.",
|
||||
tooNew, v.Config.PruneGracePeriod)
|
||||
return unreferenced, totalSize
|
||||
}
|
||||
|
||||
// deleteUnreferencedBlobs deletes the given blobs from storage and
|
||||
// populates the result.
|
||||
func (v *Vaultik) deleteUnreferencedBlobs(
|
||||
unreferencedBlobs []string, allBlobs map[string]storage.ObjectInfo,
|
||||
result *PruneBlobsResult,
|
||||
unreferencedBlobs []string, allBlobs map[string]int64, result *PruneBlobsResult,
|
||||
) {
|
||||
log.Info("Deleting unreferenced blobs")
|
||||
|
||||
@@ -460,7 +418,7 @@ func (v *Vaultik) deleteUnreferencedBlobs(
|
||||
}
|
||||
|
||||
result.BlobsDeleted++
|
||||
result.BytesFreed += allBlobs[hash].Size
|
||||
result.BytesFreed += allBlobs[hash]
|
||||
|
||||
if (i+1)%progressLogEvery == 0 || i == len(unreferencedBlobs)-1 {
|
||||
log.Info("Deletion progress",
|
||||
|
||||
@@ -1,166 +0,0 @@
|
||||
package vaultik_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/config"
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
"sneak.berlin/go/vaultik/internal/storage/faultstore"
|
||||
"sneak.berlin/go/vaultik/internal/ui"
|
||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||
)
|
||||
|
||||
// TestPruneDuringBackupKeepsItsBlobs runs a prune from another host
|
||||
// sharing the destination store at the moment a backup has uploaded its
|
||||
// blobs but not yet its manifest, so no manifest lists those blobs. The
|
||||
// prune must leave them, and the backup must then restore.
|
||||
func TestPruneDuringBackupKeepsItsBlobs(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
dataDir := filepath.Join(tempDir, "src")
|
||||
storeDir := t.TempDir()
|
||||
restoreDir := filepath.Join(tempDir, "restored")
|
||||
dbPath := filepath.Join(tempDir, "index.sqlite")
|
||||
|
||||
ctx := context.Background()
|
||||
cfg := faultTestConfig()
|
||||
testFiles := writeFaultSourceTree(t, fs, dataDir)
|
||||
|
||||
inner, err := storage.NewFileStorer(storeDir)
|
||||
require.NoError(t, err)
|
||||
|
||||
otherHostCfg := faultTestConfig()
|
||||
otherHostCfg.PruneGracePeriod = time.Hour
|
||||
otherHost := newReaderVaultik(ctx, otherHostCfg, inner, nil, fs)
|
||||
|
||||
var (
|
||||
pruned bool
|
||||
pruneErr error
|
||||
)
|
||||
|
||||
store := faultstore.New(inner)
|
||||
store.OnPut = func(key string) faultstore.PutAction {
|
||||
if strings.HasSuffix(key, "/manifest.json.zst") {
|
||||
pruned = true
|
||||
pruneErr = otherHost.PruneBlobs(&vaultik.PruneOptions{Force: true})
|
||||
}
|
||||
|
||||
return faultstore.PutNormal
|
||||
}
|
||||
|
||||
db, err := database.New(ctx, dbPath)
|
||||
require.NoError(t, err)
|
||||
|
||||
repos := database.NewRepositories(db)
|
||||
|
||||
id := fullFaultBackup(ctx, t, fs, store, cfg, repos, dataDir, dbPath, "shared")
|
||||
require.NoError(t, db.Close())
|
||||
|
||||
require.True(t, pruned, "the prune must run before the manifest upload")
|
||||
require.NoError(t, pruneErr)
|
||||
|
||||
v := newReaderVaultik(ctx, cfg, inner, nil, fs)
|
||||
require.NoError(t, v.Restore(&vaultik.RestoreOptions{
|
||||
SnapshotID: id,
|
||||
TargetDir: restoreDir,
|
||||
Verify: true,
|
||||
}), "a prune during the backup must not delete the backup's blobs")
|
||||
|
||||
assertRestoredTree(t, fs, restoreDir, testFiles)
|
||||
}
|
||||
|
||||
// TestPruneKeepsUnreferencedBlobWithNoTime checks that a blob no
|
||||
// manifest lists, whose listing gives no modification time (as an S3
|
||||
// listing entry without LastModified does), is kept and counted with
|
||||
// the blobs too new to delete: its age is unknown.
|
||||
func TestPruneKeepsUnreferencedBlobWithNoTime(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newListEnv(t)
|
||||
|
||||
// Written into the mock store's map directly rather than through
|
||||
// Put, so its listing gives no time.
|
||||
blobKey := "blobs/" + testBlobHashA[:2] + "/" + testBlobHashA[2:4] +
|
||||
"/" + testBlobHashA
|
||||
env.store.data[blobKey] = []byte("blob")
|
||||
|
||||
require.NoError(t, env.v.PruneBlobs(&vaultik.PruneOptions{JSON: true}))
|
||||
|
||||
assert.True(t, env.store.hasKey(blobKey),
|
||||
"a blob of unknown age must be kept")
|
||||
|
||||
var result vaultik.PruneBlobsResult
|
||||
|
||||
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &result))
|
||||
assert.Equal(t, 0, result.BlobsDeleted)
|
||||
assert.Equal(t, 1, result.BlobsTooNew)
|
||||
}
|
||||
|
||||
// TestPruneDeletesUnreferencedBlobOlderThanGracePeriod checks that the
|
||||
// grace period only delays deletion: a blob no manifest lists is deleted
|
||||
// once it is older than the grace period, and kept and counted until
|
||||
// then.
|
||||
func TestPruneDeletesUnreferencedBlobOlderThanGracePeriod(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
storeDir := t.TempDir()
|
||||
|
||||
store, err := storage.NewFileStorer(storeDir)
|
||||
require.NoError(t, err)
|
||||
|
||||
oldKey := "blobs/" + testBlobHashA[:2] + "/" + testBlobHashA[2:4] +
|
||||
"/" + testBlobHashA
|
||||
newKey := "blobs/" + testBlobHashB[:2] + "/" + testBlobHashB[2:4] +
|
||||
"/" + testBlobHashB
|
||||
|
||||
for _, key := range []string{oldKey, newKey} {
|
||||
require.NoError(t, store.Put(ctx, key, strings.NewReader("blob")))
|
||||
}
|
||||
|
||||
gracePeriod := time.Hour
|
||||
beforeGracePeriod := time.Now().Add(-gracePeriod - time.Minute)
|
||||
require.NoError(t, os.Chtimes(filepath.Join(storeDir, oldKey),
|
||||
beforeGracePeriod, beforeGracePeriod))
|
||||
|
||||
stdout := &bytes.Buffer{}
|
||||
v := &vaultik.Vaultik{
|
||||
Config: &config.Config{PruneGracePeriod: gracePeriod},
|
||||
Storage: store,
|
||||
Stdout: stdout,
|
||||
UI: ui.NewWithColor(stdout, false),
|
||||
}
|
||||
v.SetContext(ctx)
|
||||
|
||||
require.NoError(t, v.PruneBlobs(&vaultik.PruneOptions{JSON: true}))
|
||||
|
||||
_, err = store.Stat(ctx, oldKey)
|
||||
require.ErrorIs(t, err, storage.ErrNotFound,
|
||||
"a blob older than the grace period must be deleted")
|
||||
|
||||
_, err = store.Stat(ctx, newKey)
|
||||
require.NoError(t, err, "a blob newer than the grace period must be kept")
|
||||
|
||||
var result vaultik.PruneBlobsResult
|
||||
|
||||
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result))
|
||||
assert.Equal(t, 1, result.BlobsDeleted)
|
||||
assert.Equal(t, 1, result.BlobsTooNew)
|
||||
}
|
||||
@@ -26,14 +26,11 @@ import (
|
||||
type testStorer struct {
|
||||
mu sync.Mutex
|
||||
data map[string][]byte
|
||||
// putTimes holds when Put stored each object; ListStream reports it.
|
||||
putTimes map[string]time.Time
|
||||
}
|
||||
|
||||
func newTestStorer() *testStorer {
|
||||
return &testStorer{
|
||||
data: make(map[string][]byte),
|
||||
putTimes: make(map[string]time.Time),
|
||||
data: make(map[string][]byte),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -47,7 +44,6 @@ func (s *testStorer) Put(_ context.Context, key string, reader io.Reader) error
|
||||
}
|
||||
|
||||
s.data[key] = data
|
||||
s.putTimes[key] = time.Now().UTC()
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -91,7 +87,6 @@ func (s *testStorer) Delete(_ context.Context, key string) error {
|
||||
defer s.mu.Unlock()
|
||||
|
||||
delete(s.data, key)
|
||||
delete(s.putTimes, key)
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -125,9 +120,8 @@ func (s *testStorer) ListStream(
|
||||
for key, data := range s.data {
|
||||
if prefix == "" || strings.HasPrefix(key, prefix) {
|
||||
ch <- storage.ObjectInfo{
|
||||
Key: key,
|
||||
Size: int64(len(data)),
|
||||
ModTime: s.putTimes[key],
|
||||
Key: key,
|
||||
Size: int64(len(data)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -39,11 +39,6 @@ var (
|
||||
// caller can report it apart from a failed backup.
|
||||
var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
|
||||
|
||||
// ErrFilesUnreadable is returned when --skip-errors skipped files or
|
||||
// directories it could not read or list. The snapshot is complete and
|
||||
// kept, so the caller can report it apart from a failed backup.
|
||||
var ErrFilesUnreadable = errors.New("file(s) could not be read")
|
||||
|
||||
// listRecentLimit caps how many snapshot rows are fetched from the
|
||||
// local index for listing and purge operations.
|
||||
const listRecentLimit = 10000
|
||||
@@ -135,16 +130,12 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
||||
}
|
||||
|
||||
// Process each named snapshot
|
||||
filesUnreadable := 0
|
||||
|
||||
for snapIdx, snapName := range snapshotNames {
|
||||
unreadable, err := v.createNamedSnapshot(
|
||||
err = v.createNamedSnapshot(
|
||||
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
filesUnreadable += unreadable
|
||||
}
|
||||
|
||||
// Print overall summary if multiple snapshots
|
||||
@@ -153,34 +144,13 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
||||
len(snapshotNames), v.UI.Duration(time.Since(overallStartTime)))
|
||||
}
|
||||
|
||||
switch {
|
||||
case opts.Prune && filesUnreadable > 0:
|
||||
// An older snapshot may be the only one that still holds a file
|
||||
// this run could not read.
|
||||
log.Warn("Skipping post-backup prune",
|
||||
"files_unreadable", filesUnreadable)
|
||||
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
|
||||
"could not be read; older snapshots are kept.", filesUnreadable)
|
||||
case opts.Prune:
|
||||
if opts.Prune {
|
||||
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: %w", ErrPostBackupPruneFailed, err)
|
||||
}
|
||||
}
|
||||
|
||||
return v.finishCreateSnapshot(filesUnreadable)
|
||||
}
|
||||
|
||||
// finishCreateSnapshot ends a run whose snapshots are all complete and
|
||||
// kept. Like finishRestore, it returns an error when --skip-errors
|
||||
// skipped anything it could not read or list; otherwise it prints the
|
||||
// closing line. A file removed during the run is counted as failed in
|
||||
// the summary but is not an error, with or without --skip-errors.
|
||||
func (v *Vaultik) finishCreateSnapshot(filesUnreadable int) error {
|
||||
if filesUnreadable > 0 {
|
||||
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
|
||||
}
|
||||
|
||||
// Terminus must obey the --cron invariant: silent on total
|
||||
// success only. UI.Complete is dropped in cron/quiet mode (that's
|
||||
// the success path), but if any warnings fired during the run we
|
||||
@@ -270,22 +240,16 @@ type snapshotStats struct {
|
||||
totalBlobsUploaded int
|
||||
uploadDuration time.Duration
|
||||
|
||||
// The part of totalFilesFailed that --skip-errors skipped because it
|
||||
// could not be read or listed.
|
||||
totalFilesUnreadable int
|
||||
|
||||
// The sizes of all blobs the snapshot references, set by
|
||||
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
||||
blobSize int64
|
||||
blobUncompressedSize int64
|
||||
}
|
||||
|
||||
// createNamedSnapshot creates a single named snapshot and returns the
|
||||
// number of entries --skip-errors skipped because it could not read or
|
||||
// list them.
|
||||
// createNamedSnapshot creates a single named snapshot
|
||||
func (v *Vaultik) createNamedSnapshot(
|
||||
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
|
||||
) (int, error) {
|
||||
) error {
|
||||
snapshotStartTime := time.Now()
|
||||
|
||||
if total > 1 {
|
||||
@@ -294,12 +258,12 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
previousSnapshotID, err := v.previousSnapshotID(snapName)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
scanner := v.ScannerFactory(snapshot.ScannerParams{
|
||||
@@ -314,7 +278,7 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
|
||||
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("creating snapshot: %w", err)
|
||||
return fmt.Errorf("creating snapshot: %w", err)
|
||||
}
|
||||
|
||||
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
|
||||
@@ -322,12 +286,12 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
err = v.finalizeSnapshotMetadata(snapshotID, stats)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
|
||||
log.Info("Snapshot complete",
|
||||
@@ -340,7 +304,7 @@ func (v *Vaultik) createNamedSnapshot(
|
||||
|
||||
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
||||
|
||||
return stats.totalFilesUnreadable, nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// previousSnapshotID returns the ID of the most recent completed snapshot
|
||||
@@ -426,7 +390,6 @@ func (v *Vaultik) scanAllDirectories(
|
||||
stats.totalBlobs += result.BlobsCreated
|
||||
stats.totalFilesSkipped += result.FilesSkipped
|
||||
stats.totalFilesFailed += result.FilesFailed
|
||||
stats.totalFilesUnreadable += result.FilesUnreadable
|
||||
stats.totalBytesSkipped += result.BytesSkipped
|
||||
stats.totalFilesDeleted += result.FilesDeleted
|
||||
stats.totalBytesDeleted += result.BytesDeleted
|
||||
|
||||
@@ -178,13 +178,11 @@ func TestBackupDoesNotCountFileInUnopenableDirectoryAsDeleted(t *testing.T) {
|
||||
}
|
||||
|
||||
env.out.Reset()
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"work"},
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
||||
}))
|
||||
assert.Contains(t, env.out.String(), filesLine(1, 0, 1))
|
||||
}
|
||||
|
||||
// A backup counts only files that the previous snapshot of its own name
|
||||
|
||||
@@ -49,19 +49,13 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
||||
name string
|
||||
openErr error
|
||||
skipErrors bool
|
||||
wantErr error // nil for none
|
||||
}{
|
||||
// What a normal user gets opening a file with mode 000.
|
||||
{name: "unopenable under skip-errors",
|
||||
openErr: os.ErrPermission, skipErrors: true,
|
||||
wantErr: vaultik.ErrFilesUnreadable},
|
||||
openErr: os.ErrPermission, skipErrors: true},
|
||||
// What opening a file removed after phase 1 gives.
|
||||
{name: "removed between the phases",
|
||||
openErr: os.ErrNotExist, skipErrors: false},
|
||||
// A removal is not an error, so --skip-errors does not make the
|
||||
// run fail. See https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
{name: "removed between the phases under skip-errors",
|
||||
openErr: os.ErrNotExist, skipErrors: true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -105,11 +99,10 @@ func TestSnapshotSummaryCountsFileNotStoredAsFailed(t *testing.T) {
|
||||
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
|
||||
v.UI = ui.NewWithColor(out, false)
|
||||
|
||||
err = v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
require.NoError(t, v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: tt.skipErrors,
|
||||
Snapshots: []string{"src"},
|
||||
})
|
||||
require.ErrorIs(t, err, tt.wantErr)
|
||||
}))
|
||||
|
||||
summary := out.String()
|
||||
assert.Contains(t, summary,
|
||||
|
||||
@@ -3,7 +3,6 @@ package vaultik_test
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
@@ -74,210 +73,3 @@ func TestSkipErrorsBackupStopsOnIndexErrorRecordingDirectory(t *testing.T) {
|
||||
require.Len(t, snapshots, 1)
|
||||
assert.Nil(t, snapshots[0].CompletedAt)
|
||||
}
|
||||
|
||||
// A backup under --skip-errors that skips a file it cannot open, or a
|
||||
// directory it cannot list, keeps the snapshot, counts what it skipped
|
||||
// as failed and returns an error instead of finishing successfully. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestSkipErrorsBackupKeepsSnapshotAndFailsWhenItSkips(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
skipped string // relative to the source directory
|
||||
}{
|
||||
{name: "file it cannot open", skipped: "skipdir/file.txt"},
|
||||
{name: "directory it cannot list", skipped: "skipdir"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stored.txt": []byte("stored"),
|
||||
"skipdir/file.txt": []byte("in skipdir"),
|
||||
})
|
||||
env.setSnapshot("work", "")
|
||||
|
||||
// A mode does not lock out root, who runs the tests, so
|
||||
// opening the skipped path is made to fail instead.
|
||||
env.v.Fs = &openFailFs{
|
||||
path: filepath.Join(env.src, tt.skipped),
|
||||
err: os.ErrPermission,
|
||||
}
|
||||
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"work"},
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 1 backed up, 0 unchanged, 1 failed.")
|
||||
assert.NotContains(t, env.out.String(), "Finished successfully.")
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, snapshots, 1)
|
||||
assert.NotNil(t, snapshots[0].CompletedAt, "the snapshot is kept")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// removeOnLookupFs is the real filesystem, except that path is removed
|
||||
// just before it is looked up. The walk looks an entry up after listing
|
||||
// its directory, so path is listed but gone when the walk reaches it.
|
||||
type removeOnLookupFs struct {
|
||||
afero.OsFs
|
||||
|
||||
path string
|
||||
}
|
||||
|
||||
func (f *removeOnLookupFs) LstatIfPossible(
|
||||
name string,
|
||||
) (os.FileInfo, bool, error) {
|
||||
if name == f.path {
|
||||
_ = os.Remove(name)
|
||||
}
|
||||
|
||||
return f.OsFs.LstatIfPossible(name)
|
||||
}
|
||||
|
||||
// A file removed before the walk reaches it is counted once, as failed,
|
||||
// like a file removed before its data is read, and the backup finishes
|
||||
// successfully, with or without --skip-errors. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestBackupCountsFileRemovedBeforeWalkReachesItAsFailed(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
skipErrors bool
|
||||
}{
|
||||
{name: "under skip-errors", skipErrors: true},
|
||||
{name: "without skip-errors", skipErrors: false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stays.txt": []byte("stays"),
|
||||
"removed.txt": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("walk", "")
|
||||
|
||||
// The previous snapshot has the file, so a removal the walk
|
||||
// also counted as deleted would show.
|
||||
env.backUp(t, "walk")
|
||||
|
||||
env.v.Fs = &removeOnLookupFs{
|
||||
path: filepath.Join(env.src, "removed.txt"),
|
||||
}
|
||||
|
||||
env.out.Reset()
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: tt.skipErrors,
|
||||
Snapshots: []string{"walk"},
|
||||
}))
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Files: 2 examined, 0 backed up, 1 unchanged, 1 failed.")
|
||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A file an exclude pattern matches is not counted when it is removed
|
||||
// before the walk reaches it. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/310.
|
||||
func TestBackupIgnoresExcludedFileRemovedBeforeWalkReachesIt(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"stays.txt": []byte("stays"),
|
||||
"removed.tmp": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("walk", "", "*.tmp")
|
||||
env.v.Fs = &removeOnLookupFs{
|
||||
path: filepath.Join(env.src, "removed.tmp"),
|
||||
}
|
||||
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"walk"},
|
||||
}))
|
||||
assert.Contains(t, env.out.String(), filesLine(1, 1, 0))
|
||||
assert.Contains(t, env.out.String(), "Finished successfully.")
|
||||
}
|
||||
|
||||
// A backup under --skip-errors --prune that skipped a file it could not
|
||||
// read does not prune: the previous snapshot, the only one that holds
|
||||
// the file, is kept, and the output says the prune was skipped. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
||||
func TestSkipErrorsBackupDoesNotPruneWhenItSkips(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"other.txt": []byte("other"),
|
||||
"skipped.txt": []byte("skipped"),
|
||||
})
|
||||
env.setSnapshot("photos", "")
|
||||
env.backUp(t, "photos")
|
||||
|
||||
// An unchanged file is not read again, so the file is changed before
|
||||
// it becomes unreadable.
|
||||
skipped := filepath.Join(env.src, "skipped.txt")
|
||||
require.NoError(t, os.WriteFile(skipped, []byte("changed"), 0o600))
|
||||
env.v.Fs = &openFailFs{path: skipped, err: os.ErrPermission}
|
||||
|
||||
env.out.Reset()
|
||||
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
Prune: true,
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"photos"},
|
||||
})
|
||||
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||
assert.Contains(t, env.out.String(),
|
||||
"Skipped the post-backup prune because 1 file(s) could not be read")
|
||||
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 2, "the previous snapshot is kept")
|
||||
}
|
||||
|
||||
// A file removed during a backup under --skip-errors --prune is not a
|
||||
// skipped file, so the backup prunes as usual. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/382.
|
||||
func TestSkipErrorsBackupPrunesWhenAFileIsRemovedDuringIt(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newDeletedCountEnv(t, map[string][]byte{
|
||||
"other.txt": []byte("other"),
|
||||
"removed.txt": []byte("removed"),
|
||||
})
|
||||
env.setSnapshot("photos", "")
|
||||
env.backUp(t, "photos")
|
||||
|
||||
env.v.Fs = &removeOnLookupFs{path: filepath.Join(env.src, "removed.txt")}
|
||||
|
||||
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||
Prune: true,
|
||||
SkipErrors: true,
|
||||
Snapshots: []string{"photos"},
|
||||
}))
|
||||
|
||||
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||
context.Background(), listRecentTestLimit)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 1, "the previous snapshot is pruned")
|
||||
}
|
||||
|
||||
@@ -13,7 +13,6 @@ import (
|
||||
"github.com/spf13/afero"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"golang.org/x/sys/unix"
|
||||
"sneak.berlin/go/vaultik/internal/config"
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
@@ -189,42 +188,6 @@ func (e *summaryEnv) dataLine(total, backedUp int64) string {
|
||||
e.v.UI.Size(total), e.v.UI.Size(backedUp))
|
||||
}
|
||||
|
||||
// enumeratedLine returns the line printed when the walk of a source path
|
||||
// ends.
|
||||
func (e *summaryEnv) enumeratedLine(
|
||||
files int, total int64, toBackUp int, toBackUpSize int64,
|
||||
) string {
|
||||
return fmt.Sprintf(
|
||||
"Enumerated %d snapshot source files (%s total), %d to back up (%s).",
|
||||
files, e.v.UI.Size(total), toBackUp, e.v.UI.Size(toBackUpSize))
|
||||
}
|
||||
|
||||
// snapshotPaths returns the paths the snapshot lists, sorted.
|
||||
func (e *summaryEnv) snapshotPaths(t *testing.T, snapshotID string) []string {
|
||||
t.Helper()
|
||||
|
||||
rows, err := e.db.Conn().QueryContext(context.Background(), `
|
||||
SELECT f.path FROM snapshot_files sf JOIN files f ON f.id = sf.file_id
|
||||
WHERE sf.snapshot_id = ? ORDER BY f.path`, snapshotID)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var paths []string
|
||||
|
||||
for rows.Next() {
|
||||
var path string
|
||||
|
||||
require.NoError(t, rows.Scan(&path))
|
||||
|
||||
paths = append(paths, path)
|
||||
}
|
||||
|
||||
require.NoError(t, rows.Err())
|
||||
|
||||
return paths
|
||||
}
|
||||
|
||||
// A first backup stores copy.bin's chunks while backing up a.bin, so
|
||||
// copy.bin's chunks are deduplicated within the run. Each file and byte
|
||||
// is still counted once.
|
||||
@@ -285,85 +248,6 @@ func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
|
||||
assert.Equal(t, uncompressed, snap.BlobUncompressedSize)
|
||||
}
|
||||
|
||||
// A backup of a tree with directories and a symlink, none of which
|
||||
// changed since the last backup, backs up nothing, and the snapshot
|
||||
// still lists every one of them. Both sides of the line that ends the
|
||||
// walk count the same entries, the files and the symlink, and no
|
||||
// directory's size. See https://git.eeqj.de/sneak/vaultik/issues/305.
|
||||
func TestSnapshotSummaryUnchangedRunWithDirectoriesAndSymlink(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newSummaryEnv(t)
|
||||
|
||||
const linkTarget = "a.bin"
|
||||
|
||||
require.NoError(t, os.Symlink(linkTarget,
|
||||
filepath.Join(filepath.Dir(env.aPath), "link")))
|
||||
|
||||
// A symlink's size is the length of its target.
|
||||
total := env.totalSize + int64(len(linkTarget))
|
||||
|
||||
first := env.backUp(t, "first", false)
|
||||
assert.Contains(t, first, env.enumeratedLine(4, total, 4, total))
|
||||
assert.Contains(t, first, filesLine(4, 4, 0))
|
||||
|
||||
second := env.backUp(t, "second", false)
|
||||
assert.Contains(t, second, env.enumeratedLine(4, total, 0, 0))
|
||||
assert.Contains(t, second, filesLine(4, 0, 4))
|
||||
assert.Contains(t, second, env.dataLine(total, 0))
|
||||
|
||||
assert.Equal(t,
|
||||
env.snapshotPaths(t, env.snapshot(t, "first").ID.String()),
|
||||
env.snapshotPaths(t, env.snapshot(t, "second").ID.String()))
|
||||
}
|
||||
|
||||
// A directory whose mtime changed and a symlink whose target changed are
|
||||
// recorded again by the next backup, which counts only the symlink as
|
||||
// backed up.
|
||||
func TestSnapshotSummaryChangedDirectoryAndSymlink(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newSummaryEnv(t)
|
||||
|
||||
// The local index keeps paths with symlinks resolved.
|
||||
dir, err := filepath.EvalSymlinks(filepath.Dir(env.aPath))
|
||||
require.NoError(t, err)
|
||||
|
||||
// The new link gets the old one's mtime, and a target, which need not
|
||||
// exist, of the old target's length, a symlink's size: only the
|
||||
// target differs.
|
||||
linkTime := unix.NsecToTimeval(
|
||||
time.Date(2029, time.March, 4, 5, 6, 7, 0, time.UTC).UnixNano())
|
||||
link := filepath.Join(dir, "link")
|
||||
require.NoError(t, os.Symlink("a.bin", link))
|
||||
require.NoError(t, unix.Lutimes(link, []unix.Timeval{linkTime, linkTime}))
|
||||
|
||||
env.backUp(t, "first", false)
|
||||
|
||||
require.NoError(t, os.Remove(link))
|
||||
require.NoError(t, os.Symlink("b.bin", link))
|
||||
require.NoError(t, unix.Lutimes(link, []unix.Timeval{linkTime, linkTime}))
|
||||
|
||||
mtime := time.Date(2030, time.January, 2, 3, 4, 5, 0, time.UTC)
|
||||
require.NoError(t, os.Chtimes(dir, mtime, mtime))
|
||||
|
||||
assert.Contains(t, env.backUp(t, "second", false), filesLine(4, 1, 3))
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
dirRow, err := env.repos.Files.GetByPath(ctx, dir)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, dirRow)
|
||||
assert.Equal(t, mtime.Unix(), dirRow.MTime.Unix())
|
||||
|
||||
linkRow, err := env.repos.Files.GetByPath(ctx, link)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, linkRow)
|
||||
assert.Equal(t, "b.bin", linkRow.LinkTarget.String())
|
||||
}
|
||||
|
||||
// Under --cron the progress reporter is off; the upload figures must
|
||||
// still reach the summary and the snapshots row. The snapshot has two
|
||||
// paths, each backed up by its own scan.
|
||||
|
||||
Reference in New Issue
Block a user