quak backup --verify re-hashes stored originals and downloads again any that do not match (closes #168)
check / check (push) Failing after 1m22s
check / check (push) Failing after 1m22s
`--verify`, or `lib.backup({ verify: true })`, hashes each original already
at its save path as the download check does, streamed, a live photo as
`<imageHash>:<videoHash>`. A mismatch is logged, removed and put back in the
same run, and what is put back is hashed too, since a copy from the content
cache is not checked; one that still does not match, or a failed fetch, goes
into `failures.json`. A file with no recorded hash counts as unchecked. The
result, the summary and `--json` gain `verified`, `mismatched` and
`unchecked`.
Judgement call: a stored original that cannot be read for hashing is recorded as failed and left in place.
Judgement call: a copy put back that still does not match stays at its save path and is not counted as downloaded.
Judgement call: the summary prints the three counts only with `--verify`.
Model: opus-5-5
This commit was merged in pull request #179.
This commit is contained in:
@@ -549,7 +549,7 @@ quak collections [--json] list all collections
|
||||
quak files --collection <id> [--json] list files in a collection
|
||||
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
||||
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
||||
quak backup <dir> [--json] full incremental backup
|
||||
quak backup <dir> [--json] [--verify] full incremental backup
|
||||
quak backup-metadata <dir> [--exif] dump the metadata quak keeps as JSON
|
||||
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
|
||||
quak helper fix-missing-thumbnails [--file ids] [--json] generate + upload missing thumbnails
|
||||
@@ -582,6 +582,10 @@ reads no tag from is recorded, base64, as `exifRaw`, with the reason in
|
||||
`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails`
|
||||
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
|
||||
|
||||
`backup --verify` also hashes the originals already in the backup and replaces
|
||||
any that do not match the content hash Ente records; one it cannot replace goes
|
||||
into `failures.json` (see "Backup layout").
|
||||
|
||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||
fails, the error is logged, each of its files is written with the reason in an
|
||||
`mlDataError` field instead of `mlData`, and the dump goes on. The exit code is
|
||||
@@ -715,6 +719,25 @@ if any files failed. `quak backup` opens its library with the thumbnail and
|
||||
originals precache off, so the only file content it fetches is the originals the
|
||||
backup stores.
|
||||
|
||||
With `--verify`, or `lib.backup({ verify: true })`, a run also hashes each
|
||||
original already at its save path the way a download is checked (see "On-disk
|
||||
cache layout" below): its bytes, read in chunks, or a live photo's image and
|
||||
video, joined as `<imageHash>:<videoHash>`. An original that matches the content
|
||||
hash its metadata records is left as it is. One that does not is logged on one
|
||||
line naming the file, deleted (a live photo's image and video both), and put
|
||||
back in the same run like a missing one: downloaded, or copied from the cache if
|
||||
the cache holds it. What is put back is hashed too, because a copy from the
|
||||
cache is not checked as a download is. If it still does not match, it stays at
|
||||
its save path and the file goes into `failures.json`, as it does when the
|
||||
download fails. A file whose metadata records no hash is left as it is and
|
||||
counted as unchecked. A stored original that cannot be read is left as it is and
|
||||
counts as failed. The summary and `--json` add the counts `verified`,
|
||||
`mismatched` and `unchecked`, all of originals that were already stored; one
|
||||
first downloaded in this run is in none of them. A mismatch that was put back
|
||||
with matching bytes does not make the exit code non-zero. Without `--verify`
|
||||
nothing is hashed, the summary is unchanged, and the three counts are 0 in
|
||||
`--json`.
|
||||
|
||||
Each original is written to a temporary file in the same directory, synced to
|
||||
disk, and renamed into place, so an original is either complete or absent, even
|
||||
after a power cut. A downloaded original's temporary file is named
|
||||
@@ -945,8 +968,8 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
then counts it as present; one the cache already held is copied from there.
|
||||
`BackupOptions`: `downloadDirectory` (falls back to the library's),
|
||||
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
|
||||
`onlyAlbumNames`, `onProgress`, and `lockHeld` (default `false`). See Backup
|
||||
layout above for the tree it writes.
|
||||
`onlyAlbumNames`, `verify` (default `false`), `onProgress`, and `lockHeld`
|
||||
(default `false`). See Backup layout above for the tree it writes.
|
||||
|
||||
### Request pools
|
||||
|
||||
|
||||
Reference in New Issue
Block a user