quak backup --verify re-hashes stored originals and downloads again any that do not match (closes #168)
check / check (push) Failing after 1m22s

`--verify`, or `lib.backup({ verify: true })`, hashes each original already
at its save path as the download check does, streamed, a live photo as
`<imageHash>:<videoHash>`. A mismatch is logged, removed and put back in the
same run, and what is put back is hashed too, since a copy from the content
cache is not checked; one that still does not match, or a failed fetch, goes
into `failures.json`. A file with no recorded hash counts as unchecked. The
result, the summary and `--json` gain `verified`, `mismatched` and
`unchecked`.

Judgement call: a stored original that cannot be read for hashing is recorded as failed and left in place.
Judgement call: a copy put back that still does not match stays at its save path and is not counted as downloaded.
Judgement call: the summary prints the three counts only with `--verify`.

Model: opus-5-5
This commit was merged in pull request #179.
This commit is contained in:
2026-10-06 22:47:26 +02:00
parent bd77422965
commit ddf58af3cc
8 changed files with 445 additions and 11 deletions
+26 -3
View File
@@ -549,7 +549,7 @@ quak collections [--json] list all collections
quak files --collection <id> [--json] list files in a collection
quak get <fileID> [--out path] [--collection] download and decrypt a file
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
quak backup <dir> [--json] full incremental backup
quak backup <dir> [--json] [--verify] full incremental backup
quak backup-metadata <dir> [--exif] dump the metadata quak keeps as JSON
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
quak helper fix-missing-thumbnails [--file ids] [--json] generate + upload missing thumbnails
@@ -582,6 +582,10 @@ reads no tag from is recorded, base64, as `exifRaw`, with the reason in
`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails`
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
`backup --verify` also hashes the originals already in the backup and replaces
any that do not match the content hash Ente records; one it cannot replace goes
into `failures.json` (see "Backup layout").
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
fails, the error is logged, each of its files is written with the reason in an
`mlDataError` field instead of `mlData`, and the dump goes on. The exit code is
@@ -715,6 +719,25 @@ if any files failed. `quak backup` opens its library with the thumbnail and
originals precache off, so the only file content it fetches is the originals the
backup stores.
With `--verify`, or `lib.backup({ verify: true })`, a run also hashes each
original already at its save path the way a download is checked (see "On-disk
cache layout" below): its bytes, read in chunks, or a live photo's image and
video, joined as `<imageHash>:<videoHash>`. An original that matches the content
hash its metadata records is left as it is. One that does not is logged on one
line naming the file, deleted (a live photo's image and video both), and put
back in the same run like a missing one: downloaded, or copied from the cache if
the cache holds it. What is put back is hashed too, because a copy from the
cache is not checked as a download is. If it still does not match, it stays at
its save path and the file goes into `failures.json`, as it does when the
download fails. A file whose metadata records no hash is left as it is and
counted as unchecked. A stored original that cannot be read is left as it is and
counts as failed. The summary and `--json` add the counts `verified`,
`mismatched` and `unchecked`, all of originals that were already stored; one
first downloaded in this run is in none of them. A mismatch that was put back
with matching bytes does not make the exit code non-zero. Without `--verify`
nothing is hashed, the summary is unchanged, and the three counts are 0 in
`--json`.
Each original is written to a temporary file in the same directory, synced to
disk, and renamed into place, so an original is either complete or absent, even
after a power cut. A downloaded original's temporary file is named
@@ -945,8 +968,8 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
then counts it as present; one the cache already held is copied from there.
`BackupOptions`: `downloadDirectory` (falls back to the library's),
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
`onlyAlbumNames`, `onProgress`, and `lockHeld` (default `false`). See Backup
layout above for the tree it writes.
`onlyAlbumNames`, `verify` (default `false`), `onProgress`, and `lockHeld`
(default `false`). See Backup layout above for the tree it writes.
### Request pools