--verify keeps a mismatched original until a matching copy replaces it (closes #199)
check / check (push) Canceled after 0s

With --verify, an original whose hash does not match is no longer removed
before its replacement is downloaded. replaceOriginal, beside placeOriginal
in src/library/content.ts, has the content cache's new fetchOriginal write
the replacement to a temporary file beside the original, hashes it, and
renames it over the original only when it matches; a live photo's image and
video are both hashed before either is renamed. A failed or mismatching
download leaves the old bytes and album links, removes the temporary files,
and lands in failures.json. The replacement is always downloaded, never
copied from the cache. The leftover-file cleanup also takes a live photo's
replacement files, .quak-<pid>-<random>.<ext>.

Model: opus-5-5
This commit is contained in:
2026-10-16 03:38:17 +00:00
parent 960a7ce8a6
commit 820a22f8cb
7 changed files with 288 additions and 115 deletions
+34 -28
View File
@@ -159,10 +159,11 @@ After a `1`, the next run fetches each missing original again, fetches the ML
data that is not cached, and rebuilds the album links. An album quak could not
decrypt, or a file in `failures.json` because quak could not decrypt it, keeps
the exit code at `1` until the server sends a copy that decrypts, or deletes it
(see "Backup layout"). An original that a `--verify` run could not read, or put
back with bytes that still do not match, stays at its save path. Only a later
`--verify` run checks it again: a run without `--verify` leaves it as it is and
takes the file out of `failures.json`, so that run can exit 0.
(see "Backup layout"). An original that a `--verify` run could not read, or
could not replace with a download that matches, stays at its save path as it
was. Only a later `--verify` run checks it again: a run without `--verify`
leaves it as it is and takes the file out of `failures.json`, so that run can
exit 0.
## Examples
@@ -692,8 +693,8 @@ reads no tag from is recorded, base64, as `exifRaw`, with the reason in
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
`backup --verify` also hashes the originals already in the backup and replaces
any that do not match the content hash Ente records; one it cannot replace goes
into `failures.json` (see "Backup layout").
any that do not match the content hash Ente records; one it cannot replace stays
as it was and goes into `failures.json` (see "Backup layout").
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
fails, the error is logged, each of its files is written with the reason in an
@@ -878,33 +879,38 @@ original already at its save path the way a download is checked (see "On-disk
cache layout" below): its bytes, read in chunks, or a live photo's image and
video, joined as `<imageHash>:<videoHash>`. An original that matches the content
hash its metadata records is left as it is. One that does not is logged on one
line naming the file, deleted (a live photo's image and video both), and put
back in the same run like a missing one: downloaded, or copied from the cache if
the cache holds it. What is put back is hashed too, because a copy from the
cache is not checked as a download is. If it still does not match, it stays at
its save path and the file goes into `failures.json`, as it does when the
download fails. A file whose metadata records no hash is left as it is and
counted as unchecked. A stored original that cannot be read is left as it is and
counts as failed. The summary and `--json` add the counts `verified`,
`mismatched` and `unchecked`, all of originals that were already stored; one
first downloaded in this run is in none of them. A mismatch that was put back
with matching bytes does not make the exit code non-zero. Without `--verify`
nothing is hashed, the summary is unchanged, and the three counts are 0 in
`--json`.
line naming the file, and replaced once a matching copy is downloaded, in the
same run. The copy is downloaded even when the cache holds the original, because
a copy from the cache is not checked as a download is. It is written to a
temporary file beside the original, a live photo's image and video each to its
own, hashed the same way, and renamed over the original only when it matches; a
live photo's image and video are both hashed before either is renamed. Until
then the original stays as it was: when the download fails, or what it brings
does not match either, the temporary files are deleted, the original keeps its
bytes and its album links, and the file goes into `failures.json`. A file whose
metadata records no hash is left as it is and counted as unchecked. A stored
original that cannot be read is left as it is and counts as failed. The summary
and `--json` add the counts `verified`, `mismatched` and `unchecked`, all of
originals that were already stored; one first downloaded in this run is in none
of them. A mismatch that was replaced with matching bytes does not make the exit
code non-zero. Without `--verify` nothing is hashed, the summary is unchanged,
and the three counts are 0 in `--json`.
Each original is written to a temporary file in the same directory, synced to
disk, and renamed into place, so an original is either complete or absent, even
after a power cut. A downloaded original's temporary file is named
`.quak-<pid>-<random>.tmp`, one copied from the cache
`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A run that is
killed can leave one of these temporary files behind; the next backup deletes
those whose process is no longer running. The content cache uses the same
scheme, and opening a library deletes the temporary files in the cache whose
process is no longer running, so a download another process has in progress in
the same cache is left alone. The rename replaces whatever was at the
destination rather than writing through it: a symlink there is replaced, not
followed, and the new file has the temporary file's permissions, not those of
the file it replaced.
`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A copy that
`--verify` downloads to replace an original is written to such a
`.quak-<pid>-<random>.tmp` too, a live photo's image and video to
`.quak-<pid>-<random>.<ext>`, each with its own extension. A run that is killed
can leave one of these temporary files behind; the next backup deletes those
whose process is no longer running. The content cache uses the same scheme, and
opening a library deletes the temporary files in the cache whose process is no
longer running, so a download another process has in progress in the same cache
is left alone. The rename replaces whatever was at the destination rather than
writing through it: a symlink there is replaced, not followed, and the new file
has the temporary file's permissions, not those of the file it replaced.
## TODO