--verify keeps a mismatched original until a matching copy replaces it (closes #199)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
With --verify, an original whose hash does not match is no longer removed before its replacement is downloaded. replaceOriginal, beside placeOriginal in src/library/content.ts, has the content cache's new fetchOriginal write the replacement to a temporary file beside the original, hashes it, and renames it over the original only when it matches; a live photo's image and video are both hashed before either is renamed. A failed or mismatching download leaves the old bytes and album links, removes the temporary files, and lands in failures.json. The replacement is always downloaded, never copied from the cache. The leftover-file cleanup also takes a live photo's replacement files, .quak-<pid>-<random>.<ext>. Model: opus-5-5
This commit is contained in:
@@ -159,10 +159,11 @@ After a `1`, the next run fetches each missing original again, fetches the ML
|
||||
data that is not cached, and rebuilds the album links. An album quak could not
|
||||
decrypt, or a file in `failures.json` because quak could not decrypt it, keeps
|
||||
the exit code at `1` until the server sends a copy that decrypts, or deletes it
|
||||
(see "Backup layout"). An original that a `--verify` run could not read, or put
|
||||
back with bytes that still do not match, stays at its save path. Only a later
|
||||
`--verify` run checks it again: a run without `--verify` leaves it as it is and
|
||||
takes the file out of `failures.json`, so that run can exit 0.
|
||||
(see "Backup layout"). An original that a `--verify` run could not read, or
|
||||
could not replace with a download that matches, stays at its save path as it
|
||||
was. Only a later `--verify` run checks it again: a run without `--verify`
|
||||
leaves it as it is and takes the file out of `failures.json`, so that run can
|
||||
exit 0.
|
||||
|
||||
## Examples
|
||||
|
||||
@@ -692,8 +693,8 @@ reads no tag from is recorded, base64, as `exifRaw`, with the reason in
|
||||
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
|
||||
|
||||
`backup --verify` also hashes the originals already in the backup and replaces
|
||||
any that do not match the content hash Ente records; one it cannot replace goes
|
||||
into `failures.json` (see "Backup layout").
|
||||
any that do not match the content hash Ente records; one it cannot replace stays
|
||||
as it was and goes into `failures.json` (see "Backup layout").
|
||||
|
||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||
fails, the error is logged, each of its files is written with the reason in an
|
||||
@@ -878,33 +879,38 @@ original already at its save path the way a download is checked (see "On-disk
|
||||
cache layout" below): its bytes, read in chunks, or a live photo's image and
|
||||
video, joined as `<imageHash>:<videoHash>`. An original that matches the content
|
||||
hash its metadata records is left as it is. One that does not is logged on one
|
||||
line naming the file, deleted (a live photo's image and video both), and put
|
||||
back in the same run like a missing one: downloaded, or copied from the cache if
|
||||
the cache holds it. What is put back is hashed too, because a copy from the
|
||||
cache is not checked as a download is. If it still does not match, it stays at
|
||||
its save path and the file goes into `failures.json`, as it does when the
|
||||
download fails. A file whose metadata records no hash is left as it is and
|
||||
counted as unchecked. A stored original that cannot be read is left as it is and
|
||||
counts as failed. The summary and `--json` add the counts `verified`,
|
||||
`mismatched` and `unchecked`, all of originals that were already stored; one
|
||||
first downloaded in this run is in none of them. A mismatch that was put back
|
||||
with matching bytes does not make the exit code non-zero. Without `--verify`
|
||||
nothing is hashed, the summary is unchanged, and the three counts are 0 in
|
||||
`--json`.
|
||||
line naming the file, and replaced once a matching copy is downloaded, in the
|
||||
same run. The copy is downloaded even when the cache holds the original, because
|
||||
a copy from the cache is not checked as a download is. It is written to a
|
||||
temporary file beside the original, a live photo's image and video each to its
|
||||
own, hashed the same way, and renamed over the original only when it matches; a
|
||||
live photo's image and video are both hashed before either is renamed. Until
|
||||
then the original stays as it was: when the download fails, or what it brings
|
||||
does not match either, the temporary files are deleted, the original keeps its
|
||||
bytes and its album links, and the file goes into `failures.json`. A file whose
|
||||
metadata records no hash is left as it is and counted as unchecked. A stored
|
||||
original that cannot be read is left as it is and counts as failed. The summary
|
||||
and `--json` add the counts `verified`, `mismatched` and `unchecked`, all of
|
||||
originals that were already stored; one first downloaded in this run is in none
|
||||
of them. A mismatch that was replaced with matching bytes does not make the exit
|
||||
code non-zero. Without `--verify` nothing is hashed, the summary is unchanged,
|
||||
and the three counts are 0 in `--json`.
|
||||
|
||||
Each original is written to a temporary file in the same directory, synced to
|
||||
disk, and renamed into place, so an original is either complete or absent, even
|
||||
after a power cut. A downloaded original's temporary file is named
|
||||
`.quak-<pid>-<random>.tmp`, one copied from the cache
|
||||
`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A run that is
|
||||
killed can leave one of these temporary files behind; the next backup deletes
|
||||
those whose process is no longer running. The content cache uses the same
|
||||
scheme, and opening a library deletes the temporary files in the cache whose
|
||||
process is no longer running, so a download another process has in progress in
|
||||
the same cache is left alone. The rename replaces whatever was at the
|
||||
destination rather than writing through it: a symlink there is replaced, not
|
||||
followed, and the new file has the temporary file's permissions, not those of
|
||||
the file it replaced.
|
||||
`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A copy that
|
||||
`--verify` downloads to replace an original is written to such a
|
||||
`.quak-<pid>-<random>.tmp` too, a live photo's image and video to
|
||||
`.quak-<pid>-<random>.<ext>`, each with its own extension. A run that is killed
|
||||
can leave one of these temporary files behind; the next backup deletes those
|
||||
whose process is no longer running. The content cache uses the same scheme, and
|
||||
opening a library deletes the temporary files in the cache whose process is no
|
||||
longer running, so a download another process has in progress in the same cache
|
||||
is left alone. The rename replaces whatever was at the destination rather than
|
||||
writing through it: a symlink there is replaced, not followed, and the new file
|
||||
has the temporary file's permissions, not those of the file it replaced.
|
||||
|
||||
## TODO
|
||||
|
||||
|
||||
Reference in New Issue
Block a user