ML data that does not decrypt is recorded with its reason (closes #198)
check / check (push) Waiting to run

`fetchMLDataBatch` dropped an entry that did not decrypt, gunzip or parse,
so the file looked like one with no ML data and the run exited 0. It now
returns the reason beside the payloads, also for ML data that is not a
JSON object. The library gives the reasons from its last ML data fetch
that succeeded in `status().mlDataErrors` and the `fetchMLData` "done"
event; a fetch that fails leaves them as they were. `quak backup` writes
the reason as `mlDataError` for a file with no ML data cached and records
the file in `failures.json`. `backup-metadata` writes `mlDataError` for it
and exits 1; `runMetadataBackup` returns `filesWithMLDataError`.

Judgement call: a file whose ML data is cached keeps it in its JSON when a
newer copy does not decode, as when the fetch fails.

Model: opus-5-5
This commit is contained in:
2026-10-16 04:45:50 +00:00
parent 960a7ce8a6
commit 393ce02eb9
13 changed files with 556 additions and 85 deletions
+21 -4
View File
@@ -697,8 +697,10 @@ into `failures.json` (see "Backup layout").
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
fails, the error is logged, each of its files is written with the reason in an
`mlDataError` field instead of `mlData`, and the dump goes on. The exit code is
non-zero if any ML data request failed.
`mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML
data does not decrypt, gunzip or parse is logged and written the same way, with
why in `mlDataError`. The exit code is non-zero if any file was written with
`mlDataError`.
`helper fix-missing-thumbnails` regenerates thumbnails for JPEG images only,
because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG image
@@ -763,8 +765,15 @@ A file's JSON holds Ente's ML data for it (its faces and its CLIP embedding) as
for has no `mlData`. The backup waits for the library's ML data fetch to finish
before it writes the JSON files. If that fetch fails, each file whose ML data is
not in the cache gets the reason in `mlDataError` instead and counts as failed,
and the next run fetches it again. The JSON files are rewritten on every run, so
ML data that arrived since the last run appears.
and the next run fetches it again. The same goes for a file with no ML data in
the cache whose ML data the server sent but quak could not decrypt, gunzip or
parse. Its `mlDataError` says why, and `failures.json` records it with an error
starting `ML data: `. quak keeps that reason in memory only and caches nothing
for the file, so the next run asks for its ML data again. An ML data fetch that
fails, such as one the library's refresh starts while the backup writes the JSON
files, leaves the reasons as the last fetch that succeeded left them. The JSON
files are rewritten on every run, so ML data that arrived since the last run
appears.
A file's JSON holds, as `imageMetadata`, what `backup-metadata --exif` records
from its original, or from a live photo's image: `format`, `width` and `height`
@@ -978,6 +987,14 @@ resolves once that refresh (including its cache write), the ML data fetch and
the precache fetches already running have all finished, so the cache directory
can then be removed.
`lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML
data the last ML data fetch that succeeded got but could not decrypt, gunzip or
parse; the `fetchMLData` event with status `"done"` carries the same list. A
fetch that is still running leaves the list as it is until it ends, and a fetch
that fails leaves it as it was. Such ML data is not cached, so the next ML data
fetch asks for it again. The reasons are kept in memory only, never in the
cache.
### Default reads vs. fresh reads
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer