ML data that does not decrypt is recorded with its reason (closes #198)
check / check (push) Waiting to run
check / check (push) Waiting to run
`fetchMLDataBatch` dropped an entry that did not decrypt, gunzip or parse, so the file looked like one with no ML data and the run exited 0. It now returns the reason beside the payloads, also for ML data that is not a JSON object. The library gives the reasons from its last ML data fetch that succeeded in `status().mlDataErrors` and the `fetchMLData` "done" event; a fetch that fails leaves them as they were. `quak backup` writes the reason as `mlDataError` for a file with no ML data cached and records the file in `failures.json`. `backup-metadata` writes `mlDataError` for it and exits 1; `runMetadataBackup` returns `filesWithMLDataError`. Judgement call: a file whose ML data is cached keeps it in its JSON when a newer copy does not decode, as when the fetch fails. Model: opus-5-5
This commit is contained in:
@@ -698,8 +698,10 @@ as it was and goes into `failures.json` (see "Backup layout").
|
||||
|
||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||
fails, the error is logged, each of its files is written with the reason in an
|
||||
`mlDataError` field instead of `mlData`, and the dump goes on. The exit code is
|
||||
non-zero if any ML data request failed.
|
||||
`mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML
|
||||
data does not decrypt, gunzip or parse is logged and written the same way, with
|
||||
why in `mlDataError`. The exit code is non-zero if any file was written with
|
||||
`mlDataError`.
|
||||
|
||||
`helper fix-missing-thumbnails` regenerates thumbnails for JPEG images only,
|
||||
because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG image
|
||||
@@ -764,8 +766,15 @@ A file's JSON holds Ente's ML data for it (its faces and its CLIP embedding) as
|
||||
for has no `mlData`. The backup waits for the library's ML data fetch to finish
|
||||
before it writes the JSON files. If that fetch fails, each file whose ML data is
|
||||
not in the cache gets the reason in `mlDataError` instead and counts as failed,
|
||||
and the next run fetches it again. The JSON files are rewritten on every run, so
|
||||
ML data that arrived since the last run appears.
|
||||
and the next run fetches it again. The same goes for a file with no ML data in
|
||||
the cache whose ML data the server sent but quak could not decrypt, gunzip or
|
||||
parse. Its `mlDataError` says why, and `failures.json` records it with an error
|
||||
starting `ML data: `. quak keeps that reason in memory only and caches nothing
|
||||
for the file, so the next run asks for its ML data again. An ML data fetch that
|
||||
fails, such as one the library's refresh starts while the backup writes the JSON
|
||||
files, leaves the reasons as the last fetch that succeeded left them. The JSON
|
||||
files are rewritten on every run, so ML data that arrived since the last run
|
||||
appears.
|
||||
|
||||
A file's JSON holds, as `imageMetadata`, what `backup-metadata --exif` records
|
||||
from its original, or from a live photo's image: `format`, `width` and `height`
|
||||
@@ -987,6 +996,14 @@ resolves once that refresh (including its cache write), the ML data fetch and
|
||||
the precache fetches already running have all finished, so the cache directory
|
||||
can then be removed.
|
||||
|
||||
`lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML
|
||||
data the last ML data fetch that succeeded got but could not decrypt, gunzip or
|
||||
parse; the `fetchMLData` event with status `"done"` carries the same list. A
|
||||
fetch that is still running leaves the list as it is until it ends, and a fetch
|
||||
that fails leaves it as it was. Such ML data is not cached, so the next ML data
|
||||
fetch asks for it again. The reasons are kept in memory only, never in the
|
||||
cache.
|
||||
|
||||
### Default reads vs. fresh reads
|
||||
|
||||
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
||||
|
||||
Reference in New Issue
Block a user