ML data that does not decrypt is recorded with its reason (closes #198)

`fetchMLDataBatch` dropped an entry that did not decrypt, gunzip or parse,
so the file looked like one with no ML data and the run exited 0. It now
returns the reason beside the payloads, also for ML data that is not a
JSON object. The library gives the reasons from its last ML data fetch
that succeeded in `status().mlDataErrors` and the `fetchMLData` "done"
event; a fetch that fails leaves them as they were. `quak backup` writes
the reason as `mlDataError` for a file with no ML data cached and records
the file in `failures.json`. `backup-metadata` writes `mlDataError` for it
and exits 1; `runMetadataBackup` returns `filesWithMLDataError`.

Judgement call: a file whose ML data is cached keeps it in its JSON when a
newer copy does not decode, as when the fetch fails.

Model: opus-5-5

README.md

@@ -769,8 +769,11 @@ and the next run fetches it again. The same goes for a file with no ML data in
 the cache whose ML data the server sent but quak could not decrypt, gunzip or
 parse. Its `mlDataError` says why, and `failures.json` records it with an error
 starting `ML data: `. quak keeps that reason in memory only and caches nothing
-for the file, so the next run asks for its ML data again. The JSON files are
-rewritten on every run, so ML data that arrived since the last run appears.
+for the file, so the next run asks for its ML data again. An ML data fetch that
+fails, such as one the library's refresh starts while the backup writes the JSON
+files, leaves the reasons as the last fetch that succeeded left them. The JSON
+files are rewritten on every run, so ML data that arrived since the last run
+appears.
 
 A file's JSON holds, as `imageMetadata`, what `backup-metadata --exif` records
 from its original, or from a live photo's image: `format`, `width` and `height`
@@ -985,11 +988,12 @@ the precache fetches already running have all finished, so the cache directory
 can then be removed.
 
 `lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML
-data the last ML data fetch to finish got but could not decrypt, gunzip or
+data the last ML data fetch that succeeded got but could not decrypt, gunzip or
 parse; the `fetchMLData` event with status `"done"` carries the same list. A
-fetch that is still running leaves the list as it is until it ends. Such ML data
-is not cached, so the next ML data fetch asks for it again. The reasons are kept
-in memory only, never in the cache.
+fetch that is still running leaves the list as it is until it ends, and a fetch
+that fails leaves it as it was. Such ML data is not cached, so the next ML data
+fetch asks for it again. The reasons are kept in memory only, never in the
+cache.
 
 ### Default reads vs. fresh reads
 

TODO.md

@@ -31,13 +31,14 @@ declares one.
   no longer taken for no ML data (issue 198). `fetchMLDataBatch` returns why for
   each such file beside the payloads; ML data that parses to something other
   than a JSON object, such as `null`, is one of them. The library keeps the
-  reasons from its last ML data fetch to finish in memory, not in the cache, and
-  gives them in `status().mlDataErrors` and in the `fetchMLData` event with
-  status `"done"`; a fetch still running does not change them. `quak backup`
-  writes the reason as `mlDataError` in the file's JSON when no ML data is
-  cached for it, and records the file in `failures.json` with an error starting
-  `ML data: `; `backup-metadata` writes `mlDataError` too. Both exit 1. Tests
-  serve ML data encrypted under a key that is not the file's.
+  reasons from its last ML data fetch that succeeded in memory, not in the
+  cache, and gives them in `status().mlDataErrors` and in the `fetchMLData`
+  event with status `"done"`; a fetch still running or one that fails does not
+  change them. `quak backup` writes the reason as `mlDataError` in the file's
+  JSON when no ML data is cached for it, and records the file in `failures.json`
+  with an error starting `ML data: `; `backup-metadata` writes `mlDataError`
+  too. Both exit 1. Tests serve ML data encrypted under a key that is not the
+  file's.
 
 - 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
   connection and a name that does not resolve, and the backup records either as

src/backup.ts

@@ -203,9 +203,10 @@ export interface BackupLibrary {
     // A file's cached ML data, as `lib.mldata.forFile()` returns it, or
     // undefined when none is cached.
     mlData(fileID: number): Promise<MLData | undefined>;
-    // Why the ML data the last ML data fetch to finish got for a file did not
-    // decrypt, gunzip or parse, or undefined when it did or the fetch got
-    // none. A fetch still running does not change it.
+    // Why the ML data the last ML data fetch that succeeded got for a file did
+    // not decrypt, gunzip or parse, or undefined when it did or the fetch got
+    // none. A fetch still running does not change it. A fetch that fails
+    // leaves it as it was.
     mlDataError(fileID: number): unknown;
 }
 

src/library/index.ts

@@ -233,10 +233,10 @@ export interface LibraryStatus {
     lastMLFetchAt?: number;
     // The most recent ML fetch pass's error, set only while it failed.
     lastMLError?: string;
-    // Each file whose ML data the last ML fetch pass to finish got but could
-    // not decrypt, gunzip or parse, with the reason. A pass still running
-    // leaves it as it is. Such ML data is not cached, so the next pass asks
-    // for it again.
+    // Each file whose ML data the last ML fetch pass that succeeded got but
+    // could not decrypt, gunzip or parse, with the reason. A pass still
+    // running leaves it as it is. A pass that fails leaves it as it was. Such
+    // ML data is not cached, so the next pass asks for it again.
     mlDataErrors: { fileID: number; error: string }[];
     // ML payloads stored on disk and CLIP embeddings in the index; undefined
     // when ML fetching is disabled.
@@ -305,11 +305,11 @@ export class Library {
     private lastError?: string;
     private lastMLFetchAt?: number;
     private lastMLError?: string;
-    // For each file whose ML data the last ML fetch pass to finish got but
-    // could not decrypt, gunzip or parse, why not, by fileID. A pass gathers
-    // its own and puts them here when it ends, so `status()` and `backup()`
-    // never see the list of a pass that is still running. Kept in RAM only,
-    // never in the cache.
+    // For each file whose ML data the last ML fetch pass that succeeded got
+    // but could not decrypt, gunzip or parse, why not, by fileID. A pass
+    // gathers its own and puts them here when it ends without failing, so
+    // `status()` and `backup()` never see the list of a pass that is still
+    // running or that failed. Kept in RAM only, never in the cache.
     private mlDataErrors = new Map<number, unknown>();
     // The plain-record projection as of the last refresh, and the GUI change
     // subscribers. A refresh that alters the projection notifies each with the
@@ -860,8 +860,8 @@ export class Library {
     // advanced), through the metadata pool, and update the CLIP index. For a
     // file whose ML data does not decrypt, gunzip or parse, it gathers the
     // reason instead, and puts the reasons it gathered in `mlDataErrors` when
-    // it ends. A failure is reported through `status()` and `onProgress`,
-    // then thrown.
+    // it ends without failing. A failure leaves `mlDataErrors` as it was and
+    // is reported through `status()` and `onProgress`, then thrown.
     private async runMLFetch(): Promise<void> {
         const mldata = this.mlStore;
         // Bind so the call keeps the client as its receiver when invoked
@@ -871,9 +871,9 @@ export class Library {
 
         const files = this.uniqueFiles();
         const needed = mldata.neededFor(files);
-        // This pass's reasons, put in `this.mlDataErrors` when it ends. A
-        // file whose ML data did not decode in the pass before is still among
-        // `needed`, so this pass gives its reason anew.
+        // This pass's reasons, put in `this.mlDataErrors` when it ends without
+        // failing. A file whose ML data did not decode in the pass before is
+        // still among `needed`, so this pass gives its reason anew.
         const mlDataErrors = new Map<number, unknown>();
         if (needed.length === 0) {
             this.mlDataErrors = mlDataErrors;
@@ -914,7 +914,6 @@ export class Library {
                 mlDataErrors: this.describeMLDataErrors(),
             });
         } catch (err) {
-            this.mlDataErrors = mlDataErrors;
             const error = err instanceof Error ? err.message : String(err);
             this.lastMLError = error;
             this.emit({ operation: "fetchMLData", status: "failed", error });

test/cli/backup-end-to-end.test.ts

@@ -9,7 +9,7 @@
  * data this file encrypts with libsodium, as Ente's clients encrypt it. So
  * quak's own code fetches each original, decrypts it, checks it against the
  * hash its uploader recorded and unpacks a live photo, and quak's own retries
- * repeat a failed request. Two tests run `lib.backup()` instead, as a program
+ * repeat a failed request. Three tests run `lib.backup()` instead, as a program
  * using quak as a library does.
  *
  * The account, user 42, has three albums:
@@ -927,6 +927,83 @@ describe("quak backup", () => {
         }
     });
 
+    // A program using quak as a library backs the account up, and another ML
+    // data fetch, as one the library's refresh starts, fails while the backup
+    // writes the JSON files. The backup's own fetch got beach.jpg's ML data
+    // under a key that is not the file's; the fetch that fails leaves that
+    // reason in place, so beach.jpg's JSON and `failures.json` still give it.
+    it("keeps the reason in beach.jpg's JSON and in failures.json when an ML data fetch fails while lib.backup() writes the JSON files", async () => {
+        const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        const mlData = encryptStreamBlob(
+            gzipSync(JSON.stringify(ML_DATA)),
+            otherKey,
+        );
+        let mlDataFetchFails = false;
+        const fetch: typeof globalThis.fetch = async (input, init) => {
+            if (new URL(String(input)).pathname !== "/files/data/fetch") {
+                return server(input, init);
+            }
+            if (mlDataFetchFails) throw new Error("the connection was reset");
+            const data = [
+                {
+                    fileID: BEACH.id,
+                    encryptedData: toBase64(mlData.ciphertext),
+                    decryptionHeader: toBase64(mlData.header),
+                },
+            ];
+            return new Response(JSON.stringify({ data }), {
+                headers: { "content-type": "application/json" },
+            });
+        };
+        const dir = mkdtempSync(join(root, "backup-"));
+        const lib = await Library.open({
+            client: loadSession(join(sessionDir, "session.json"), {
+                fetch,
+                retry: { ...UNATTENDED_RETRY_OPTIONS, sleep: async () => {} },
+            })!,
+            cacheDirectory: `${dir}-cache`,
+            downloadDirectory: dir,
+            refreshIntervalSeconds: Infinity,
+            precacheThumbnails: false,
+            precacheOriginals: false,
+        });
+        try {
+            // The backup reads a file's cached ML data just before it writes
+            // that file's JSON. The first time, a refresh starts an ML data
+            // fetch whose request fails, and the backup goes on once that
+            // fetch has failed.
+            const forFile = lib.mldata.forFile.bind(lib.mldata);
+            vi.spyOn(lib.mldata, "forFile").mockImplementationOnce(
+                async (args) => {
+                    mlDataFetchFails = true;
+                    await lib.fresh();
+                    await vi.waitFor(
+                        () => expect(lib.status().lastMLError).toBeDefined(),
+                        { timeout: 2000, interval: 5 },
+                    );
+                    return forFile(args);
+                },
+            );
+
+            const result = await lib.backup();
+
+            expect(result.failed).toBe(1);
+            const reason = "secretstream chunk authentication failed";
+            const beachJSON = readJSON(join(dir, DAY, "2024-03-01.100.json"));
+            expect(beachJSON).not.toHaveProperty("mlData");
+            expect(beachJSON.mlDataError).toBe(reason);
+            const failures = readJSON(join(dir, "failures.json"));
+            expect(Object.keys(failures.files)).toEqual(["100"]);
+            expect(failures.files["100"]).toMatchObject({
+                fileID: 100,
+                title: "beach.jpg",
+                error: `ML data: ${reason}`,
+            });
+        } finally {
+            await lib.close();
+        }
+    });
+
     // Something changed beach.jpg on disk after it was backed up. A run with
     // --verify hashes every stored original, finds that one no longer matches
     // the hash Ente recorded, says so, and downloads it again; it put back the