ML data that does not decrypt is recorded with its reason (closes #198)

`fetchMLDataBatch` dropped an entry that did not decrypt, gunzip or parse,
so the file looked like one with no ML data and the run exited 0. It now
returns the reason beside the payloads, also for ML data that is not a
JSON object. The library gives the reasons from its last finished ML data
fetch in `status().mlDataErrors` and the `fetchMLData` "done" event.
`quak backup` writes the reason as `mlDataError` for a file with no ML
data cached and records the file in `failures.json`. `backup-metadata`
writes `mlDataError` for it and exits 1; `runMetadataBackup` returns
`filesWithMLDataError`.

Judgement call: a file whose ML data is cached keeps it in its JSON when a
newer copy does not decode, as when the fetch fails.

Model: opus-5-5

README.md

@@ -985,10 +985,11 @@ the precache fetches already running have all finished, so the cache directory
 can then be removed.
 
 `lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML
-data the last ML data fetch got but could not decrypt, gunzip or parse; the
+data the last ML data fetch to finish got but could not decrypt, gunzip or
-`fetchMLData` event with status `"done"` carries the same list. Such ML data is
+parse; the `fetchMLData` event with status `"done"` carries the same list. A
-not cached, so the next ML data fetch asks for it again. The reasons are kept in
+fetch that is still running leaves the list as it is until it ends. Such ML data
-memory only, never in the cache.
+is not cached, so the next ML data fetch asks for it again. The reasons are kept
+in memory only, never in the cache.
 
 ### Default reads vs. fresh reads
 

TODO.md

@@ -29,13 +29,15 @@ declares one.
 
 - 2026-10-16: ML data the server sends that does not decrypt, gunzip or parse is
   no longer taken for no ML data (issue 198). `fetchMLDataBatch` returns why for
-  each such file beside the payloads. The library keeps those reasons from its
+  each such file beside the payloads; ML data that parses to something other
-  last ML data fetch in memory, not in the cache, and gives them in
+  than a JSON object, such as `null`, is one of them. The library keeps the
-  `status().mlDataErrors` and in the `fetchMLData` event with status `"done"`.
+  reasons from its last ML data fetch to finish in memory, not in the cache, and
-  `quak backup` writes the reason as `mlDataError` in the file's JSON when no ML
+  gives them in `status().mlDataErrors` and in the `fetchMLData` event with
-  data is cached for it, and records the file in `failures.json` with an error
+  status `"done"`; a fetch still running does not change them. `quak backup`
-  starting `ML data: `; `backup-metadata` writes `mlDataError` too. Both exit 1.
+  writes the reason as `mlDataError` in the file's JSON when no ML data is
-  Tests serve ML data encrypted under a key that is not the file's.
+  cached for it, and records the file in `failures.json` with an error starting
+  `ML data: `; `backup-metadata` writes `mlDataError` too. Both exit 1. Tests
+  serve ML data encrypted under a key that is not the file's.
 
 - 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
   connection and a name that does not resolve, and the backup records either as

src/backup.ts

@@ -203,8 +203,9 @@ export interface BackupLibrary {
     // A file's cached ML data, as `lib.mldata.forFile()` returns it, or
     // undefined when none is cached.
     mlData(fileID: number): Promise<MLData | undefined>;
-    // Why the ML data the last ML data fetch got for a file did not decrypt,
+    // Why the ML data the last ML data fetch to finish got for a file did not
-    // gunzip or parse, or undefined when it did or the fetch got none.
+    // decrypt, gunzip or parse, or undefined when it did or the fetch got
+    // none. A fetch still running does not change it.
     mlDataError(fileID: number): unknown;
 }
 

src/cli-commands.ts

@@ -394,11 +394,11 @@ export const backupMetadataCommand = async (
         // Refresh first so the dump holds current account state, not what the
         // cache last held; a failed refresh throws.
         await lib.fresh();
-        const { mlDataErrors } = await runMetadataBackup(lib, client, dir, {
+        const result = await runMetadataBackup(lib, client, dir, {
             exif: opts.exif || opts.all,
             onProgress: (msg) => ctx.stderr.write(msg + "\n"),
         });
-        return mlDataErrors > 0 ? 1 : 0;
+        return result.filesWithMLDataError > 0 ? 1 : 0;
     } finally {
         await lib.close();
     }

src/library/index.ts

@@ -233,9 +233,10 @@ export interface LibraryStatus {
     lastMLFetchAt?: number;
     // The most recent ML fetch pass's error, set only while it failed.
     lastMLError?: string;
-    // Each file whose ML data the most recent ML fetch pass got but could not
+    // Each file whose ML data the last ML fetch pass to finish got but could
-    // decrypt, gunzip or parse, with the reason. Such ML data is not cached,
+    // not decrypt, gunzip or parse, with the reason. A pass still running
-    // so the next pass asks for it again.
+    // leaves it as it is. Such ML data is not cached, so the next pass asks
+    // for it again.
     mlDataErrors: { fileID: number; error: string }[];
     // ML payloads stored on disk and CLIP embeddings in the index; undefined
     // when ML fetching is disabled.
@@ -304,10 +305,12 @@ export class Library {
     private lastError?: string;
     private lastMLFetchAt?: number;
     private lastMLError?: string;
-    // For each file whose ML data the most recent ML fetch pass got but could
+    // For each file whose ML data the last ML fetch pass to finish got but
-    // not decrypt, gunzip or parse, why not, by fileID. Kept in RAM only,
+    // could not decrypt, gunzip or parse, why not, by fileID. A pass gathers
+    // its own and puts them here when it ends, so `status()` and `backup()`
+    // never see the list of a pass that is still running. Kept in RAM only,
     // never in the cache.
-    private readonly mlDataErrors = new Map<number, unknown>();
+    private mlDataErrors = new Map<number, unknown>();
     // The plain-record projection as of the last refresh, and the GUI change
     // subscribers. A refresh that alters the projection notifies each with the
     // delta; `lastRecords` is kept current every refresh so a subscriber that
@@ -855,9 +858,10 @@ export class Library {
     // One ML fetch pass: fetch, decrypt and store the ML data for every file
     // the store knows about that is not cached (or whose `updationTime` has
     // advanced), through the metadata pool, and update the CLIP index. For a
-    // file whose ML data does not decrypt, gunzip or parse, it keeps the
+    // file whose ML data does not decrypt, gunzip or parse, it gathers the
-    // reason in `mlDataErrors` instead. A failure is reported through
+    // reason instead, and puts the reasons it gathered in `mlDataErrors` when
-    // `status()` and `onProgress`, then thrown.
+    // it ends. A failure is reported through `status()` and `onProgress`,
+    // then thrown.
     private async runMLFetch(): Promise<void> {
         const mldata = this.mlStore;
         // Bind so the call keeps the client as its receiver when invoked
@@ -867,10 +871,14 @@ export class Library {
 
         const files = this.uniqueFiles();
         const needed = mldata.neededFor(files);
-        // A file whose ML data did not decode in the pass before is still
+        // This pass's reasons, put in `this.mlDataErrors` when it ends. A
-        // among `needed`, so this pass gives its reason anew.
+        // file whose ML data did not decode in the pass before is still among
-        this.mlDataErrors.clear();
+        // `needed`, so this pass gives its reason anew.
-        if (needed.length === 0) return;
+        const mlDataErrors = new Map<number, unknown>();
+        if (needed.length === 0) {
+            this.mlDataErrors = mlDataErrors;
+            return;
+        }
 
         this.emit({ operation: "fetchMLData", status: "started" });
         try {
@@ -890,12 +898,13 @@ export class Library {
                     { priority: "background" },
                 );
                 for (const [id, error] of errors) {
-                    this.mlDataErrors.set(id, error);
+                    mlDataErrors.set(id, error);
                 }
                 stored += (await mldata.storeFetched(payloads, updation))
                     .stored;
             }
 
+            this.mlDataErrors = mlDataErrors;
             this.lastMLFetchAt = Date.now();
             this.lastMLError = undefined;
             this.emit({
@@ -905,6 +914,7 @@ export class Library {
                 mlDataErrors: this.describeMLDataErrors(),
             });
         } catch (err) {
+            this.mlDataErrors = mlDataErrors;
             const error = err instanceof Error ? err.message : String(err);
             this.lastMLError = error;
             this.emit({ operation: "fetchMLData", status: "failed", error });

src/metadata-backup.ts

@@ -100,7 +100,7 @@ export const runMetadataBackup = async (
     client: Client,
     outDir: string,
     opts?: MetadataBackupOptions,
-): Promise<{ mlDataErrors: number }> => {
+): Promise<{ filesWithMLDataError: number }> => {
     const log = opts?.onProgress ?? (() => {});
     const wantExif = opts?.exif ?? false;
 
@@ -235,5 +235,5 @@ export const runMetadataBackup = async (
     }
 
     log("Metadata backup complete.");
-    return { mlDataErrors: mlDataErrors.size };
+    return { filesWithMLDataError: mlDataErrors.size };
 };

src/mldata-fetch.ts

@@ -29,7 +29,8 @@ interface RawRemoteFileData {
 }
 
 // Decrypt one entry with its file key and gunzip the JSON payload. Throws why
-// when the entry does not decrypt, gunzip or parse.
+// when the entry does not decrypt, gunzip or parse, or parses to something
+// other than a JSON object, such as `null`, a number or an array.
 const decodeEntry = (entry: RawRemoteFileData, key: Uint8Array): MLData => {
     const decrypted = decryptBlob(
         fromBase64(entry.encryptedData),
@@ -37,7 +38,15 @@ const decodeEntry = (entry: RawRemoteFileData, key: Uint8Array): MLData => {
         key,
     );
     const json = gunzipSync(Buffer.from(decrypted)).toString("utf-8");
-    return JSON.parse(json) as MLData;
+    const payload: unknown = JSON.parse(json);
+    if (
+        typeof payload !== "object" ||
+        payload === null ||
+        Array.isArray(payload)
+    ) {
+        throw new Error("not a JSON object");
+    }
+    return payload as MLData;
 };
 
 // Fetch ML data for up to `MLDATA_BATCH_SIZE` ids in a single request. This is

test/library/mldata.test.ts

@@ -13,9 +13,10 @@
  *   2. `Library` wiring: after each refresh the library fetches ML data through
  *      the metadata pool for every known file not yet cached, is incremental on
  *      later refreshes, and refetches a file whose `updationTime` advanced. A
- *      file whose ML data did not decode is reported with the reason, and
+ *      file whose ML data did not decode is reported with the reason, which
- *      asked for again. Opening a cache directory written by another account
+ *      stays listed until the next pass ends, and asked for again. Opening a
- *      starts empty, its ML data included (issue #104).
+ *      cache directory written by another account starts empty, its ML data
+ *      included (issue #104).
  *
  * Embedding values are chosen to be exactly representable as float32 so the
  * round-trip through `clip.f32` compares equal.
@@ -526,6 +527,69 @@ describe("Library ML-data fetch on refresh", () => {
         }
     });
 
+    it("still lists the last pass's reasons while the next pass runs", async () => {
+        const client = new MLMockClient();
+        client.collectionsQueue.push({
+            collections: [collection(1, 100)],
+            deleted: [],
+            cursor: 100,
+        });
+        client.filesFor(1, {
+            files: [file(1002, 1, 95)],
+            deleted: [],
+            cursor: 95,
+        });
+        client.mlErrorByFile.set(
+            1002,
+            new Error("secretstream chunk authentication failed"),
+        );
+        const reasons = [
+            { fileID: 1002, error: "secretstream chunk authentication failed" },
+        ];
+        let release!: () => void;
+        const held = new Promise<void>((r) => (release = r));
+
+        const lib = await Library.open({
+            client,
+            cacheDirectory,
+            refreshIntervalSeconds: Infinity,
+        });
+        try {
+            await vi.waitFor(
+                () => expect(lib.status().lastMLFetchAt).toBeGreaterThan(0),
+                { timeout: 2000, interval: 5 },
+            );
+            expect(lib.status().mlDataErrors).toEqual(reasons);
+
+            // Hold the next ML data fetch open until the test releases it.
+            let fetchStarted!: () => void;
+            const started = new Promise<void>((r) => (fetchStarted = r));
+            const realFetch = client.fetchMLData.bind(client);
+            client.fetchMLData = async (args) => {
+                fetchStarted();
+                await held;
+                return realFetch(args);
+            };
+            client.mlErrorByFile.clear();
+            client.mlByFile.set(1002, payload([1, -2, 0.5]));
+
+            // The pass a refresh starts asks for 1002 again. While it is
+            // held, the reason from the pass before is still listed; once it
+            // ends, its own empty list takes the place of that one.
+            await lib.fresh();
+            await started;
+            expect(lib.status().mlDataErrors).toEqual(reasons);
+            release();
+            await vi.waitFor(
+                () => expect(lib.status().mlDataErrors).toEqual([]),
+                { timeout: 2000, interval: 5 },
+            );
+        } finally {
+            release();
+            await lib.close();
+        }
+    });
+
     it("close() resolves only after a running ML data fetch has stored its payloads", async () => {
         const client = new MLMockClient();
         client.collectionsQueue.push({

test/mldata-fetch/mldata-fetch.test.ts

@@ -0,0 +1,54 @@
+// `fetchMLDataBatch`, against a server that sends one file's ML data.
+//
+// ML data that decrypts and gunzips but is not a JSON object is reported with
+// the reason, as ML data that does not parse is, and is never returned as a
+// payload: the library would fail to store it.
+
+import { gzipSync } from "node:zlib";
+import sodium from "libsodium-wrappers-sumo";
+import { beforeAll, describe, expect, it } from "vitest";
+
+import { ApiClient } from "../../src/api/client.js";
+import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
+import { fetchMLDataBatch } from "../../src/mldata-fetch.js";
+
+beforeAll(async () => {
+    await init();
+    await sodium.ready;
+});
+
+// Fetch the ML data of file 1 from a server that sends `json`, gzipped and
+// encrypted with the file's key.
+const fetchServing = async (json: string) => {
+    const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+    const { header, ciphertext } = encryptBlob(
+        new Uint8Array(gzipSync(json)),
+        key,
+    );
+    const data = [
+        {
+            fileID: 1,
+            encryptedData: toBase64(ciphertext),
+            decryptionHeader: toBase64(header),
+        },
+    ];
+    const api = new ApiClient({
+        fetch: async () =>
+            new Response(JSON.stringify({ data }), {
+                headers: { "content-type": "application/json" },
+            }),
+    });
+    return fetchMLDataBatch(api, [1], new Map([[1, key]]));
+};
+
+describe("fetchMLDataBatch", () => {
+    it.each(["null", "5", "[]"])(
+        "gives a reason for ML data that is %s",
+        async (json) => {
+            const { payloads, errors } = await fetchServing(json);
+
+            expect(payloads.size).toBe(0);
+            expect(errors.get(1)).toEqual(new Error("not a JSON object"));
+        },
+    );
+});