ML data that does not decrypt is recorded with its reason (closes #198)

`fetchMLDataBatch` dropped an entry that did not decrypt, gunzip or parse,
so the file looked like one with no ML data and the run exited 0. It now
returns the reason beside the payloads. The library keeps the reasons from
its last ML data fetch in memory only and gives them in
`status().mlDataErrors` and the `fetchMLData` "done" event. `quak backup`
writes the reason as `mlDataError` for a file with no ML data cached and
records the file in `failures.json` as `ML data: REASON`. `backup-metadata`
writes `mlDataError` for it and exits 1; `runMetadataBackup` now returns how
many files have `mlDataError`.

Judgement call: a file whose ML data is cached keeps it in its JSON when a
newer copy does not decode, as when the fetch fails.

Model: opus-5-5

README.md

@@ -985,11 +985,10 @@ the precache fetches already running have all finished, so the cache directory
 can then be removed.
 
 `lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML
-data the last ML data fetch to finish got but could not decrypt, gunzip or
+data the last ML data fetch got but could not decrypt, gunzip or parse; the
-parse; the `fetchMLData` event with status `"done"` carries the same list. A
+`fetchMLData` event with status `"done"` carries the same list. Such ML data is
-fetch that is still running leaves the list as it is until it ends. Such ML data
+not cached, so the next ML data fetch asks for it again. The reasons are kept in
-is not cached, so the next ML data fetch asks for it again. The reasons are kept
+memory only, never in the cache.
-in memory only, never in the cache.
 
 ### Default reads vs. fresh reads
 

TODO.md

@@ -29,15 +29,13 @@ declares one.
 
 - 2026-10-16: ML data the server sends that does not decrypt, gunzip or parse is
   no longer taken for no ML data (issue 198). `fetchMLDataBatch` returns why for
-  each such file beside the payloads; ML data that parses to something other
+  each such file beside the payloads. The library keeps those reasons from its
-  than a JSON object, such as `null`, is one of them. The library keeps the
+  last ML data fetch in memory, not in the cache, and gives them in
-  reasons from its last ML data fetch to finish in memory, not in the cache, and
+  `status().mlDataErrors` and in the `fetchMLData` event with status `"done"`.
-  gives them in `status().mlDataErrors` and in the `fetchMLData` event with
+  `quak backup` writes the reason as `mlDataError` in the file's JSON when no ML
-  status `"done"`; a fetch still running does not change them. `quak backup`
+  data is cached for it, and records the file in `failures.json` with an error
-  writes the reason as `mlDataError` in the file's JSON when no ML data is
+  starting `ML data: `; `backup-metadata` writes `mlDataError` too. Both exit 1.
-  cached for it, and records the file in `failures.json` with an error starting
+  Tests serve ML data encrypted under a key that is not the file's.
-  `ML data: `; `backup-metadata` writes `mlDataError` too. Both exit 1. Tests
-  serve ML data encrypted under a key that is not the file's.
 
 - 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
   connection and a name that does not resolve, and the backup records either as

src/backup.ts

@@ -203,9 +203,8 @@ export interface BackupLibrary {
     // A file's cached ML data, as `lib.mldata.forFile()` returns it, or
     // undefined when none is cached.
     mlData(fileID: number): Promise<MLData | undefined>;
-    // Why the ML data the last ML data fetch to finish got for a file did not
+    // Why the ML data the last ML data fetch got for a file did not decrypt,
-    // decrypt, gunzip or parse, or undefined when it did or the fetch got
+    // gunzip or parse, or undefined when it did or the fetch got none.
-    // none. A fetch still running does not change it.
     mlDataError(fileID: number): unknown;
 }
 

src/cli-commands.ts

@@ -394,11 +394,11 @@ export const backupMetadataCommand = async (
         // Refresh first so the dump holds current account state, not what the
         // cache last held; a failed refresh throws.
         await lib.fresh();
-        const result = await runMetadataBackup(lib, client, dir, {
+        const { mlDataErrors } = await runMetadataBackup(lib, client, dir, {
             exif: opts.exif || opts.all,
             onProgress: (msg) => ctx.stderr.write(msg + "\n"),
         });
-        return result.filesWithMLDataError > 0 ? 1 : 0;
+        return mlDataErrors > 0 ? 1 : 0;
     } finally {
         await lib.close();
     }

src/library/index.ts

@@ -233,10 +233,9 @@ export interface LibraryStatus {
     lastMLFetchAt?: number;
     // The most recent ML fetch pass's error, set only while it failed.
     lastMLError?: string;
-    // Each file whose ML data the last ML fetch pass to finish got but could
+    // Each file whose ML data the most recent ML fetch pass got but could not
-    // not decrypt, gunzip or parse, with the reason. A pass still running
+    // decrypt, gunzip or parse, with the reason. Such ML data is not cached,
-    // leaves it as it is. Such ML data is not cached, so the next pass asks
+    // so the next pass asks for it again.
-    // for it again.
     mlDataErrors: { fileID: number; error: string }[];
     // ML payloads stored on disk and CLIP embeddings in the index; undefined
     // when ML fetching is disabled.
@@ -305,12 +304,10 @@ export class Library {
     private lastError?: string;
     private lastMLFetchAt?: number;
     private lastMLError?: string;
-    // For each file whose ML data the last ML fetch pass to finish got but
+    // For each file whose ML data the most recent ML fetch pass got but could
-    // could not decrypt, gunzip or parse, why not, by fileID. A pass gathers
+    // not decrypt, gunzip or parse, why not, by fileID. Kept in RAM only,
-    // its own and puts them here when it ends, so `status()` and `backup()`
-    // never see the list of a pass that is still running. Kept in RAM only,
     // never in the cache.
-    private mlDataErrors = new Map<number, unknown>();
+    private readonly mlDataErrors = new Map<number, unknown>();
     // The plain-record projection as of the last refresh, and the GUI change
     // subscribers. A refresh that alters the projection notifies each with the
     // delta; `lastRecords` is kept current every refresh so a subscriber that
@@ -858,10 +855,9 @@ export class Library {
     // One ML fetch pass: fetch, decrypt and store the ML data for every file
     // the store knows about that is not cached (or whose `updationTime` has
     // advanced), through the metadata pool, and update the CLIP index. For a
-    // file whose ML data does not decrypt, gunzip or parse, it gathers the
+    // file whose ML data does not decrypt, gunzip or parse, it keeps the
-    // reason instead, and puts the reasons it gathered in `mlDataErrors` when
+    // reason in `mlDataErrors` instead. A failure is reported through
-    // it ends. A failure is reported through `status()` and `onProgress`,
+    // `status()` and `onProgress`, then thrown.
-    // then thrown.
     private async runMLFetch(): Promise<void> {
         const mldata = this.mlStore;
         // Bind so the call keeps the client as its receiver when invoked
@@ -871,14 +867,10 @@ export class Library {
 
         const files = this.uniqueFiles();
         const needed = mldata.neededFor(files);
-        // This pass's reasons, put in `this.mlDataErrors` when it ends. A
+        // A file whose ML data did not decode in the pass before is still
-        // file whose ML data did not decode in the pass before is still among
+        // among `needed`, so this pass gives its reason anew.
-        // `needed`, so this pass gives its reason anew.
+        this.mlDataErrors.clear();
-        const mlDataErrors = new Map<number, unknown>();
+        if (needed.length === 0) return;
-        if (needed.length === 0) {
-            this.mlDataErrors = mlDataErrors;
-            return;
-        }
 
         this.emit({ operation: "fetchMLData", status: "started" });
         try {
@@ -898,13 +890,12 @@ export class Library {
                     { priority: "background" },
                 );
                 for (const [id, error] of errors) {
-                    mlDataErrors.set(id, error);
+                    this.mlDataErrors.set(id, error);
                 }
                 stored += (await mldata.storeFetched(payloads, updation))
                     .stored;
             }
 
-            this.mlDataErrors = mlDataErrors;
             this.lastMLFetchAt = Date.now();
             this.lastMLError = undefined;
             this.emit({
@@ -914,7 +905,6 @@ export class Library {
                 mlDataErrors: this.describeMLDataErrors(),
             });
         } catch (err) {
-            this.mlDataErrors = mlDataErrors;
             const error = err instanceof Error ? err.message : String(err);
             this.lastMLError = error;
             this.emit({ operation: "fetchMLData", status: "failed", error });

src/metadata-backup.ts

@@ -100,7 +100,7 @@ export const runMetadataBackup = async (
     client: Client,
     outDir: string,
     opts?: MetadataBackupOptions,
-): Promise<{ filesWithMLDataError: number }> => {
+): Promise<{ mlDataErrors: number }> => {
     const log = opts?.onProgress ?? (() => {});
     const wantExif = opts?.exif ?? false;
 
@@ -235,5 +235,5 @@ export const runMetadataBackup = async (
     }
 
     log("Metadata backup complete.");
-    return { filesWithMLDataError: mlDataErrors.size };
+    return { mlDataErrors: mlDataErrors.size };
 };

src/mldata-fetch.ts

@@ -29,8 +29,7 @@ interface RawRemoteFileData {
 }
 
 // Decrypt one entry with its file key and gunzip the JSON payload. Throws why
-// when the entry does not decrypt, gunzip or parse, or parses to something
+// when the entry does not decrypt, gunzip or parse.
-// other than a JSON object, such as `null`, a number or an array.
 const decodeEntry = (entry: RawRemoteFileData, key: Uint8Array): MLData => {
     const decrypted = decryptBlob(
         fromBase64(entry.encryptedData),
@@ -38,15 +37,7 @@ const decodeEntry = (entry: RawRemoteFileData, key: Uint8Array): MLData => {
         key,
     );
     const json = gunzipSync(Buffer.from(decrypted)).toString("utf-8");
-    const payload: unknown = JSON.parse(json);
+    return JSON.parse(json) as MLData;
-    if (
-        typeof payload !== "object" ||
-        payload === null ||
-        Array.isArray(payload)
-    ) {
-        throw new Error("not a JSON object");
-    }
-    return payload as MLData;
 };
 
 // Fetch ML data for up to `MLDATA_BATCH_SIZE` ids in a single request. This is

test/library/mldata.test.ts

@@ -13,10 +13,9 @@
  *   2. `Library` wiring: after each refresh the library fetches ML data through
  *      the metadata pool for every known file not yet cached, is incremental on
  *      later refreshes, and refetches a file whose `updationTime` advanced. A
- *      file whose ML data did not decode is reported with the reason, which
+ *      file whose ML data did not decode is reported with the reason, and
- *      stays listed until the next pass ends, and asked for again. Opening a
+ *      asked for again. Opening a cache directory written by another account
- *      cache directory written by another account starts empty, its ML data
+ *      starts empty, its ML data included (issue #104).
- *      included (issue #104).
  *
  * Embedding values are chosen to be exactly representable as float32 so the
  * round-trip through `clip.f32` compares equal.
@@ -527,69 +526,6 @@ describe("Library ML-data fetch on refresh", () => {
         }
     });
 
-    it("still lists the last pass's reasons while the next pass runs", async () => {
-        const client = new MLMockClient();
-        client.collectionsQueue.push({
-            collections: [collection(1, 100)],
-            deleted: [],
-            cursor: 100,
-        });
-        client.filesFor(1, {
-            files: [file(1002, 1, 95)],
-            deleted: [],
-            cursor: 95,
-        });
-        client.mlErrorByFile.set(
-            1002,
-            new Error("secretstream chunk authentication failed"),
-        );
-        const reasons = [
-            { fileID: 1002, error: "secretstream chunk authentication failed" },
-        ];
-        let release!: () => void;
-        const held = new Promise<void>((r) => (release = r));
-
-        const lib = await Library.open({
-            client,
-            cacheDirectory,
-            refreshIntervalSeconds: Infinity,
-        });
-        try {
-            await vi.waitFor(
-                () => expect(lib.status().lastMLFetchAt).toBeGreaterThan(0),
-                { timeout: 2000, interval: 5 },
-            );
-            expect(lib.status().mlDataErrors).toEqual(reasons);
-
-            // Hold the next ML data fetch open until the test releases it.
-            let fetchStarted!: () => void;
-            const started = new Promise<void>((r) => (fetchStarted = r));
-            const realFetch = client.fetchMLData.bind(client);
-            client.fetchMLData = async (args) => {
-                fetchStarted();
-                await held;
-                return realFetch(args);
-            };
-            client.mlErrorByFile.clear();
-            client.mlByFile.set(1002, payload([1, -2, 0.5]));
-
-            // The pass a refresh starts asks for 1002 again. While it is
-            // held, the reason from the pass before is still listed; once it
-            // ends, its own empty list takes the place of that one.
-            await lib.fresh();
-            await started;
-            expect(lib.status().mlDataErrors).toEqual(reasons);
-            release();
-            await vi.waitFor(
-                () => expect(lib.status().mlDataErrors).toEqual([]),
-                { timeout: 2000, interval: 5 },
-            );
-        } finally {
-            release();
-            await lib.close();
-        }
-    });
-
     it("close() resolves only after a running ML data fetch has stored its payloads", async () => {
         const client = new MLMockClient();
         client.collectionsQueue.push({

test/mldata-fetch/mldata-fetch.test.ts

@@ -1,54 +0,0 @@
-// `fetchMLDataBatch`, against a server that sends one file's ML data.
-//
-// ML data that decrypts and gunzips but is not a JSON object is reported with
-// the reason, as ML data that does not parse is, and is never returned as a
-// payload: the library would fail to store it.
-
-import { gzipSync } from "node:zlib";
-import sodium from "libsodium-wrappers-sumo";
-import { beforeAll, describe, expect, it } from "vitest";
-
-import { ApiClient } from "../../src/api/client.js";
-import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
-import { fetchMLDataBatch } from "../../src/mldata-fetch.js";
-
-beforeAll(async () => {
-    await init();
-    await sodium.ready;
-});
-
-// Fetch the ML data of file 1 from a server that sends `json`, gzipped and
-// encrypted with the file's key.
-const fetchServing = async (json: string) => {
-    const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
-    const { header, ciphertext } = encryptBlob(
-        new Uint8Array(gzipSync(json)),
-        key,
-    );
-    const data = [
-        {
-            fileID: 1,
-            encryptedData: toBase64(ciphertext),
-            decryptionHeader: toBase64(header),
-        },
-    ];
-    const api = new ApiClient({
-        fetch: async () =>
-            new Response(JSON.stringify({ data }), {
-                headers: { "content-type": "application/json" },
-            }),
-    });
-    return fetchMLDataBatch(api, [1], new Map([[1, key]]));
-};
-
-describe("fetchMLDataBatch", () => {
-    it.each(["null", "5", "[]"])(
-        "gives a reason for ML data that is %s",
-        async (json) => {
-            const { payloads, errors } = await fetchServing(json);
-
-            expect(payloads.size).toBe(0);
-            expect(errors.get(1)).toEqual(new Error("not a JSON object"));
-        },
-    );
-});