ML data that does not decrypt is recorded with its reason (closes #198)

`fetchMLDataBatch` dropped an entry that did not decrypt, gunzip or parse, so the file looked like one with no ML data and the run exited 0. It now returns the reason beside the payloads. The library keeps the reasons from its last ML data fetch in memory only and gives them in `status().mlDataErrors` and the `fetchMLData` "done" event. `quak backup` writes the reason as `mlDataError` for a file with no ML data cached and records the file in `failures.json` as `ML data: REASON`. `backup-metadata` writes `mlDataError` for it and exits 1; `runMetadataBackup` now returns how many files have `mlDataError`. Judgement call: a file whose ML data is cached keeps it in its JSON when a newer copy does not decode, as when the fetch fails. Model: opus-5-5
2026-10-16 03:10:55 +00:00
parent 20410f8f71
commit 4633150de9
12 changed files with 332 additions and 84 deletions
@@ -841,6 +841,56 @@ describe("quak backup", () => {
        ]);
    });

+    // The server sends beach.jpg's ML data encrypted under a key that is not
+    // the file's, so quak cannot decrypt it. The file's JSON has the reason
+    // in place of the ML data, and the file goes into `failures.json` with
+    // it, so the exit code is 1. The other two files have no ML data, which
+    // is no failure.
+    it("records a file whose ML data does not decrypt in its JSON and in failures.json, and exits 1", async () => {
+        const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        const mlData = encryptStreamBlob(
+            gzipSync(JSON.stringify(ML_DATA)),
+            otherKey,
+        );
+        const fetch: typeof globalThis.fetch = async (input, init) => {
+            if (new URL(String(input)).pathname !== "/files/data/fetch") {
+                return server(input, init);
+            }
+            const data = [
+                {
+                    fileID: BEACH.id,
+                    encryptedData: toBase64(mlData.ciphertext),
+                    decryptionHeader: toBase64(mlData.header),
+                },
+            ];
+            return new Response(JSON.stringify({ data }), {
+                headers: { "content-type": "application/json" },
+            });
+        };
+        const dir = mkdtempSync(join(root, "backup-"));
+
+        const { code, stderr } = await backup(dir, {}, fetch);
+
+        expect(code).toBe(1);
+        const reason = "secretstream chunk authentication failed";
+        const beachJSON = readJSON(join(dir, DAY, "2024-03-01.100.json"));
+        expect(beachJSON).not.toHaveProperty("mlData");
+        expect(beachJSON.mlDataError).toBe(reason);
+        const failures = readJSON(join(dir, "failures.json"));
+        expect(Object.keys(failures.files)).toEqual(["100"]);
+        expect(failures.files["100"]).toMatchObject({
+            fileID: 100,
+            title: "beach.jpg",
+            error: `ML data: ${reason}`,
+        });
+        expect(stderr).toContain(`beach.jpg (id 100): ML data: ${reason}\n`);
+        for (const fileID of [200, 300]) {
+            const json = readJSON(join(dir, DAY, `2024-03-01.${fileID}.json`));
+            expect(json).not.toHaveProperty("mlData");
+            expect(json).not.toHaveProperty("mlDataError");
+        }
+    });
+
    // Something changed beach.jpg on disk after it was backed up. A run with
    // --verify hashes every stored original, finds that one no longer matches
    // the hash Ente recorded, says so, and downloads it again; it put back the