Test keywords in place of the CLIP embedding in a file's JSON
check / check (push) Canceled after 0s

clipKeywords takes a file's ML data and whether it is a video, and gives the
titles of the Discover categories of Ente's photos app that its CLIP embedding
falls in, highest score first. The tests cover the rule with made-up vectors
and Ente's real categories with the CLIP embedding of a public-domain photo of
a sunset. The backup and backup-metadata tests now expect keywords in a file's
JSON and no CLIP embedding in its mlData. These tests fail until the
implementation lands.

Model: opus-5-5
This commit is contained in:
2026-10-22 18:36:30 +00:00
parent ae0f72971d
commit 4863833f0a
5 changed files with 290 additions and 16 deletions
+14 -3
View File
@@ -86,10 +86,17 @@ const DAY = "2024/2024-03/2024-03-01";
const TOAST = new TextEncoder().encode("a few seconds of the wedding toast");
// Ente's ML data for beach.jpg: a face it found, and the photo's CLIP
// embedding.
// embedding, here that of a photo of a sunset (see clip-keywords.test.ts).
const ML_DATA = {
face: { faces: [{ faceID: "100_1", score: 0.98 }] },
clip: { embedding: [0.5, 0.6, 0.7] },
clip: {
embedding: JSON.parse(
readFileSync(
new URL("../clip-keywords-sunset.json", import.meta.url),
"utf-8",
),
).embedding,
},
};
// Each file's ID, its owner, its original as uploaded, and the metadata its
@@ -486,7 +493,10 @@ describe("the first backup of the account", () => {
},
updationTime: TAKEN,
});
expect(beach.mlData).toEqual(ML_DATA);
// The JSON holds the ML data without its CLIP embedding, and the
// keywords the embedding gives in its place.
expect(beach.mlData).toEqual({ face: ML_DATA.face });
expect(beach.keywords).toEqual(["Sunset"]);
expect(beach.imageMetadata.exif.Make.description).toBe("Canon");
expect(beach.imageMetadata.exif.Model.description).toBe("EOS R5");
@@ -497,6 +507,7 @@ describe("the first backup of the account", () => {
metadata: { title: "toast.mp4", fileType: "video" },
});
expect(toast).not.toHaveProperty("mlData");
expect(toast).not.toHaveProperty("keywords");
expect(toast).not.toHaveProperty("imageMetadata");
expect(toast).not.toHaveProperty("imageMetadataError");