A file's JSON holds keywords in place of its CLIP embedding (closes #218)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
`quak backup` and `quak backup-metadata` write each file's ML data without its CLIP embedding, and `keywords` in its place: the titles of the Discover categories of Ente's photos app that the file falls in, highest score first. A file falls in a category when the cosine similarity of its CLIP embedding and the text embedding of the category's query reaches the category's minimum score; as in Ente's app, five categories leave out videos. The 16 text embeddings ship in `src/clip-keyword-vectors.ts`, written by `bin/clip-keyword-vectors` from Ente's list and CLIP text model. quak runs no model and gains no dependency. Disclosure: the text embeddings are committed model output, against REPO_POLICIES' rule on generated files. Model: opus-5-5
This commit was merged in pull request #220.
This commit is contained in:
@@ -86,10 +86,17 @@ const DAY = "2024/2024-03/2024-03-01";
|
||||
const TOAST = new TextEncoder().encode("a few seconds of the wedding toast");
|
||||
|
||||
// Ente's ML data for beach.jpg: a face it found, and the photo's CLIP
|
||||
// embedding.
|
||||
// embedding, here that of a photo of a sunset (see clip-keywords.test.ts).
|
||||
const ML_DATA = {
|
||||
face: { faces: [{ faceID: "100_1", score: 0.98 }] },
|
||||
clip: { embedding: [0.5, 0.6, 0.7] },
|
||||
clip: {
|
||||
embedding: JSON.parse(
|
||||
readFileSync(
|
||||
new URL("../clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
).embedding,
|
||||
},
|
||||
};
|
||||
|
||||
// Each file's ID, its owner, its original as uploaded, and the metadata its
|
||||
@@ -486,7 +493,10 @@ describe("the first backup of the account", () => {
|
||||
},
|
||||
updationTime: TAKEN,
|
||||
});
|
||||
expect(beach.mlData).toEqual(ML_DATA);
|
||||
// The JSON holds the ML data without its CLIP embedding, and the
|
||||
// keywords the embedding gives in its place.
|
||||
expect(beach.mlData).toEqual({ face: ML_DATA.face });
|
||||
expect(beach.keywords).toEqual(["Sunset"]);
|
||||
expect(beach.imageMetadata.exif.Make.description).toBe("Canon");
|
||||
expect(beach.imageMetadata.exif.Model.description).toBe("EOS R5");
|
||||
|
||||
@@ -497,6 +507,7 @@ describe("the first backup of the account", () => {
|
||||
metadata: { title: "toast.mp4", fileType: "video" },
|
||||
});
|
||||
expect(toast).not.toHaveProperty("mlData");
|
||||
expect(toast).not.toHaveProperty("keywords");
|
||||
expect(toast).not.toHaveProperty("imageMetadata");
|
||||
expect(toast).not.toHaveProperty("imageMetadataError");
|
||||
|
||||
|
||||
Reference in New Issue
Block a user