A file's JSON holds keywords in place of its CLIP embedding (closes #218)
check / check (push) Canceled after 0s

`quak backup` and `quak backup-metadata` write each file's ML data without its
CLIP embedding, and `keywords` in its place: the titles of the Discover
categories of Ente's photos app that the file falls in, highest score first. A
file falls in a category when the cosine similarity of its CLIP embedding and
the text embedding of the category's query reaches the category's minimum
score; as in Ente's app, five categories leave out videos.

The 16 text embeddings ship in `src/clip-keyword-vectors.ts`, written by
`bin/clip-keyword-vectors` from Ente's list and CLIP text model. quak runs no
model and gains no dependency.

Disclosure: the text embeddings are committed model output, against
REPO_POLICIES' rule on generated files.

Model: opus-5-5
This commit was merged in pull request #220.
This commit is contained in:
2026-10-22 23:32:25 +02:00
parent ae0f72971d
commit 657266a60e
14 changed files with 2694 additions and 68 deletions
+14 -3
View File
@@ -86,10 +86,17 @@ const DAY = "2024/2024-03/2024-03-01";
const TOAST = new TextEncoder().encode("a few seconds of the wedding toast");
// Ente's ML data for beach.jpg: a face it found, and the photo's CLIP
// embedding.
// embedding, here that of a photo of a sunset (see clip-keywords.test.ts).
const ML_DATA = {
face: { faces: [{ faceID: "100_1", score: 0.98 }] },
clip: { embedding: [0.5, 0.6, 0.7] },
clip: {
embedding: JSON.parse(
readFileSync(
new URL("../clip-keywords-sunset.json", import.meta.url),
"utf-8",
),
).embedding,
},
};
// Each file's ID, its owner, its original as uploaded, and the metadata its
@@ -486,7 +493,10 @@ describe("the first backup of the account", () => {
},
updationTime: TAKEN,
});
expect(beach.mlData).toEqual(ML_DATA);
// The JSON holds the ML data without its CLIP embedding, and the
// keywords the embedding gives in its place.
expect(beach.mlData).toEqual({ face: ML_DATA.face });
expect(beach.keywords).toEqual(["Sunset"]);
expect(beach.imageMetadata.exif.Make.description).toBe("Canon");
expect(beach.imageMetadata.exif.Model.description).toBe("EOS R5");
@@ -497,6 +507,7 @@ describe("the first backup of the account", () => {
metadata: { title: "toast.mp4", fileType: "video" },
});
expect(toast).not.toHaveProperty("mlData");
expect(toast).not.toHaveProperty("keywords");
expect(toast).not.toHaveProperty("imageMetadata");
expect(toast).not.toHaveProperty("imageMetadataError");