Share clipEmbedding, and shorten the README's keyword sentences

clipKeywords takes the CLIP embedding out of a file's ML data with
clipEmbedding, now exported from src/library/mldata.ts, and adds only its own
checks: 512 numbers, not all zero. bin/clip-keyword-vectors puts its body in
a main function called at the bottom, as the scripts under script/ do; it
writes the same files. The README lists the Discover categories as bullets
and states the score, the rule and the video rule in short sentences of their
own.

Model: opus-5-5

README.md

@@ -791,10 +791,11 @@ as it was and goes into `failures.json` (see "Backup layout").
 
 `backup-metadata <dir>` writes `account.json` and, for each collection, a
 directory `collections/<id>-<name>/` holding `_collection.json` and a
-`<fileID>.json` for each of its files, with the file's ML data and keywords as
+`<fileID>.json` for each of its files. Each `<fileID>.json` holds the file's ML
-`quak backup` writes them (see "Backup layout"). The collection's name has
+data and keywords as `quak backup` writes them (see "Backup layout"). The
-unsafe characters replaced and is cut short between two characters so that the
+collection's name has unsafe characters replaced and is cut short between two
-directory's name fits in 255 bytes; the ID keeps the directories apart.
+characters so that the directory's name fits in 255 bytes; the ID keeps the
+directories apart.
 
 `backup-metadata` fetches ML data in requests of up to 200 files. When a request
 fails, the error is logged, each of its files is written with the reason in an
@@ -863,17 +864,38 @@ and its `magicMetadata` and `pubMagicMetadata` when it has them. Update times
 are in microseconds, as Ente records them.
 
 A file's JSON holds Ente's ML data for it as `mlData`, such as the faces Ente
-found, without its CLIP embedding. In place of the embedding it holds
+found, without its CLIP embedding. It holds `keywords` in place of the
-`keywords`, the titles of the categories of the Discover section of Ente's
+embedding.
-photos app that the file falls in, highest score first. The categories are
+
-Identity, Screenshots, QR Codes, Receipts, Notes, Memes, Visiting Cards, Babies,
+The Discover section of Ente's photos app has 16 categories:
-Pets, Selfies, Wallpapers, Food, Celebrations, Sunset, Hills and Greenery. As in
+
-Ente's app, a file falls in a category when the cosine similarity of its CLIP
+- Identity (no videos)
-embedding and the text embedding of the category's query is at least the
+- Screenshots (no videos)
-category's minimum score, and a video stays out of Identity, Screenshots, QR
+- QR Codes (no videos)
-Codes, Visiting Cards and Wallpapers. A file with no ML data has neither field,
+- Receipts
-and one whose ML data holds no usable CLIP embedding has no `keywords`.
+- Notes
-`backup-metadata` writes both the same way.
+- Memes
+- Visiting Cards (no videos)
+- Babies
+- Pets
+- Selfies
+- Wallpapers (no videos)
+- Food
+- Celebrations
+- Sunset
+- Hills
+- Greenery
+
+A file's keywords are the titles of the categories it falls in. A file's score
+for a category is the cosine similarity of its CLIP embedding and the text
+embedding of the category's query. As in Ente's app, the file falls in the
+category when that score is at least the category's minimum score. A video never
+falls in a category marked "no videos". The keywords are listed highest score
+first.
+
+A file with no ML data has neither `mlData` nor `keywords`. A file whose ML data
+holds no usable CLIP embedding has no `keywords`. `backup-metadata` writes both
+fields the same way.
 
 The backup waits for the library's ML data fetch to finish before it writes the
 JSON files. If that fetch fails, each file whose ML data is not in the cache
@@ -1289,18 +1311,18 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
   not already at its save path there as `photo.download()` does (and, with
   `includeThumbnails`, fetches thumbnails) through the content cache, waits for
   an ML data fetch, and rebuilds the on-disk backup tree with a durable failure
-  ledger. Each file's JSON holds its ML data and keywords and, for an image (for
+  ledger. Each file's JSON holds its ML data and keywords. An image's JSON also
-  a live photo, its image), its original's EXIF and XMP, with its dimensions for
+  holds the EXIF and XMP of its original, and its dimensions if it is a JPEG. A
-  a JPEG only; a video's JSON holds none of these three. A fetched original is
+  live photo's JSON holds these for its image. A video's JSON holds none of
-  written straight to its save path and not into the cache, which then counts it
+  them. A fetched original is written straight to its save path and not into the
-  as present; one the cache already held is copied from there. `BackupOptions`:
+  cache, which then counts it as present; one the cache already held is copied
-  `downloadDirectory` (falls back to the library's), `includeOriginals` (default
+  from there. `BackupOptions`: `downloadDirectory` (falls back to the
-  `true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
+  library's), `includeOriginals` (default `true`), `includeThumbnails` (default
-  (default `false`), `onProgress`, and `lock`. See Backup layout above for the
+  `false`), `onlyAlbumNames`, `verify` (default `false`), `onProgress`, and
-  tree it writes, and for the albums `BackupResult` lists in
+  `lock`. See Backup layout above for the tree it writes, and for the albums
-  `undecryptableAlbums` and `failedAlbums`. Whether the backup took the lock or
+  `BackupResult` lists in `undecryptableAlbums` and `failedAlbums`. Whether the
-  its caller did, once the lock is lost the backup rejects with
+  backup took the lock or its caller did, once the lock is lost the backup
-  `lost the lock on <dir>: <reason>` before it starts another file.
+  rejects with `lost the lock on <dir>: <reason>` before it starts another file.
 
 ### Request pools
 
@@ -1385,11 +1407,11 @@ against the upstream implementations in `web/packages/base/`,
 
 The keywords come from Ente too. `src/clip-keyword-vectors.ts` holds the
 Discover categories of Ente's photos app, from
-`mobile/apps/photos/assets/discover.json`, each with the text embedding Ente's
+`mobile/apps/photos/assets/discover.json`, each with its query's text embedding
-CLIP text model gives its query. `bin/clip-keyword-vectors` writes that file,
+from Ente's CLIP text model. `bin/clip-keyword-vectors` writes that file and the
-and the CLIP embedding of a public-domain photo that the tests use, from Ente's
+CLIP embedding of a public-domain photo that the tests use. It runs Ente's
-list and models; it needs docker, pins every download by sha256, and runs only
+models in Docker, with every download pinned by sha256. It is run by hand only,
-by hand, followed by `make fmt`.
+followed by `make fmt`.
 
 ## For LLMs
 

bin/clip-keyword-vectors

@@ -5,21 +5,26 @@
 # no gate runs it. It downloads about 215 MB, most of it Ente's two CLIP models.
 set -eu
 
-cd "$(dirname "$0")/.."
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
 
 # python:3.13-slim (Python 3.13.17 on Debian 13.7), 2026-10-21. The wheels
 # below are the ones for this image on linux/amd64.
-image=python@sha256:57b29a7c050c4e679e4bea572e14b3965cf4b5839de2da49d55e49ffc4024f39
+IMAGE=python@sha256:57b29a7c050c4e679e4bea572e14b3965cf4b5839de2da49d55e49ffc4024f39
 
-docker run --rm -i --platform linux/amd64 \
+main() {
-    --user "$(id -u):$(id -g)" -e HOME=/tmp \
+    cd "$ROOT"
-    -v "$PWD:/repo" -w /repo "$image" sh -c '
+    docker run --rm -i --platform linux/amd64 \
-        python -m venv /tmp/venv &&
+        --user "$(id -u):$(id -g)" -e HOME=/tmp \
-        /tmp/venv/bin/pip install --quiet --disable-pip-version-check \
+        -v "$ROOT:/repo" -w /repo "$IMAGE" sh -c '
-            --no-cache-dir --no-deps --require-hashes -r /dev/stdin &&
+            python -m venv /tmp/venv &&
-        /tmp/venv/bin/python bin/clip-keyword-vectors.py' <<'EOF'
+            /tmp/venv/bin/pip install --quiet --disable-pip-version-check \
+                --no-cache-dir --no-deps --require-hashes -r /dev/stdin &&
+            /tmp/venv/bin/python bin/clip-keyword-vectors.py' <<'EOF'
 numpy==2.5.3 --hash=sha256:a5fa86b80fd24bcd1aff83ad23be44ea323de3f787be8f8b15d4a65621e25321
 onnxruntime==1.30.0 --hash=sha256:86f940afc801ea9681a4da8af84fbe95e1d9ea7d80903952cc1bfad54faad38f
 pillow==12.3.0 --hash=sha256:0847a763afefb695bc912d7c131e7e0632d4edc1d8698f58ddabec8e46b8b6d3
 tokenizers==0.23.2 --hash=sha256:41c2f84d172449b4dadb9cdc508e3e364076613c35b16e76ecfe47a60d1e3305
 EOF
+}
+
+main "$@"

src/clip-keywords.ts

@@ -10,6 +10,7 @@
 // their text embeddings are in `clip-keyword-vectors.ts`.
 
 import { DISCOVER_CATEGORIES } from "./clip-keyword-vectors.js";
+import { clipEmbedding } from "./library/mldata.js";
 import type { MLData } from "./mldata-fetch.js";
 
 // How many numbers a CLIP embedding from Ente's image model has.
@@ -33,13 +34,10 @@ export const clipKeywords = (
     isVideo: boolean,
     categories = DISCOVER_CATEGORIES,
 ): string[] | undefined => {
-    const clip = mlData.clip;
+    const embedding = clipEmbedding(mlData);
-    if (typeof clip !== "object" || clip === null) return undefined;
-    const embedding = (clip as { embedding?: unknown }).embedding;
     if (
-        !Array.isArray(embedding) ||
+        !embedding ||
         embedding.length !== EMBEDDING_LENGTH ||
-        !embedding.every((v) => typeof v === "number" && Number.isFinite(v)) ||
         length(embedding) === 0
     ) {
         return undefined;

src/library/mldata.ts

@@ -61,7 +61,7 @@ export interface MLIndex {
 // Pull the CLIP embedding out of a payload, or undefined when it is absent or
 // misshapen. Kept strict so a bad payload is skipped rather than corrupting the
 // packed index.
-const clipEmbedding = (payload: MLData): number[] | undefined => {
+export const clipEmbedding = (payload: MLData): number[] | undefined => {
     const clip = payload.clip;
     if (typeof clip !== "object" || clip === null) return undefined;
     const embedding = (clip as { embedding?: unknown }).embedding;

test/cli/backup.test.ts

@@ -52,6 +52,7 @@ import { tmpdir } from "node:os";
 import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
 
 import { runBackup, type BackupLibrary } from "../../src/backup.js";
+import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js";
 import { Library } from "../../src/library/index.js";
 import type { ContentSource } from "../../src/library/content.js";
 import type { CollectionsPage, FilesPage } from "../../src/client.js";
@@ -995,6 +996,58 @@ describe("ML data in each file's JSON", () => {
         await lib.close();
     });
 
+    // Ente's photos app keeps videos out of some Discover categories, Identity
+    // among them. Here a photo and a video share one CLIP embedding, Identity's
+    // own text embedding, so the photo gets the keyword Identity and the video
+    // does not.
+    it("keeps a video out of a category that leaves out videos", async () => {
+        const identity = DISCOVER_CATEGORIES.find(
+            (c) => c.title === "Identity",
+        )!;
+        expect(identity.showVideo).toBe(false);
+        const toast = file(102, 1, "toast.mp4");
+        const video = {
+            ...toast,
+            metadata: { ...toast.metadata, fileType: "video" as const },
+        };
+
+        // Serves beach.jpg and the video in the Vacation album, and gives
+        // each ML data holding only that embedding.
+        class VideoClient extends MockClient {
+            override async filesSince(args: {
+                collectionID: number;
+            }): Promise<FilesPage> {
+                const files =
+                    args.collectionID === 1
+                        ? [file(100, 1, "beach.jpg"), video]
+                        : [];
+                return { files, deleted: [], cursor: 1 };
+            }
+            async fetchMLData(args: { fileIDs: number[] }): Promise<{
+                payloads: Map<number, MLData>;
+                errors: Map<number, unknown>;
+            }> {
+                const payloads = new Map<number, MLData>();
+                for (const id of args.fileIDs) {
+                    payloads.set(id, {
+                        clip: { embedding: identity.textEmbedding },
+                    });
+                }
+                return { payloads, errors: new Map() };
+            }
+        }
+        const lib = await openLibrary(stubSource(), new VideoClient());
+        const outDir = join(root, "backup");
+
+        const result = await lib.backup({ downloadDirectory: outDir });
+
+        expect(result.failed).toBe(0);
+        expect(fileJSON(outDir, 100).keywords).toContain("Identity");
+        expect(fileJSON(outDir, 102)).toHaveProperty("keywords");
+        expect(fileJSON(outDir, 102).keywords).not.toContain("Identity");
+        await lib.close();
+    });
+
     it("gives each file with no cached ML data the reason when the fetch fails, and fails the run", async () => {
         const client = new MLClient();
         const outDir = join(root, "backup");

test/cli/metadata-backup.test.ts

@@ -41,6 +41,7 @@ import { beforeAll, afterAll, describe, expect, it, vi } from "vitest";
 import { init, toBase64, encryptBlob } from "../../src/crypto/index.js";
 import * as jpegJs from "jpeg-js";
 import { Client } from "../../src/client.js";
+import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js";
 import { Library } from "../../src/library/index.js";
 import {
     runMetadataBackup,
@@ -56,8 +57,8 @@ import {
     type MetaMockState,
 } from "../ente-server.js";
 
-// One file per ML data request, so the two files of the mock account are
+// One file per ML data request, so each file of the mock account is fetched in
-// fetched in two requests and one of them can fail on its own.
+// a request of its own and one of them can fail on its own.
 vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
     ...(await importOriginal<typeof import("../../src/mldata-fetch.js")>()),
     MLDATA_BATCH_SIZE: 1,
@@ -75,6 +76,9 @@ const SUNSET_EMBEDDING: number[] = JSON.parse(
     ),
 ).embedding;
 
+// Identity, a Discover category of Ente's photos app that leaves out videos.
+const IDENTITY = DISCOVER_CATEGORIES.find((c) => c.title === "Identity")!;
+
 let mock: MetaMockState;
 let testDir: string;
 
@@ -213,6 +217,63 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
         updationTime: 1710000000000000,
     };
 
+    // A file in coll 10 whose ML data holds only a CLIP embedding, Identity's
+    // own text embedding. `fileType` is Ente's number: 0 an image, 1 a video.
+    const identityFile = (id: number, title: string, fileType: number) => {
+        const fileKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        const { ciphertext: encKey, nonce } = encryptSecretbox(fileKey, ck1);
+        const meta = encryptStreamBlob(
+            new TextEncoder().encode(
+                JSON.stringify({
+                    title,
+                    fileType,
+                    creationTime: 1700000000000000,
+                    modificationTime: 1700000000000000,
+                }),
+            ),
+            fileKey,
+        );
+        const ml = encryptBlob(
+            new Uint8Array(
+                gzipSync(
+                    Buffer.from(
+                        JSON.stringify({
+                            clip: { embedding: IDENTITY.textEmbedding },
+                        }),
+                    ),
+                ),
+            ),
+            fileKey,
+        );
+        return {
+            raw: {
+                id,
+                collectionID: 10,
+                ownerID: 42,
+                encryptedKey: toBase64(encKey),
+                keyDecryptionNonce: toBase64(nonce),
+                metadata: {
+                    encryptedData: toBase64(meta.ciphertext),
+                    decryptionHeader: toBase64(meta.header),
+                },
+                file: {
+                    decryptionHeader: toBase64(sodium.randombytes_buf(24)),
+                },
+                thumbnail: {
+                    decryptionHeader: toBase64(sodium.randombytes_buf(24)),
+                },
+                updationTime: 1700000000000000,
+            },
+            mlData: {
+                encryptedData: toBase64(ml.ciphertext),
+                decryptionHeader: toBase64(ml.header),
+            },
+        };
+    };
+    // Files 300 and 301 in coll 10: a photo and a video.
+    const identityPhoto = identityFile(300, "passport.jpg", 0);
+    const identityVideo = identityFile(301, "passport.mp4", 1);
+
     // Encrypt ML data for file 100 (gzipped JSON, encrypted with file key)
     const mlPayload = JSON.stringify({
         face: {
@@ -279,12 +340,17 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
         keyAttributes: account.keyAttributes,
         encryptedToken: account.encryptedToken,
         collections: [rawColl1, rawColl2],
-        filesByCollection: { 10: [rawFile1], 20: [rawFile2] },
+        filesByCollection: {
+            10: [rawFile1, identityPhoto.raw, identityVideo.raw],
+            20: [rawFile2],
+        },
         encryptedMLData: {
             100: {
                 encryptedData: toBase64(mlCiphertext),
                 decryptionHeader: toBase64(mlHeader),
             },
+            300: identityPhoto.mlData,
+            301: identityVideo.mlData,
         },
         fileCiphertexts: { 100: encFileBody1 },
     };
@@ -467,6 +533,23 @@ describe("quak backup-metadata", () => {
         expect(fileMeta.keywords).toEqual(["Sunset"]);
     });
 
+    // Ente's photos app keeps videos out of some Discover categories, Identity
+    // among them. Files 300 and 301, a photo and a video, share one CLIP
+    // embedding, Identity's own text embedding, so the photo gets the keyword
+    // Identity and the video does not.
+    it("keeps a video out of a category that leaves out videos", async () => {
+        expect(IDENTITY.showVideo).toBe(false);
+        const outDir = join(testDir, "video-keywords");
+        await runBackup(outDir);
+
+        const vacation = join(outDir, "collections", "10-Vacation");
+        const fileJSON = (fileID: number) =>
+            JSON.parse(readFileSync(join(vacation, `${fileID}.json`), "utf-8"));
+        expect(fileJSON(300).keywords).toContain("Identity");
+        expect(fileJSON(301)).toHaveProperty("keywords");
+        expect(fileJSON(301).keywords).not.toContain("Identity");
+    });
+
     it("extracts EXIF from downloaded files when --exif is set", async () => {
         const outDir = join(testDir, "exif-data");
         await runBackup(outDir, { exif: true });