A file's JSON holds keywords in place of its CLIP embedding (closes #218)

`quak backup` and `quak backup-metadata` write each file's ML data without its
CLIP embedding, and `keywords` in its place: the titles of the Discover
categories of Ente's photos app that the file falls in, highest score first. A
file falls in a category when the cosine similarity of its CLIP embedding and
the text embedding of the category's query reaches the category's minimum
score; as in Ente's app, five categories leave out videos.

The 16 text embeddings ship in `src/clip-keyword-vectors.ts`, written by
`bin/clip-keyword-vectors` from Ente's list and CLIP text model. quak runs no
model and gains no dependency.

Disclosure: the text embeddings are committed model output, against
REPO_POLICIES' rule on generated files.

Model: opus-5-5

README.md

@@ -10,12 +10,13 @@ account into a deduplicated local directory tree, skipping files that already
 exist on disk and continuing past individual download failures instead of
 crashing. For each file it persists the basic metadata fields quak keeps (title,
 file type, creation and modification time, latitude, longitude, content hash),
-its update time, the private and public magic metadata in full, Ente's ML data
-for it when there is any, and, for an image (for a live photo, its image), its
-original's EXIF and XMP, with its dimensions for a JPEG only; for a video it
-keeps none of these three. It runs unattended from cron (see "Running the backup
-from cron"). A helper subcommand can detect and regenerate missing thumbnails,
-encrypting and uploading them back to the server.
+its update time, and the private and public magic metadata in full. It keeps any
+ML data Ente has for the file, except the CLIP embedding, and writes the file's
+keywords in place of the embedding. For an image (for a live photo, its image),
+it keeps its original's EXIF and XMP, with its dimensions for a JPEG only. For a
+video it keeps none of these three. It runs unattended from cron (see "Running
+the backup from cron"). A helper subcommand can detect and regenerate missing
+thumbnails, encrypting and uploading them back to the server.
 
 ## Getting Started
 
@@ -371,6 +372,11 @@ quak/
                         backup-metadata: the metadata quak keeps, as JSON
         exif.ts         EXIF read from an image's bytes with exifreader
         mldata-fetch.ts fetch + decrypt per-file ML data
+        clip-keywords.ts
+                        a file's keywords from its CLIP embedding
+        clip-keyword-vectors.ts
+                        Ente's Discover categories with their text
+                        embeddings, written by bin/clip-keyword-vectors
         filename.ts     safe file names from server metadata
         errors.ts       error types shared across layers, and how an error
                         is described
@@ -387,6 +393,9 @@ quak/
         index.ts        public library exports
     bin/
         quak.ts         CLI entrypoint (commander.js)
+        clip-keyword-vectors, clip-keyword-vectors.py
+                        write src/clip-keyword-vectors.ts and a test
+                        fixture (see Source attribution)
     examples/
         download-albums.ts
                         download every album's photos and metadata
@@ -783,9 +792,11 @@ as it was and goes into `failures.json` (see "Backup layout").
 
 `backup-metadata <dir>` writes `account.json` and, for each collection, a
 directory `collections/<id>-<name>/` holding `_collection.json` and a
-`<fileID>.json` for each of its files. The collection's name has unsafe
-characters replaced and is cut short between two characters so that the
-directory's name fits in 255 bytes; the ID keeps the directories apart.
+`<fileID>.json` for each of its files. Each `<fileID>.json` holds the file's ML
+data and keywords as `quak backup` writes them (see "Backup layout"). The
+collection's name has unsafe characters replaced and is cut short between two
+characters so that the directory's name fits in 255 bytes; the ID keeps the
+directories apart.
 
 `backup-metadata` fetches ML data in requests of up to 200 files. When a request
 fails, the error is logged, each of its files is written with the reason in an
@@ -818,11 +829,11 @@ the smallest does not.
                                     below)
         YYYY-MM-DD.<fileID>.json    the file's basic metadata fields quak
                                     keeps, its update time, its private and
-                                    public magic metadata, its ML data, and,
-                                    for an image (for a live photo, its
-                                    image), its original's EXIF and XMP, with
-                                    dimensions for a JPEG only; none of these
-                                    three for a video
+                                    public magic metadata, its ML data and
+                                    keywords, and, for an image (for a live
+                                    photo, its image), its original's EXIF
+                                    and XMP, with dimensions for a JPEG
+                                    only; none of these three for a video
         YYYY-MM-DD.<fileID>.livephoto.json
                                     which of a live photo's two files is which
     collections/
@@ -853,20 +864,51 @@ JSON holds its `id`, `collectionID`, `ownerID`, `metadata` and `updationTime`,
 and its `magicMetadata` and `pubMagicMetadata` when it has them. Update times
 are in microseconds, as Ente records them.
 
-A file's JSON holds Ente's ML data for it (its faces and its CLIP embedding) as
-`mlData`, the same payload `backup-metadata` writes; a file Ente has no ML data
-for has no `mlData`. The backup waits for the library's ML data fetch to finish
-before it writes the JSON files. If that fetch fails, each file whose ML data is
-not in the cache gets the reason in `mlDataError` instead and counts as failed,
-and the next run fetches it again. The same goes for a file with no ML data in
-the cache whose ML data the server sent but quak could not decrypt, gunzip or
-parse. Its `mlDataError` says why, and `failures.json` records it with an error
-starting `ML data: `. quak keeps that reason in memory only and caches nothing
-for the file, so the next run asks for its ML data again. An ML data fetch that
-fails, such as one the library's refresh starts while the backup writes the JSON
-files, leaves the reasons as the last fetch that succeeded left them. The JSON
-files are rewritten on every run, so ML data that arrived since the last run
-appears.
+A file's JSON holds Ente's ML data for it as `mlData`, such as the faces Ente
+found, without its CLIP embedding. It holds `keywords` in place of the
+embedding.
+
+The Discover section of Ente's photos app has 16 categories:
+
+- Identity (no videos)
+- Screenshots (no videos)
+- QR Codes (no videos)
+- Receipts
+- Notes
+- Memes
+- Visiting Cards (no videos)
+- Babies
+- Pets
+- Selfies
+- Wallpapers (no videos)
+- Food
+- Celebrations
+- Sunset
+- Hills
+- Greenery
+
+A file's keywords are the titles of the categories it falls in. A file's score
+for a category is the cosine similarity of its CLIP embedding and the text
+embedding of the category's query. As in Ente's app, the file falls in the
+category when that score is at least the category's minimum score. A video never
+falls in a category marked "no videos". The keywords are listed highest score
+first.
+
+A file with no ML data has neither `mlData` nor `keywords`. A file whose ML data
+holds no usable CLIP embedding has no `keywords`. `backup-metadata` writes both
+fields the same way.
+
+The backup waits for the library's ML data fetch to finish before it writes the
+JSON files. If that fetch fails, each file whose ML data is not in the cache
+gets the reason in `mlDataError` instead and counts as failed, and the next run
+fetches it again. The same goes for a file with no ML data in the cache whose ML
+data the server sent but quak could not decrypt, gunzip or parse. Its
+`mlDataError` says why, and `failures.json` records it with an error starting
+`ML data: `. quak keeps that reason in memory only and caches nothing for the
+file, so the next run asks for its ML data again. An ML data fetch that fails,
+such as one the library's refresh starts while the backup writes the JSON files,
+leaves the reasons as the last fetch that succeeded left them. The JSON files
+are rewritten on every run, so ML data that arrived since the last run appears.
 
 A file's JSON holds, as `imageMetadata`, what `backup-metadata --exif` records
 from its original, or from a live photo's image: `format`, `width` and `height`
@@ -1270,18 +1312,18 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
   not already at its save path there as `photo.download()` does (and, with
   `includeThumbnails`, fetches thumbnails) through the content cache, waits for
   an ML data fetch, and rebuilds the on-disk backup tree with a durable failure
-  ledger. Each file's JSON holds its ML data and, for an image (for a live
-  photo, its image), its original's EXIF and XMP, with its dimensions for a JPEG
-  only; a video's JSON holds none of these three. A fetched original is written
-  straight to its save path and not into the cache, which then counts it as
-  present; one the cache already held is copied from there. `BackupOptions`:
-  `downloadDirectory` (falls back to the library's), `includeOriginals` (default
-  `true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
-  (default `false`), `onProgress`, and `lock`. See Backup layout above for the
-  tree it writes, and for the albums `BackupResult` lists in
-  `undecryptableAlbums` and `failedAlbums`. Whether the backup took the lock or
-  its caller did, once the lock is lost the backup rejects with
-  `lost the lock on <dir>: <reason>` before it starts another file.
+  ledger. Each file's JSON holds its ML data and keywords. An image's JSON also
+  holds the EXIF and XMP of its original, and its dimensions if it is a JPEG. A
+  live photo's JSON holds these for its image. A video's JSON holds no EXIF, XMP
+  or dimensions. A fetched original is written straight to its save path and not
+  into the cache, which then counts it as present; one the cache already held is
+  copied from there. `BackupOptions`: `downloadDirectory` (falls back to the
+  library's), `includeOriginals` (default `true`), `includeThumbnails` (default
+  `false`), `onlyAlbumNames`, `verify` (default `false`), `onProgress`, and
+  `lock`. See Backup layout above for the tree it writes, and for the albums
+  `BackupResult` lists in `undecryptableAlbums` and `failedAlbums`. Whether the
+  backup took the lock or its caller did, once the lock is lost the backup
+  rejects with `lost the lock on <dir>: <reason>` before it starts another file.
 
 ### Request pools
 
@@ -1364,6 +1406,14 @@ is rewritten in TypeScript in this repository. Protocol fidelity is verified
 against the upstream implementations in `web/packages/base/`,
 `mobile/apps/photos/lib/`, and `cli/`.
 
+The keywords come from Ente too. `src/clip-keyword-vectors.ts` holds the
+Discover categories of Ente's photos app, from
+`mobile/apps/photos/assets/discover.json`, each with its query's text embedding
+from Ente's CLIP text model. `bin/clip-keyword-vectors` writes that file and the
+CLIP embedding of a public-domain photo that the tests use. It runs Ente's
+models in Docker, with every download pinned by sha256. It is run by hand only,
+followed by `make fmt`.
+
 ## For LLMs
 
 If you are an LLM agent working on this repository, read and follow these