Share clipEmbedding, and shorten the README's keyword sentences
check / check (push) Waiting to run

clipKeywords takes the CLIP embedding out of a file's ML data with
clipEmbedding, now exported from src/library/mldata.ts, and adds only its own
checks: 512 numbers, not all zero. bin/clip-keyword-vectors puts its body in
a main function called at the bottom, as the scripts under script/ do; it
writes the same files. The README lists the Discover categories as bullets
and states the score, the rule and the video rule in short sentences of their
own.

Model: opus-5-5
This commit is contained in:
2026-10-22 19:25:40 +00:00
parent 791f17b84e
commit c65b0fae79
4 changed files with 72 additions and 47 deletions
+54 -32
View File
@@ -791,10 +791,11 @@ as it was and goes into `failures.json` (see "Backup layout").
`backup-metadata <dir>` writes `account.json` and, for each collection, a
directory `collections/<id>-<name>/` holding `_collection.json` and a
`<fileID>.json` for each of its files, with the file's ML data and keywords as
`quak backup` writes them (see "Backup layout"). The collection's name has
unsafe characters replaced and is cut short between two characters so that the
directory's name fits in 255 bytes; the ID keeps the directories apart.
`<fileID>.json` for each of its files. Each `<fileID>.json` holds the file's ML
data and keywords as `quak backup` writes them (see "Backup layout"). The
collection's name has unsafe characters replaced and is cut short between two
characters so that the directory's name fits in 255 bytes; the ID keeps the
directories apart.
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
fails, the error is logged, each of its files is written with the reason in an
@@ -863,17 +864,38 @@ and its `magicMetadata` and `pubMagicMetadata` when it has them. Update times
are in microseconds, as Ente records them.
A file's JSON holds Ente's ML data for it as `mlData`, such as the faces Ente
found, without its CLIP embedding. In place of the embedding it holds
`keywords`, the titles of the categories of the Discover section of Ente's
photos app that the file falls in, highest score first. The categories are
Identity, Screenshots, QR Codes, Receipts, Notes, Memes, Visiting Cards, Babies,
Pets, Selfies, Wallpapers, Food, Celebrations, Sunset, Hills and Greenery. As in
Ente's app, a file falls in a category when the cosine similarity of its CLIP
embedding and the text embedding of the category's query is at least the
category's minimum score, and a video stays out of Identity, Screenshots, QR
Codes, Visiting Cards and Wallpapers. A file with no ML data has neither field,
and one whose ML data holds no usable CLIP embedding has no `keywords`.
`backup-metadata` writes both the same way.
found, without its CLIP embedding. It holds `keywords` in place of the
embedding.
The Discover section of Ente's photos app has 16 categories:
- Identity (no videos)
- Screenshots (no videos)
- QR Codes (no videos)
- Receipts
- Notes
- Memes
- Visiting Cards (no videos)
- Babies
- Pets
- Selfies
- Wallpapers (no videos)
- Food
- Celebrations
- Sunset
- Hills
- Greenery
A file's keywords are the titles of the categories it falls in. A file's score
for a category is the cosine similarity of its CLIP embedding and the text
embedding of the category's query. As in Ente's app, the file falls in the
category when that score is at least the category's minimum score. A video never
falls in a category marked "no videos". The keywords are listed highest score
first.
A file with no ML data has neither `mlData` nor `keywords`. A file whose ML data
holds no usable CLIP embedding has no `keywords`. `backup-metadata` writes both
fields the same way.
The backup waits for the library's ML data fetch to finish before it writes the
JSON files. If that fetch fails, each file whose ML data is not in the cache
@@ -1289,18 +1311,18 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
not already at its save path there as `photo.download()` does (and, with
`includeThumbnails`, fetches thumbnails) through the content cache, waits for
an ML data fetch, and rebuilds the on-disk backup tree with a durable failure
ledger. Each file's JSON holds its ML data and keywords and, for an image (for
a live photo, its image), its original's EXIF and XMP, with its dimensions for
a JPEG only; a video's JSON holds none of these three. A fetched original is
written straight to its save path and not into the cache, which then counts it
as present; one the cache already held is copied from there. `BackupOptions`:
`downloadDirectory` (falls back to the library's), `includeOriginals` (default
`true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
(default `false`), `onProgress`, and `lock`. See Backup layout above for the
tree it writes, and for the albums `BackupResult` lists in
`undecryptableAlbums` and `failedAlbums`. Whether the backup took the lock or
its caller did, once the lock is lost the backup rejects with
`lost the lock on <dir>: <reason>` before it starts another file.
ledger. Each file's JSON holds its ML data and keywords. An image's JSON also
holds the EXIF and XMP of its original, and its dimensions if it is a JPEG. A
live photo's JSON holds these for its image. A video's JSON holds none of
them. A fetched original is written straight to its save path and not into the
cache, which then counts it as present; one the cache already held is copied
from there. `BackupOptions`: `downloadDirectory` (falls back to the
library's), `includeOriginals` (default `true`), `includeThumbnails` (default
`false`), `onlyAlbumNames`, `verify` (default `false`), `onProgress`, and
`lock`. See Backup layout above for the tree it writes, and for the albums
`BackupResult` lists in `undecryptableAlbums` and `failedAlbums`. Whether the
backup took the lock or its caller did, once the lock is lost the backup
rejects with `lost the lock on <dir>: <reason>` before it starts another file.
### Request pools
@@ -1385,11 +1407,11 @@ against the upstream implementations in `web/packages/base/`,
The keywords come from Ente too. `src/clip-keyword-vectors.ts` holds the
Discover categories of Ente's photos app, from
`mobile/apps/photos/assets/discover.json`, each with the text embedding Ente's
CLIP text model gives its query. `bin/clip-keyword-vectors` writes that file,
and the CLIP embedding of a public-domain photo that the tests use, from Ente's
list and models; it needs docker, pins every download by sha256, and runs only
by hand, followed by `make fmt`.
`mobile/apps/photos/assets/discover.json`, each with its query's text embedding
from Ente's CLIP text model. `bin/clip-keyword-vectors` writes that file and the
CLIP embedding of a public-domain photo that the tests use. It runs Ente's
models in Docker, with every download pinned by sha256. It is run by hand only,
followed by `make fmt`.
## For LLMs