Recorded: quak shows the image's keywords, not the raw list of numbers from the image model. quak's manager does this alongside the fix for listing one collection, when the fleet restarts at 23:00 UTC on 25 October.
Model: opus-5-5
Recorded: quak shows the image's keywords, not the raw list of numbers from the image model. quak's manager does this alongside the fix for listing one collection, when the fleet restarts at 23:00 UTC on 25 October.
Model: opus-5-5
clawbot
self-assigned this 2026-10-22 19:36:31 +02:00
Ente stores no labels or keywords. The ML data quak fetches for a file holds its face detections and one CLIP embedding (512 numbers). Ente's apps turn text into the same kind of vector on the device, with a text model, each time someone searches. Ente's one fixed word list is the Discover section of its photos app: 16 categories in mobile/apps/photos/assets/discover.json (https://github.com/ente-io/ente/blob/f5ad70a860ed25d0c8057b0efb4bd6eaefe4b0c5/mobile/apps/photos/assets/discover.json, sha256 cf70ac7116b592f7921ca4cf67cdbb421a8e6b287fbe44b2be678b8d198e5c24). Each has a title (Identity, Screenshots, QR Codes, Receipts, Notes, Memes, Visiting Cards, Babies, Pets, Selfies, Wallpapers, Food, Celebrations, Sunset, Hills, Greenery), a search phrase and a minimum score. The app puts a file in a category when the cosine similarity of the file's CLIP embedding and the phrase's text vector is at least that score, and keeps videos out of the categories marked "showVideo": false. quak uses that list and that rule: a file's keywords are the titles of the categories it falls in.
The 16 phrase vectors are computed once with the text model Ente's apps use, https://models.ente.com/mobileclip_s2_text_opset18_quant.onnx (sha256 d92f33dfcff83077fc2e0d3414250710efbb51795dfd89767bdbefb5fdc47322), and its vocabulary https://models.ente.com/bpe_simple_vocab_16e6.txt (sha256 67603cfda2e032ad77b5f8808af37789d590db664b26df8705d2bf8b3c553fc8): 77 token ids in, the output scaled to length 1, as in Ente's rust/crates/ml/src/clip/. They ship in quak's source. quak runs no model and gains no dependency.
Build, one PR into next:
src/clip-keywords.ts, a new top-level module beside mldata-fetch.ts (no existing module's job covers this): one function taking a file's ML data and whether the file is a video, returning the matching titles, highest score first, or undefined when the ML data has no CLIP embedding of 512 finite numbers that is not all zeros.
src/clip-keyword-vectors.ts, written only by the generator (item 4): the 16 categories with title, phrase, minimum score, whether videos can have it, and vector.
quak backup (src/backup.ts) and quak backup-metadata (src/metadata-backup.ts): each file's JSON gets keywords (the list, possibly empty) whenever that function returns one, and mlData is written without its clip part, so no CLIP numbers appear in the JSON. Face data is unchanged.
A generator under bin/, run by hand only, never by a gate. It runs in a Docker image pinned by digest, and every download it makes (packages, Ente's list, model, vocabulary) is pinned by sha256. It tokenizes with a widely used library, not a hand-written tokenizer. It fetches Ente's list at the commit above and writes item 2. It also writes the test fixture in item 5.
Tests: the rule with made-up vectors (at and just below a minimum score, the video rule, the order, unusable embeddings); one CLIP embedding of a public-domain photo with obvious content (a sunset, say), made once with Ente's image model mobileclip_s2_image_gelu_opset20.onnx (sha256 205a430af825e501c5138e5bb9abea942482a7a4fd4a680e98e47cf0830dce7e) and Ente's preparation (scale to cover 256×256, crop the centre, RGB divided by 255), committed with the photo's URL and sha256, gets the expected keyword; the backup and backup-metadata tests check keywords is there and mlData.clip is not.
README: Backup layout, backup-metadata, Layout, and Source attribution (the list and the vectors come from Ente); a dated TODO.md entry as the others are.
Disclosure: the vectors are model output committed to the repo, against REPO_POLICIES' rule on generated files; making them in every build would mean downloading and running a 67 MB model each time.
Model: opus-5-5
Plan, and where the keywords come from.
Ente stores no labels or keywords. The ML data quak fetches for a file holds its face detections and one CLIP embedding (512 numbers). Ente's apps turn text into the same kind of vector on the device, with a text model, each time someone searches. Ente's one fixed word list is the Discover section of its photos app: 16 categories in `mobile/apps/photos/assets/discover.json` (https://github.com/ente-io/ente/blob/f5ad70a860ed25d0c8057b0efb4bd6eaefe4b0c5/mobile/apps/photos/assets/discover.json, sha256 `cf70ac7116b592f7921ca4cf67cdbb421a8e6b287fbe44b2be678b8d198e5c24`). Each has a title (Identity, Screenshots, QR Codes, Receipts, Notes, Memes, Visiting Cards, Babies, Pets, Selfies, Wallpapers, Food, Celebrations, Sunset, Hills, Greenery), a search phrase and a minimum score. The app puts a file in a category when the cosine similarity of the file's CLIP embedding and the phrase's text vector is at least that score, and keeps videos out of the categories marked `"showVideo": false`. quak uses that list and that rule: a file's keywords are the titles of the categories it falls in.
The 16 phrase vectors are computed once with the text model Ente's apps use, `https://models.ente.com/mobileclip_s2_text_opset18_quant.onnx` (sha256 `d92f33dfcff83077fc2e0d3414250710efbb51795dfd89767bdbefb5fdc47322`), and its vocabulary `https://models.ente.com/bpe_simple_vocab_16e6.txt` (sha256 `67603cfda2e032ad77b5f8808af37789d590db664b26df8705d2bf8b3c553fc8`): 77 token ids in, the output scaled to length 1, as in Ente's `rust/crates/ml/src/clip/`. They ship in quak's source. quak runs no model and gains no dependency.
Build, one PR into `next`:
1. `src/clip-keywords.ts`, a new top-level module beside `mldata-fetch.ts` (no existing module's job covers this): one function taking a file's ML data and whether the file is a video, returning the matching titles, highest score first, or undefined when the ML data has no CLIP embedding of 512 finite numbers that is not all zeros.
2. `src/clip-keyword-vectors.ts`, written only by the generator (item 4): the 16 categories with title, phrase, minimum score, whether videos can have it, and vector.
3. `quak backup` (`src/backup.ts`) and `quak backup-metadata` (`src/metadata-backup.ts`): each file's JSON gets `keywords` (the list, possibly empty) whenever that function returns one, and `mlData` is written without its `clip` part, so no CLIP numbers appear in the JSON. Face data is unchanged.
4. A generator under `bin/`, run by hand only, never by a gate. It runs in a Docker image pinned by digest, and every download it makes (packages, Ente's list, model, vocabulary) is pinned by sha256. It tokenizes with a widely used library, not a hand-written tokenizer. It fetches Ente's list at the commit above and writes item 2. It also writes the test fixture in item 5.
5. Tests: the rule with made-up vectors (at and just below a minimum score, the video rule, the order, unusable embeddings); one CLIP embedding of a public-domain photo with obvious content (a sunset, say), made once with Ente's image model `mobileclip_s2_image_gelu_opset20.onnx` (sha256 `205a430af825e501c5138e5bb9abea942482a7a4fd4a680e98e47cf0830dce7e`) and Ente's preparation (scale to cover 256×256, crop the centre, RGB divided by 255), committed with the photo's URL and sha256, gets the expected keyword; the backup and backup-metadata tests check `keywords` is there and `mlData.clip` is not.
6. README: Backup layout, `backup-metadata`, Layout, and Source attribution (the list and the vectors come from Ente); a dated TODO.md entry as the others are.
Disclosure: the vectors are model output committed to the repo, against REPO_POLICIES' rule on generated files; making them in every build would mean downloading and running a 67 MB model each time.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
i want a list of keywords, not a list of floats
Recorded: quak shows the image's keywords, not the raw list of numbers from the image model. quak's manager does this alongside the fix for listing one collection, when the fleet restarts at 23:00 UTC on 25 October.
Model: opus-5-5
implement this using the primary, now
Started on the primary, as you asked: quak's manager is building this now, one reviewed PR into
next.Model: opus-5-5
Plan, and where the keywords come from.
Ente stores no labels or keywords. The ML data quak fetches for a file holds its face detections and one CLIP embedding (512 numbers). Ente's apps turn text into the same kind of vector on the device, with a text model, each time someone searches. Ente's one fixed word list is the Discover section of its photos app: 16 categories in
mobile/apps/photos/assets/discover.json(https://github.com/ente-io/ente/blob/f5ad70a860ed25d0c8057b0efb4bd6eaefe4b0c5/mobile/apps/photos/assets/discover.json, sha256cf70ac7116b592f7921ca4cf67cdbb421a8e6b287fbe44b2be678b8d198e5c24). Each has a title (Identity, Screenshots, QR Codes, Receipts, Notes, Memes, Visiting Cards, Babies, Pets, Selfies, Wallpapers, Food, Celebrations, Sunset, Hills, Greenery), a search phrase and a minimum score. The app puts a file in a category when the cosine similarity of the file's CLIP embedding and the phrase's text vector is at least that score, and keeps videos out of the categories marked"showVideo": false. quak uses that list and that rule: a file's keywords are the titles of the categories it falls in.The 16 phrase vectors are computed once with the text model Ente's apps use,
https://models.ente.com/mobileclip_s2_text_opset18_quant.onnx(sha256d92f33dfcff83077fc2e0d3414250710efbb51795dfd89767bdbefb5fdc47322), and its vocabularyhttps://models.ente.com/bpe_simple_vocab_16e6.txt(sha25667603cfda2e032ad77b5f8808af37789d590db664b26df8705d2bf8b3c553fc8): 77 token ids in, the output scaled to length 1, as in Ente'srust/crates/ml/src/clip/. They ship in quak's source. quak runs no model and gains no dependency.Build, one PR into
next:src/clip-keywords.ts, a new top-level module besidemldata-fetch.ts(no existing module's job covers this): one function taking a file's ML data and whether the file is a video, returning the matching titles, highest score first, or undefined when the ML data has no CLIP embedding of 512 finite numbers that is not all zeros.src/clip-keyword-vectors.ts, written only by the generator (item 4): the 16 categories with title, phrase, minimum score, whether videos can have it, and vector.quak backup(src/backup.ts) andquak backup-metadata(src/metadata-backup.ts): each file's JSON getskeywords(the list, possibly empty) whenever that function returns one, andmlDatais written without itsclippart, so no CLIP numbers appear in the JSON. Face data is unchanged.bin/, run by hand only, never by a gate. It runs in a Docker image pinned by digest, and every download it makes (packages, Ente's list, model, vocabulary) is pinned by sha256. It tokenizes with a widely used library, not a hand-written tokenizer. It fetches Ente's list at the commit above and writes item 2. It also writes the test fixture in item 5.mobileclip_s2_image_gelu_opset20.onnx(sha256205a430af825e501c5138e5bb9abea942482a7a4fd4a680e98e47cf0830dce7e) and Ente's preparation (scale to cover 256×256, crop the centre, RGB divided by 255), committed with the photo's URL and sha256, gets the expected keyword; the backup and backup-metadata tests checkkeywordsis there andmlData.clipis not.backup-metadata, Layout, and Source attribution (the list and the vectors come from Ente); a dated TODO.md entry as the others are.Disclosure: the vectors are model output committed to the repo, against REPO_POLICIES' rule on generated files; making them in every build would mean downloading and running a 67 MB model each time.
Model: opus-5-5