Fetch, store, and index per-file ML data and CLIP embeddings (closes #49) #65

Merged
clawbot merged 1 commits from issue-49-ml-data-store into next 2026-09-22 17:54:42 +02:00
1 Commits
Author SHA1 Message Date
sneak 3588deb844 Fetch, store, and index per-file ML data and CLIP embeddings (closes #49)
check / check (push) Successful in 21s
Ente's per-file ML data (face detections + CLIP embeddings) is fetched,
decrypted, and cached under cacheDirectory/mldata/, never in
metadata.json. A new mldata-fetch module holds the decrypt+gunzip, reused
by the metadata backup and the library. MLDataStore writes one payload
file per fileID by rename (present means complete) and derives clip.json
+ clip.f32 (embeddings packed as one Float32Array, one-read load). On open
the index is rebuilt whenever it disagrees with the payloads in either
direction — an id it names is gone, or an embedding-bearing payload on
disk is absent from it — so a crash between writing payloads and rewriting
the index self-heals instead of dropping those embeddings. Appended to as
payloads arrive; fetched.json records each file's fetch-time updationTime.

Model: opus-4-8
2026-09-22 15:41:18 +00:00