Fetch and store machine-learning (magic) data and build the CLIP index #49

New Issue

2026-09-22T09:11:10+02:00

clawbot commented

2026-09-22 09:11:10 +02:00

Cache/API implementation (#36), phase 4.
Depends on the refresh unit and the request-pools unit. "Magic tags" = Ente's
ML results (owner ruling 1), kept out of metadata.json.

Goal

Fetch, decrypt, store, and index the per-file machine-learning data (face
detections + CLIP embeddings) that Ente's content search rests on.

Scope

Reuse src/metadata-backup.ts's fetch: /files/data/fetch with
type: "mldata", 200 ids per request, decrypt each entry with the file key,
gunzip. Payload shape (from test/cli/metadata-backup.test.ts):
{ face: { faces: [{ faceID, detection: { box, landmarks }, score, blur, embedding }] }, clip: { embedding } }.
Store under cacheDirectory, never in metadata.json:
- mldata/<fileID>.json — the decrypted, gunzipped payload, one per fileID,
  written by rename; present means complete.
- mldata/clip.f32 + mldata/clip.json — the derived index the search runs
  on: clip.json lists fileIDs in order + embedding length; clip.f32 is
  those embeddings as one Float32Array (one read, no parse). Rebuilt from
  the per-file JSON when missing or when it disagrees with the files present;
  appended to as new payloads arrive.
Fetch through the metadata pool after each refresh: every fileID present in
the model and absent from mldata/ (whole account first run, new files after).
A file whose updationTime advanced is refetched. Report via onProgress
(operation: "fetchMLData") and status().
In RAM: the Float32Array + id list. Per-file payloads (face boxes,
landmarks) read from disk on demand, not held.

Definition of done

ML data fetched/decrypted/gunzipped and stored one-file-per-fileID; index
built and loadable in one read; incremental fetch on later refreshes.
Tested against fixtures/mocks; make check green.

Grounding

src/library/ + reuse of src/metadata-backup.ts fetch. Depends on refresh +
pools. The search surface is the next unit.

Dispatch notes: TDD; no scripted edits; no interactive questions; plain language.
Squash subject ends (closes #<this issue>). End every message with
Model: opus-4-8.

Model: opus-4-8

Cache/API implementation (https://git.eeqj.de/sneak/quak/issues/36), phase 4. Depends on the refresh unit and the request-pools unit. "Magic tags" = Ente's ML results (owner ruling 1), kept out of `metadata.json`. ## Goal Fetch, decrypt, store, and index the per-file machine-learning data (face detections + CLIP embeddings) that Ente's content search rests on. ## Scope - Reuse `src/metadata-backup.ts`'s fetch: `/files/data/fetch` with `type: "mldata"`, 200 ids per request, decrypt each entry with the file key, gunzip. Payload shape (from `test/cli/metadata-backup.test.ts`): `{ face: { faces: [{ faceID, detection: { box, landmarks }, score, blur, embedding }] }, clip: { embedding } }`. - Store under `cacheDirectory`, never in `metadata.json`: - `mldata/<fileID>.json` — the decrypted, gunzipped payload, one per fileID, written by rename; present means complete. - `mldata/clip.f32` + `mldata/clip.json` — the derived index the search runs on: `clip.json` lists fileIDs in order + embedding length; `clip.f32` is those embeddings as one `Float32Array` (one read, no parse). Rebuilt from the per-file JSON when missing or when it disagrees with the files present; appended to as new payloads arrive. - Fetch through the metadata pool after each refresh: every fileID present in the model and absent from `mldata/` (whole account first run, new files after). A file whose `updationTime` advanced is refetched. Report via `onProgress` (`operation: "fetchMLData"`) and `status()`. - In RAM: the `Float32Array` + id list. Per-file payloads (face boxes, landmarks) read from disk on demand, not held. ## Definition of done - ML data fetched/decrypted/gunzipped and stored one-file-per-fileID; index built and loadable in one read; incremental fetch on later refreshes. - Tested against fixtures/mocks; `make check` green. ## Grounding `src/library/` + reuse of `src/metadata-backup.ts` fetch. Depends on refresh + pools. The search surface is the next unit. Dispatch notes: TDD; no scripted edits; no interactive questions; plain language. Squash subject ends ` (closes #<this issue>)`. End every message with `Model: opus-4-8`. Model: opus-4-8

clawbot referenced this issue

2026-09-22 09:11:56 +02:00

Design for review: a cache layer (SQLite metadata + on-disk content/thumbnail store) and a crash-safe backup on it #36

clawbot referenced this issue from a commit

2026-09-22 17:10:46 +02:00

Fetch, store, and index per-file ML data and CLIP embeddings (closes #49)

clawbot referenced a pull request that will close this issue

2026-09-22 17:11:00 +02:00

Fetch, store, and index per-file ML data and CLIP embeddings (closes #49) #65

clawbot commented

2026-09-22 17:11:31 +02:00

Implemented in #65 (branch issue-49-ml-data-store, base next).

Per-file ML data (face detections + CLIP embeddings) is now fetched, decrypted, gunzipped, and cached under cacheDirectory/mldata/ — one payload file per fileID, written by rename (present means complete), never in metadata.json. A derived index clip.json (fileIDs in order + embedding length) + clip.f32 (embeddings as one Float32Array, one read no parse) is rebuilt when missing or inconsistent with the files present and appended to as payloads arrive. After each refresh the library fetches, through the metadata pool, the ML data for every known file absent from mldata/ or whose updationTime advanced, reporting via onProgress (operation: "fetchMLData") and status(). RAM holds only the id list and Float32Array; payloads are read on demand.

The /files/data/fetch fetch is shared with src/metadata-backup.ts (extracted to src/mldata-fetch.ts). The search surface (similar/searchByEmbedding) remains for #50.

Judgement call: updationTime refetch bookkeeping lives in a small mldata/fetched.json, separate from the index, because the payload files carry no updationTime and the index is defined as rebuildable purely from them.

make check green.

Model: opus-4-8

Implemented in https://git.eeqj.de/sneak/quak/pulls/65 (branch `issue-49-ml-data-store`, base `next`). Per-file ML data (face detections + CLIP embeddings) is now fetched, decrypted, gunzipped, and cached under `cacheDirectory/mldata/` — one payload file per fileID, written by rename (present means complete), never in `metadata.json`. A derived index `clip.json` (fileIDs in order + embedding length) + `clip.f32` (embeddings as one `Float32Array`, one read no parse) is rebuilt when missing or inconsistent with the files present and appended to as payloads arrive. After each refresh the library fetches, through the metadata pool, the ML data for every known file absent from `mldata/` or whose `updationTime` advanced, reporting via `onProgress` (`operation: "fetchMLData"`) and `status()`. RAM holds only the id list and `Float32Array`; payloads are read on demand. The `/files/data/fetch` fetch is shared with `src/metadata-backup.ts` (extracted to `src/mldata-fetch.ts`). The search surface (similar/searchByEmbedding) remains for https://git.eeqj.de/sneak/quak/issues/50. Judgement call: `updationTime` refetch bookkeeping lives in a small `mldata/fetched.json`, separate from the index, because the payload files carry no `updationTime` and the index is defined as rebuildable purely from them. `make check` green. Model: opus-4-8

clawbot referenced this issue from a commit

2026-09-22 17:27:32 +02:00

Fetch, store, and index per-file ML data and CLIP embeddings (closes #49)

clawbot referenced this issue from a commit

2026-09-22 17:41:31 +02:00

Fetch, store, and index per-file ML data and CLIP embeddings (closes #49)

clawbot closed this issue

2026-09-22 17:54:42 +02:00

clawbot referenced this issue from a commit

2026-09-22 17:54:43 +02:00

Fetch, store, and index per-file ML data (closes #49)

clawbot referenced this issue from a commit