A file's JSON holds keywords in place of its CLIP embedding (closes #218)
check / check (push) Canceled after 0s

`quak backup` and `quak backup-metadata` write each file's ML data without its
CLIP embedding, and `keywords` in its place: the titles of the Discover
categories of Ente's photos app that the file falls in, highest score first. A
file falls in a category when the cosine similarity of its CLIP embedding and
the text embedding of the category's query reaches the category's minimum
score; as in Ente's app, five categories leave out videos.

The 16 text embeddings ship in `src/clip-keyword-vectors.ts`, written by
`bin/clip-keyword-vectors` from Ente's list and CLIP text model. quak runs no
model and gains no dependency.

Disclosure: the text embeddings are committed model output, against
REPO_POLICIES' rule on generated files.

Model: opus-5-5
This commit was merged in pull request #220.
This commit is contained in:
2026-10-22 23:32:25 +02:00
parent ae0f72971d
commit 657266a60e
14 changed files with 2694 additions and 68 deletions
+12 -2
View File
@@ -1,6 +1,7 @@
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import * as jpeg from "jpeg-js";
import { clipKeywords } from "./clip-keywords.js";
import type { Client } from "./client.js";
import { describeError } from "./errors.js";
import { readExifTags } from "./exif.js";
@@ -97,7 +98,8 @@ const extractExif = async (
// Dump every decrypted metadata layer the account holds into a directory tree
// of plain JSON: account, per-collection, and per-file records including the
// private and public magic metadata and (by default) the ML data. Collections
// private and public magic metadata and the ML data, with the keywords of its
// CLIP embedding in place of the embedding (see clip-keywords.ts). Collections
// and files are enumerated from the library's cache, which the caller refreshes
// first. Returns how many files have no ML data because their ML data request
// failed or their ML data did not decrypt, gunzip or parse; they are still
@@ -230,8 +232,16 @@ export const runMetadataBackup = async (
if (file.pubMagicMetadata)
fileMeta.pubMagicMetadata = file.pubMagicMetadata;
// The ML data without the CLIP embedding, and the keywords that
// embedding gives.
const ml = mlDataMap.get(file.id);
if (ml) fileMeta.mlData = ml;
if (ml) {
const { clip: _clip, ...mlDataWithoutClip } = ml;
fileMeta.mlData = mlDataWithoutClip;
const isVideo = file.metadata.fileType === "video";
const keywords = clipKeywords(ml, isVideo);
if (keywords) fileMeta.keywords = keywords;
}
const mlError = mlDataErrors.get(file.id);
if (mlError) fileMeta.mlDataError = mlError;