A file's JSON holds keywords in place of its CLIP embedding (closes #218)
check / check (push) Canceled after 0s

`quak backup` and `quak backup-metadata` write each file's ML data without its
CLIP embedding, and `keywords` in its place: the titles of the Discover
categories of Ente's photos app that the file falls in, highest score first. A
file falls in a category when the cosine similarity of its CLIP embedding and
the text embedding of the category's query reaches the category's minimum
score; as in Ente's app, five categories leave out videos.

The 16 text embeddings ship in `src/clip-keyword-vectors.ts`, written by
`bin/clip-keyword-vectors` from Ente's list and CLIP text model. quak runs no
model and gains no dependency.

Disclosure: the text embeddings are committed model output, against
REPO_POLICIES' rule on generated files.

Model: opus-5-5
This commit was merged in pull request #220.
This commit is contained in:
2026-10-22 23:32:25 +02:00
parent ae0f72971d
commit 657266a60e
14 changed files with 2694 additions and 68 deletions
+14 -6
View File
@@ -14,9 +14,9 @@
// YYYY/YYYY-MM/YYYY-MM-DD/
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path)
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar, with
// the file's ML data and its
// original's EXIF, XMP and
// dimensions
// the file's ML data, its
// keywords, and its original's
// EXIF, XMP and dimensions
// collections/<name>/<title> symlink to the original
// collections/<name>.json per-collection metadata
// account.json the account's email and user ID
@@ -77,6 +77,7 @@ import { readFile } from "node:fs/promises";
import { dirname, extname, join, relative, resolve } from "node:path";
import lockfile from "proper-lockfile";
import { clipKeywords } from "./clip-keywords.js";
import { removeLeftoverTempFiles } from "./download/index.js";
import { describeError } from "./errors.js";
import {
@@ -497,8 +498,9 @@ const imageMetadataFor = async (
}
};
// The file's JSON: its basic fields, its magic metadata, its ML data or the
// reason the ML data is missing, and its image metadata.
// The file's JSON: its basic fields, its magic metadata, its ML data without
// the CLIP embedding and the keywords that embedding gives, or the reason the
// ML data is missing, and its image metadata.
const writeSidecar = (
path: string,
file: EnteFile,
@@ -514,7 +516,13 @@ const writeSidecar = (
};
if (file.magicMetadata) meta.magicMetadata = file.magicMetadata;
if (file.pubMagicMetadata) meta.pubMagicMetadata = file.pubMagicMetadata;
if (ml.mlData) meta.mlData = ml.mlData;
if (ml.mlData) {
const { clip: _clip, ...mlDataWithoutClip } = ml.mlData;
meta.mlData = mlDataWithoutClip;
const isVideo = file.metadata.fileType === "video";
const keywords = clipKeywords(ml.mlData, isVideo);
if (keywords) meta.keywords = keywords;
}
if (ml.mlDataError) meta.mlDataError = ml.mlDataError;
if (image.imageMetadata) meta.imageMetadata = image.imageMetadata;
if (image.imageMetadataError) {