Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
5034ae94e8 |
@@ -10,13 +10,12 @@ account into a deduplicated local directory tree, skipping files that already
|
||||
exist on disk and continuing past individual download failures instead of
|
||||
crashing. For each file it persists the basic metadata fields quak keeps (title,
|
||||
file type, creation and modification time, latitude, longitude, content hash),
|
||||
its update time, and the private and public magic metadata in full. It keeps any
|
||||
ML data Ente has for the file, except the CLIP embedding, and writes the file's
|
||||
keywords in place of the embedding. For an image (for a live photo, its image),
|
||||
it keeps its original's EXIF and XMP, with its dimensions for a JPEG only. For a
|
||||
video it keeps none of these three. It runs unattended from cron (see "Running
|
||||
the backup from cron"). A helper subcommand can detect and regenerate missing
|
||||
thumbnails, encrypting and uploading them back to the server.
|
||||
its update time, the private and public magic metadata in full, Ente's ML data
|
||||
for it when there is any, and, for an image (for a live photo, its image), its
|
||||
original's EXIF and XMP, with its dimensions for a JPEG only; for a video it
|
||||
keeps none of these three. It runs unattended from cron (see "Running the backup
|
||||
from cron"). A helper subcommand can detect and regenerate missing thumbnails,
|
||||
encrypting and uploading them back to the server.
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -372,11 +371,6 @@ quak/
|
||||
backup-metadata: the metadata quak keeps, as JSON
|
||||
exif.ts EXIF read from an image's bytes with exifreader
|
||||
mldata-fetch.ts fetch + decrypt per-file ML data
|
||||
clip-keywords.ts
|
||||
a file's keywords from its CLIP embedding
|
||||
clip-keyword-vectors.ts
|
||||
Ente's Discover categories with their text
|
||||
embeddings, written by bin/clip-keyword-vectors
|
||||
filename.ts safe file names from server metadata
|
||||
errors.ts error types shared across layers, and how an error
|
||||
is described
|
||||
@@ -393,9 +387,6 @@ quak/
|
||||
index.ts public library exports
|
||||
bin/
|
||||
quak.ts CLI entrypoint (commander.js)
|
||||
clip-keyword-vectors, clip-keyword-vectors.py
|
||||
write src/clip-keyword-vectors.ts and a test
|
||||
fixture (see Source attribution)
|
||||
examples/
|
||||
download-albums.ts
|
||||
download every album's photos and metadata
|
||||
@@ -792,11 +783,9 @@ as it was and goes into `failures.json` (see "Backup layout").
|
||||
|
||||
`backup-metadata <dir>` writes `account.json` and, for each collection, a
|
||||
directory `collections/<id>-<name>/` holding `_collection.json` and a
|
||||
`<fileID>.json` for each of its files. Each `<fileID>.json` holds the file's ML
|
||||
data and keywords as `quak backup` writes them (see "Backup layout"). The
|
||||
collection's name has unsafe characters replaced and is cut short between two
|
||||
characters so that the directory's name fits in 255 bytes; the ID keeps the
|
||||
directories apart.
|
||||
`<fileID>.json` for each of its files. The collection's name has unsafe
|
||||
characters replaced and is cut short between two characters so that the
|
||||
directory's name fits in 255 bytes; the ID keeps the directories apart.
|
||||
|
||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||
fails, the error is logged, each of its files is written with the reason in an
|
||||
@@ -829,11 +818,11 @@ the smallest does not.
|
||||
below)
|
||||
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
|
||||
keeps, its update time, its private and
|
||||
public magic metadata, its ML data and
|
||||
keywords, and, for an image (for a live
|
||||
photo, its image), its original's EXIF
|
||||
and XMP, with dimensions for a JPEG
|
||||
only; none of these three for a video
|
||||
public magic metadata, its ML data, and,
|
||||
for an image (for a live photo, its
|
||||
image), its original's EXIF and XMP, with
|
||||
dimensions for a JPEG only; none of these
|
||||
three for a video
|
||||
YYYY-MM-DD.<fileID>.livephoto.json
|
||||
which of a live photo's two files is which
|
||||
collections/
|
||||
@@ -864,51 +853,20 @@ JSON holds its `id`, `collectionID`, `ownerID`, `metadata` and `updationTime`,
|
||||
and its `magicMetadata` and `pubMagicMetadata` when it has them. Update times
|
||||
are in microseconds, as Ente records them.
|
||||
|
||||
A file's JSON holds Ente's ML data for it as `mlData`, such as the faces Ente
|
||||
found, without its CLIP embedding. It holds `keywords` in place of the
|
||||
embedding.
|
||||
|
||||
The Discover section of Ente's photos app has 16 categories:
|
||||
|
||||
- Identity (no videos)
|
||||
- Screenshots (no videos)
|
||||
- QR Codes (no videos)
|
||||
- Receipts
|
||||
- Notes
|
||||
- Memes
|
||||
- Visiting Cards (no videos)
|
||||
- Babies
|
||||
- Pets
|
||||
- Selfies
|
||||
- Wallpapers (no videos)
|
||||
- Food
|
||||
- Celebrations
|
||||
- Sunset
|
||||
- Hills
|
||||
- Greenery
|
||||
|
||||
A file's keywords are the titles of the categories it falls in. A file's score
|
||||
for a category is the cosine similarity of its CLIP embedding and the text
|
||||
embedding of the category's query. As in Ente's app, the file falls in the
|
||||
category when that score is at least the category's minimum score. A video never
|
||||
falls in a category marked "no videos". The keywords are listed highest score
|
||||
first.
|
||||
|
||||
A file with no ML data has neither `mlData` nor `keywords`. A file whose ML data
|
||||
holds no usable CLIP embedding has no `keywords`. `backup-metadata` writes both
|
||||
fields the same way.
|
||||
|
||||
The backup waits for the library's ML data fetch to finish before it writes the
|
||||
JSON files. If that fetch fails, each file whose ML data is not in the cache
|
||||
gets the reason in `mlDataError` instead and counts as failed, and the next run
|
||||
fetches it again. The same goes for a file with no ML data in the cache whose ML
|
||||
data the server sent but quak could not decrypt, gunzip or parse. Its
|
||||
`mlDataError` says why, and `failures.json` records it with an error starting
|
||||
`ML data: `. quak keeps that reason in memory only and caches nothing for the
|
||||
file, so the next run asks for its ML data again. An ML data fetch that fails,
|
||||
such as one the library's refresh starts while the backup writes the JSON files,
|
||||
leaves the reasons as the last fetch that succeeded left them. The JSON files
|
||||
are rewritten on every run, so ML data that arrived since the last run appears.
|
||||
A file's JSON holds Ente's ML data for it (its faces and its CLIP embedding) as
|
||||
`mlData`, the same payload `backup-metadata` writes; a file Ente has no ML data
|
||||
for has no `mlData`. The backup waits for the library's ML data fetch to finish
|
||||
before it writes the JSON files. If that fetch fails, each file whose ML data is
|
||||
not in the cache gets the reason in `mlDataError` instead and counts as failed,
|
||||
and the next run fetches it again. The same goes for a file with no ML data in
|
||||
the cache whose ML data the server sent but quak could not decrypt, gunzip or
|
||||
parse. Its `mlDataError` says why, and `failures.json` records it with an error
|
||||
starting `ML data: `. quak keeps that reason in memory only and caches nothing
|
||||
for the file, so the next run asks for its ML data again. An ML data fetch that
|
||||
fails, such as one the library's refresh starts while the backup writes the JSON
|
||||
files, leaves the reasons as the last fetch that succeeded left them. The JSON
|
||||
files are rewritten on every run, so ML data that arrived since the last run
|
||||
appears.
|
||||
|
||||
A file's JSON holds, as `imageMetadata`, what `backup-metadata --exif` records
|
||||
from its original, or from a live photo's image: `format`, `width` and `height`
|
||||
@@ -1312,18 +1270,18 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
not already at its save path there as `photo.download()` does (and, with
|
||||
`includeThumbnails`, fetches thumbnails) through the content cache, waits for
|
||||
an ML data fetch, and rebuilds the on-disk backup tree with a durable failure
|
||||
ledger. Each file's JSON holds its ML data and keywords. An image's JSON also
|
||||
holds the EXIF and XMP of its original, and its dimensions if it is a JPEG. A
|
||||
live photo's JSON holds these for its image. A video's JSON holds no EXIF, XMP
|
||||
or dimensions. A fetched original is written straight to its save path and not
|
||||
into the cache, which then counts it as present; one the cache already held is
|
||||
copied from there. `BackupOptions`: `downloadDirectory` (falls back to the
|
||||
library's), `includeOriginals` (default `true`), `includeThumbnails` (default
|
||||
`false`), `onlyAlbumNames`, `verify` (default `false`), `onProgress`, and
|
||||
`lock`. See Backup layout above for the tree it writes, and for the albums
|
||||
`BackupResult` lists in `undecryptableAlbums` and `failedAlbums`. Whether the
|
||||
backup took the lock or its caller did, once the lock is lost the backup
|
||||
rejects with `lost the lock on <dir>: <reason>` before it starts another file.
|
||||
ledger. Each file's JSON holds its ML data and, for an image (for a live
|
||||
photo, its image), its original's EXIF and XMP, with its dimensions for a JPEG
|
||||
only; a video's JSON holds none of these three. A fetched original is written
|
||||
straight to its save path and not into the cache, which then counts it as
|
||||
present; one the cache already held is copied from there. `BackupOptions`:
|
||||
`downloadDirectory` (falls back to the library's), `includeOriginals` (default
|
||||
`true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
|
||||
(default `false`), `onProgress`, and `lock`. See Backup layout above for the
|
||||
tree it writes, and for the albums `BackupResult` lists in
|
||||
`undecryptableAlbums` and `failedAlbums`. Whether the backup took the lock or
|
||||
its caller did, once the lock is lost the backup rejects with
|
||||
`lost the lock on <dir>: <reason>` before it starts another file.
|
||||
|
||||
### Request pools
|
||||
|
||||
@@ -1406,14 +1364,6 @@ is rewritten in TypeScript in this repository. Protocol fidelity is verified
|
||||
against the upstream implementations in `web/packages/base/`,
|
||||
`mobile/apps/photos/lib/`, and `cli/`.
|
||||
|
||||
The keywords come from Ente too. `src/clip-keyword-vectors.ts` holds the
|
||||
Discover categories of Ente's photos app, from
|
||||
`mobile/apps/photos/assets/discover.json`, each with its query's text embedding
|
||||
from Ente's CLIP text model. `bin/clip-keyword-vectors` writes that file and the
|
||||
CLIP embedding of a public-domain photo that the tests use. It runs Ente's
|
||||
models in Docker, with every download pinned by sha256. It is run by hand only,
|
||||
followed by `make fmt`.
|
||||
|
||||
## For LLMs
|
||||
|
||||
If you are an LLM agent working on this repository, read and follow these
|
||||
|
||||
@@ -26,18 +26,6 @@ declares one.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-22: The JSON `quak backup` and `quak backup-metadata` write for a file
|
||||
holds `keywords` in place of the CLIP embedding in its ML data
|
||||
(https://git.eeqj.de/sneak/quak/issues/218). The keywords are the titles of
|
||||
the Discover categories of Ente's photos app that the file falls in, by the
|
||||
app's rule: the cosine similarity of the file's CLIP embedding and the text
|
||||
embedding of a category's query is at least the category's minimum score, and
|
||||
videos stay out of the categories that leave them out. `src/clip-keywords.ts`
|
||||
applies the rule. `src/clip-keyword-vectors.ts` holds the 16 categories with
|
||||
their text embeddings. `bin/clip-keyword-vectors` writes it with Ente's CLIP
|
||||
text model, and writes the CLIP embedding of a public-domain photo of a sunset
|
||||
that the tests use.
|
||||
|
||||
- 2026-10-18: `-v`/`--verbose` raises the CLI's log level once for each repeat,
|
||||
and `-q`/`--quiet` prints only fatal errors
|
||||
(https://git.eeqj.de/sneak/quak/issues/212). The levels are fatal, error,
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
#!/bin/sh
|
||||
# Writes src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json by
|
||||
# running bin/clip-keyword-vectors.py in a Python image pinned by digest, with
|
||||
# its packages pinned by hash. Run it by hand from a checkout, then `make fmt`;
|
||||
# no gate runs it. Besides the Python image, it downloads about 265 MB, most of
|
||||
# it Ente's two CLIP models.
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
# python:3.13-slim (Python 3.13.17 on Debian 13.7), 2026-10-21. The wheels
|
||||
# below are the ones for this image on linux/amd64.
|
||||
IMAGE=python@sha256:57b29a7c050c4e679e4bea572e14b3965cf4b5839de2da49d55e49ffc4024f39
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
docker run --rm -i --platform linux/amd64 \
|
||||
--user "$(id -u):$(id -g)" -e HOME=/tmp \
|
||||
-v "$ROOT:/repo" -w /repo "$IMAGE" sh -c '
|
||||
python -m venv /tmp/venv &&
|
||||
/tmp/venv/bin/pip install --quiet --disable-pip-version-check \
|
||||
--no-cache-dir --no-deps --require-hashes -r /dev/stdin &&
|
||||
/tmp/venv/bin/python bin/clip-keyword-vectors.py' <<'EOF'
|
||||
numpy==2.5.3 --hash=sha256:a5fa86b80fd24bcd1aff83ad23be44ea323de3f787be8f8b15d4a65621e25321
|
||||
onnxruntime==1.30.0 --hash=sha256:86f940afc801ea9681a4da8af84fbe95e1d9ea7d80903952cc1bfad54faad38f
|
||||
pillow==12.3.0 --hash=sha256:0847a763afefb695bc912d7c131e7e0632d4edc1d8698f58ddabec8e46b8b6d3
|
||||
tokenizers==0.23.2 --hash=sha256:41c2f84d172449b4dadb9cdc508e3e364076613c35b16e76ecfe47a60d1e3305
|
||||
EOF
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -1,158 +0,0 @@
|
||||
"""Write src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json.
|
||||
|
||||
bin/clip-keyword-vectors runs this in a pinned Python image. It downloads
|
||||
Ente's Discover list, Ente's CLIP text and image models, the CLIP tokenizer
|
||||
and a public-domain photo of a sunset, checks each against its sha256, and
|
||||
writes:
|
||||
|
||||
- src/clip-keyword-vectors.ts: each Discover category of Ente's photos app,
|
||||
with the text embedding Ente's CLIP text model gives its query;
|
||||
- test/clip-keywords-sunset.json: the CLIP embedding Ente's image model gives
|
||||
the photo, prepared as Ente's apps prepare a photo.
|
||||
|
||||
Both embeddings are scaled to length 1, as Ente's apps scale them.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import urllib.request
|
||||
|
||||
import numpy as np
|
||||
import onnxruntime
|
||||
from PIL import Image, ImageOps
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
ENTE_COMMIT = "f5ad70a860ed25d0c8057b0efb4bd6eaefe4b0c5"
|
||||
DISCOVER_PATH = "mobile/apps/photos/assets/discover.json"
|
||||
DISCOVER_URL = (
|
||||
f"https://raw.githubusercontent.com/ente-io/ente/{ENTE_COMMIT}/{DISCOVER_PATH}"
|
||||
)
|
||||
DISCOVER_SHA256 = "cf70ac7116b592f7921ca4cf67cdbb421a8e6b287fbe44b2be678b8d198e5c24"
|
||||
|
||||
TEXT_MODEL = "mobileclip_s2_text_opset18_quant.onnx"
|
||||
TEXT_MODEL_SHA256 = "d92f33dfcff83077fc2e0d3414250710efbb51795dfd89767bdbefb5fdc47322"
|
||||
IMAGE_MODEL = "mobileclip_s2_image_gelu_opset20.onnx"
|
||||
IMAGE_MODEL_SHA256 = "205a430af825e501c5138e5bb9abea942482a7a4fd4a680e98e47cf0830dce7e"
|
||||
|
||||
# The CLIP tokenizer Hugging Face publishes for OpenAI's CLIP. Its vocabulary
|
||||
# and merges are those of bpe_simple_vocab_16e6.txt, the vocabulary Ente's
|
||||
# apps tokenize with.
|
||||
TOKENIZER_URL = (
|
||||
"https://huggingface.co/openai/clip-vit-base-patch32/resolve/"
|
||||
"3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268/tokenizer.json"
|
||||
)
|
||||
TOKENIZER_SHA256 = "b556ac8c99757ffb677208af34bc8c6721572114111a6e0aaf5fa69ff0b8d842"
|
||||
|
||||
# "Sunset off the Massachusetts coast", a NOAA photo in the public domain:
|
||||
# https://commons.wikimedia.org/wiki/File:Sunset_off_the_Massachusetts_coast_-_NOAA.jpg
|
||||
PHOTO_URL = (
|
||||
"https://upload.wikimedia.org/wikipedia/commons/d/db/"
|
||||
"Sunset_off_the_Massachusetts_coast_-_NOAA.jpg"
|
||||
)
|
||||
PHOTO_SHA256 = "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3"
|
||||
|
||||
# Ente's text model takes 77 token ids, padded with 0; its image model takes
|
||||
# a 256x256 RGB image.
|
||||
TOKEN_COUNT = 77
|
||||
IMAGE_SIZE = 256
|
||||
|
||||
|
||||
def download(url, sha256):
|
||||
# Wikimedia refuses requests without a User-Agent.
|
||||
request = urllib.request.Request(url, headers={"User-Agent": "quak"})
|
||||
with urllib.request.urlopen(request) as response:
|
||||
data = response.read()
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
if digest != sha256:
|
||||
raise SystemExit(f"{url} has sha256 {digest}, not {sha256}")
|
||||
return data
|
||||
|
||||
|
||||
def ente_model(name, sha256):
|
||||
return onnxruntime.InferenceSession(
|
||||
download(f"https://models.ente.com/{name}", sha256),
|
||||
providers=["CPUExecutionProvider"],
|
||||
)
|
||||
|
||||
|
||||
def run(session, tensor):
|
||||
(output,) = session.run(None, {session.get_inputs()[0].name: tensor})
|
||||
embedding = output[0]
|
||||
return embedding / np.linalg.norm(embedding)
|
||||
|
||||
|
||||
def text_embedding(session, tokenizer, query):
|
||||
ids = tokenizer.encode(query).ids
|
||||
ids += [0] * (TOKEN_COUNT - len(ids))
|
||||
return run(session, np.array([ids], dtype=np.int32))
|
||||
|
||||
|
||||
def image_embedding(session, photo):
|
||||
image = ImageOps.exif_transpose(Image.open(io.BytesIO(photo))).convert("RGB")
|
||||
# Scale to cover IMAGE_SIZE x IMAGE_SIZE and crop the centre.
|
||||
square = ImageOps.fit(image, (IMAGE_SIZE, IMAGE_SIZE), Image.Resampling.BILINEAR)
|
||||
pixels = np.asarray(square, dtype=np.float32) / 255
|
||||
return run(session, pixels.transpose(2, 0, 1)[np.newaxis])
|
||||
|
||||
|
||||
def numbers(embedding):
|
||||
return ", ".join(str(value) for value in embedding)
|
||||
|
||||
|
||||
def write_categories(prompts, embeddings):
|
||||
lines = [
|
||||
"// Written by bin/clip-keyword-vectors; do not edit by hand.",
|
||||
"//",
|
||||
"// The Discover categories of Ente's photos app, in their order, from",
|
||||
f"// {DISCOVER_PATH} in https://github.com/ente-io/ente",
|
||||
f"// at commit {ENTE_COMMIT}. Each has the text",
|
||||
f"// embedding Ente's CLIP text model ({TEXT_MODEL})",
|
||||
"// gives its query, scaled to length 1. `showVideo` is false for a category",
|
||||
"// that leaves out videos.",
|
||||
"",
|
||||
"export const DISCOVER_CATEGORIES = [",
|
||||
]
|
||||
for prompt, embedding in zip(prompts, embeddings):
|
||||
lines += [
|
||||
" {",
|
||||
f" title: {json.dumps(prompt['title'])},",
|
||||
f" query: {json.dumps(prompt['query'])},",
|
||||
f" minScore: {json.dumps(prompt['minScore'])},",
|
||||
f" showVideo: {json.dumps(prompt.get('showVideo', True))},",
|
||||
f" textEmbedding: [{numbers(embedding)}],",
|
||||
" },",
|
||||
]
|
||||
lines.append("];")
|
||||
with open("src/clip-keyword-vectors.ts", "w", encoding="utf-8") as file:
|
||||
file.write("\n".join(lines) + "\n")
|
||||
|
||||
|
||||
def write_fixture(embedding):
|
||||
with open("test/clip-keywords-sunset.json", "w", encoding="utf-8") as file:
|
||||
file.write(
|
||||
"{\n"
|
||||
f' "photo": {json.dumps(PHOTO_URL)},\n'
|
||||
f' "sha256": {json.dumps(PHOTO_SHA256)},\n'
|
||||
f' "embedding": [{numbers(embedding)}]\n'
|
||||
"}\n"
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
prompts = json.loads(download(DISCOVER_URL, DISCOVER_SHA256))["prompts"]
|
||||
tokenizer = Tokenizer.from_str(
|
||||
download(TOKENIZER_URL, TOKENIZER_SHA256).decode("utf-8")
|
||||
)
|
||||
text_model = ente_model(TEXT_MODEL, TEXT_MODEL_SHA256)
|
||||
embeddings = [
|
||||
text_embedding(text_model, tokenizer, prompt["query"]) for prompt in prompts
|
||||
]
|
||||
write_categories(prompts, embeddings)
|
||||
|
||||
image_model = ente_model(IMAGE_MODEL, IMAGE_MODEL_SHA256)
|
||||
write_fixture(image_embedding(image_model, download(PHOTO_URL, PHOTO_SHA256)))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+6
-14
@@ -14,9 +14,9 @@
|
||||
// YYYY/YYYY-MM/YYYY-MM-DD/
|
||||
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path)
|
||||
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar, with
|
||||
// the file's ML data, its
|
||||
// keywords, and its original's
|
||||
// EXIF, XMP and dimensions
|
||||
// the file's ML data and its
|
||||
// original's EXIF, XMP and
|
||||
// dimensions
|
||||
// collections/<name>/<title> symlink to the original
|
||||
// collections/<name>.json per-collection metadata
|
||||
// account.json the account's email and user ID
|
||||
@@ -77,7 +77,6 @@ import { readFile } from "node:fs/promises";
|
||||
import { dirname, extname, join, relative, resolve } from "node:path";
|
||||
import lockfile from "proper-lockfile";
|
||||
|
||||
import { clipKeywords } from "./clip-keywords.js";
|
||||
import { removeLeftoverTempFiles } from "./download/index.js";
|
||||
import { describeError } from "./errors.js";
|
||||
import {
|
||||
@@ -498,9 +497,8 @@ const imageMetadataFor = async (
|
||||
}
|
||||
};
|
||||
|
||||
// The file's JSON: its basic fields, its magic metadata, its ML data without
|
||||
// the CLIP embedding and the keywords that embedding gives, or the reason the
|
||||
// ML data is missing, and its image metadata.
|
||||
// The file's JSON: its basic fields, its magic metadata, its ML data or the
|
||||
// reason the ML data is missing, and its image metadata.
|
||||
const writeSidecar = (
|
||||
path: string,
|
||||
file: EnteFile,
|
||||
@@ -516,13 +514,7 @@ const writeSidecar = (
|
||||
};
|
||||
if (file.magicMetadata) meta.magicMetadata = file.magicMetadata;
|
||||
if (file.pubMagicMetadata) meta.pubMagicMetadata = file.pubMagicMetadata;
|
||||
if (ml.mlData) {
|
||||
const { clip: _clip, ...mlDataWithoutClip } = ml.mlData;
|
||||
meta.mlData = mlDataWithoutClip;
|
||||
const isVideo = file.metadata.fileType === "video";
|
||||
const keywords = clipKeywords(ml.mlData, isVideo);
|
||||
if (keywords) meta.keywords = keywords;
|
||||
}
|
||||
if (ml.mlData) meta.mlData = ml.mlData;
|
||||
if (ml.mlDataError) meta.mlDataError = ml.mlDataError;
|
||||
if (image.imageMetadata) meta.imageMetadata = image.imageMetadata;
|
||||
if (image.imageMetadataError) {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,57 +0,0 @@
|
||||
// Keywords for a file from its CLIP embedding.
|
||||
//
|
||||
// Ente stores no keywords. A file's ML data holds a CLIP embedding, the 512
|
||||
// numbers Ente's image model gave the photo, and Ente's photos app sorts files
|
||||
// into the categories of its Discover section with it: a file falls in a
|
||||
// category when the cosine similarity of its CLIP embedding and the text
|
||||
// embedding of the category's query is at least the category's `minScore`,
|
||||
// unless the file is a video and the category leaves out videos. A file's
|
||||
// keywords are the titles of the categories it falls in. The categories and
|
||||
// their text embeddings are in `clip-keyword-vectors.ts`.
|
||||
|
||||
import { DISCOVER_CATEGORIES } from "./clip-keyword-vectors.js";
|
||||
import { clipEmbedding } from "./library/mldata.js";
|
||||
import type { MLData } from "./mldata-fetch.js";
|
||||
|
||||
// How many numbers a CLIP embedding from Ente's image model has.
|
||||
const EMBEDDING_LENGTH = 512;
|
||||
|
||||
const norm = (vector: number[]): number => Math.hypot(...vector);
|
||||
|
||||
const cosineSimilarity = (a: number[], b: number[]): number => {
|
||||
let dot = 0;
|
||||
a.forEach((value, i) => {
|
||||
dot += value * (b[i] ?? 0);
|
||||
});
|
||||
return dot / (norm(a) * norm(b));
|
||||
};
|
||||
|
||||
// The titles of the Discover categories a file falls in, highest score first,
|
||||
// or undefined when its ML data has no CLIP embedding of 512 finite numbers
|
||||
// that are not all zero. Tests pass their own `categories`.
|
||||
export const clipKeywords = (
|
||||
mlData: MLData,
|
||||
isVideo: boolean,
|
||||
categories = DISCOVER_CATEGORIES,
|
||||
): string[] | undefined => {
|
||||
const embedding = clipEmbedding(mlData);
|
||||
if (
|
||||
!embedding ||
|
||||
embedding.length !== EMBEDDING_LENGTH ||
|
||||
norm(embedding) === 0
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const matches: { title: string; score: number }[] = [];
|
||||
for (const category of categories) {
|
||||
if (isVideo && !category.showVideo) continue;
|
||||
const score = cosineSimilarity(embedding, category.textEmbedding);
|
||||
if (score >= category.minScore) {
|
||||
matches.push({ title: category.title, score });
|
||||
}
|
||||
}
|
||||
return matches
|
||||
.sort((a, b) => b.score - a.score)
|
||||
.map((match) => match.title);
|
||||
};
|
||||
@@ -61,7 +61,7 @@ export interface MLIndex {
|
||||
// Pull the CLIP embedding out of a payload, or undefined when it is absent or
|
||||
// misshapen. Kept strict so a bad payload is skipped rather than corrupting the
|
||||
// packed index.
|
||||
export const clipEmbedding = (payload: MLData): number[] | undefined => {
|
||||
const clipEmbedding = (payload: MLData): number[] | undefined => {
|
||||
const clip = payload.clip;
|
||||
if (typeof clip !== "object" || clip === null) return undefined;
|
||||
const embedding = (clip as { embedding?: unknown }).embedding;
|
||||
|
||||
+2
-12
@@ -1,7 +1,6 @@
|
||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import * as jpeg from "jpeg-js";
|
||||
import { clipKeywords } from "./clip-keywords.js";
|
||||
import type { Client } from "./client.js";
|
||||
import { describeError } from "./errors.js";
|
||||
import { readExifTags } from "./exif.js";
|
||||
@@ -98,8 +97,7 @@ const extractExif = async (
|
||||
|
||||
// Dump every decrypted metadata layer the account holds into a directory tree
|
||||
// of plain JSON: account, per-collection, and per-file records including the
|
||||
// private and public magic metadata and the ML data, with the keywords of its
|
||||
// CLIP embedding in place of the embedding (see clip-keywords.ts). Collections
|
||||
// private and public magic metadata and (by default) the ML data. Collections
|
||||
// and files are enumerated from the library's cache, which the caller refreshes
|
||||
// first. Returns how many files have no ML data because their ML data request
|
||||
// failed or their ML data did not decrypt, gunzip or parse; they are still
|
||||
@@ -232,16 +230,8 @@ export const runMetadataBackup = async (
|
||||
if (file.pubMagicMetadata)
|
||||
fileMeta.pubMagicMetadata = file.pubMagicMetadata;
|
||||
|
||||
// The ML data without the CLIP embedding, and the keywords that
|
||||
// embedding gives.
|
||||
const ml = mlDataMap.get(file.id);
|
||||
if (ml) {
|
||||
const { clip: _clip, ...mlDataWithoutClip } = ml;
|
||||
fileMeta.mlData = mlDataWithoutClip;
|
||||
const isVideo = file.metadata.fileType === "video";
|
||||
const keywords = clipKeywords(ml, isVideo);
|
||||
if (keywords) fileMeta.keywords = keywords;
|
||||
}
|
||||
if (ml) fileMeta.mlData = ml;
|
||||
const mlError = mlDataErrors.get(file.id);
|
||||
if (mlError) fileMeta.mlDataError = mlError;
|
||||
|
||||
|
||||
@@ -86,17 +86,10 @@ const DAY = "2024/2024-03/2024-03-01";
|
||||
const TOAST = new TextEncoder().encode("a few seconds of the wedding toast");
|
||||
|
||||
// Ente's ML data for beach.jpg: a face it found, and the photo's CLIP
|
||||
// embedding, here that of a photo of a sunset (see clip-keywords.test.ts).
|
||||
// embedding.
|
||||
const ML_DATA = {
|
||||
face: { faces: [{ faceID: "100_1", score: 0.98 }] },
|
||||
clip: {
|
||||
embedding: JSON.parse(
|
||||
readFileSync(
|
||||
new URL("../clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
).embedding,
|
||||
},
|
||||
clip: { embedding: [0.5, 0.6, 0.7] },
|
||||
};
|
||||
|
||||
// Each file's ID, its owner, its original as uploaded, and the metadata its
|
||||
@@ -493,10 +486,7 @@ describe("the first backup of the account", () => {
|
||||
},
|
||||
updationTime: TAKEN,
|
||||
});
|
||||
// The JSON holds the ML data without its CLIP embedding, and the
|
||||
// keywords the embedding gives in its place.
|
||||
expect(beach.mlData).toEqual({ face: ML_DATA.face });
|
||||
expect(beach.keywords).toEqual(["Sunset"]);
|
||||
expect(beach.mlData).toEqual(ML_DATA);
|
||||
expect(beach.imageMetadata.exif.Make.description).toBe("Canon");
|
||||
expect(beach.imageMetadata.exif.Model.description).toBe("EOS R5");
|
||||
|
||||
@@ -507,7 +497,6 @@ describe("the first backup of the account", () => {
|
||||
metadata: { title: "toast.mp4", fileType: "video" },
|
||||
});
|
||||
expect(toast).not.toHaveProperty("mlData");
|
||||
expect(toast).not.toHaveProperty("keywords");
|
||||
expect(toast).not.toHaveProperty("imageMetadata");
|
||||
expect(toast).not.toHaveProperty("imageMetadataError");
|
||||
|
||||
|
||||
+7
-101
@@ -52,7 +52,6 @@ import { tmpdir } from "node:os";
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
|
||||
import { runBackup, type BackupLibrary } from "../../src/backup.js";
|
||||
import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import type { ContentSource } from "../../src/library/content.js";
|
||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||
@@ -947,19 +946,9 @@ describe("the backup lock", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// Also serves ML data: file 100 has `ML_PAYLOAD`, whose CLIP embedding is that
|
||||
// of a photo of a sunset (see clip-keywords.test.ts), and the other two have
|
||||
// none. While `mlError` is set, every ML data request fails with it.
|
||||
const SUNSET_EMBEDDING: number[] = JSON.parse(
|
||||
readFileSync(
|
||||
new URL("../clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
).embedding;
|
||||
const ML_PAYLOAD = {
|
||||
face: { faces: [] },
|
||||
clip: { embedding: SUNSET_EMBEDDING },
|
||||
};
|
||||
// Also serves ML data: file 100 has `ML_PAYLOAD`, the other two have none.
|
||||
// While `mlError` is set, every ML data request fails with it.
|
||||
const ML_PAYLOAD = { face: { faces: [] }, clip: { embedding: [0.5, 0.25] } };
|
||||
|
||||
class MLClient extends MockClient {
|
||||
mlError?: string;
|
||||
@@ -979,99 +968,16 @@ const fileJSON = (outDir: string, fileID: number): Record<string, unknown> =>
|
||||
JSON.parse(readFileSync(saved(outDir, `${fileID}.json`), "utf-8"));
|
||||
|
||||
describe("ML data in each file's JSON", () => {
|
||||
// The JSON holds the ML data without its CLIP embedding, and in its place
|
||||
// the keywords the embedding gives.
|
||||
it("writes a file's ML data and keywords, and no ML field for a file that has none", async () => {
|
||||
it("writes a file's ML data, and no ML field for a file that has none", async () => {
|
||||
const lib = await openLibrary(stubSource(), new MLClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
|
||||
expect(fileJSON(outDir, 100).keywords).toEqual(["Sunset"]);
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("mlData");
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("mlDataError");
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("keywords");
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
// A file's ML data can hold faces and no CLIP embedding. Its JSON then has
|
||||
// the ML data and no keywords.
|
||||
it("writes no keywords for ML data without a CLIP embedding", async () => {
|
||||
const faces = { face: { faces: [{ faceID: "101_1", score: 0.9 }] } };
|
||||
|
||||
// Gives file 101 ML data holding only faces.
|
||||
class FacesClient extends MockClient {
|
||||
async fetchMLData(args: { fileIDs: number[] }): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
const payloads = new Map<number, MLData>();
|
||||
if (args.fileIDs.includes(101)) payloads.set(101, faces);
|
||||
return { payloads, errors: new Map() };
|
||||
}
|
||||
}
|
||||
const lib = await openLibrary(stubSource(), new FacesClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(fileJSON(outDir, 101).mlData).toEqual(faces);
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("keywords");
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
// Ente's photos app keeps videos out of some Discover categories, Identity
|
||||
// among them. Here a photo and a video share one CLIP embedding, Identity's
|
||||
// own text embedding, so the photo gets the keyword Identity and the video
|
||||
// does not.
|
||||
it("keeps a video out of a category that leaves out videos", async () => {
|
||||
const identity = DISCOVER_CATEGORIES.find(
|
||||
(c) => c.title === "Identity",
|
||||
)!;
|
||||
expect(identity.showVideo).toBe(false);
|
||||
const toast = file(102, 1, "toast.mp4");
|
||||
const video = {
|
||||
...toast,
|
||||
metadata: { ...toast.metadata, fileType: "video" as const },
|
||||
};
|
||||
|
||||
// Serves beach.jpg and the video in the Vacation album, and gives
|
||||
// each ML data holding only that embedding.
|
||||
class VideoClient extends MockClient {
|
||||
override async filesSince(args: {
|
||||
collectionID: number;
|
||||
}): Promise<FilesPage> {
|
||||
const files =
|
||||
args.collectionID === 1
|
||||
? [file(100, 1, "beach.jpg"), video]
|
||||
: [];
|
||||
return { files, deleted: [], cursor: 1 };
|
||||
}
|
||||
async fetchMLData(args: { fileIDs: number[] }): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
const payloads = new Map<number, MLData>();
|
||||
for (const id of args.fileIDs) {
|
||||
payloads.set(id, {
|
||||
clip: { embedding: identity.textEmbedding },
|
||||
});
|
||||
}
|
||||
return { payloads, errors: new Map() };
|
||||
}
|
||||
}
|
||||
const lib = await openLibrary(stubSource(), new VideoClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(fileJSON(outDir, 100).keywords).toContain("Identity");
|
||||
expect(fileJSON(outDir, 102)).toHaveProperty("keywords");
|
||||
expect(fileJSON(outDir, 102).keywords).not.toContain("Identity");
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
@@ -1087,7 +993,7 @@ describe("ML data in each file's JSON", () => {
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
// File 100's ML data was cached by the first run and is kept.
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
expect(fileJSON(outDir, 100)).not.toHaveProperty("mlDataError");
|
||||
for (const fileID of [101, 200]) {
|
||||
expect(fileJSON(outDir, fileID).mlDataError).toBe(
|
||||
@@ -1120,7 +1026,7 @@ describe("ML data in each file's JSON", () => {
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(result.skipped).toBe(3);
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
for (const fileID of [100, 101, 200]) {
|
||||
expect(fileJSON(outDir, fileID)).not.toHaveProperty("mlDataError");
|
||||
}
|
||||
|
||||
@@ -41,7 +41,6 @@ import { beforeAll, afterAll, describe, expect, it, vi } from "vitest";
|
||||
import { init, toBase64, encryptBlob } from "../../src/crypto/index.js";
|
||||
import * as jpegJs from "jpeg-js";
|
||||
import { Client } from "../../src/client.js";
|
||||
import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import {
|
||||
runMetadataBackup,
|
||||
@@ -57,8 +56,8 @@ import {
|
||||
type MetaMockState,
|
||||
} from "../ente-server.js";
|
||||
|
||||
// One file per ML data request, so each file of the mock account is fetched in
|
||||
// a request of its own and one of them can fail on its own.
|
||||
// One file per ML data request, so the two files of the mock account are
|
||||
// fetched in two requests and one of them can fail on its own.
|
||||
vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../../src/mldata-fetch.js")>()),
|
||||
MLDATA_BATCH_SIZE: 1,
|
||||
@@ -67,21 +66,6 @@ vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
|
||||
const TEST_EMAIL = "metabackup@example.com";
|
||||
const TEST_PASSWORD = "metapass";
|
||||
|
||||
// The CLIP embedding of a photo of a sunset (see clip-keywords.test.ts), which
|
||||
// the ML data of beach.jpg holds.
|
||||
const SUNSET_EMBEDDING: number[] = JSON.parse(
|
||||
readFileSync(
|
||||
new URL("../clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
).embedding;
|
||||
|
||||
// Identity, a Discover category of Ente's photos app that leaves out videos.
|
||||
const IDENTITY = DISCOVER_CATEGORIES.find((c) => c.title === "Identity")!;
|
||||
|
||||
// ML data holding faces and no CLIP embedding, which file 302 has.
|
||||
const FACES_ONLY = { face: { faces: [{ faceID: "face-def", score: 0.9 }] } };
|
||||
|
||||
let mock: MetaMockState;
|
||||
let testDir: string;
|
||||
|
||||
@@ -220,65 +204,6 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
updationTime: 1710000000000000,
|
||||
};
|
||||
|
||||
// A file in coll 10 with the ML data given. `fileType` is Ente's number: 0
|
||||
// an image, 1 a video.
|
||||
const vacationFile = (
|
||||
id: number,
|
||||
title: string,
|
||||
fileType: number,
|
||||
mlData: object,
|
||||
) => {
|
||||
const fileKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { ciphertext: encKey, nonce } = encryptSecretbox(fileKey, ck1);
|
||||
const meta = encryptStreamBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
title,
|
||||
fileType,
|
||||
creationTime: 1700000000000000,
|
||||
modificationTime: 1700000000000000,
|
||||
}),
|
||||
),
|
||||
fileKey,
|
||||
);
|
||||
const ml = encryptBlob(
|
||||
new Uint8Array(gzipSync(Buffer.from(JSON.stringify(mlData)))),
|
||||
fileKey,
|
||||
);
|
||||
return {
|
||||
raw: {
|
||||
id,
|
||||
collectionID: 10,
|
||||
ownerID: 42,
|
||||
encryptedKey: toBase64(encKey),
|
||||
keyDecryptionNonce: toBase64(nonce),
|
||||
metadata: {
|
||||
encryptedData: toBase64(meta.ciphertext),
|
||||
decryptionHeader: toBase64(meta.header),
|
||||
},
|
||||
file: {
|
||||
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
|
||||
},
|
||||
thumbnail: {
|
||||
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
|
||||
},
|
||||
updationTime: 1700000000000000,
|
||||
},
|
||||
mlData: {
|
||||
encryptedData: toBase64(ml.ciphertext),
|
||||
decryptionHeader: toBase64(ml.header),
|
||||
},
|
||||
};
|
||||
};
|
||||
// Files 300 and 301 in coll 10: a photo and a video whose ML data holds
|
||||
// only a CLIP embedding, Identity's own text embedding.
|
||||
const identityML = { clip: { embedding: IDENTITY.textEmbedding } };
|
||||
const identityPhoto = vacationFile(300, "passport.jpg", 0, identityML);
|
||||
const identityVideo = vacationFile(301, "passport.mp4", 1, identityML);
|
||||
// File 302 in coll 10: a photo whose ML data holds faces and no CLIP
|
||||
// embedding.
|
||||
const facesPhoto = vacationFile(302, "party.jpg", 0, FACES_ONLY);
|
||||
|
||||
// Encrypt ML data for file 100 (gzipped JSON, encrypted with file key)
|
||||
const mlPayload = JSON.stringify({
|
||||
face: {
|
||||
@@ -305,7 +230,7 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
clip: {
|
||||
version: 1,
|
||||
client: "test",
|
||||
embedding: SUNSET_EMBEDDING,
|
||||
embedding: [0.5, 0.6, 0.7],
|
||||
},
|
||||
});
|
||||
const gzipped = gzipSync(Buffer.from(mlPayload));
|
||||
@@ -345,23 +270,12 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
keyAttributes: account.keyAttributes,
|
||||
encryptedToken: account.encryptedToken,
|
||||
collections: [rawColl1, rawColl2],
|
||||
filesByCollection: {
|
||||
10: [
|
||||
rawFile1,
|
||||
identityPhoto.raw,
|
||||
identityVideo.raw,
|
||||
facesPhoto.raw,
|
||||
],
|
||||
20: [rawFile2],
|
||||
},
|
||||
filesByCollection: { 10: [rawFile1], 20: [rawFile2] },
|
||||
encryptedMLData: {
|
||||
100: {
|
||||
encryptedData: toBase64(mlCiphertext),
|
||||
decryptionHeader: toBase64(mlHeader),
|
||||
},
|
||||
300: identityPhoto.mlData,
|
||||
301: identityVideo.mlData,
|
||||
302: facesPhoto.mlData,
|
||||
},
|
||||
fileCiphertexts: { 100: encFileBody1 },
|
||||
};
|
||||
@@ -518,9 +432,7 @@ describe("quak backup-metadata", () => {
|
||||
expect(account.email).toBe(TEST_EMAIL);
|
||||
});
|
||||
|
||||
// The CLIP embedding is left out of the ML data, and the keywords it gives
|
||||
// are written in its place.
|
||||
it("fetches and decrypts ML data by default, with keywords for the CLIP embedding", async () => {
|
||||
it("fetches and decrypts ML data by default", async () => {
|
||||
const outDir = join(testDir, "ml-data");
|
||||
await runBackup(outDir);
|
||||
|
||||
@@ -540,41 +452,8 @@ describe("quak backup-metadata", () => {
|
||||
expect(fileMeta.mlData.face.faces[0].faceID).toBe("face-abc");
|
||||
expect(fileMeta.mlData.face.faces[0].score).toBeCloseTo(0.98);
|
||||
expect(fileMeta.mlData.face.faces[0].detection.box.x).toBeCloseTo(0.1);
|
||||
expect(fileMeta.mlData).not.toHaveProperty("clip");
|
||||
expect(fileMeta.keywords).toEqual(["Sunset"]);
|
||||
});
|
||||
|
||||
// Ente's photos app keeps videos out of some Discover categories, Identity
|
||||
// among them. Files 300 and 301, a photo and a video, share one CLIP
|
||||
// embedding, Identity's own text embedding, so the photo gets the keyword
|
||||
// Identity and the video does not.
|
||||
it("keeps a video out of a category that leaves out videos", async () => {
|
||||
expect(IDENTITY.showVideo).toBe(false);
|
||||
const outDir = join(testDir, "video-keywords");
|
||||
await runBackup(outDir);
|
||||
|
||||
const vacation = join(outDir, "collections", "10-Vacation");
|
||||
const fileJSON = (fileID: number) =>
|
||||
JSON.parse(readFileSync(join(vacation, `${fileID}.json`), "utf-8"));
|
||||
expect(fileJSON(300).keywords).toContain("Identity");
|
||||
expect(fileJSON(301)).toHaveProperty("keywords");
|
||||
expect(fileJSON(301).keywords).not.toContain("Identity");
|
||||
});
|
||||
|
||||
// A file's ML data can hold faces and no CLIP embedding. File 302's JSON
|
||||
// then has the ML data and no keywords.
|
||||
it("writes no keywords for ML data without a CLIP embedding", async () => {
|
||||
const outDir = join(testDir, "no-clip");
|
||||
await runBackup(outDir);
|
||||
|
||||
const fileMeta = JSON.parse(
|
||||
readFileSync(
|
||||
join(outDir, "collections", "10-Vacation", "302.json"),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(fileMeta.mlData).toEqual(FACES_ONLY);
|
||||
expect(fileMeta).not.toHaveProperty("keywords");
|
||||
expect(fileMeta.mlData.clip).toBeDefined();
|
||||
expect(fileMeta.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
|
||||
});
|
||||
|
||||
it("extracts EXIF from downloaded files when --exif is set", async () => {
|
||||
@@ -662,7 +541,7 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(ok.keywords).toEqual(["Sunset"]);
|
||||
expect(ok.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
|
||||
expect(ok.mlDataError).toBeUndefined();
|
||||
|
||||
const failed = JSON.parse(
|
||||
@@ -720,7 +599,7 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(ok.keywords).toEqual(["Sunset"]);
|
||||
expect(ok.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
|
||||
});
|
||||
|
||||
it("exits 0 when every ML data request succeeds", async () => {
|
||||
|
||||
@@ -1,109 +0,0 @@
|
||||
{
|
||||
"photo": "https://upload.wikimedia.org/wikipedia/commons/d/db/Sunset_off_the_Massachusetts_coast_-_NOAA.jpg",
|
||||
"sha256": "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3",
|
||||
"embedding": [
|
||||
-0.0021123015, -0.04437615, -0.062354308, 0.006249992, 0.022141185,
|
||||
0.011159019, 0.01227392, -0.00030532264, 0.03186437, 0.09196922,
|
||||
0.039111234, -0.037084445, -0.032962628, -0.019820206, -0.013141825,
|
||||
0.0047183544, 0.017137334, 0.046581306, 0.0047622235, 0.04982521,
|
||||
0.023128154, 0.032080866, 0.037709303, 0.019241406, 0.0032100384,
|
||||
-0.014275843, 0.049337693, 0.025953151, 0.014263995, -0.0062328726,
|
||||
-0.040504098, -0.015083303, -0.008123875, -0.006233639, -0.05352683,
|
||||
-0.0048083956, 0.0763486, -0.013155067, 0.025114346, 0.022016674,
|
||||
0.0001588023, -0.010341572, -0.0037518279, -0.019181008, 0.04549337,
|
||||
-0.0010725886, 0.004636866, 0.017177915, -0.0117317885, 0.0026097358,
|
||||
0.030738998, 0.013158572, -0.08856403, -0.0121391015, 0.011477229,
|
||||
-0.0033104012, 0.035524756, -0.018607879, -0.0071739843, -0.031906743,
|
||||
0.0056713386, -0.0044195866, 0.00292471, 0.00061686034, -0.043075223,
|
||||
0.09040427, 0.001522537, -0.011445652, 0.009609886, -0.0053201183,
|
||||
-0.0057452377, 0.00972853, -0.0024397348, -0.021255083, 0.015893811,
|
||||
-0.006020548, -0.09428023, 0.014277583, -0.017975299, -0.011878315,
|
||||
0.011825521, 0.00014203975, -0.0020437292, -0.05305343, -0.024840152,
|
||||
-0.011336872, -0.002796541, -0.00036171873, 0.004530499, 0.044486605,
|
||||
0.021602718, -0.01292857, 0.024530916, 0.019559948, -0.039928272,
|
||||
-0.0681174, 0.049101, 0.02461532, 0.0365531, -0.04070358, -0.011011939,
|
||||
-0.0027188526, -0.020370876, 0.012904856, 0.043157287, -0.051658224,
|
||||
-0.3726012, 0.019549074, -0.0499252, -0.04800559, -0.011493859,
|
||||
0.02893256, 0.060341954, 0.0013929688, -0.051283866, -0.024286803,
|
||||
0.044124287, -0.005995806, -0.062817656, 0.007369614, -0.021470634,
|
||||
-0.028147545, 0.00411659, 0.04234773, 0.03787647, 0.008579682,
|
||||
-0.038568925, -0.027498025, -0.028416378, 0.0022012168, 0.010906191,
|
||||
-0.013745787, 0.018267838, 0.013531097, -0.061559845, 0.004682517,
|
||||
-0.005993303, 0.0028113858, -0.0028286583, 0.0026029332, -0.06693403,
|
||||
-0.0072381413, -0.02419584, 0.0037009937, -0.0011296495, -0.030310456,
|
||||
0.0005999609, 0.009945354, 0.024915686, 0.038672045, 0.023009432,
|
||||
-0.05093051, -0.09101409, -0.06686167, -0.06211481, -0.022600474,
|
||||
0.0024902367, -0.0067171026, -0.016769778, -0.0032803628, 0.008623083,
|
||||
-0.0016110836, 0.014925073, 0.033811048, 0.012829449, -0.052575592,
|
||||
0.03728281, -0.0051201633, 0.021238064, -0.031086605, -0.0043402985,
|
||||
0.25054494, -0.028396713, -0.010725523, -0.023445267, 0.021751683,
|
||||
0.003130971, 0.012270548, -0.014836475, 0.014466885, 0.0021478906,
|
||||
-0.088153616, 0.015356315, 0.039639276, -0.040283255, 0.014923924,
|
||||
-0.029303031, -0.004475201, 0.047299225, -0.06726505, -0.012491676,
|
||||
-0.002080582, 0.03506341, 0.04146718, 0.0074114986, -0.009582633,
|
||||
0.006623749, -0.05454898, -0.0034387247, -0.014780821, 0.03698328,
|
||||
-0.034441486, 0.22032379, -0.021792699, 0.008001907, 0.0341019,
|
||||
0.0025784625, -0.0071689896, -0.014151225, -0.050547987, -0.030372301,
|
||||
-0.03288388, -0.042987447, -0.0033656165, 0.021502906, -0.0030374376,
|
||||
-0.024413, -0.03506603, 0.030891743, 0.0069994032, 0.004887366,
|
||||
0.030707076, 0.0065348223, 0.0008240555, 0.016809713, -0.024151647,
|
||||
-0.0040458064, 0.044567548, 0.03908212, -0.061782435, 0.0026002831,
|
||||
0.02003242, -0.0115451, -0.045135964, -0.0663542, 0.005240975,
|
||||
0.0010777109, -0.009276002, -0.011919081, 0.027654119, -0.013634551,
|
||||
0.0129154315, -0.013195512, 0.040571272, -0.099290505, -0.04656693,
|
||||
-0.20979854, 0.04181004, -0.06769166, -0.04121946, 0.018260444,
|
||||
0.005617506, -0.15694222, 0.032472793, 0.014189598, 0.008345103,
|
||||
-0.041118417, 0.0074527003, -0.011494104, 0.004906474, -0.0027100458,
|
||||
0.0009994594, -0.041061398, -0.045274835, 0.015187793, 0.034275036,
|
||||
-0.02813289, 0.02098019, -0.015043198, 0.03716667, 0.013542642,
|
||||
-0.008104835, 0.014965486, -0.027978558, 0.021163663, -0.006942737,
|
||||
-0.034969814, -0.038961593, 0.016802894, -0.012318175, -0.047821593,
|
||||
-0.01933216, 0.0064432374, 0.0322358, -0.020437784, 0.02659447,
|
||||
-0.03022022, -0.019443465, -0.094178624, -0.017009107, 0.03860671,
|
||||
-0.00023695534, -0.033089854, -0.014677462, 0.012356399, 0.018040499,
|
||||
-0.19360113, 0.011710817, -0.019871691, 0.020124774, 0.024948105,
|
||||
-0.024573864, -0.028349072, -0.041731454, 0.032646254, 0.015098165,
|
||||
0.0018447012, -0.01129152, -0.02481687, 0.036048, 0.20797811,
|
||||
0.04317757, -0.017239755, 0.033431605, -0.011384422, 0.0099174455,
|
||||
-0.19739223, -0.008901062, -0.056426458, 0.014591864, 0.0044474266,
|
||||
0.008209326, -0.012241819, 0.042553645, -0.021552453, 0.008611734,
|
||||
0.048944127, -0.013872049, 0.019179989, 0.008183264, -0.053223096,
|
||||
-0.027327595, -0.0033542586, -0.012353858, -0.010941209, 0.010212546,
|
||||
0.029714134, -0.011385593, -0.0013529479, 0.014505385, 0.01148663,
|
||||
0.012613636, -0.0074430504, -0.014008101, -0.028146038, -0.042778917,
|
||||
0.02143165, 0.016812252, -0.06286215, 0.02085532, -0.035128277,
|
||||
0.0241282, 0.01397908, -0.034992814, 0.013153766, 0.0068236147,
|
||||
0.013594184, -0.06859393, 0.051849406, 0.029675119, -0.0037699796,
|
||||
-0.00013216575, 0.02916647, 0.031859048, -0.015006051, 0.0001439966,
|
||||
0.0026584554, -0.0046948595, 0.039754897, 0.034873154, 0.0012870439,
|
||||
-0.011516082, -0.074089095, 0.019241294, 0.00076195505, 0.0027334464,
|
||||
-0.036206458, -0.0011109262, -0.048838835, -0.00204685, 0.01179991,
|
||||
0.0068136845, 0.009852495, 0.017308988, 0.03159545, 0.058803353,
|
||||
0.038116634, 0.008836143, -0.010460637, -0.010994099, -0.0012926942,
|
||||
-0.048370842, 0.037696566, -0.08093239, -0.0041242186, -0.005866373,
|
||||
0.02048607, -0.12549639, 0.011705739, -0.0021425558, 0.018078592,
|
||||
-0.038228087, 0.004245971, -0.11560507, 0.01381203, -0.02734893,
|
||||
0.029595908, 0.101082794, -0.042842254, -0.08397301, -0.1546237,
|
||||
-0.037624564, 0.0069606556, -0.008020479, 0.0043268185, 0.016504597,
|
||||
0.044554908, -0.05240209, 0.020519335, -0.050994243, -0.019861093,
|
||||
0.022066997, -0.0188911, 0.012919696, -0.022872692, -0.012961374,
|
||||
-0.020071112, 0.0074553066, 0.06402014, 0.04065211, 0.014139608,
|
||||
0.06221546, -0.02574805, 0.010114545, 0.020619567, 0.032264773,
|
||||
-0.009148828, 0.022849955, -0.016593235, -0.032456633, -0.006737303,
|
||||
-0.0037493357, -0.065912865, -0.0024181998, -0.0037048983, -0.019304488,
|
||||
0.009085037, 0.057529982, -0.0378338, 0.01792042, 0.01885012,
|
||||
-0.0043600523, 0.017382368, -0.027792772, 0.006749118, -0.0053297766,
|
||||
-0.011628587, 0.015736576, 0.024801968, 0.011543913, -0.01159873,
|
||||
-0.000759976, 0.042183932, 0.05764386, -0.06422416, 0.010577374,
|
||||
-0.02261249, 0.02282793, -0.05141075, -0.11287078, -0.020670144,
|
||||
-0.020953743, 0.030885734, -0.012597732, -0.121472664, 0.028447151,
|
||||
-0.0415664, -0.010061708, 0.06441338, 0.0271207, -0.0016693748,
|
||||
-0.0045694243, 0.020342957, 0.030528478, 0.007129517, -0.0029600926,
|
||||
-0.007061397, 0.06036486, -0.004736018, -0.0090786135, -0.07601116,
|
||||
-0.039556272, -0.051269546, -0.019473685, 0.029858151, 0.028464556,
|
||||
0.012424476, 0.013742254, 0.008742927, 0.006065751, -0.009892663,
|
||||
0.07699999, -0.003443197, 0.0027863425, 0.029197995, 0.0211013,
|
||||
0.01098379, 0.016684106, -0.028917195, 0.008122037, 0.0265241,
|
||||
-0.04461223
|
||||
]
|
||||
}
|
||||
@@ -1,129 +0,0 @@
|
||||
// `clipKeywords`: the keywords quak writes for a file in place of its CLIP
|
||||
// embedding.
|
||||
//
|
||||
// Ente's ML data for a file holds a CLIP embedding, 512 numbers its image
|
||||
// model gave the photo. Ente's photos app puts a file in a category of its
|
||||
// Discover section when the cosine similarity of that embedding and the text
|
||||
// embedding of the category's query is at least the category's minimum score,
|
||||
// and keeps videos out of some categories. `clipKeywords` applies that rule
|
||||
// and returns the titles of the categories a file falls in, highest score
|
||||
// first; `quak backup` and `quak backup-metadata` write them as `keywords`.
|
||||
|
||||
import { readFileSync } from "node:fs";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { DISCOVER_CATEGORIES } from "../src/clip-keyword-vectors.js";
|
||||
import { clipKeywords } from "../src/clip-keywords.js";
|
||||
|
||||
// 512 numbers: the ones given, then zeros.
|
||||
const vector = (...start: number[]): number[] => [
|
||||
...start,
|
||||
...new Array<number>(512 - start.length).fill(0),
|
||||
];
|
||||
|
||||
const withEmbedding = (embedding: unknown) => ({
|
||||
face: { faces: [] },
|
||||
clip: { embedding },
|
||||
});
|
||||
|
||||
// Two made-up categories, each with a text embedding along one axis. The
|
||||
// embedding [3, 4] has a cosine similarity of 3/5 with "Across" and 4/5 with
|
||||
// "Up". Videos are kept out of "Across".
|
||||
const ACROSS = {
|
||||
title: "Across",
|
||||
query: "across",
|
||||
minScore: 0.6,
|
||||
showVideo: false,
|
||||
textEmbedding: vector(1),
|
||||
};
|
||||
const UP = {
|
||||
title: "Up",
|
||||
query: "up",
|
||||
minScore: 0.5,
|
||||
showVideo: true,
|
||||
textEmbedding: vector(0, 1),
|
||||
};
|
||||
|
||||
describe("clipKeywords", () => {
|
||||
it("gives each category whose minimum score the similarity reaches, highest score first", () => {
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(3, 4)), false, [ACROSS, UP]),
|
||||
).toEqual(["Up", "Across"]);
|
||||
});
|
||||
|
||||
it("leaves out a category when the similarity is just below its minimum score", () => {
|
||||
const stricter = { ...ACROSS, minScore: 0.6 + 1e-9 };
|
||||
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(3, 4)), false, [stricter, UP]),
|
||||
).toEqual(["Up"]);
|
||||
});
|
||||
|
||||
it("keeps a video out of a category that leaves out videos", () => {
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(3, 4)), true, [ACROSS, UP]),
|
||||
).toEqual(["Up"]);
|
||||
});
|
||||
|
||||
it("gives an empty list for an embedding that falls in no category", () => {
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(0, 0, 1)), false, [ACROSS, UP]),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["no CLIP embedding", { face: { faces: [] } }],
|
||||
["a CLIP part that is not an object", { clip: null }],
|
||||
["a CLIP part without an embedding", { clip: {} }],
|
||||
["an embedding of 511 numbers", withEmbedding(vector(3, 4).slice(1))],
|
||||
["an embedding with NaN in it", withEmbedding(vector(3, NaN))],
|
||||
["an embedding with Infinity in it", withEmbedding(vector(Infinity))],
|
||||
["an embedding of strings", withEmbedding(vector(3, 4).map(String))],
|
||||
["an embedding of zeros", withEmbedding(vector())],
|
||||
])("gives no keywords for ML data with %s", (_, mlData) => {
|
||||
expect(clipKeywords(mlData, false, [ACROSS, UP])).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
// The CLIP embedding Ente's image model gives a public-domain NOAA photo of a
|
||||
// sunset, made by bin/clip-keyword-vectors from the photo at `photo`.
|
||||
const SUNSET: { photo: string; sha256: string; embedding: number[] } =
|
||||
JSON.parse(
|
||||
readFileSync(
|
||||
new URL("clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
|
||||
describe("Ente's Discover categories", () => {
|
||||
it("are the 16 of Ente's photos app, each with a text embedding of norm 1", () => {
|
||||
expect(DISCOVER_CATEGORIES.map((c) => c.title)).toEqual([
|
||||
"Identity",
|
||||
"Screenshots",
|
||||
"QR Codes",
|
||||
"Receipts",
|
||||
"Notes",
|
||||
"Memes",
|
||||
"Visiting Cards",
|
||||
"Babies",
|
||||
"Pets",
|
||||
"Selfies",
|
||||
"Wallpapers",
|
||||
"Food",
|
||||
"Celebrations",
|
||||
"Sunset",
|
||||
"Hills",
|
||||
"Greenery",
|
||||
]);
|
||||
for (const category of DISCOVER_CATEGORIES) {
|
||||
expect(category.textEmbedding).toHaveLength(512);
|
||||
expect(Math.hypot(...category.textEmbedding)).toBeCloseTo(1, 5);
|
||||
}
|
||||
});
|
||||
|
||||
it("give a photo of a sunset the keyword Sunset and no other", () => {
|
||||
expect(
|
||||
clipKeywords({ clip: { embedding: SUNSET.embedding } }, false),
|
||||
).toEqual(["Sunset"]);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user