sneak/quak
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions 9 Packages Projects Releases Wiki Activity

A file's JSON holds keywords in place of its CLIP embedding (closes #218)
check / check (push) Waiting to run
Details

Browse Source 
quak backup and quak backup-metadata write each file's ML data without its
CLIP embedding, and keywords in its place: the titles of the Discover
categories of Ente's photos app that the file falls in, by the app's rule.
src/clip-keywords.ts applies the rule. src/clip-keyword-vectors.ts holds the
16 categories from Ente's discover.json with the text embedding Ente's CLIP
text model gives each query. bin/clip-keyword-vectors writes that file and the
test fixture in a digest-pinned Python image, with every download pinned by
sha256; it runs by hand only. quak gains no dependency.

Model: opus-5-5
This commit is contained in:
clawbot
2026-10-22 18:36:38 +00:00
parent 4863833f0a
commit 5ec176550f
8 changed files with 2169 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+53 -26
View File
@@ -371,6 +371,11 @@ quak/
backup-metadata: the metadata quak keeps, as JSON
exif.ts EXIF read from an image's bytes with exifreader
mldata-fetch.ts fetch + decrypt per-file ML data
clip-keywords.ts
a file's keywords from its CLIP embedding
clip-keyword-vectors.ts
Ente's Discover categories with their text
embeddings, written by bin/clip-keyword-vectors
filename.ts safe file names from server metadata
errors.ts error types shared across layers, and how an error
is described
@@ -387,6 +392,9 @@ quak/
index.ts public library exports
bin/
quak.ts CLI entrypoint (commander.js)
clip-keyword-vectors, clip-keyword-vectors.py
write src/clip-keyword-vectors.ts and a test
fixture (see Source attribution)
examples/
download-albums.ts
download every album's photos and metadata
@@ -783,8 +791,9 @@ as it was and goes into `failures.json` (see "Backup layout").
`backup-metadata <dir>` writes `account.json` and, for each collection, a
directory `collections/<id>-<name>/` holding `_collection.json` and a
`<fileID>.json` for each of its files. The collection's name has unsafe
characters replaced and is cut short between two characters so that the
`<fileID>.json` for each of its files, with the file's ML data and keywords as
`quak backup` writes them (see "Backup layout"). The collection's name has
unsafe characters replaced and is cut short between two characters so that the
directory's name fits in 255 bytes; the ID keeps the directories apart.
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
@@ -818,11 +827,11 @@ the smallest does not.
below)
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
keeps, its update time, its private and
public magic metadata, its ML data, and,
for an image (for a live photo, its
image), its original's EXIF and XMP, with
dimensions for a JPEG only; none of these
three for a video
public magic metadata, its ML data and
keywords, and, for an image (for a live
photo, its image), its original's EXIF
and XMP, with dimensions for a JPEG
only; none of these three for a video
YYYY-MM-DD.<fileID>.livephoto.json
which of a live photo's two files is which
collections/
@@ -853,20 +862,30 @@ JSON holds its `id`, `collectionID`, `ownerID`, `metadata` and `updationTime`,
and its `magicMetadata` and `pubMagicMetadata` when it has them. Update times
are in microseconds, as Ente records them.
A file's JSON holds Ente's ML data for it (its faces and its CLIP embedding) as
`mlData`, the same payload `backup-metadata` writes; a file Ente has no ML data
for has no `mlData`. The backup waits for the library's ML data fetch to finish
before it writes the JSON files. If that fetch fails, each file whose ML data is
not in the cache gets the reason in `mlDataError` instead and counts as failed,
and the next run fetches it again. The same goes for a file with no ML data in
the cache whose ML data the server sent but quak could not decrypt, gunzip or
parse. Its `mlDataError` says why, and `failures.json` records it with an error
starting `ML data: `. quak keeps that reason in memory only and caches nothing
for the file, so the next run asks for its ML data again. An ML data fetch that
fails, such as one the library's refresh starts while the backup writes the JSON
files, leaves the reasons as the last fetch that succeeded left them. The JSON
files are rewritten on every run, so ML data that arrived since the last run
appears.
A file's JSON holds Ente's ML data for it as `mlData`, such as the faces Ente
found, without its CLIP embedding. In place of the embedding it holds
`keywords`, the titles of the categories of the Discover section of Ente's
photos app that the file falls in, highest score first. The categories are
Identity, Screenshots, QR Codes, Receipts, Notes, Memes, Visiting Cards, Babies,
Pets, Selfies, Wallpapers, Food, Celebrations, Sunset, Hills and Greenery. As in
Ente's app, a file falls in a category when the cosine similarity of its CLIP
embedding and the text embedding of the category's query is at least the
category's minimum score, and a video stays out of Identity, Screenshots, QR
Codes, Visiting Cards and Wallpapers. A file with no ML data has neither field,
and one whose ML data holds no usable CLIP embedding has no `keywords`.
`backup-metadata` writes both the same way.
The backup waits for the library's ML data fetch to finish before it writes the
JSON files. If that fetch fails, each file whose ML data is not in the cache
gets the reason in `mlDataError` instead and counts as failed, and the next run
fetches it again. The same goes for a file with no ML data in the cache whose ML
data the server sent but quak could not decrypt, gunzip or parse. Its
`mlDataError` says why, and `failures.json` records it with an error starting
`ML data: `. quak keeps that reason in memory only and caches nothing for the
file, so the next run asks for its ML data again. An ML data fetch that fails,
such as one the library's refresh starts while the backup writes the JSON files,
leaves the reasons as the last fetch that succeeded left them. The JSON files
are rewritten on every run, so ML data that arrived since the last run appears.
A file's JSON holds, as `imageMetadata`, what `backup-metadata --exif` records
from its original, or from a live photo's image: `format`, `width` and `height`
@@ -1270,11 +1289,11 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
not already at its save path there as `photo.download()` does (and, with
`includeThumbnails`, fetches thumbnails) through the content cache, waits for
an ML data fetch, and rebuilds the on-disk backup tree with a durable failure
ledger. Each file's JSON holds its ML data and, for an image (for a live
photo, its image), its original's EXIF and XMP, with its dimensions for a JPEG
only; a video's JSON holds none of these three. A fetched original is written
straight to its save path and not into the cache, which then counts it as
present; one the cache already held is copied from there. `BackupOptions`:
ledger. Each file's JSON holds its ML data and keywords and, for an image (for
a live photo, its image), its original's EXIF and XMP, with its dimensions for
a JPEG only; a video's JSON holds none of these three. A fetched original is
written straight to its save path and not into the cache, which then counts it
as present; one the cache already held is copied from there. `BackupOptions`:
`downloadDirectory` (falls back to the library's), `includeOriginals` (default
`true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
(default `false`), `onProgress`, and `lock`. See Backup layout above for the
@@ -1364,6 +1383,14 @@ is rewritten in TypeScript in this repository. Protocol fidelity is verified
against the upstream implementations in `web/packages/base/`,
`mobile/apps/photos/lib/`, and `cli/`.
The keywords come from Ente too. `src/clip-keyword-vectors.ts` holds the
Discover categories of Ente's photos app, from
`mobile/apps/photos/assets/discover.json`, each with the text embedding Ente's
CLIP text model gives its query. `bin/clip-keyword-vectors` writes that file,
and the CLIP embedding of a public-domain photo that the tests use, from Ente's
list and models; it needs docker, pins every download by sha256, and runs only
by hand, followed by `make fmt`.
## For LLMs
If you are an LLM agent working on this repository, read and follow these