diff --git a/README.md b/README.md index 06e77ea..57ae839 100644 --- a/README.md +++ b/README.md @@ -1,27 +1,37 @@ # quak -quak is a WTFPL-licensed TypeScript client library and CLI by -[@sneak](https://sneak.berlin) for the [Ente](https://ente.io) end-to-end -encrypted photo hosting service. It logs in, enumerates collections and files, -and downloads individual images while decrypting them on the way to disk. +quak is a WTFPL-licensed TypeScript library and command-line tool by +[@sneak](https://sneak.berlin) for reading and backing up an +[Ente](https://ente.io) photo account. -quak also includes a resilient backup command that downloads every file in the -account into a deduplicated local directory tree, skipping files that already -exist on disk and continuing past individual download failures instead of -crashing. For each file it persists the basic metadata fields quak keeps (title, -file type, creation and modification time, latitude, longitude, content hash), -its update time, the private and public magic metadata in full, Ente's ML data -for it when there is any, and, for an image (for a live photo, its image), its -original's EXIF and XMP, with its dimensions for a JPEG only; for a video it -keeps none of these three. It runs unattended from cron (see "Running the backup -from cron"). A helper subcommand can detect and regenerate missing thumbnails, -encrypting and uploading them back to the server. +Ente is a photo service that encrypts photos end to end. The server never sees a +photo, its name or its date unencrypted. Only a client that holds the account's +keys can read them. quak is such a client, built for scripts and for jobs that +run unattended. It logs in with the account's email and password, derives the +keys, and keeps a decrypted copy of the account's album and file lists in a +local cache. + +With the command line you can: + +- list the account's albums and the files in them; +- download and decrypt one file, or its thumbnail; +- back up every file in the account into a directory tree that a photo viewer + can open, and keep that backup current from cron; +- write the metadata quak keeps about the account, its albums and its files as + JSON; +- find files whose thumbnails are missing on the server, and repair them. + +The library underneath lets a program of your own read and back up the account. +It keeps the account's album and file lists on disk and answers from that copy +without asking the server. + +A backup run never prompts and never stops on one bad file. It skips files that +are already on disk, records each failure in a file, and tries the failed files +again on the next run. Every download is decrypted on its way to disk. ## Getting Started -The commands below need git, node 22 or later, and yarn. `make install`, which -builds the single binary that "Running the backup from cron" uses, also needs -[nix](https://nixos.org/download). +The commands below need `git`, Node 22 or later and `yarn`. ```bash git clone https://git.eeqj.de/sneak/quak.git @@ -29,43 +39,64 @@ cd quak yarn install yarn build -# Log in (prompts for email, password, and OTP/TOTP if required). +# Log in. quak asks for the email and the password, and for a code when +# the account needs one. yarn quak login -# List the user's collections (albums). +# List the account's collections. Ente calls an album a collection. yarn quak collections # List every file in the account. yarn quak files -# List the files in a collection. +# List the files in one collection. yarn quak files 12345 -# Download and decrypt a single file. +# Download and decrypt one file. yarn quak get 67890 --out ./photo.jpg # Back up every file in the account. yarn quak backup ./my-backup ``` -For library use, the primary surface is the cache-backed `Library`: +`yarn quak` runs the command line that `yarn build` wrote to `dist/`. To run +quak from cron, build it as one binary instead; see "Running the backup from +cron". + +### Using the library + +The library keeps a local copy of the account's album and file lists, the cache. +It brings the cache up to date with the server in rounds called refreshes. + +quak is not published to a package registry: `package.json` marks it `private`, +and `dist/` is not committed. Clone and build it as above, then add the clone to +your own program's `package.json` by its path, for example +`yarn add file:../quak`. The package's `main`, `types` and `exports` fields +point into `dist/`, so the clone must be built first. The package is an ES +module only, so a CommonJS program cannot `require` it. ```ts import { Client, Library } from "quak"; -// Log in once; the client satisfies the library's client interface. +// Log in once. The client holds the account's keys in memory. const client = await Client.login({ email: "you@example.com", password: "your-password", }); -// Open a cache-backed library. On an empty cache this awaits one server -// refresh; on an existing cache it returns immediately and refreshes in the -// background. Later refreshes start `refreshIntervalSeconds` (default 3) after -// the previous one ends. -const lib = await Library.open({ client }); +// Open the library. On an empty cache this waits for one refresh. On an +// existing cache it returns at once and refreshes in the background. +const lib = await Library.open({ + client, + // Without these two, open() also starts downloading every thumbnail in + // the account, and the originals of the favorites album and the latest + // week, into the cache. It fetches machine-learning data after each + // refresh either way. + precacheThumbnails: false, + precacheOriginals: false, +}); -// Default reads answer synchronously from the local cache — no network. +// Default reads answer from the cache and never touch the network. for (const album of lib.albums.list()) { console.log(album.collectionID, album.name); for (const photo of album.photos.list()) { @@ -73,11 +104,11 @@ for (const album of lib.albums.list()) { } } -// Fresh reads await a server round-trip, joining one already running. +// A fresh read waits for a refresh first. const { albums } = await lib.fresh(); console.log(`${albums.list().length} albums as of now`); -// Fetch (and cache) one photo's full-resolution bytes. +// Fetch one photo's original. The bytes land in the cache. const photo = lib.photos.byID({ fileID: 12345 }); if (photo) { const { path } = await photo.original(); @@ -87,46 +118,59 @@ if (photo) { await lib.close(); ``` -The lower-level `Client` (login, session serialization, and the raw -enumeration/download calls) is exported too and documented under Design below. +`Library.open({ client })` with no other options does more than the example asks +for. It refreshes every 3 seconds until `close()`. It also downloads, in the +background, every thumbnail in the account and the originals of the favorites +album and of the latest week (see "Opening a library"). The example turns those +two downloads off. After each refresh that succeeds, the library also fetches +each file's machine-learning data into the cache. That is the faces and the +other data Ente's apps compute for a photo (see "The JSON beside each +original"). No option in `LibraryOptions` turns this off. A `client` without a +`fetchMLData` method fetches none, which is how quak's own read commands open +their library. + +The `Client` under the library handles login, saved sessions and the raw +requests. The "Library reference" below covers both. ## Running the backup from cron -`quak backup` never prompts, so cron can run it. `make install` builds quak as a -single binary and copies it to `~/bin/quak`. Besides what Getting Started needs, -it needs [nix](https://nixos.org/download), even where bun is already installed: -`make install` runs `make build-bin`, which bundles the built `dist/` with bun -run through `nix-shell -p bun`. Log in once with `~/bin/quak`, as the user the -cron job will run as; the session is saved in that user's data directory (see -"Session handling"): +`quak backup` never prompts, so cron can run it. `make install` builds quak as +one binary and copies it to `~/bin/quak`. Besides what Getting Started needs, +`make install` needs `make` and [nix](https://nixos.org/download). It uses bun, +a JavaScript runtime, to bundle the built `dist/` into one executable. It runs +bun through `nix-shell -p bun`. + +Log in once with `~/bin/quak`, as the user the cron job will run as: ```bash ~/bin/quak login ``` +The session is saved in that user's data directory (see "The session file"). Then add the backup to that user's crontab with `crontab -e`. These two lines -back up the account to `~/photos-backup` at 03:30 every night, with `--verify` -on Sundays, and append all output to `~/quak-backup.log`: +back up the account to `~/photos-backup` at 03:30 every night and append all +output to `~/quak-backup.log`. Sunday's run adds `--verify`: ``` 30 3 * * 1-6 $HOME/bin/quak backup $HOME/photos-backup >> $HOME/quak-backup.log 2>&1 30 3 * * 0 $HOME/bin/quak backup --verify $HOME/photos-backup >> $HOME/quak-backup.log 2>&1 ``` -A run with `--verify` does all a plain run does, and also checks each original -already in the backup against the content hash Ente records for it, and replaces -any that do not match (see "Backup layout"). Sunday's run is the `--verify` one, -not a second job that night, because a backup that starts while another backup -of the same directory is running exits 2 without backing anything up. +A run with `--verify` does all a plain run does. It also hashes each original +already in the backup and compares the result with the content hash. The content +hash is the hash of the file's bytes that Ente recorded at upload. The run +fetches a new copy of each original that does not match, and replaces the +original once the copy matches. Sunday's run is the `--verify` run, rather than +a second job that night, because two backups of one directory cannot run at +once. The second one exits 2 without doing anything. Cron runs the job with a short `PATH`, usually `/usr/bin:/bin`, so the crontab -names quak by its full path. To find the saved session, quak needs the same -`HOME` as when you logged in, which cron sets from the password file, and on -Linux the same `XDG_DATA_HOME`: quak looks for the session in -`$XDG_DATA_HOME/quak`, or in `~/.local/share/quak` when that is not set. Cron -does not set `XDG_DATA_HOME`, so if your login shell does, set it at the top of -the crontab too. Cron does not expand variables in such a line, so give the full -path: +names quak by its full path. quak finds the saved session through `HOME`, which +cron sets from `/etc/passwd`. On Linux it also reads `XDG_DATA_HOME`: the +session is in `$XDG_DATA_HOME/quak`, or in `~/.local/share/quak` when that +variable is not set. Cron does not set `XDG_DATA_HOME`. If your login shell sets +it, set it at the top of the crontab too, as a full path, because cron does not +expand variables there: ``` XDG_DATA_HOME=/home/you/.data @@ -135,1203 +179,1344 @@ XDG_DATA_HOME=/home/you/.data On macOS the session is in `~/Library/Application Support/quak`, and only `HOME` matters. +A backup that ends with failures exits 1, and its last line says how many files +and albums failed. The next run fetches each missing original again and rebuilds +the album links. The exit codes are listed under "Exit codes", and the backup's +files under "The backup directory". + +## Command reference + +Every command has the form `quak [options] [arguments]`. The options +below belong to quak itself, and go before or after the command's name. + +### Global options + +- `--cache-dir `: where the local cache lives. Without it, each account + gets its own directory under the per-user cache directory (see "The cache"). +- `-v`, `--verbose`: log more on stderr. `-v` adds debug lines, and `-vv` adds + trace lines too (see "What quak prints"). +- `-q`, `--quiet`: print nothing on stderr but the reason a failing command + exits. `-q` with `-v` is refused with one line, + `error: option '-q, --quiet' cannot be used with option '-v, --verbose'`, and + exit status 1. +- `-V`, `--version`: print the version and exit (see "Version"). +- `-h`, `--help`: print the usage and exit. + +### The commands + +``` +quak login +quak whoami +quak logout +quak collections [--json] +quak files [collectionID] [--json] +quak get [--out ] +quak get-thumb [--out ] +quak backup [--json] [--verify] +quak backup-metadata [--exif] +quak helper list-missing-thumbnails [--json] +quak helper fix-missing-thumbnails [--file ...] [--json] +``` + +Every command but `login` and `logout` needs a saved session. Every command but +`login`, `whoami` and `logout` opens the library. How each one gets current data +from the server is described under "How a command gets current data". + +#### `quak login` + +Logs in to an account and saves the session. It reads the email from +`QUAK_EMAIL` and the password from `QUAK_PASSWORD`. For each one that is not +set, it asks on the terminal. The password shows as `*` characters while it is +typed. + +When the account has a second factor, quak asks for the code. It asks +`TOTP code: ` for a time-based code from an authenticator app. An account whose +only second factor is a passkey cannot log in with quak. When the account has +email verification on, quak asks `Email verification code: ` for the code sent +by email. The password is then used only to unlock the keys. After the email +code quak expects the login to complete. When the server asks for a second +factor at that point, login fails. + +On success it logs `Logged in as (user )` and +`Session saved to `. On failure it logs `Login failed: `, and the +exit status is 1. `login` is the only command that prompts, and its prompts go +to stdout. + +#### `quak whoami` + +Prints the saved session's account on stdout as one line of JSON: +`{"email":"you@example.com","userID":12345}`. It reads the session file only and +sends no request, so it succeeds even when the server no longer accepts the +session. + +#### `quak logout` + +Ends the session on the server and deletes the session file. Ending it on the +server makes the saved token stop working everywhere, even in a copy of the +file. With no session file, the command logs `No session found.` and exits 0. + +When the server call fails, or the session file is corrupt, the file is still +deleted. The command then logs +`Could not end the session on the server: ` and exits 1. + +`logout` does not delete the cache. It warns that the cache directory still +holds decrypted data. It knows the directory from `--cache-dir` or from a +session file it can read. Delete that directory yourself if you want the data +gone. + +#### `quak collections [--json]` + +Lists every collection in the account, in the server's order, after one request +to the server. + +Without `--json`, each line holds the collection's ID, its type and its name, +separated by tabs. A collection another account owns gets ` (shared)` after its +name. A collection's type is one of `album`, `folder`, `favorites`, +`uncategorized` and `unknown`. + +With `--json`, the output is a JSON array with one object per collection. Each +object holds `id`, `name`, `type`, `ownerID`, `isShared` and `updationTime`. +Update times are in microseconds, as Ente records them. + +#### `quak files [collectionID] [--json]` + +With a collection ID, lists the files in that collection, in the server's order. +Without one, lists every file in the account once, collection by collection. A +file that is in several collections is listed under the first collection that +holds it. + +Without `--json`, each line holds the file's ID, its type and its title, +separated by tabs. A file's type is one of `image`, `video`, `livePhoto` and +`unknown`. With `--json`, each object holds `id`, `title`, `fileType`, +`creationTime` in microseconds and `collectionID`. + +A collection ID that does not exist gives `Collection not found` and exit +status 1. + +#### `quak get [--out ]` + +Downloads and decrypts one file's original, the full-size file as uploaded. +Without `--out`, it writes the file to the working directory under the file's +title, with unsafe characters replaced (see "File names from the server"). With +`--out`, it writes to that path as given. It logs ` bytes -> `. The +original is kept in the cache as well. + +A file ID that no collection holds gives `File not found`, and the exit +status is 1. The option `--collection ` is accepted and ignored. + +A live photo is a still image and a short video that Ente stores as one file. +`get` writes both, each named after the title with its own extension, as Ente's +own clients name them: `IMG_0001.heic` and `IMG_0001.mov`. With `--out `, +the image goes to `` and the video beside it, with the same name and the +video's extension. A `` that already ends in the video's extension is +refused, because both files would land on it. + +#### `quak get-thumb [--out ]` + +Downloads and decrypts one file's thumbnail, the small JPEG Ente shows in a +grid. It writes one file, also for a live photo. Without `--out`, the name is +`thumb_` followed by the title. In all else it works as `get` does. + +#### `quak backup [--json] [--verify]` + +Backs up every file in the account to ``. The layout is described under +"The backup directory". The command downloads only the originals that are not +yet in the backup, and it goes on past a file that fails. A file is downloaded +once, however many collections hold it. An original the cache already holds is +copied from there instead. + +Progress goes to stderr: one line per original fetched, and a `FAILED` line for +each original or album that failed. The run ends with a summary of counts and, +when any file failed, a `Failed files:` list of every failed file. With +`--json`, the summary and the list are replaced by one JSON object on stdout. +Its fields are: + +- `totalFiles`: how many distinct files the run covered, however many + collections hold each; +- `downloaded`: how many originals this run fetched or copied from the cache; +- `skipped`: how many originals were already in the backup and left alone; +- `verified`, `mismatched` and `unchecked`: the counts a `--verify` run makes + (see "`--verify`"), all 0 without it; +- `failed`: how many files are in `failures.json` after this run; +- `errors`: this run's failures, one object per file, with `fileID`, `title`, + `collection` and `error`; +- `undecryptableAlbums`: the IDs of the collections quak could not decrypt; +- `failedAlbums`: the IDs of the collections whose directory or JSON could not + be written. + +With `--verify`, the run also hashes each original already in the backup and +replaces any whose bytes do not match the content hash Ente records. "The backup +directory" describes what it checks and what it changes. + +A backup refreshes the library as it starts and not again while it runs. Each +request it sends retries for longer than other commands' requests do (see +"Retries and timeouts"). + +The exit code is 0 when every original is in the backup and nothing failed. It +is 1 when files or albums failed, 2 when another backup of `` is running, +and 3 when there is no usable session (see "Exit codes"). + +#### `quak backup-metadata [--exif]` + +Writes the metadata quak keeps about the account, its collections and its files +as JSON files under ``. The layout is described under "The metadata dump +directory". Photo content is not written. With `--exif`, or its alias `--all`, +the command also fetches each file's original and records the image's own +metadata from it (see "The metadata dump directory"). Each original it fetches +stays in the cache. + +Ente's apps compute machine-learning data for each photo, such as the faces in +it (see "The JSON beside each original"). The command fetches it in requests of +up to 200 files. When a request fails, each of its files is written with the +reason in `mlDataError` instead of `mlData`, and the dump goes on. A file whose +data does not decrypt, decompress or parse also gets `mlDataError`. When any +file was written with `mlDataError`, the last line says how many and the exit +status is 1. + +A collection or file that quak cannot decrypt is left out of the dump without a +word, and does not change the exit status. + +#### `quak helper list-missing-thumbnails [--json]` + +Finds the files whose thumbnail is missing on the server. It downloads the +thumbnail of every file you own, one collection at a time, newest updated first, +so on a large account it takes a while. A thumbnail counts as missing when the +server answers with an empty body or with HTTP 404. Any other failure is logged +as `[] Could not check : <reason> (not reported as missing)`, +and the file is left out of the list. The command exits 0 even when some files +could not be checked. + +The server lets only a file's owner replace its thumbnail. A file another +account owns, in a collection shared with you, is skipped and not checked. + +Without `--json`, the command logs `<n> file(s) with missing thumbnails:` and +prints one line per file on stdout. Each line holds the file's ID, its title, +its collection and the reason, separated by tabs. With nothing missing it logs +`No missing thumbnails found.` With `--json`, the output is a JSON array; each +object holds `fileID`, `title`, `collection` and `reason`. + +#### `quak helper fix-missing-thumbnails [--file <id>...] [--json]` + +Makes a new thumbnail for each file that lacks one, and uploads it. Without +`--file`, it first finds the files as `list-missing-thumbnails` does. With +`--file`, it works on the files named and checks nothing else. + +For each file, the helper: + +1. downloads the original; +2. makes a JPEG thumbnail from it; +3. encrypts the thumbnail under the file's own key; +4. uploads it. + +The server refuses a thumbnail larger than the one it already records for the +file. The helper therefore tries five encodings in turn and uploads the first +that fits. They run from 720 pixels on the long side at JPEG quality 50 down to +160 pixels at quality 20. + +Each file ends in one of three states: + +- `fixed`: a thumbnail was uploaded. +- `skipped`: the helper cannot make a thumbnail for the file, or the server + would refuse it. This covers: + - a file another account owns; + - a file that is not an image; + - an image that is not a JPEG; + - a file whose recorded thumbnail size is 0 or unknown; + - a file for which even the smallest encoding is too large. +- `failed`: the download, the encoding or the upload failed, or an ID given with + `--file` is held by no collection. + +Without `--json`, the command ends with counts of fixed, skipped and failed +files. A `Skipped:` list and a `Failed files:` list follow when they have +entries. With `--json`, one JSON array on stdout replaces them; each object +holds `fileID`, `title`, `collection`, `status` and, for a skipped or failed +file, `reason`. When any file failed, the exit status is 1. A skipped file does +not change it. When nothing needs fixing, the command logs +`No missing thumbnails found.` and exits 0. With `--json` it then writes nothing +on stdout. + +quak refuses the whole command, before any request, when one `--file` ID is +invalid (see "IDs"). + +### How a command gets current data + +Every command that reads the account answers for the account as it is now, not +for whatever the cache last held. Each one gets current data this way: + +- `collections` asks for the collection list alone, in one request. It fetches + no collection's files and changes nothing in the cache. +- `files` with a collection ID asks for the same list, then for that one + collection's files. It changes nothing in the cache either. +- `get`, `get-thumb` and `fix-missing-thumbnails --file` look the file up in the + cache after a refresh. On an empty cache, they ask for the collection list + instead, then for one collection's files at a time, and stop at the collection + that holds the file. `fix-missing-thumbnails --file` stops once it has found + every file named. These searches store nothing in the cache. The original or + thumbnail fetched is kept in the cache. +- `files` without a collection ID, `backup`, `backup-metadata` and the two + thumbnail helpers without `--file` wait for a refresh. + +A refresh asks the server which collections changed since the last refresh and +fetches the files of each one, one collection after another. On an empty cache +that is every collection. On a large account this takes a while. Nothing is +stored until every collection is in. Meanwhile the command logs a line for each +collection fetched, such as `Fetched the files of 3 of 120 collections`. + +When a refresh or a search fails, the command logs the error on one line and +exits 1, or 3 when the server no longer accepts the saved session. A collection +or file that quak cannot decrypt does not stop a refresh or a search: the +command leaves it out and answers from what did decrypt (see "Default reads and +fresh reads"). `backup` counts it as failed (see "What quak cannot decrypt"). + +### IDs + +A collection ID or file ID given to `files`, `get`, `get-thumb` or +`fix-missing-thumbnails --file` is written in decimal digits and is at +most 9007199254740991. Anything else, such as `0x10`, `1e3` or `-1`, is refused +before any request is sent. The command logs `Invalid file ID: <arg>` or +`Invalid collection ID: <arg>` and exits 1. + +### What quak prints + +stdout holds a command's output, such as the listing or the JSON, and the +prompts of `login`. Every other line goes to stderr as a log line at one of six +levels. From the quietest, the levels are fatal, error, warning, info, debug and +trace. A run prints the lines at its own level and at every quieter one. The +level is info unless `-v`, `-vv` or `-q` says otherwise. + +- Fatal: why a command exits non-zero. A command that runs and exits non-zero + logs at least one fatal line. A command that exits 1 because files or albums + failed logs it last, saying how many failed and where to look. +- Error: something that failed without ending the run. These lines are: + - the `FAILED` lines of `backup` and `fix-missing-thumbnails`; + - their `Failed files:` lists; + - the `ML data` lines of `backup-metadata`; + - `[<collection>] Could not check <title>` from the thumbnail helpers. +- Warning: something that did not fail but needs your attention. The only + warning is the one `logout` prints: the cache directory still holds decrypted + data. +- Info: everything else quak prints by default, such as a refresh's progress, + each original a backup fetches, and the summaries. +- Debug, with `-v`: each retry of a request, with why the attempt failed and how + long quak waits. Also the start and end of each refresh and of each fetch of + machine-learning data, with the error of one that failed. +- Trace, with `-vv`: each request quak sends, as its method, its URL and the + HTTP status of the answer or the error it failed with. The URL of a thumbnail + upload is logged without its query, which holds the upload's signature. + +`-q` lowers the level to fatal: a run that exits 0 prints nothing on stderr, and +one that exits non-zero prints only its fatal lines. The prompts of `login` are +not log lines, and `-q` leaves them alone. + +Two things on stderr are not log lines. A command line that quak cannot parse, +such as a missing argument, an unknown option or `-q` with `-v`, is refused with +an `error: …` message, and quak exits 1. `-q` does not hide the message. `quak` +with no command prints the usage on stderr and exits 1. + +`yarn quak` prints yarn's own lines on stdout as well. To pipe a command's +output, run `yarn --silent quak files --json` or +`node dist/bin/quak.js files --json` instead. + +With `--json`, a command's JSON on stdout takes the place of its summary on +stderr. For `backup`, that is the `--- Backup complete ---` line, the counts and +the `Failed files:` list. For `list-missing-thumbnails`, it is +`No missing thumbnails found.` or `<n> file(s) with missing thumbnails:`. For +`fix-missing-thumbnails`, it is the `--- Done ---` line, the counts, the +`Skipped:` list and the `Failed files:` list. Every other line is logged as +without `--json`. + ### Exit codes -| Code | Meaning | -| ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `0` | The backup is complete: every original is at its save path, and no file failed. | -| `1` | The run finished with files in `failures.json` or albums it could not decrypt or write (see "Backup layout"), and its last line says how many, or it stopped on another error, printed as one line `quak: <message>`. The next run tries again. | -| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. | -| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. | +- `0`: the command succeeded. For `backup`, every original is in the backup and + no file or album failed. +- `1`: the command failed. For `backup`, the run finished with files in + `failures.json` or with albums it could not decrypt or write, and its last + line says how many. The next backup run tries again. An unexpected error is + printed as one line, `quak: <message>` (see "Error messages"). +- `2`: another backup of the same directory is running. This run sent no request + and changed nothing. Only `backup` exits 2. +- `3`: there is no usable session. None is saved, the saved one is corrupt, or + the server no longer accepts it. Run `quak login` as the user the command runs + as. -`<message>` is the error's message, followed by the message of each error that -caused it. Under Node, as `yarn quak` runs quak, `fetch` reports every failure -below HTTP as just `fetch failed` and gives the reason as its cause, so a run -that could not reach the server prints, for example, +### Error messages + +An unexpected error is printed as one line, `quak: <message>`, with no stack +trace. `<message>` is the error's own message, followed by the message of each +error that caused it, separated by `: `. Some lines a command logs itself, such +as `File <id> not found` and `Login failed: <reason>`, have no prefix. + +`yarn quak` runs quak under Node. Node's `fetch` reports every network failure +as `fetch failed`, with the reason as its cause. A run that could not reach the +server therefore prints, for example, `quak: fetch failed: connect ECONNREFUSED 1.2.3.4:443`. When the server has -several addresses and none of them could be reached, the line gives the reason -for each, separated by `, `. A file whose download failed that way is in -`failures.json` with that error, as a transient failure. So is each file with no -ML data cached when the ML data fetch failed that way, its error starting -`ML data: `. The `~/bin/quak` that `make install` builds runs under bun, and -prints bun's own wording for such a failure, with no reason after it, such as -`quak: Unable to connect. Is the computer able to access the url?` for a refused -connection. It records such a failure in `failures.json` as transient too. +several addresses and none could be reached, the line gives the reason for each +one, separated by `, `. -After a `1`, the next run fetches each missing original again, fetches the ML -data that is not cached, and rebuilds the album links. An album quak could not -decrypt, or a file in `failures.json` because quak could not decrypt it, keeps -the exit code at `1` until the server sends a copy that decrypts, or deletes it -(see "Backup layout"). An original that a `--verify` run could not read, or -could not replace with a download that matches, stays at its save path as it -was. Only a later `--verify` run checks it again: a run without `--verify` -leaves it as it is and takes the file out of `failures.json`, so that run can -exit 0. +The `~/bin/quak` that `make install` builds runs under bun. Bun gives its own +wording for such a failure, with no reason after it. A refused connection, for +example, prints +`quak: Unable to connect. Is the computer able to access the url?`. -## Examples +A file whose download failed that way goes into the backup's `failures.json` as +a transient failure, with that error. So does each file whose machine-learning +data could not be fetched that way, with an error starting `ML data: `. -`examples/download-albums.ts` downloads every album's photos and their metadata -into a directory, `photos` in the working directory unless you name another. The -build compiles it; run it after `yarn install`: +## Files quak writes -```bash -yarn build -QUAK_EMAIL=… QUAK_PASSWORD=… node dist/examples/download-albums.js [dir] -``` +### The session file -It opens the library with `precacheThumbnails` and `precacheOriginals` off, as -`quak backup` does, so the only file content it fetches is the originals it -saves. It asks on the terminal for a two-factor or email code when the account -requires one, and writes: +`quak login` writes the session to `session.json` in quak's data directory. On +Linux that is `$XDG_DATA_HOME/quak/session.json`, or +`~/.local/share/quak/session.json` when the variable is not set. On macOS it is +`~/Library/Application Support/quak/session.json`. quak creates the file with +mode `0600`, in a directory it creates with mode `0700`. -- each photo's original at its save path under `dir`, as `photo.download()` - writes it: `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, and for a live - photo its image, its video and the `.livephoto.json` file naming them -- beside each original, a JSON file named after it with `.json` added, for - example `2026-03-01.12345.jpg.json`: the photo's record (`photo.record()`) - without its cache paths, and every EXIF tag of the photo (`photo.exif()`) - under `exif` -- `albums/<collectionID>.json` for each album: its `collectionID`, its `name`, - and under `savePaths` the save paths of its photos relative to `dir`, newest - first +The file holds the account's email, its user ID, the session token and the +account's three keys, in base64 and unencrypted. Treat it as you would treat the +password. A missing file means "not logged in". A file that cannot be read back +is reported as corrupt, with the reason. Run `quak logout`, then `quak login`. -A photo in several albums is downloaded once. A second run downloads nothing and -rewrites only the JSON files whose content changed. A failed download stops the -run; running it again carries on, since every photo already saved is skipped. +A command that needs a session exits 3 when the file is missing or corrupt. It +also exits 3 when the server no longer accepts the session, and prints +`quak: the saved session is no longer valid; run "quak login"`. `quak backup` +learns this at its first refresh. By then it has written only its lock. When the +session stops working partway through a backup, each remaining file goes into +`failures.json`. That run exits 1, and the next run stops at its refresh with +status 3. -## Entrypoints +### The cache -This repository adheres to the -[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) -standard: normalized scripts in `script/` are the entrypoints for the -development workflow, and most Makefile targets are thin shims that call them. -The scripts are POSIX sh (not bash) so they run in minimal containers such as -alpine. We provide: +Without `--cache-dir`, each account has its own cache directory: -- `script/bootstrap` — install all dependencies (node/yarn if missing, then - `yarn install --frozen-lockfile`) -- `script/setup` — set up the repo for development after a fresh clone: runs - `script/bootstrap`, then `script/install-precommit` -- `script/projectname` — output the project name (our own extension); used by - `script/docker` for the image tag -- `script/build` — compile the TypeScript sources into `dist/`, stamp the - version into `dist/package.json`, then verify that the entrypoints - `package.json` declares (`main`, `types`, `bin`) are among the files the - compiler wrote, and make the CLI executable (our own extension) -- `script/version` — print the version `script/build` stamps (our own - extension); see Version below -- `script/test` — run the test suite, by building the `test` phase of the - `Dockerfile` (vitest, 90s timeout, verbose rerun on failure); requires docker -- `script/lint` — run eslint and a prettier check, by building the `lint` phase - of the `Dockerfile`; requires docker (see Linting and testing below) -- `script/fmt` — format all files with prettier (writes) -- `script/fmt-check` — check formatting on the host (read-only) -- `script/check` — run all checks: `test`, `lint`, `fmt-check` (our own - extension) -- `script/docker` — build the image, tagged via `script/projectname` -- `script/cibuild` — what CI runs: `script/bootstrap`, then `script/check`, then - the image build -- `script/precommit` — run by the git pre-commit hook (our own extension); runs - `script/lint` and `script/fmt-check` but deliberately not the tests, so the - TDD red-phase commit can land -- `script/install-precommit` — installs the git pre-commit hook (our own - extension); `make hooks` shims to it +- on Linux, `$XDG_CACHE_HOME/quak/<userID>`, or `~/.cache/quak/<userID>` when + the variable is not set; +- on macOS, `~/Library/Caches/quak/<userID>`. -`make hooks` installs the pre-commit hook that runs `script/precommit`. - -### Linting and testing - -Linting and testing are phases of the `Dockerfile`. The `lint` phase copies the -repo into a digest-pinned node image and runs eslint and `prettier --check .`; -the `test` phase does the same with the suite. `script/lint` and `script/test` -each build one phase with `docker build --no-cache --target <phase>`. Neither -script runs the tools on the host: docker is required, and that also works where -the docker daemon is remote and bind mounts are impossible. - -The last stage of the `Dockerfile` compiles the package, and it copies a file -from each phase, so it cannot be built unless lint and the tests pass. The image -build in `script/cibuild` therefore runs lint and the tests a second time, after -`script/check` has run them. - -Every `docker build` in `script/` passes `--no-cache`. On an unchanged tree -Docker would otherwise serve the lint and test steps from cache, nothing would -run, and the build would still exit 0. - -`script/fmt-check` runs prettier on the host. Its verdict matches the `lint` -phase's: prettier is pinned to an exact version, installed from `yarn.lock` -under `--frozen-lockfile` in both places, and reads `.gitignore` as its default -ignore file — which is why `.dockerignore` keeps `.gitignore` in the build -context. - -### Version - -`quak --version` reports the `version` of `dist/package.json`, which -`script/build` writes after compiling; the repo's own `package.json` keeps -`0.0.0`, and that is what the tests, which run from source, report. -`script/version` decides what is written: - -- the `VERSION` environment variable, or the `Dockerfile`'s `VERSION` build arg - (`--build-arg VERSION=...`), when one is given and not empty; -- otherwise, in a checkout with `.git`, `git describe --tags --always`: the tag - on a tagged commit; the tag, the commits since it and the short commit on a - later commit (`v1.2.3-4-gabc1234`); the short commit when no tag is reachable; -- otherwise, as in a source tarball, the version `package.json` declares. - -The build fails if the checkout has `.git` and the version still comes out -empty, `dev` or `unknown`: such a build could not be traced back to its commit. - -`.dockerignore` therefore does not leave out `.git`, so any `docker build .` of -a clone stamps the commit it was built from; a shallow clone stamps a tag only -when the cloned commit itself carries one, and otherwise the short commit. It -leaves out `.git/config`, which holds the clone's remote URL and any credential -in it, so the image carries `.git` without its config; `git describe` does not -need that file. `script/docker` (and so `make docker`) and `script/cibuild` pass -the version they resolve on the host, with `--dirty`, as the build arg, which -takes precedence. The image's `org.opencontainers.image.version` label carries -that build arg only, so a build given none leaves it empty. `make build-bin` -bundles the built `dist/`, so the single binary reports the stamped version too. - -## Rationale - -Ente is one of very few photo services with a credible end-to-end encryption -story. The shipping clients (mobile Flutter, web React, desktop Electron, and Go -CLI) work, but they are slow, buggy, and difficult to script against. The -Flutter app fails to sync reliably. The web app is heavy. The desktop app is the -web app inside a slow Electron wrapper. The Go CLI is the closest thing to a -usable tool, but it is awkward to integrate from anything that is not a shell. -The Go CLI's backup mode crashes entirely when a single file download fails, -which makes it useless as an actual backup tool. - -quak fixes these problems. This repo ships a correct, well-tested implementation -of Ente's cryptographic protocol and API surface, plus a CLI that proves the -library is enough to do real work without a UI. The backup command is resilient -by design: per-file errors are logged and the run continues. - -The longer-term goal of this project is a simple desktop client for Ente, built -on this library in Electron (or a comparable runtime), with two priorities above -everything else: correctness and stability. Performance and simplicity follow -from those. Features will be added only after the protocol layer is correct, the -local cache is reliable, and the UI is responsive on a five-year-old laptop. - -## Development workflow - -All work on quak is test-driven. No exceptions. - -1. Every change starts on a feature branch off `next`, and its pull request - targets `next`. -2. The first commit on the branch is the test suite for what is being added or - changed. Those tests must fail at that commit; the branch is red until the - implementation lands. -3. Subsequent commits add the implementation and any refactors needed to make - the tests pass. -4. A pull request can only be merged into `next` when `make check` is green. - Once it has passed review, the repository manager squash-merges it into - `next`. Only sneak merges `next` into `main`. `main` and `next` are always - green. CI runs `script/cibuild`, which builds the `Dockerfile`: its `lint` - and `test` phases, then the compile, so neither a red branch nor one that - does not compile can pass CI. -5. Tests are the canonical API documentation for this library. Every test file - is commented thoroughly enough that a reader who has never seen quak can - learn how to use it from the tests alone. Comments explain why a behavior - matters, not just what the assertion checks. -6. Test fixtures (cryptographic vectors, recorded HTTP responses, sample files) - are committed alongside their tests. Where possible they are generated by - deterministic helpers in the `test/` tree so any reviewer can reproduce them - by running the helper. -7. `git rebase -i` is allowed on a feature branch before merge to clean up the - test-then-implementation sequence into reviewable commits, but the final - history must still show tests landing before (or with) the matching - implementation. -8. The pre-commit hook installed by `make hooks` runs `script/precommit`, which - runs `script/lint` and `script/fmt-check` but not the tests, and so not the - full `make check`. This is deliberate so the TDD red-phase commit (failing - tests, no implementation yet) can land. CI executes `script/cibuild`, which - runs the tests, so a red branch still cannot reach `next`. - -## Design - -quak is a TypeScript library with a thin CLI wrapper. The library does the work; -the CLI is for humans. - -### Layout +The directory is readable by its owner only (mode `0700`), because it holds +decrypted data, file keys included. ``` -quak/ - src/ - crypto/ libsodium primitives (boxes, secretstreams, KDF, hash) - api/ HTTP client (ApiClient class) - auth/ login flow (SRP + email OTP + TOTP), key unwrap - model/ decrypted Collection, File, Metadata types + decrypt fns - download/ streaming file/thumbnail download + decryption - library/ the cache-backed Library: metadata store, read - surface, records, content cache, precache, ML data - and search, request pools - backup.ts resilient full-account backup with dedup - metadata-backup.ts - backup-metadata: the metadata quak keeps, as JSON - exif.ts EXIF read from an image's bytes with exifreader - mldata-fetch.ts fetch + decrypt per-file ML data - filename.ts safe file names from server metadata - errors.ts error types shared across layers, and how an error - is described - retry.ts retry classifier + exponential backoff with jitter - thumbnails.ts detect + regenerate missing thumbnails - client.ts high-level Client class assembled from the above - cli-commands.ts the CLI's commands as functions returning exit codes - cli-log.ts the CLI's log level, and the one function the - commands write stderr lines through - cli-output.ts how the CLI prints a file's title and time - cli-read.ts fresh reads for the CLI's read commands - cli-run.ts run a command, print its error, exit with its code - cli-session.ts read the saved session file back into a Client - index.ts public library exports - bin/ - quak.ts CLI entrypoint (commander.js) - examples/ - download-albums.ts - download every album's photos and metadata - test/ unit + integration tests (vitest) - Makefile - Dockerfile lint phase, test phase, compile - package.json - tsconfig.json +<cacheDirectory>/ + metadata.json the decrypted collection and file records, + the refresh cursor, and the list of what + did not decrypt + originals/<fileID>.<ext> cached originals + originals/<fileID>.livephoto.json + which of a live photo's two files is which + thumbnails/<fileID>.jpg cached thumbnails + mldata/ + <fileID>.json one decrypted machine-learning payload per + file + clip.f32, clip.json the packed CLIP index and its list of IDs + fetched.json per-file fetch bookkeeping ``` -`make build` compiles that tree into `dist/`, preserving its shape: the library -lands in `dist/src/`, the examples in `dist/examples/`, and the CLI in -`dist/bin/quak.js`, which is what `package.json` points `main`, `types` and -`bin` at. The compiler's `rootDir` is the repository root rather than `src/`, -because `bin/` is compiled too and `rootDir` has to contain everything that is -compiled. +When `metadata.json` belongs to another account than the one logged in, opening +the library deletes it and `mldata/` and starts from an empty cache. Cached +originals and thumbnails are kept. They are reached only through the records of +the current account, so another account's files are never served. -### Cryptography +A file appears in the cache only once it is complete: each one is written to a +temporary file and renamed into place. Ente's clients record a hash of each +file's bytes at upload, the content hash. Every original whose metadata records +one is hashed as it is written. One that does not match is not stored: the +download fails with an error naming the file, so `quak get` exits 1. The hash is +Ente's: unkeyed BLAKE2b with a 64-byte output, in standard base64. A live photo +arrives as a ZIP of its image and its video and is unpacked as it is written. +Its two files are hashed separately and compared as `<imageHash>:<videoHash>`, +and neither is stored unless both match. quak refuses a live photo ZIP that +holds a second image or video. It also refuses one whose files unpack to more +than 20 times its size plus 16 MiB. An original with no recorded hash, uploaded +by a very old client, is stored unchecked. -All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is -required for `crypto_pwhash` / Argon2id), except the SRP handshake, which uses -`fast-srp-hap`, and the MD5 checksum sent with a thumbnail upload, which uses -Node's built-in `node:crypto`. No hand-rolled crypto. +The cache has a size limit for originals (see "Opening a library"). Thumbnails +have none. A live photo's image and video leave the cache together. -The key hierarchy, derived during login, is: +### The backup directory -1. The user enters their password. -2. Argon2id (`crypto_pwhash`) over the password and a server-issued `kekSalt`, - with server-issued `memLimit` and `opsLimit`, produces a 32-byte Key - Encryption Key (KEK). -3. SRP login: `crypto_kdf_derive_from_key` (BLAKE2b) derives a 32-byte subkey - from the KEK with subkey id 1 and context `loginctx`. Its first 16 bytes are - the SRP password. -4. When SRP completes, the server returns a blob of "key attributes" plus an - encrypted auth token, or first asks for a second factor. quak answers a TOTP - request with the code (`POST /users/two-factor/verify`), after which the - server returns them, and cannot answer a passkey request. When the account - has email MFA on (`isEmailMFAEnabled`), an email OTP replaces SRP and the - server returns them after it. -5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK - yields the 32-byte master key. -6. `crypto_secretbox_open_easy` over the encrypted secret key with the master - key yields the user's X25519 private key. The matching public key is - delivered in cleartext. -7. `crypto_box_seal_open` over the encrypted token with the user's keypair - yields the auth token's bytes. Encoded as URL-safe base64 with padding, they - are the `X-Auth-Token` value for all subsequent calls. - -Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the -master key (for owned collections). Per-file keys are decrypted with -`crypto_secretbox_open_easy` using the collection key. File metadata is a -secretstream blob (single chunk, TAG_FINAL) under the file key. File content is -a chunked `crypto_secretstream_xchacha20poly1305` stream under the file key, -with a 4 MiB plaintext chunk size and a 17-byte authentication overhead per -chunk. Thumbnails use the same secretstream blob format as metadata. - -For upload (thumbnail repair), `encryptBlob` performs the push side: a single -secretstream chunk with TAG_FINAL, returning the header and ciphertext. - -### HTTP API - -Production endpoints: - -- API: `https://api.ente.io` -- File download CDN: `https://files.ente.io/?fileID=<id>` -- Thumbnail CDN: `https://thumbnails.ente.io/?fileID=<id>` - -A custom API endpoint is configurable for self-hosted servers via the -constructor option `apiOrigin`. When set, file downloads route through -`<apiOrigin>/files/download/<id>` instead of the dedicated CDN host. - -The constructor option `onRequest` is called once for each request sent, a -retried request once for each attempt, with its method, its URL and the status -of the answer or the error the request failed with. The URL of a presigned -upload is given without its query, which holds the upload's signature. The CLI -logs each request with `-vv` (see "CLI surface"). - -Required request headers on every authenticated call: - -- `X-Auth-Token`: the decrypted auth token from login. -- `X-Client-Package`: identifies the client. quak uses `berlin.sneak.quak`. - -Endpoints used: - -- `GET /users/srp/attributes?email=<email>`: fetch SRP and KDF parameters. -- `POST /users/srp/create-session`: begin SRP handshake. -- `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or the - encrypted token plus key attributes. -- `POST /users/ott` and `POST /users/verify-email`: email OTP, used instead of - SRP when the SRP attributes have `isEmailMFAEnabled` set. -- `POST /users/two-factor/verify`: TOTP second factor. -- `POST /users/logout`: end the calling token's session (`quak logout`). -- `GET /collections/v2?sinceTime=<usec>`: list collections changed since - microsecond timestamp; pass 0 for a full enumeration. -- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a - collection; paginate while `hasMore` is true. -- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes. -- `POST /files/data/fetch`: fetch encrypted ML data for a batch of files. -- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair). -- `PUT /files/thumbnail`: register an uploaded thumbnail's object key. - -### Retries and timeouts - -Every request in the library goes through one policy, in `src/retry.ts`. A -request is repeated only when repeating it could produce a different answer: - -- `ApiError` with a 5xx status: retried. So are `408` and `429`, the two 4xx - codes that are statements about timing rather than about the request. -- Every other 4xx: not retried. A 404 in particular is an answer, and - `listMissingThumbnails` depends on getting it promptly and once. -- Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or - TLS failure — and deadline aborts: retried. Node's `fetch` rejects with a - plain `TypeError`, so every `TypeError` is retried. The errno is looked for in - the error's `cause` chain, because that is where Node's `fetch` puts it. -- A truncated download: retried. -- Anything else, including a secretstream authentication failure that is not - truncation: not retried. The default answer is no. For a backup tool, retrying - a permanent failure spends round trips and delays every remaining file, while - declining to retry a transient one costs a single file that the next run picks - up. - -The `~/bin/quak` that `make install` builds runs under bun, whose `fetch` -rejects with a plain `Error` that has no `cause` and carries a code of bun's -own. quak retries a refused connection, a name that does not resolve, a reset -connection and a deadline under bun as it does under Node. Bun reports a refused -connection, and a TLS handshake that failed, as `ConnectionRefused`. For a name -that does not resolve it gives `ConnectionRefused` on some requests and -`FailedToOpenSocket` on others. A connection reset or closed before the response -ended is `ECONNRESET`, as under Node, and a deadline is the same `TimeoutError`. -A certificate bun rejects, reported with a code such as -`DEPTH_ZERO_SELF_SIGNED_CERT`, is not retried. - -Backoff is exponential with full jitter: the delay before retry _n_ is -`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term -is the ceiling and the wait is drawn below it, so a client that lost many -parallel downloads to one CDN blip does not send them all again at the same -instant. The numbers, configurable through `ApiClientOptions.retry`: - -| Option | Default | `quak backup` | Meaning | -| ------------- | ------- | ------------- | ----------------------------------- | -| `attempts` | `4` | `10` | total calls, not retries | -| `baseDelayMs` | `500` | `1000` | ceiling for the first retry's delay | -| `maxDelayMs` | `10000` | `60000` | upper bound on that ceiling | - -With the defaults a file that is going to fail gives up after at most three and -a half seconds of waiting. `quak backup` usually runs from cron with nobody -watching, so every request it makes uses the `quak backup` column instead, -exported as `UNATTENDED_RETRY_OPTIONS`: a request that keeps failing gives up -after at most 243 seconds of waiting, and usually after about half that, since -each wait is drawn at random below its ceiling. Every other command uses the -defaults. A library user gets the same budget by passing -`UNATTENDED_RETRY_OPTIONS` as `ApiClientOptions.retry`. `sleep` and `random` are -injectable through the same option, which is how the test suite exercises the -whole policy without waiting. `onRetry`, given there too, is called before each -wait with the error the attempt failed with, the attempt's number and the wait; -the CLI logs it with `-v` (see "CLI surface"). - -Two deadlines, renewed for each attempt: - -| Option | Default | Applies to | Kind | -| ------------------- | ------- | ------------------------------------------- | ------------------------------------- | -| `requestTimeoutMs` | `30000` | `getJSON`, `postJSON`, `putJSON`, `putFile` | the whole request | -| `downloadTimeoutMs` | `60000` | file and thumbnail downloads | idle: no bytes received for this long | - -They are different kinds because a download's length depends on the file and the -link: a whole-transfer deadline short enough to catch a hung connection would -cancel a large video on a slow link that is still making progress. The download -deadline restarts every time bytes arrive, so a slow download runs as long as it -keeps moving, and one that stalls is aborted after 60 seconds of silence. It -covers the wait for the headers and the body — `getFileStream` returns as soon -as headers arrive, so a deadline that only guarded the initial request would -leave the same hang one layer down. There is no limit on the total length of a -download. - -**Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON` -send every `POST` and `PUT` in the endpoint list above; some of them change -server state, and `/users/two-factor/verify` consumes one of a small number of -second-factor attempts. They are retried only when every errno in the error's -`cause` chain is one of the three that establish no TCP connection to the server -ever existed, so no request byte can have been transmitted: `ENOTFOUND` and -`EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer -refused the connection). A 5xx, a mid-flight reset and a deadline are all left -to the caller, because each of them can happen after the server has already -acted. These two do not follow redirects either: a redirect means the server -already received the request, so it is reported as an error and not retried. The -routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the -same reason, despite looking like connect-time failures: on Linux an ICMP -unreachable arriving mid-flight, or a local interface going down after the -request was written, delivers them on an already-established socket. They stay -retryable for the idempotent calls. So do bun's `ConnectionRefused` and -`FailedToOpenSocket`, on which a `POST` or `PUT` is not replayed either. When a -connection bun kept open from an earlier request closes before any answer, bun -sends the request again on a new connection, and if that one fails it gives one -of these codes although the server may already have the request. `putFile` is -exempt: a presigned PUT stores one whole object at one key in one request, so -replaying it has no partial state to damage. - -A download is retried as a whole — request, stream consumption, and decryption — -because a socket reset after the response headers have arrived surfaces in the -download layer rather than in `ApiClient`, and that is the common failure for -multi-megabyte photos over a CDN. The secretstream pull state is not resumable -and these endpoints have no Range support, so a retry starts the file over. The -atomic write is part of the retried unit: each attempt writes its own temporary -files, one for most files and two for a live photo (its image and its video), -and removes them if it fails. Only the attempt that completes renames anything -into place. The retry sits below `runBackup` and `runMetadataBackup`. -`runBackup` logs, counts and steps over a file that still fails after its -retries; `runMetadataBackup`, which downloads only with `--exif`, records the -error in that file's JSON as `imageMetadataError` and goes on. - -One imprecision is deliberate and worth knowing about. When a body ends part-way -through a secretstream chunk, Poly1305 fails and carries no framing signal, so a -cut connection and genuinely corrupt bytes are indistinguishable. quak reports -that as truncation, which means it is retried. For a body of more than one chunk -the distinction is real — a chunk that failed while the stream carried on past -it stays an authentication failure and is not retried — but for a single-chunk -body, which is most thumbnails and every small file, a wrong key, server-side -corruption and a mid-chunk cutoff all present alike and all get retried. The -cost is bounded by the attempt count, and it buys never silently keeping a -truncated file. - -### Session handling - -The `Client` class holds the auth token, master key, secret key, and public key -in memory. There is no on-disk session store in the library; the consumer -decides how to persist sessions. - -`client.toJSON()` returns a `ClientSnapshot` (a plain serializable object with -base64-encoded keys) that the consumer can write to disk, a database, or -whatever else fits their use case. `Client.fromJSON(snapshot)` restores a -working client from that snapshot without re-authenticating; it checks every -field and each key's length first, and throws an error naming the bad field. -`client.logout()` clears the token and zeroes the key buffers in place; after -it, every other method on that client throws. It does not contact the server, so -the token stays valid there and in any saved snapshot; -`await client.logoutOnServer()` first ends the session on the server -(`POST /users/logout`). - -The CLI stores the snapshot at the platform-appropriate data directory via -`env-paths`: `~/Library/Application Support/quak/session.json` on macOS, -`$XDG_DATA_HOME/quak/session.json` on Linux. The file is written with mode -`0600`. The key material is stored in cleartext in the JSON; treat this file as -you would treat the password itself. A missing file is reported as "not logged -in"; a file that exists but is corrupt is reported as such, naming the bad -field. When the refresh a command starts with gets HTTP 401 from the server, -because it no longer accepts the saved session's token, the command prints one -line, `quak: the saved session is no longer valid; run "quak login"`, with no -stack trace. All three exit with status 3, which means the user must run -`quak login` again. `quak logout` is the exception: with no file it says there -is no session and exits 0, and it handles a corrupt file or a failed server call -as described below. `quak backup` meets an expired session on the refresh that -starts every run, before it touches any file but its lock (see "Backup layout"). -A session that stops working partway through a backup instead fails each -remaining file into `failures.json`, so that run exits 1 and the next one stops -at its refresh with status 3. No command but `quak login` ever prompts. - -`quak logout` ends the session on the server, so the token in `session.json` -stops working even in a copy of the file, and then deletes the file. If the -server call fails (or the file is corrupt), the file is still deleted, the -command says the server session could not be ended, and it exits with status 1. -It does not delete the cache. When it knows the cache directory, from -`--cache-dir` or from a session file it could read, it prints it and says it -still holds decrypted data (file keys in `metadata.json`, cached originals and -thumbnails), for the user to delete if they want it gone. - -### CLI surface - -``` -quak [--cache-dir <path>] <command> global: local metadata/content cache location -quak [-v | -vv | -q] <command> global: log more on stderr, or only fatal errors -quak login interactive or QUAK_EMAIL/QUAK_PASSWORD -quak whoami print logged-in account as JSON -quak logout end the session, delete it -quak collections [--json] list all collections -quak files [collectionID] [--json] list files in a collection, or every file -quak get <fileID> [--out path] [--collection] download and decrypt a file -quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail -quak backup <dir> [--json] [--verify] full incremental backup -quak backup-metadata <dir> [--exif] dump the metadata quak keeps as JSON -quak helper list-missing-thumbnails [--json] find files with missing thumbnails -quak helper fix-missing-thumbnails [--file ids] [--json] generate + upload missing thumbnails -``` - -Every command except `login`, `whoami` and `logout` runs on the cache-backed -library. The read commands — `collections`, `files`, `get`, `get-thumb`, -`backup-metadata`, `helper list-missing-thumbnails` and -`helper fix-missing-thumbnails` — force a fresh server round-trip before they -answer, so they report current account state rather than whatever the cache last -held. `collections` asks the server for the collection list alone, in one -request, and neither fetches any collection's files nor changes the cache. The -others wait for a refresh. With an empty cache, as on a first run, that refresh -fetches the files of every collection, one collection after another, and stores -nothing until it has them all, which on a large account takes a while. Meanwhile -the command logs a line on stderr for each collection whose files it has -fetched, such as `Fetched the files of 3 of 120 collections`, with `--json` too; -`backup` does the same while the refresh it starts with runs. stdout holds only -the command's output. If that round-trip fails, the command prints the error on -one line and exits 1, or 3 when the server no longer accepts the saved session -(see "Session handling"). An album or file quak cannot decrypt does not fail it; -the command answers from what did decrypt (see "Default reads vs. fresh reads"). -`--cache-dir` overrides where the cache lives; without it each account gets its -own directory under the per-user cache path. - -`-v` (`--verbose`) and `-q` (`--quiet`) set how much quak logs on stderr. Like -`--cache-dir`, they go before or after any command's name. Every line quak -prints on stderr is a log line at one of six levels, from the quietest: fatal, -error, warning, info, debug and trace. A run prints the lines at its own level -and at every quieter one. stdout holds only the command's output, at every -level. - -- Every command that exits non-zero logs a fatal line saying why, such as - `quak: <message>`, `Not logged in`, an invalid ID or a collection that does - not exist. A command that exits 1 because files or albums failed logs it last, - saying how many failed and where to look: `backup` points to `failures.json` - and its `FAILED` lines, `backup-metadata` to the `mlDataError` in the JSON of - each such file, and `helper fix-missing-thumbnails` to its `Failed files:` - list, or to its JSON with `--json`. -- An error line reports something that failed: the `FAILED` lines that `backup` - and `helper fix-missing-thumbnails` log as they work, - `ML data of file <id> could not be read` and - `ML data request for <n> file(s) failed` from `backup-metadata`, - `Could not check <title>` from both thumbnail helpers, and the `Failed files:` - list at the end of `backup` and of `helper fix-missing-thumbnails`. -- A warning reports something that did not fail but needs your attention: after - `logout`, that the cache directory still holds decrypted data. -- Every other line quak logs by default is info. - -With `--json`, a command's result on stdout takes the place of its summary on -stderr: for `backup`, the lines from `--- Backup complete ---` to the end of its -`Failed files:` list; for `helper list-missing-thumbnails`, -`No missing thumbnails found.` or `<n> file(s) with missing thumbnails:`; and -for `helper fix-missing-thumbnails`, the lines from `--- Done ---` to the end of -its `Failed files:` list. Every other line is logged as it is without `--json`. - -- With neither option the level is info: the lines quak prints as it works, such - as a refresh's progress, the files a backup fetches, its summary and the files - that failed. -- `-v` raises the level to debug, which adds each retry of a request, with why - the attempt failed and how long quak waits before the next, and the start and - end of each refresh and each ML data fetch, with the error of one that failed. -- `-vv` raises it to trace, which adds each request quak sends: its method, its - URL and the status of the answer, or the error the request failed with. The - URL of an upload of a thumbnail is logged without its query, which holds the - upload's signature. More repeats stay at trace, and `--verbose --verbose` is - `-vv`. -- `-q` lowers it to fatal: a run that exits 0 prints nothing on stderr, and one - that exits non-zero prints only its fatal lines. The prompts of `login` are - not log lines, and `-q` leaves them as they are. - -`-q` with `-v` is refused with one line, -`error: option '-q, --quiet' cannot be used with option '-v, --verbose'`, and -exit status 1. - -`files <collectionID>` lists the files in that collection. `files` with no -argument lists every file in the account, collection by collection, each file -once: a file in several collections is listed with the first of them, which is -its `collectionID` in `--json`. - -An ID given to `files`, `get`, `get-thumb` or -`helper fix-missing-thumbnails --file` is written in decimal digits, no larger -than 9007199254740991. Anything else, such as `0x10`, `1e3` or `-1`, is refused -before any request with one line, `Invalid file ID: <arg>` or -`Invalid collection ID: <arg>`, and exit status 1. One bad `--file` ID refuses -the whole command, so nothing is fixed. - -Without `--out`, `get` writes the file to the working directory under its title, -and `get-thumb` writes the thumbnail there under `thumb_` followed by the title, -each with unsafe characters replaced. A name over 255 bytes, which most file -systems refuse, is cut short between two characters before its extension, so the -extension is kept whenever it fits. An extension too long to leave room for the -name's first character is cut short too, so the name never starts with a dot. - -`get` and `get-thumb` resolve the file by ID directly, so `--collection` is -accepted for backward compatibility but ignored. For a live photo, `get` writes -its image and its video, each named after the title with its own extension, as -Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`), and each name is -cut short as above once its own extension is in it. With `--out PATH`, the image -is written to `PATH` and the video beside it, with `PATH`'s name and the video's -extension; a `PATH` with the video's extension is refused. -`backup-metadata --exif` (alias `--all`) additionally fetches each file's -original through the cache and records, from it or a live photo's image, its XMP -metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is read with -[exifreader](https://github.com/mattiasw/ExifReader) from any image format it -reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's `exif` field -is exifreader's EXIF tag output: each tag by name, with its `value`, -`description` and `computed` value. An EXIF block exifreader finds but reads no -tag from is recorded, base64, as `exifRaw`, with the reason in `exifError`. -`collections`, `files`, `backup`, `helper list-missing-thumbnails` and -`helper fix-missing-thumbnails` take `--json` for machine-readable output. - -`backup --verify` also hashes the originals already in the backup and replaces -any that do not match the content hash Ente records; one it cannot replace stays -as it was and goes into `failures.json` (see "Backup layout"). - -`backup-metadata <dir>` writes `account.json` and, for each collection, a -directory `collections/<id>-<name>/` holding `_collection.json` and a -`<fileID>.json` for each of its files. The collection's name has unsafe -characters replaced and is cut short between two characters so that the -directory's name fits in 255 bytes; the ID keeps the directories apart. - -`backup-metadata` fetches ML data in requests of up to 200 files. When a request -fails, the error is logged, each of its files is written with the reason in an -`mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML -data does not decrypt, gunzip or parse is logged and written the same way, with -why in `mlDataError`. The exit code is non-zero if any file was written with -`mlDataError`, and the last line then says how many were. - -`helper fix-missing-thumbnails` regenerates thumbnails for JPEG images only, -because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG image -(PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept -distinct from a `failed` repair, and does not affect the exit code; a genuine -failure still exits non-zero. The server accepts a new thumbnail only from the -file's owner and only when it is no larger than the thumbnail size it records -for the file. So a file another account owns, in an album shared with you, is -skipped by both thumbnail helpers without being fetched, and the fixer skips a -file whose recorded thumbnail size is 0 or unknown. Otherwise the fixer lowers -the quality and size of the thumbnail until it fits, and skips the file if even -the smallest does not. - -### Backup layout - -`quak backup <dir>` produces: +`quak backup <dir>` produces this tree: ``` <dir>/ YYYY/YYYY-MM/YYYY-MM-DD/ - YYYY-MM-DD.<fileID>.<ext> actual file content, at its save path (one - per unique file, two for a live photo: see - below) - YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak - keeps, its update time, its private and - public magic metadata, its ML data, and, - for an image (for a live photo, its - image), its original's EXIF and XMP, with - dimensions for a JPEG only; none of these - three for a video + YYYY-MM-DD.<fileID>.<ext> the original, at its save path (one per + file, two for a live photo) + YYYY-MM-DD.<fileID>.json the metadata quak keeps about the file, + plus its machine-learning data, its + keywords and its image metadata YYYY-MM-DD.<fileID>.livephoto.json which of a live photo's two files is which collections/ <name>/ <title> -> ../../YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext> - (symlink) - <name>.json collection metadata + file list + a symlink per file + <name>.json the collection's metadata and file list account.json the account's email and user ID - backup.lock the lock a running backup holds (see below) - failures.json files that failed and have not yet succeeded + backup.lock the lock a running backup holds + failures.json the files that failed and have not yet + succeeded ``` -Each original is saved at its save path, the same path `photo.savePath` gives -and `photo.download()` writes (see Read surface below). The date is the photo's -`takenAt` (the date set in Ente if it was edited, else its creation time) in the -time zone of the machine running quak. The extension is the one in the file's -name as uploaded, case kept, or `.bin` when it has none, it holds anything but -letters and digits, or it is more than 16 of them long. When the date or the -time zone changes, the next run saves the original at its new path and leaves -the old copy where it is. +#### Save paths -`account.json` holds the account's `email` and `userID`, as `backup-metadata` -writes it. A collection's JSON holds its `id`, `name`, `type`, `ownerID`, -`isShared` and `updationTime`, its `magicMetadata`, `pubMagicMetadata` and -`sharedMagicMetadata` when it has them, as `backup-metadata`'s -`_collection.json` does, and `files`, each file's `id` and `metadata`. A file's -JSON holds its `id`, `collectionID`, `ownerID`, `metadata` and `updationTime`, -and its `magicMetadata` and `pubMagicMetadata` when it has them. Update times -are in microseconds, as Ente records them. +Each original is stored once, at its save path: +`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under `<dir>`. The date is +the day the photo was taken, in the time zone of the machine running quak. That +is the date set in Ente when the date was edited, and otherwise the file's +creation time. The extension is the one in the file's title, with its case kept. +It is `.bin` when the title has no extension, when the extension holds anything +but letters and digits, or when it is longer than 16 characters. -A file's JSON holds Ente's ML data for it (its faces and its CLIP embedding) as -`mlData`, the same payload `backup-metadata` writes; a file Ente has no ML data -for has no `mlData`. The backup waits for the library's ML data fetch to finish -before it writes the JSON files. If that fetch fails, each file whose ML data is -not in the cache gets the reason in `mlDataError` instead and counts as failed, -and the next run fetches it again. The same goes for a file with no ML data in -the cache whose ML data the server sent but quak could not decrypt, gunzip or -parse. Its `mlDataError` says why, and `failures.json` records it with an error -starting `ML data: `. quak keeps that reason in memory only and caches nothing -for the file, so the next run asks for its ML data again. An ML data fetch that -fails, such as one the library's refresh starts while the backup writes the JSON -files, leaves the reasons as the last fetch that succeeded left them. The JSON -files are rewritten on every run, so ML data that arrived since the last run -appears. +When the date or the time zone changes, the next run stores the original at its +new path and leaves the old copy where it is. -A file's JSON holds, as `imageMetadata`, what `backup-metadata --exif` records -from its original, or from a live photo's image: `format`, `width` and `height` -for a JPEG, `exif` (or `exifRaw` and `exifError`), and `xmp`. An original with -none of these gets `{}`. A video gets no `imageMetadata`: reading a whole video -to look for tags is not worth it. When the original cannot be read, the reason -is in `imageMetadataError` instead; neither the file nor the run fails. The -original is read when the run stores it, or when the JSON beside it has neither -field, as one an earlier version wrote. Otherwise the field is taken from that -JSON when it is rewritten, so a run does not read every stored original again. +The library gives the same path as `photo.savePath` and writes it with +`photo.download()` (see "A photo's content"). -`failures.json` records each failed file with the kind of failure, how many -times it has been tried and when it was last tried. A file leaves it once it -succeeds, or once it is no longer in the library or in the backup's scope. The -library's `lib.backup({ includeThumbnails: true })` also writes -`thumbnails/<fileID>.jpg` beside `collections/`; `quak backup` does not. +#### The JSON beside each original -A file in the backup's scope that the library could not decrypt (see "Default -reads vs. fresh reads") goes into `failures.json` with an error that starts -`could not be decrypted:` and gives the reason. If a copy of it in another album -decrypted, it does not: its original is stored from that copy, and the album -where it does not decrypt neither links nor lists it. An album the library could -not decrypt has no name quak can read and no files it can list. `quak backup` -prints one line for it on stderr, -`FAILED album <id>: could not be decrypted: <reason>`, and `--json` and the -`BackupResult` list its ID in `undecryptableAlbums`. Such a file in -`failures.json`, or such an album, keeps the exit code at 1, run after run, -until the server sends a copy that decrypts or deletes it; the next run then -drops it. +The JSON file beside an original holds: + +- `id`, `collectionID`, `ownerID` and `updationTime`; +- `metadata`: the fields quak keeps from the file's own metadata. They are + `title`, `fileType`, `creationTime`, `modificationTime`, `latitude`, + `longitude` and `hash`. The last three are present only when the file has + them. +- `magicMetadata` and `pubMagicMetadata`, when the file has them. The magic + metadata is Ente's name for the fields its clients add later, such as an + edited name, an edited date or a caption. + +Update times are in microseconds, as Ente records them. + +The JSON also holds the machine-learning data Ente computed for the photo, as +`mlData`. Ente's apps detect faces in each photo and compute a CLIP embedding +for it, a list of 512 numbers that describes what the photo shows. `mlData` +holds everything but the `clip` block, which holds that embedding. In its place +the JSON holds `keywords`, derived from the embedding as described under +"Keywords". A file with no machine-learning data has neither field. A file whose +data holds no usable embedding has no `keywords`. + +The backup waits for the library's fetch of machine-learning data to finish +before it writes the JSON files. When a file's data cannot be fetched, +decrypted, decompressed or parsed, the JSON holds the reason in `mlDataError` +instead of `mlData`. Such a file counts as failed and goes into `failures.json` +with an error starting `ML data: `. Nothing is cached for it, so the next run +asks for its data again. The JSON files are rewritten on every run, so data that +arrived since the last run appears. + +Every file but a video also holds `imageMetadata`: what the original itself +records. That is its `format`, `width` and `height` for a JPEG, its EXIF tags as +`exif`, and its XMP packet as `xmp`. EXIF and XMP are the two blocks of metadata +a camera or an editor writes inside an image file. For a live photo, these come +from its image. An original with none of them gets `{}`. When the original +cannot be read, the reason is in `imageMetadataError` instead, and neither the +file nor the run fails. + +EXIF is read with [exifreader](https://github.com/mattiasw/ExifReader) from any +image format it reads, such as JPEG, HEIC, AVIF, PNG and WebP. `exif` holds each +tag by name, with its `value`, `description` and `computed` value. An EXIF block +that exifreader finds but reads no tag from is kept in base64 as `exifRaw`, with +the reason in `exifError`. + +The original is read for this when the run stores it, or when the JSON beside it +has neither `imageMetadata` nor `imageMetadataError`. Otherwise the two fields +are copied from the JSON being rewritten, so a run does not read every stored +original again. + +#### Keywords + +The Discover section of Ente's photos app sorts photos into 16 categories. quak +derives a file's keywords the way the app sorts it. The categories are: + +- Identity (no videos) +- Screenshots (no videos) +- QR Codes (no videos) +- Receipts +- Notes +- Memes +- Visiting Cards (no videos) +- Babies +- Pets +- Selfies +- Wallpapers (no videos) +- Food +- Celebrations +- Sunset +- Hills +- Greenery + +Each category has a query, and a text embedding of that query from Ente's CLIP +text model. A file's score for a category is the cosine similarity of the file's +embedding and the category's. That is a number from -1 to 1; the more alike the +two embeddings are, the higher it is. The file falls in the category when its +score reaches the category's minimum score. The keywords are the titles of the +categories the file falls in, highest score first. A video never falls in a +category marked "no videos", as in the app. + +#### Collection directories + +Each collection gets a directory under `collections/`, named after the +collection, and a JSON file beside it, `<name>.json`. The directory holds one +symlink per file, named after the file's title, pointing at the original's save +path. A live photo gets two symlinks, one to its image and one to its video, +each named after the title with that file's extension. + +The JSON file holds: + +- `id`, `name`, `type`, `ownerID`, `isShared` and `updationTime`; +- `magicMetadata`, `pubMagicMetadata` and `sharedMagicMetadata`, when the + collection has them; +- `files`: each file's `id` and `metadata`. + +Names come from the server, so unsafe characters in them are replaced (see "File +names from the server"). When two collections would get the same directory name, +ignoring case, each gets its ID added: two albums named `Trip` become +`Trip (10)/` and `Trip (11)/`. Two files in one collection whose titles match, +ignoring case, get their IDs added before the extension: `IMG_0001 (12345).JPG` +and `IMG_0001 (12346).JPG`. A collection named like another collection's JSON +file, such as `Trip.json` beside `Trip`, also gets its ID added. A file name +longer than 255 bytes, the limit on most file systems, gets its ID added too. So +does a collection name longer than 250 bytes, which leaves room for `.json`. +Either name is then cut short before the ID, just far enough to fit. IDs never +change, so a name stays the same from run to run until such a clash appears or +goes away. + +Each run removes every symlink in a collection's directory that points into the +date folders and is not one of its current links, yours included. A file or +symlink of yours at the name of one of its links is replaced by the link. A +directory of yours at that name stays, and the photo whose link it blocks goes +into `failures.json`. Anything else you put under `collections/` stays. A run +removes the directory and JSON of a collection that was deleted or renamed, once +the directory holds nothing else. One that still holds something of yours stays, +with its JSON. + +A collection whose directory or JSON cannot be written does not stop the run. +For example, a file you put there may already have the directory's name. The run +logs `FAILED album <name>: <reason>`, lists the collection's ID in +`failedAlbums`, and exits 1. The other collections and `failures.json` are still +written. + +#### What quak cannot decrypt + +A collection or file that quak cannot decrypt is left out of the library (see +"Default reads and fresh reads"). The backup treats the two cases this way: + +- A file quak cannot decrypt goes into `failures.json` with an error starting + `could not be decrypted:` and the reason. It stays out of `failures.json` when + its copy in another collection decrypts. The original is then stored from that + copy, and the collection where it does not decrypt neither links nor lists it. +- A collection quak cannot decrypt has no name quak can read and no files it can + list. The run logs `FAILED album <id>: could not be decrypted: <reason>` and + lists the ID in `undecryptableAlbums`. + +Every run exits 1 until the server either sends a copy that decrypts or deletes +the record. The run after that drops the entry. A collection that stops +decrypting loses its directory and JSON as a deleted one does, and a file that +stops decrypting loses its symlink in that collection. The originals stay at +their save paths. + +#### Live photos + +Ente stores a live photo as one ZIP of its image and its video. The backup +stores the two files, which a photo viewer can open: each is +`YYYY-MM-DD.<fileID>.<ext>` with the extension it has inside the ZIP, such as +`2026-03-01.12345.heic` and `2026-03-01.12345.mov`. The file +`YYYY-MM-DD.<fileID>.livephoto.json` beside them names which is which. The live +photo counts as stored only when its `.livephoto.json` and both files it names +are present and not empty. + +#### `failures.json` + +`failures.json` records each file that failed and has not succeeded since. It +holds `version`, which is 1, and `files`, one entry per file keyed by its ID. +Each entry holds: + +- `fileID` and `title`; +- `classification`: the kind of failure; +- `attempts`: how many runs in a row have failed the file; +- `lastTriedAt`: when the run that last tried it finished its refresh, as a Unix + time in milliseconds; +- `error`: the error, as quak printed it. + +The kinds are: + +- `transient`: a network or server problem; +- `permanent`: a refusal, or a local file-system or decryption error; +- `unknown`: anything else. + +A file leaves the list once it succeeds, or once it is no longer in the library. +A library caller that backs up only some albums (`onlyAlbumNames`) also drops a +file outside them. `failures.json` is deleted when the list is empty. + +#### `backup.lock` `backup.lock` keeps two backups of the same directory from running at once, such as a cron run that starts while the previous one is still going. A backup -creates `<dir>` if it is missing and takes the lock before its refresh, and -removes the lock when it ends, whether it succeeds or fails, unless it lost the -lock (see below); `quak backup` takes it before it opens its library, so a -refused run sends no request. The lock is a directory that -[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates -and keeps touching while the backup runs. A second backup of the directory, from -another process or the same one, fails at once: `quak backup` prints -`quak: another backup of <dir> is running` and exits with status 2. A run -stopped with Ctrl-C or `kill` removes the lock as it exits. Only a run that -cannot, such as one killed with SIGKILL or cut off by a crash or power loss, -leaves it behind; once it has gone 10 seconds untouched, the next run takes it -over, so nobody has to remove it. The lock is outside the date folders and -`collections/`, so it is never taken for an original, and the removal of old -album directories never touches it. +creates `<dir>` if it is missing and takes the lock before its refresh, so a +refused run sends no request. A run that still holds the lock removes it when it +ends, whether it succeeded or failed. -A running backup loses its lock when someone removes `backup.lock`, or when -another backup takes it over because the running one left it untouched for 10 -seconds, as a stopped or stalled process does. The backup finds this out the -next time it goes to touch the lock, which it does every 5 seconds, and then -stops before it starts another file. The originals it is downloading at that -moment are still stored. It leaves `failures.json` and `backup.lock` as they -are, since another backup may be using them. `quak backup` prints -`quak: lost the lock on <dir>: <reason>` as one line and exits 1, and -`lib.backup()` rejects with `lost the lock on <dir>: <reason>`. +The lock is a directory that +[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates. +While the backup runs, it updates the modification time of `backup.lock` every 5 +seconds. A second backup of `<dir>` fails at once: it prints +`quak: another backup of <dir> is running` and exits 2. A run that is killed, or +cut off by a crash, can leave the lock behind. Once the lock has gone 10 seconds +without an update, the next run takes it over, so nobody has to remove it by +hand. -A collection's directory and JSON are named after the collection, and a symlink -after the file's title, both with unsafe characters replaced. When two -collections would get the same name, or two symlinks in one collection the same -name (ignoring case in both), each of them gets its ID added: two albums named -`Trip` become `Trip (10)/` and `Trip (11)/`, and two files titled `IMG_0001.JPG` -become `IMG_0001 (12345).JPG` and `IMG_0001 (12346).JPG`. A collection whose -directory would have the name of another collection's JSON, again ignoring case, -gets its ID added as well: beside an album `x`, whose JSON is `x.json`, an album -named `x.json` gets the directory `x.json (12)/`. A name too long for a file -name, which most file systems limit to 255 bytes, gets its ID added too. A name -with its ID added is cut short between two characters in the part before the ID, -just far enough that it fits. When a symlink's extension leaves no room for the -ID and the first character of the name, the extension is cut short, and the part -before the ID keeps that character and any bytes the cut leaves over. A -collection's directory name also leaves room for the `.json` its JSON adds. IDs -never change, so a name stays the same from run to run until such a clash -appears or goes away. +A running backup loses its lock when someone removes `backup.lock`. Another +backup takes the lock over when this one has not updated it for 10 seconds, for +example because this process was stopped. The backup notices at its next update +of the lock, and stops before it starts another file. The originals it is +downloading at that moment are still stored. It leaves `failures.json` and +`backup.lock` as they are, since another backup may be using them. It prints +`quak: lost the lock on <dir>: <reason>` and exits 1. -A collection whose directory or JSON cannot be written, such as one whose -directory name is taken by a file you put in `collections/`, does not stop the -run. `quak backup` prints `FAILED album <name>: <reason>` for it on stderr, -`--json` and the `BackupResult` list its ID in `failedAlbums`, and the exit code -is 1. The other collections and `failures.json` are still written. +#### `--verify` -A live photo, which Ente stores as one ZIP of its image and its video, is stored -as those two files, which a photo viewer can open: each is -`YYYY-MM-DD.<fileID>.<ext>` with the extension it has inside the ZIP (for -example `2026-03-01.12345.heic` and `2026-03-01.12345.mov`), and -`YYYY-MM-DD.<fileID>.livephoto.json` names the two. The live photo counts as -stored only when both files are present and not empty. Its album folder links -both, each named after the title with that file's extension (`IMG_0001.heic` and -`IMG_0001.mov`). +With `--verify`, a run hashes each original already at its save path the way a +download is checked. For a live photo it hashes the image and the video and +joins the two as `<imageHash>:<videoHash>`. An original that matches the content +hash in its metadata is left as it is. One that does not is logged on one line +naming the file, downloaded again in the same run, and replaced once the new +copy matches. The new copy is downloaded even when the cache holds the original, +because a copy from the cache is not checked as a download is. -Each run removes the symlinks into the date folders that no longer belong in -their collection's directory, and the directories (and JSON) of collections that -were deleted or renamed. An album quak can no longer decrypt loses its directory -and JSON the same way, and a file whose copy in an album quak can no longer -decrypt loses its symlink there and its entry in that album's JSON; the -originals stay at their save paths. Nothing else in `collections/` is touched: a -file or a symlink you put there stays, and a directory that still holds one -after its symlinks are removed stays too, with its JSON. +The new copy is written to a temporary file beside the original, hashed the same +way, and renamed over the original only when it matches. A live photo's image +and video are both hashed before either is renamed. Until then the original +stays as it was. When the new copy fails or does not match: -Each file is downloaded exactly once regardless of how many collections it -appears in, and written once: straight to its save path, with no copy left in -the cache. An original the cache already held is copied from there instead. On -subsequent runs, existing originals are skipped. If a download fails, the error -is logged and the backup continues with the next file. The exit code is non-zero -if any files failed. `quak backup` opens its library with the thumbnail and -originals precache off, so the only file content it fetches is the originals the -backup stores. It also opens it with no background timer, so it asks the server -for changes only as it starts, not again while it downloads. +- the temporary files are deleted; +- the original keeps its bytes and its album links; +- the file goes into `failures.json`. -With `--verify`, or `lib.backup({ verify: true })`, a run also hashes each -original already at its save path the way a download is checked (see "On-disk -cache layout" below): its bytes, read in chunks, or a live photo's image and -video, joined as `<imageHash>:<videoHash>`. An original that matches the content -hash its metadata records is left as it is. One that does not is logged on one -line naming the file, and replaced once a matching copy is downloaded, in the -same run. The copy is downloaded even when the cache holds the original, because -a copy from the cache is not checked as a download is. It is written to a -temporary file beside the original, a live photo's image and video each to its -own, hashed the same way, and renamed over the original only when it matches; a -live photo's image and video are both hashed before either is renamed. Until -then the original stays as it was: when the download fails, or what it brings -does not match either, the temporary files are deleted, the original keeps its -bytes and its album links, and the file goes into `failures.json`. A run that -stops between renaming a live photo's new image and its new video leaves the new -image beside the old video, which a later `--verify` run replaces if the video -was the part that did not match. A file whose metadata records no hash is left -as it is and counted as unchecked. A stored original that cannot be read is left -as it is and counts as failed. The summary and `--json` add the counts -`verified`, `mismatched` and `unchecked`, all of originals that were already -stored; one first downloaded in this run is in none of them. A mismatch that was -replaced with matching bytes does not make the exit code non-zero. Without -`--verify` nothing is hashed, the summary is unchanged, and the three counts are -0 in `--json`. +A run that stops between renaming a live photo's new image and its new video +leaves the new image beside the old video. A later `--verify` run then replaces +the video, if the video was the part that did not match. + +A file whose metadata records no hash is left as it is and counted as unchecked. +A stored original that cannot be read is left as it is and counted as failed. +The summary and the `--json` output add the counts `verified`, `mismatched` and +`unchecked`. Each counts only originals that were already stored. A mismatch +that was replaced with matching bytes does not make the exit code non-zero. + +A run without `--verify` drops two kinds of failure from `failures.json`: + +- a mismatched original whose new copy failed; +- a stored original that cannot be read. + +That run finds the original in place, counts it as skipped and drops the entry, +so it can exit 0. Only a `--verify` run hashes the original again. + +Without `--verify`, no stored original is hashed, the summary is unchanged, and +the three counts are 0 in the JSON. + +#### Temporary files and crash safety Each original is written to a temporary file in the same directory, synced to -disk, and renamed into place, so an original is either complete or absent, even -after a power cut. A downloaded original's temporary file is named -`.quak-<pid>-<random>.tmp`, one copied from the cache -`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A copy that -`--verify` downloads to replace an original is written to such a -`.quak-<pid>-<random>.tmp` too, a live photo's image and video to -`.quak-<pid>-<random>.<ext>`, each with its own extension. A run that is killed -can leave one of these temporary files behind; the next backup deletes those -whose process is no longer running. The content cache uses the same scheme, and -opening a library deletes the temporary files in the cache whose process is no -longer running, so a download another process has in progress in the same cache -is left alone. The rename replaces whatever was at the destination rather than -writing through it: a symlink there is replaced, not followed, and the new file -has the temporary file's permissions, not those of the file it replaced. +disk, and renamed into place. So an original is either complete or absent, even +after a power cut. The temporary files are named: -## TODO +- `.quak-<pid>-<random>.tmp` for a downloaded original, and for the copy that + `--verify` downloads; +- `.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp` for an original + copied from the cache; +- `.quak-<pid>-<random>.<ext>` for the image and the video of a live photo that + `--verify` downloads, each with its own extension. -- [x] Retry policy: no retry on 4xx (except `408` and `429`), exponential - backoff on 5xx and network errors -- [x] Update the API reference section below to match the current implementation -- [x] `make docker` green -- [x] Store live photos in a form a photo viewer can open - (https://git.eeqj.de/sneak/quak/issues/107): unpacked into the image and - the video +A run that is killed can leave one of these files behind. The next backup +deletes those whose process is no longer running. The cache uses the same +scheme. Opening a library deletes such leftovers in the cache's `originals/` and +`thumbnails/`. A download another process has in progress is left alone. -Tagging and releases are decided by sneak alone, and happen only when he -declares one. +The rename replaces whatever was at the destination rather than writing through +it. A symlink there is replaced, not followed, and the new file has the +temporary file's permissions, not those of the file it replaced. Each original +is written with mode `0600`, readable by its owner only. Everything else the +backup writes, its folders and JSON files included, gets the mode your umask +allows, which usually lets every user on the machine read it. The JSON beside an +original holds the file's location and faces. The `backup-metadata` tree is +written the same way. -Future (desktop client, separate repo): +### The metadata dump directory -- [ ] Electron app skeleton consuming this library -- [ ] Local cache (SQLite) keyed on `(collectionID, fileID, updationTime)` -- [ ] Background sync worker that streams new files into the cache -- [ ] Gallery UI: thumbnails, full-image view, basic search -- [ ] Upload, delete, and share operations in the library +`quak backup-metadata <dir>` produces this tree: -## API reference +``` +<dir>/ + account.json the account's email and user ID + collections/ + <id>-<name>/ + _collection.json the collection's metadata + <fileID>.json one file's metadata, one per file in + the collection +``` -The library's primary surface is the cache-backed `Library`; the lower-level -`Client` sits underneath it and is covered by the Design sections above. The -test suite is the canonical, executable documentation — `test/library/` and -`test/client/usage.test.ts` walk most operations, `test/cli/backup.test.ts` -walks `lib.backup()`, `test/cli/backup-end-to-end.test.ts` runs `quak backup` -against a fake of the Ente HTTP API, and `yarn test` verifies them. +Each run writes into `<dir>` and deletes nothing. `account.json` holds `email` +and `userID`. In `<id>-<name>`, the name has unsafe characters replaced. It is +cut short so the directory name fits in 255 bytes. The ID keeps two collections +of the same name apart. + +`_collection.json` holds the fields a backup's collection JSON holds, except +`files`. Each `<fileID>.json` holds the fields a backup's JSON beside an +original holds, with these differences: + +- `imageMetadata` is written only with `--exif`. The original of every file, + videos included, is then downloaded and read for its EXIF, XMP and JPEG + dimensions. +- A file whose original holds no image metadata gets no `imageMetadata` field. +- An original that cannot be downloaded or read gives `imageMetadataError` + instead. +- A file in several collections has its original read once; its image metadata + is written with the first of them. + +### File names from the server + +A file's title and a collection's name come from the server, so a name taken +from them is made safe before it is used on disk. Each path separator, each +character Windows forbids in a file name and each control character becomes `_`. +A leading run of dots becomes one `_`. A name Windows reserves for a device, +such as `CON`, gets a leading `_`. An empty title becomes `file-<id>`. An empty +collection name becomes `collection-<id>` in a backup and `unnamed` in the +metadata dump. + +A file name over 255 bytes is shortened before its extension, without splitting +a character, so the extension is kept when it fits. An extension too long to +leave room for the name's first character is cut short too, so the name never +starts with a dot. A collection's directory name is cut at its end. A path you +give with `--out` is used as it is. + +## Library reference + +The library's main class is `Library`. The `Client` sits underneath it and +handles login, saved sessions and the raw requests. The test suite is the +executable documentation: + +- `test/library/` shows most of what `Library` does; +- `test/client/usage.test.ts` shows the `Client`; +- `test/cli/backup.test.ts` shows `lib.backup()`; +- `test/cli/backup-end-to-end.test.ts` runs `quak backup` against a fake of the + Ente HTTP API. ### Opening a library -`Library.open(options)` loads the on-disk cache, starts the background refresh -loop, and resolves to a `Library`. On an empty cache it awaits the first -refresh, so it opens onto the account's data whenever the server is reachable; -if that refresh fails, it opens with no data and records the error in -`lib.status()`. On an existing cache it returns immediately and refreshes in the -background, so an unreachable server does not block opening. With -`refreshOnOpen: false` it starts no refresh and returns at once onto what the -cache holds, empty or not; a refresh then starts only when `fresh()` or -`lib.backup()` asks, or the background timer fires. The CLI's read commands open -it so (see "CLI surface"). +`Library.open(options)` loads the on-disk cache and resolves to a `Library`. On +an empty cache it waits for the first refresh, so it opens onto the account's +data whenever the server is reachable. If that refresh fails, it opens with no +data and records the error in `lib.status()`. On an existing cache it returns at +once and refreshes in the background, so an unreachable server does not block +opening. With `refreshOnOpen: false` it starts no refresh and returns at once +onto what the cache holds, empty or not. A refresh then starts only when +`fresh()` or `lib.backup()` asks, or when the background timer fires. `LibraryOptions`: -| Option | Default | Meaning | -| ------------------------ | -------------------------------------------------- | -------------------------------------------------------------------- | -| `client` | required | the account client (a `Client`, or any `LibraryClient`) | -| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live | -| `downloadDirectory` | `photos` in the working directory | root of the save paths; an original stored there counts as cached | -| `refreshIntervalSeconds` | `3` | background refresh cadence; `Infinity` for no background timer | -| `refreshOnOpen` | `true` | start a refresh as the library opens | -| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first | -| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals | -| `precacheOriginalsDays` | `7` | length in days of that latest window | -| `cacheOriginalsMaxBytes` | 100 GiB | size limit on the originals cache; pinned originals can exceed it | -| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down | -| `isOriginalPinned` | none | extra predicate for originals that must never be evicted | -| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) | -| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) | -| `contentSource` | the client's own | override the byte source (mainly for tests) | +- `client`, required: the account's `Client`, or anything that satisfies + `LibraryClient`. +- `cacheDirectory`: where `metadata.json` and the content cache live. Default: + `quak/<userID>` under the per-user cache directory (see "The cache"). +- `downloadDirectory`: the root of the save paths. Default: `photos` in the + working directory, resolved once when the library opens; + `lib.downloadDirectory` holds the result. An original stored at its save path + counts as cached. +- `refreshIntervalSeconds`: how long after one refresh ends the background timer + starts the next. Default `3`. `Infinity` starts no timer. +- `refreshOnOpen`: whether `open()` starts a refresh. Default `true`. +- `precacheThumbnails`: fetch every thumbnail in the account in the background, + newest first, until all are on disk. Default `true`. +- `precacheOriginals`: fetch, in the background, the originals of the favorites + album and of every file taken in the latest days (see the next option). These + originals are pinned: they are never evicted from the cache. Default `true`. +- `precacheOriginalsDays`: how many days count as the latest, counted back from + the newest file in the account. Default `7`. +- `cacheOriginalsMaxBytes`: the size limit on the originals cache. Default 100 + GiB. The limit in force is this, or less when the disk is fuller (see the next + option). Pinned originals can push the cache over it. +- `freeBelowBytes`: the free space to keep on the cache's volume. Default 50 + GiB. When the volume cannot hold the limit above and this much free space too, + the limit in force shrinks to what fits. `lib.status().originalsLimitBytes` + reports the limit in force. +- `isOriginalPinned`: a function that names more originals to pin. +- `pools`: the request pools, which set how many requests run at once (see + "Request pools"). Default: a fresh `RequestPools`. +- `onProgress`: a callback for the progress of refreshes, machine-learning + fetches and the background downloads, called with a `RefreshEvent`. +- `contentSource`: a replacement for the client's own byte source. Mainly for + tests. -The default `downloadDirectory` is resolved against the working directory once, -when the library opens; `lib.downloadDirectory` holds the result. +In the content and thumbnail pools, your own requests take the next free slot +ahead of the two background downloads. A file already in the cache is not +fetched again. When the originals cache grows past its limit, the least recently +used originals are evicted, but never a pinned one, and never one whose fetch is +still in flight. -`onProgress` receives a `RefreshEvent` with status `"started"` as a refresh, an -ML data fetch or a precache sweep starts, and `"done"` or `"failed"` as it ends. -In between, a refresh sends `"progress"` after each collection whose files it -fetched, with `fetched`, how many collections it has fetched the files of so -far, and `total`, how many it will. A refresh fetches the files of each -collection that is new or changed since the cache's copy, so on an empty cache -it fetches those of every collection. +`onProgress` receives a `RefreshEvent`. Its `operation` names the work: +`"refresh"`, `"fetchMLData"` for a machine-learning fetch, and +`"precacheThumbnails"` or `"precacheOriginals"` for the two background +downloads. Its `status` is `"started"` as the work starts and `"done"` as it +ends. A refresh or a machine-learning fetch that fails ends with `"failed"` +instead, with the message in `error`. In between, a refresh sends `"progress"` +after each collection whose files it fetched: `fetched` is the number of +collections fetched so far, and `total` the number it will fetch. A refresh +fetches the files of each collection that is new or changed since the cache's +copy, so on an empty cache it fetches those of every collection. The `"done"` +event of a machine-learning fetch, and of each background download, also carries +`fetched`: how many items it newly stored. -Concurrency is set through `pools`: construct -`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })` -and pass it. The three pools default to 10 / 5 / 25 (see Request pools below). +`lib.status()` returns a `LibraryStatus`. It holds the collection and file +counts. `lastRefreshAt` is the time of the last refresh that succeeded, and +`lastError` the message of the latest refresh, set only while that refresh +failed. `lastMLFetchAt` and `lastMLError` do the same for machine-learning +fetches. It also holds the originals cache's usage and limit in force, the +background downloads' progress and `closed`. `lib.status().mlDataErrors` lists +each file whose data arrived in the last successful fetch but could not be +decrypted, decompressed or parsed. Each entry is `{ fileID, error }`. The +`fetchMLData` event with status `"done"` carries the same list. Such data is not +cached, so the next fetch asks for it again. The reasons are kept in memory +only. -`lib.status()` returns a `LibraryStatus` (collection/file counts, last -refresh/ML times and errors, originals usage and effective limit, precache -progress, and `closed`). `lib.close()` stops the background timer; it is -idempotent, and an in-flight refresh is left to finish. The promise it returns -resolves once that refresh (including its cache write), the ML data fetch and -the precache fetches already running have all finished, so the cache directory -can then be removed. +`lib.close()` stops the background timer. It can be called more than once. A +refresh in flight is left to finish, with its write to the cache. `close()` +returns a promise. It resolves once that refresh, the machine-learning fetch and +the background downloads already running have all finished. The cache directory +can then be removed. After `close()`, `fresh()`, `freshCollections()`, +`freshFiles()` and `lib.backup()` reject with `the library is closed`. -`lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML -data the last ML data fetch that succeeded got but could not decrypt, gunzip or -parse; the `fetchMLData` event with status `"done"` carries the same list. A -fetch that is still running leaves the list as it is until it ends, and a fetch -that fails leaves it as it was. Such ML data is not cached, so the next ML data -fetch asks for it again. The reasons are kept in memory only, never in the -cache. +### Default reads and fresh reads -### Default reads vs. fresh reads +Default reads, through `lib.albums`, `lib.photos` and `lib.timeline`, answer +synchronously from the copy held in memory and never touch the network. A +refresh replaces that copy only once all its server requests have succeeded. So +a default read is immediate, but only as current as the last refresh whose +requests all succeeded. Right after opening an existing cache, it is the copy on +disk. -Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer -synchronously from the copy held in RAM and never touch the network. A refresh -changes that copy only once all its server requests have succeeded, and the -background timer starts the next refresh `refreshIntervalSeconds` after the -previous one ends. So a default read is immediate, but only as current as the -last refresh whose requests all succeeded; right after opening an existing -cache, it is the copy on disk. - -`await lib.fresh()` waits for a refresh to complete and persist, and returns the -same `{ albums, photos, timeline }` namespaces, which then reflect a completed -server round-trip. When a refresh is already running, background or not, -`fresh()` waits for that one, so its answer can come from requests made before -the call; only when none is running does it start one. A refresh that fails -rejects the caller (default reads stay silent and keep serving the last good -copy). The CLI's read commands use fresh reads (issue -https://git.eeqj.de/sneak/quak/issues/75). +`await lib.fresh()` waits for a refresh to complete and persist, then returns +the same `{ albums, photos, timeline }`, which then reflect a completed server +round-trip. When a refresh is already running, `fresh()` waits for that one, so +its answer can come from requests made before the call. Only when none is +running does it start one. When the refresh fails, `fresh()` rejects. Default +reads stay silent and keep serving the last good copy. `await lib.freshCollections()` is the fresh read of the collection list alone: -one collections request, answered with every live collection that decrypts, as -the `Collection` objects `listCollections()` returns, in the server's order. It -fetches no collection's files and stores nothing, not even the list: a refresh -fetches a collection's files only when the collection is new or has changed -since the stored copy, so a collection stored without its files would never get -them. `quak collections` uses it (issue -https://git.eeqj.de/sneak/quak/issues/210). +one request, answered with every collection that is not deleted and decrypts, as +`Collection` objects, in the server's order. It fetches no collection's files +and stores nothing, not even the list. It stores nothing because a refresh +fetches a collection's files only when the collection is new or changed. A +collection stored without its files would never get them. -An album or file in the server's answer that quak cannot decrypt, or whose -metadata is not JSON, does not fail the refresh. The refresh leaves it out, an -album with all its files, and drops any older copy the cache held, so reads -answer from what did decrypt. The library keeps a list of what it left out, with -the reason, in `metadata.json`: the next refresh asks the server only for what -changed, so it would not see the record again. A record stays on that list until -a later refresh gets a copy that decrypts or learns that the server deleted it. -`lib.backup()` reports each such album, and each such file in its scope that has -no copy that decrypted (see "Backup layout"). +`await lib.freshFiles(collection)` is the fresh read of one collection's files, +as `EnteFile` objects, in the server's order, each file once. `collection` is an +object that `freshCollections()` returned. The method fetches no other +collection's files and stores nothing. -### Read surface +The refresh does not fail on a record it cannot decrypt, or whose metadata is +not JSON. It leaves the record out; a collection left out takes its files with +it. It also drops any older copy the cache held, so reads answer from what did +decrypt. The library keeps a list of what it left out, with the reason, in +`metadata.json`. It needs the list because the next refresh asks only for what +changed, and would not see the record again. A record stays on that list until a +later refresh gets a copy that decrypts or learns that the server deleted it. -- `lib.albums.list()` → `Album[]`, newest-updated first. - `lib.albums.byID({ collectionID })` and `byName({ albumName })` → - `Album | undefined`. -- `lib.photos.byID({ fileID })` → `Photo | undefined`. - `lib.photos.records({ fileIDs })` → `PhotoRecord[]` in the requested order, - each id once, unknown ids dropped. -- `lib.timeline.groups({ groupBy, filter? })` → `TimelineGroup[]`, grouped by - `"day" | "week" | "month"` (keys `YYYY-MM-DD`, ISO `YYYY-Www`, `YYYY-MM`), - newest group first. A `PhotoFilter` combines `albumID`, `text` - (title/caption/album-name substring), `fileTypes`, `hasLocation`, and - `includeArchived`; hidden photos are always excluded. +### Albums, photos and the timeline -An `Album` exposes its record fields and `album.photos.list()` → `Photo[]` -(newest first). A `Photo` exposes its record fields other than `thumbnailPath` -and `originalPath`, and `photo.year`, the local-time year of `takenAt`, all as -synchronous getters read from RAM; `photo.record()` → `PhotoRecord`. Two more -synchronous getters look at the disk and never touch the network: +- `lib.albums.list()` returns `Album[]`, newest updated first. + `lib.albums.byID({ collectionID })` and `lib.albums.byName({ albumName })` + return `Album | undefined`. Names are not unique in Ente, so `byName` picks + the newest updated album of that name. +- `lib.photos.byID({ fileID })` returns `Photo | undefined`. + `lib.photos.records({ fileIDs })` returns `PhotoRecord[]` in the order asked + for, each ID once, unknown IDs dropped. +- `lib.timeline.groups({ groupBy, filter? })` returns `TimelineGroup[]`, newest + group first. `groupBy` is `"day"`, `"week"` or `"month"`, and the group keys + are `YYYY-MM-DD`, `YYYY-Www` (the ISO week) and `YYYY-MM`. Each group holds + `key`, `startsAt` and `fileIDs`. `key` is that group key. `startsAt` is the + start of the period, in local-time milliseconds. `fileIDs` lists its files, + newest first. Weeks start on Monday. -- `photo.savePath` → `string` — where `photo.download()` and `lib.backup()` put - the original, whether or not it is there yet: - `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's - `downloadDirectory` (see Backup layout above for the date and the extension). - For a live photo already stored, its image. For a live photo not yet stored, - it carries the title's extension, and the image may be stored under a - different one, found inside the live photo. It needs no content source. -- `photo.isLocal` → `boolean` — whether the whole original is at `savePath`. A - copy only in the cache does not count. +A `PhotoFilter` narrows the timeline. Its fields combine with AND: -These async methods may download: +- `albumID`: only photos in that album. +- `text`: a case-insensitive substring of the title, the caption or the name of + an album the photo is in. +- `fileTypes`: only photos of one of these types. +- `hasLocation`: `true` keeps only photos with coordinates, `false` only those + without. +- `includeArchived`: archived photos are left out unless this is `true`. -- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the - full-resolution file. For a live photo, `path` and `bytes` are its image's and - `videoPath` is its video. -- `await photo.download()` → `{ path, bytes, videoPath? }`, as `original()` — - puts the original at `savePath`, creating its folders. When it is already - there, nothing is written. When the cache holds it, it is copied from the - cache; otherwise it is fetched straight to `savePath`, with no copy left in - the cache. Afterwards `isLocal` is true. -- `await photo.thumbnail(opts?)` → `{ path, bytes }`. -- `await photo.content(opts?)` → `Uint8Array` — the original's bytes, read - through `original()`; for a live photo, its image's. -- `await photo.exif(opts?)` → `ExifTags` — every EXIF tag in the file, keyed by - tag name, each as exifreader decodes it, with its `id`, `value`, `description` - and `computed` value: for example `Make` is - `{ id: 271, value: ["Canon"], description: "Canon", computed: "Canon" }`. A - tag exifreader has no name for is keyed `undefined-<tag number>`. The embedded - thumbnail's tags are under `Thumbnail`, so they cannot hide the main image's - tags of the same name; the thumbnail image itself is left out. EXIF is read - from any image format exifreader reads (such as JPEG, HEIC/HEIF, AVIF, PNG, - WebP and TIFF), a live photo's image included. Any other original gives `{}`, - and a video gives `{}` without being downloaded. -- `await photo.make(opts?)`, and likewise `model()`, `lensModel()`, - `dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`, - `iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()` - and `gpsAltitude()` → one common field each, picked from the tags `exif()` - returns and typed as in `PhotoExif`, or `undefined` when the file lacks it. - GPS values are signed decimal degrees and metres. `dateTimeOriginal()` is the - camera's clock reading held in the `Date`'s UTC fields; - `offsetTimeOriginal()`, when present, is that clock's offset from UTC. Each - calls `exif()` with its `opts`, so each call reads the original again. +Hidden photos are always left out. -They serve from the on-disk content cache when the bytes are present and -otherwise fetch through the pools; `original()`, `content()` and `exif()` also -serve an original already stored at its save path. `opts.onProgress` reports -per-file progress. They throw when the library was opened without a content -source. An original that `original()`, `content()` or `exif()` downloads lands -in the cache, which does not make `isLocal` true; only `download()` and -`lib.backup()` do. +An `Album` exposes its record's fields and `album.photos.list()`, which returns +`Photo[]`, newest first. A `Photo` exposes its record's fields except the cache +paths, plus `photo.year`, the local-time year of `takenAt`. All of these are +synchronous getters read from memory. `photo.record()` returns the +`PhotoRecord`. -Lower-level accessors that return decrypted model objects (which hold key -material) are also available: `listCollections()`, `getCollection(id)`, -`listFiles(collectionID)`, `getFile(collectionID, fileID)`, and -`getFileByID(fileID)`. +Two more getters look at the disk and never touch the network: + +- `photo.savePath`: where `photo.download()` and `lib.backup()` put the + original, whether or not it is there yet (see "Save paths"). For a live photo + already stored, it is the image. For a live photo not yet stored, it carries + the title's extension, and the image may be stored under another one. +- `photo.isLocal`: whether the whole original is at `savePath`. A copy only in + the cache does not count. + +### A photo's content + +These methods may download. They serve from the cache when the bytes are present +and otherwise fetch through the pools. They throw when the library was opened +without a content source. `opts.onProgress`, where accepted, reports the +progress of that one file. + +- `await photo.original(opts?)` returns `{ path, bytes, videoPath? }`, the + full-size file. `bytes` is the file's size. For a live photo, `path` and + `bytes` are its image's and `videoPath` is its video. `original()` also serves + an original already stored at its save path. +- `await photo.download()` returns the same shape and puts the original at + `savePath`, creating its folders. When it is already there, nothing is + written. When the cache holds it, it is copied from the cache. Otherwise it is + fetched straight to `savePath`, with no copy left in the cache. Afterwards + `isLocal` is true. +- `await photo.thumbnail(opts?)` returns `{ path, bytes }`. +- `await photo.content(opts?)` returns the original's bytes as a `Uint8Array`, + read through `original()`. For a live photo, its image's. +- `await photo.exif(opts?)` returns `ExifTags`: every EXIF tag in the file, + keyed by tag name, each as exifreader decodes it with its `id`, `value`, + `description` and `computed` value. For example, `Make` is + `{ id: 271, value: ["Example"], description: "Example", computed: "Example" }`. + A tag that exifreader has no name for is keyed `undefined-<tag number>`. The + tags of the thumbnail embedded in the file are under `Thumbnail`, so they + cannot hide the main image's tags of the same name. The thumbnail image itself + is left out. Any file exifreader cannot read gives `{}`, and a video gives + `{}` without being downloaded. +- `await photo.make(opts?)` returns one common EXIF field, picked from the tags + `exif()` returns and typed as in `PhotoExif`, or `undefined` when the file + lacks it. These methods do the same for their fields: + - `model()` and `lensModel()`; + - `dateTimeOriginal()` and `offsetTimeOriginal()`; + - `exposureTime()`, `fNumber()`, `iso()` and `focalLength()`; + - `orientation()`; + - `gpsLatitude()`, `gpsLongitude()` and `gpsAltitude()`, as signed decimal + degrees and metres. + +`dateTimeOriginal()` is the camera's clock reading, held in the `Date`'s UTC +fields. `offsetTimeOriginal()`, when present, is that clock's offset from UTC. +Each of these methods calls `exif()`, so each call reads the original again. + +An original that `original()`, `content()` or `exif()` downloads lands in the +cache, which does not make `isLocal` true. Only `download()` and `lib.backup()` +do. + +`await lib.originalOf(file, opts?)` and `await lib.thumbnailOf(file, opts?)` +fetch and cache as `photo.original()` and `photo.thumbnail()` do. `file` is an +`EnteFile` the caller holds, and the library need not hold it. `quak get` and +`quak get-thumb` use them; on an empty cache they pass a file that +`lib.freshFiles()` returned. + +Lower-level reads return the decrypted model objects, which hold key material: +`lib.listCollections()`, `lib.getCollection(id)`, `lib.listFiles(collectionID)`, +`lib.getFile(collectionID, fileID)` and `lib.getFileByID(fileID)`. ### Records and change notifications -The GUI-facing records hold no key material and no binary, so they survive -`structuredClone`/JSON across the Electron IPC boundary: +The records hold no key material and no binary, so they survive +`structuredClone` and JSON, for example between processes: -- `PhotoRecord`: `fileID`, `albumIDs`, `title`, `takenAt` and `modifiedAt` - (milliseconds), `fileType`, optional `caption` / `width` / `height` / - `latitude` / `longitude`, optional `hash` (the content hash recorded at - upload; very old files have none), `isArchived`, `isHidden`, and - `thumbnailPath` / `originalPath` once the bytes are cached (for a live photo, - `originalPath` is its image). -- `AlbumRecord`: `collectionID`, `name`, `type`, `isShared`, `updationTime`, and - `fileIDs` (newest first). -- `LibrarySnapshot`: `{ albums, photos, takenAt }`. +- `PhotoRecord` holds: + - `fileID` and `albumIDs`; + - `title`; + - `takenAt` and `modifiedAt`, in milliseconds; + - `fileType`, `isArchived` and `isHidden`; + - optional: `caption`, `width`, `height`, `latitude` and `longitude`; + - optional: `hash`, the content hash recorded at upload, which very old + files lack; + - optional: `thumbnailPath` and `originalPath`, once the bytes are cached. A + live photo's `originalPath` is its image. +- `AlbumRecord` holds `collectionID`, `name`, `type`, `isShared`, `updationTime` + and `fileIDs`, newest first. +- `LibrarySnapshot` is `{ albums, photos, takenAt }`. Here `takenAt` is when the + snapshot was taken, in milliseconds. -`lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first, -photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange` -(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`, -`refreshedAt`) whenever a refresh alters the projection, and returns -`{ unsubscribe }`; a refresh that changes nothing delivers nothing. +`lib.snapshot()` returns a `LibrarySnapshot`, albums newest updated first and +photos newest first. `lib.subscribe({ onChange })` delivers a `LibraryChange` +whenever a refresh changes the records, and returns `{ unsubscribe }`. A +`LibraryChange` holds `albumsChanged`, `photosChanged`, `fileIDsRemoved`, +`albumIDsRemoved` and `refreshedAt`. A refresh that changes nothing delivers +nothing. -### Thumbnails, ML search, and backup +### Thumbnails, machine-learning data and backup -- `lib.thumbnails.ensure({ fileIDs, priority, signal?, onProgress? })` - prefetches thumbnails through the thumbnail pool, deduped by fileID, returning - one `EnsureResult` (`{ fileID, path?, error? }`) per file. `priority` is - `"visible" | "ahead" | "background"`; only `"visible"` preempts background - work. -- `lib.mldata` searches the CLIP index built from Ente's per-file ML data: - `forFile({ fileID })` → `Promise<MLData | undefined>` (the whole stored - payload — face boxes, landmarks, embedding — read from disk on demand); - `similar({ fileID, limit? })` and `searchByEmbedding({ embedding, limit? })` → - `SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first, - default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes - a query vector the caller produced elsewhere. -- `await lib.backup(opts?)` → `BackupResult`. It takes the lock in the download - directory, and fails at once with an error whose `code` is `ELOCKED` while - another backup of it runs. A caller can instead take the lock itself, before - it opens its library, as `quak backup` does: `await lockBackupDirectory(dir)` - takes it, failing the same way, and returns a `BackupLock`, whose `release()` - removes the lock unless it was lost and whose `lost` holds why it was lost. - The caller passes it to the backup as `lock`, and releases it once the backup - ends. It waits for a refresh as `fresh()` does, puts every in-scope original - not already at its save path there as `photo.download()` does (and, with - `includeThumbnails`, fetches thumbnails) through the content cache, waits for - an ML data fetch, and rebuilds the on-disk backup tree with a durable failure - ledger. Each file's JSON holds its ML data and, for an image (for a live - photo, its image), its original's EXIF and XMP, with its dimensions for a JPEG - only; a video's JSON holds none of these three. A fetched original is written - straight to its save path and not into the cache, which then counts it as - present; one the cache already held is copied from there. `BackupOptions`: - `downloadDirectory` (falls back to the library's), `includeOriginals` (default - `true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify` - (default `false`), `onProgress`, and `lock`. See Backup layout above for the - tree it writes, and for the albums `BackupResult` lists in - `undecryptableAlbums` and `failedAlbums`. Whether the backup took the lock or - its caller did, once the lock is lost the backup rejects with - `lost the lock on <dir>: <reason>` before it starts another file. +`lib.thumbnails.ensure({ fileIDs, priority, signal?, onProgress? })` fetches +thumbnails ahead of need through the thumbnail pool, each file once, and returns +one `EnsureResult` per file: `{ fileID, path?, error? }`. `priority` is +`"visible"`, `"ahead"` or `"background"`. Only `"visible"` goes ahead of +background work; `"ahead"` is treated as `"background"`. + +`lib.mldata` searches the CLIP index built from Ente's per-file machine-learning +data: + +- `forFile({ fileID })` returns `Promise<MLData | undefined>`, the whole stored + payload (faces, landmarks and embedding), read from disk on demand. `MLData` + is a plain object type and is not exported by name. +- `similar({ fileID, limit? })` and `searchByEmbedding({ embedding, limit? })` + return `SimilarResult[]`, each `{ fileID, score }`, most similar first. The + score is the cosine similarity, from -1 to 1 (see "Keywords"). `limit` + defaults to 20. quak bundles no text encoder, so `searchByEmbedding` takes a + query vector the caller produced elsewhere. + +`await lib.backup(opts?)` returns a `BackupResult` and writes the tree described +under "The backup directory". It takes the lock in the download directory and +fails at once, with an error whose `code` is `ELOCKED`, while another backup of +it runs. Then it waits for a refresh as `fresh()` does. It puts the original of +every file in the albums it backs up at its save path, as `photo.download()` +does. It waits for a machine-learning fetch, and rebuilds the JSON files and +symlinks. Once the lock is lost, it rejects with +`lost the lock on <dir>: <reason>` before it starts another file. + +`BackupOptions`: + +- `downloadDirectory`: where the tree lives. Default: the library's. +- `includeOriginals`: fetch and store originals. Default `true`. With `false`, + the backup also fetches no machine-learning data, and writes no JSON beside + the originals and no symlinks. +- `includeThumbnails`: also store thumbnails as `thumbnails/<fileID>.jpg` beside + `collections/`. Default `false`. `quak backup` does not set it. +- `onlyAlbumNames`: back up only the albums with these names. +- `verify`: as `--verify`. Default `false`. +- `onProgress`: called with each progress line and, for a line that reports a + failure, the level `"error"`. +- `lock`: a lock the caller took itself. + +A caller can take the lock before it opens its library, as `quak backup` does: +`await lockBackupDirectory(dir)` takes it, failing the same way, and returns a +`BackupLock`. Its `release()` removes the lock unless it was lost, and its +`lost` holds why it was lost. The caller passes it to the backup as `lock` and +releases it once the backup ends. ### Request pools -`RequestPools` holds three independent bounded pools — metadata (10), content -(5), thumbnails (25) — because Ente meters these traffic classes differently. -Each pool orders on-demand work ahead of background/precache work and dedups -in-flight fetches by key, and an idle pool never lends its slots to a busy one. +`RequestPools` holds three independent pools, each with its own limit on how +many requests run at once, because Ente meters these three kinds of traffic +differently: -`lib.backup()`, and so `quak backup`, works on up to the content pool's limit of -originals at once, 5 unless the library was opened with other `pools`. The limit -covers everything it does for one original, hashing it for `verify` included, so -however large the account is, no more originals than that are open at once. +- metadata, with 10 slots: only the machine-learning data requests; +- content, with 5: originals; +- thumbnails, with 25: thumbnails. -### On-disk cache layout +Only the requests named above go through the pools. A refresh, +`freshCollections()` and `freshFiles()` send theirs outside them. Construct one +with +`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })` +and pass it as `pools`. Each pool serves on-demand work ahead of background +work. Two tasks given to `BoundedPool.run()` with the same `key` run once while +the first is pending, and both get its result. An idle pool never lends its +slots to a busy one. -Under `cacheDirectory`: +`lib.backup()` works on as many originals at once as the content pool allows, 5 +by default. `quak backup` uses the default. The limit covers everything it does +for one original, hashing for `verify` included. -``` -<cacheDirectory>/ - metadata.json decrypted account state + refresh cursor - originals/<fileID>.<ext> cached full-resolution files - originals/<fileID>.livephoto.json - which of a live photo's two files is which - thumbnails/<fileID>.jpg cached thumbnails - mldata/ - <fileID>.json one decrypted ML payload per file - clip.f32, clip.json the packed CLIP index and its id list - fetched.json per-file fetch bookkeeping -``` +### Retries and timeouts -When `metadata.json` belongs to a different account than the client's, -`Library.open` deletes it and `mldata/` and starts from an empty cache. Cached -originals and thumbnails are kept; they are reached only through the files the -current account's records name. +Every request in the library goes through one policy. A request is repeated only +when repeating it could produce a different answer: -A live photo's original is cached as at its save path: its image and its video, -each `originals/<fileID>.<ext>` with its own extension, and -`originals/<fileID>.livephoto.json` naming them; the two are evicted together. +- An `ApiError` with a 5xx status is retried. So are `408` and `429`, the two + 4xx codes that are statements about timing rather than about the request. +- Every other 4xx is not retried. A 404 in particular is an answer, and + `list-missing-thumbnails` depends on getting it promptly and once. +- A transport failure is retried: a rejected `fetch`, `ECONNRESET`, `ETIMEDOUT`, + a DNS or TLS failure, and one of quak's own deadlines. Node's `fetch` rejects + with a plain `TypeError`, so every `TypeError` is retried. The error code is + looked for along the error's `cause` chain, which is where Node's `fetch` puts + it. +- A truncated download is retried. +- Anything else is not retried. This includes an authentication failure in the + encrypted stream that is not truncation. -A stored file appears only via an atomic temp-then-rename, so its presence means -it is complete. Every downloaded original (by `quak get`, the cache, or -`backup`) whose metadata records a content hash (`FileMetadata.hash`) is hashed -as it is written: unkeyed BLAKE2b with a 64-byte output, standard base64. A live -photo arrives as a ZIP and is unpacked as it is written; its image and its video -are hashed separately and joined as `<imageHash>:<videoHash>`, and neither is -stored unless both are complete and match. A live photo ZIP holding a second -image or video, or whose two files together come to more than 20 times its size -plus 16 MiB, is refused. A mismatch or a refusal stores nothing and fails the -download with an error naming the file ID. An original with no recorded hash, -from a very old client, is stored unchecked. +The binary that `make install` builds runs under bun. Bun's `fetch` rejects with +a plain `Error` that has no `cause`. That error carries a code of bun's own. +quak retries a refused connection, a name that does not resolve, a reset +connection and a deadline under bun as it does under Node. Bun reports a refused +connection, and a failed TLS handshake, as `ConnectionRefused`. For a name that +does not resolve it gives `ConnectionRefused` on some requests and +`FailedToOpenSocket` on others. A connection reset or closed before the response +ended is `ECONNRESET`, as under Node, and a deadline is the same `TimeoutError`. +A certificate bun rejects, with a code such as `DEPTH_ZERO_SELF_SIGNED_CERT`, is +not retried. + +Backoff is exponential with full jitter. The delay before retry _n_ is +`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term +is the ceiling, and the wait is drawn at random below it. So a client that lost +many parallel downloads to one blip does not send them all again at the same +instant. The numbers are set through `ApiClientOptions.retry`: + +- `attempts`: total calls, not retries. Default `4`; `quak backup` uses `10`. +- `baseDelayMs`: the ceiling for the first retry's delay. Default `500`; + `quak backup` uses `1000`. +- `maxDelayMs`: the upper bound on that ceiling. Default `10000`; `quak backup` + uses `60000`. + +With the defaults, a request that keeps failing gives up after at most three and +a half seconds of waiting. `quak backup` usually runs from cron with nobody +watching, so every request it makes uses its own numbers, exported as +`UNATTENDED_RETRY_OPTIONS`. Under them, a request that keeps failing gives up +after at most 243 seconds of waiting. It usually gives up after about half that, +since each wait is drawn at random below its ceiling. Every other command uses +the defaults. A library user gets the backup's budget by passing +`UNATTENDED_RETRY_OPTIONS` as `ApiClientOptions.retry`. `sleep` and `random` can +be injected through the same option, which is how the test suite exercises the +policy without waiting. `onRetry`, given there too, is called before each wait +with the error, the attempt's number and the wait. + +Two deadlines apply, each renewed for every attempt: + +- `requestTimeoutMs`, default `30000`, bounds a whole JSON request or upload. +- `downloadTimeoutMs`, default `60000`, is an idle deadline on a file or + thumbnail download: the download is aborted when no bytes have arrived for + that long. + +They differ because a download's length depends on the file and the link. A +whole-transfer deadline short enough to catch a hung connection would cancel a +large video on a slow link that is still making progress. The download deadline +restarts each time bytes arrive, so a slow download runs as long as it keeps +moving, and one that stalls is aborted after 60 seconds of silence. The download +deadline covers the wait for the headers as well as the body. There is no limit +on the total length of a download. + +A request that changes state is not blindly replayed. Every `POST` and `PUT` in +the endpoint list under "HTTP API" goes through this rule. Verifying a second +factor, for one, uses up one of a small number of attempts. Such a request is +retried only when every error code in its `cause` chain proves that no TCP +connection to the server ever existed. Three codes prove that: + +- `ENOTFOUND`: name resolution found no address; +- `EAI_AGAIN`: name resolution failed for the moment; +- `ECONNREFUSED`: the peer refused the connection. + +A 5xx, a reset mid-flight and a deadline are all left to the caller, because +each of them can happen after the server has already acted. These requests do +not follow redirects either: a redirect means the server already received the +request, so it is reported as an error and not retried. + +The routing errors `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are left out of +that rule although they look like connect-time failures. On Linux they are also +delivered on an established socket, after the request may have been written. +They stay retryable for requests that change nothing. Bun's `ConnectionRefused` +and `FailedToOpenSocket` are treated the same way: a request that changes +nothing is retried on them, and a `POST` or `PUT` is not replayed on them. Bun +reuses a connection it kept open from an earlier request. When that connection +closes before any answer, bun sends the request again on a new one. If that one +fails, bun gives one of these codes although the server may already have the +request. The upload of a thumbnail to its presigned URL is exempt from the rule. +It stores one whole object at one key in one request, so replaying it has no +partial state to damage. + +A download is retried as a whole: the request, the reading of the body and the +decryption. A socket reset after the response headers have arrived surfaces in +the download layer rather than in the HTTP client. That is the common failure +for a large file over a CDN. The encrypted stream cannot be resumed and the +endpoints have no Range support, so a retry starts the file over. Each attempt +writes its own temporary files and removes them if it fails. Only the attempt +that completes renames anything into place. + +One imprecision is deliberate. When a body ends partway through an encrypted +chunk, the authentication check fails and carries no sign of why, so a cut +connection and corrupt bytes look the same. quak reports that as truncation, +which means it is retried. quak can tell the two apart only in a body of more +than one chunk. There, a chunk that fails while the stream carries on past it +stays an authentication failure and is not retried. Most thumbnails and every +small file are one chunk. There a wrong key, corrupt bytes and a cut connection +look alike, and all are retried. The attempt count limits the extra retries. In +return, quak never keeps a truncated file. + +### The `Client` + +`Client.login({ email, password, totp?, emailOTP?, apiOptions? })` logs in and +resolves to a `Client`. `totp` and `emailOTP` are callbacks that return the code +when the account asks for one. Without the callback the account needs, login +throws. A passkey challenge always throws. After an email code, login expects +the key attributes and the token. When the server asks for a second factor +instead, login throws. `apiOptions` configures the HTTP client (see below). + +The `Client` holds the session token, the master key, the secret key and the +public key in memory. There is no on-disk session store in the library; the +caller decides how to keep a session. `client.toJSON()` returns a +`ClientSnapshot`, a plain object with the keys in base64, which the caller can +write wherever it likes. `Client.fromJSON(snapshot, apiOptions?)` restores a +working client from that snapshot without logging in again. It checks every +field and each key's length first, and throws an error naming the bad field. + +`Client.fromJSON` works only once libsodium, the cryptography library quak uses, +is loaded. `Client.login` loads it, and `Client.fromJSON` does not. A program +that restores a session in a new process first runs `await sodium.ready`, with +`sodium` imported from `libsodium-wrappers-sumo`, which quak depends on. The +exported `beginLogin`, `unwrapAuth`, `decryptCollection`, `decryptFile`, +`downloadFile` and `downloadThumbnail` need the same. + +`client.logout()` clears the token and zeroes the key buffers in place. After +it, every other method on that client throws. `logout()` does not contact the +server, so the token stays valid there and in any saved snapshot. Call +`await client.logoutOnServer()` before `logout()` to end the session on the +server. + +`client.whoami()` returns `{ email, userID }`. + +`client.collectionsSince({ sinceTime })` and +`client.filesSince({ collectionID, collectionKey, sinceTime })` enumerate the +changes since a microsecond timestamp, with `0` for everything. Each returns: + +- `collections` or `files`: the decrypted records that are not deleted; +- `deleted`: the IDs of deleted records; +- `undecryptable`: each record that did not decrypt, as `{ id, error }`; +- `cursor`: the timestamp to pass next time. + +`client.listCollections()` and `client.listFiles(collectionID, collectionKey)` +return only the decrypted records that are not deleted, from `sinceTime` `0`. +`client.fetchMLData({ fileIDs, fileKeys })` fetches and decrypts the +machine-learning data of up to 200 files. `client.downloadFile(file, outPath?)` +and `client.downloadThumbnail(file, outPath?)` download and decrypt one file to +`outPath`. For a live photo, `downloadFile` writes the image and the video, each +named `outPath` with its own extension. `quak get --out` instead puts the image +at the path itself. Without `outPath`, they write to the working directory under +the names `quak get` and `quak get-thumb` would use. + +`ApiClientOptions`: + +- `apiOrigin`: the API server. Default `https://api.ente.io`. A self-hosted + server is set here; file and thumbnail downloads then go through it too. +- `filesOrigin` and `thumbsOrigin`: the download hosts used with the default API + server. +- `authToken`: a session token to send from the start. +- `fetch`: a replacement for the global `fetch`. +- `retry`, `requestTimeoutMs` and `downloadTimeoutMs`: see "Retries and + timeouts". +- `onRequest`: called once for each request sent, and once for each attempt of a + retried one. It gets the method, the URL and the status of the answer or the + error. The URL of a thumbnail upload is given without its query, which holds + the upload's signature. + +The package also exports the pieces the `Client` and the `Library` are built +from, and a few constants: + +- the HTTP client: `ApiClient`, `ApiError` and `TruncatedStreamError`, with + `DEFAULT_REQUEST_TIMEOUT_MS` and `DEFAULT_DOWNLOAD_TIMEOUT_MS`; +- the retry policy: `withRetry`, `isRetryable`, `isSafeToReplay`, + `resolveRetryOptions`, `DEFAULT_RETRY_OPTIONS` and `UNATTENDED_RETRY_OPTIONS`; +- the login steps: `beginLogin`, `submitTOTP`, `requestEmailOTP`, + `submitEmailOTP` and `unwrapAuth`; +- the record decryption: `decryptCollection` and `decryptFile`; +- the downloads: `downloadFile` and `downloadThumbnail`; +- the backup: `runBackup` and `lockBackupDirectory`; +- the pools: `RequestPools` and `BoundedPool`, with + `DEFAULT_METADATA_CONCURRENCY`, `DEFAULT_CONTENT_CONCURRENCY` and + `DEFAULT_THUMBNAIL_CONCURRENCY`; +- the library: `Library`, `Album`, `Photo` and + `DEFAULT_REFRESH_INTERVAL_SECONDS`; +- `VERSION`: the version string `quak --version` prints. ### Key types by source file @@ -1344,8 +1529,8 @@ from a very old client, is stored unchecked. - `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`, `LibraryChange` - `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult` -- `src/exif.ts`: `ExifTags`, `PhotoExif` - `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool` +- `src/exif.ts`: `ExifTags`, `PhotoExif` - `src/backup.ts`: `BackupLock`, `BackupOptions`, `BackupResult`, `BackupError` - `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot` - `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `StreamOptions` @@ -1353,54 +1538,419 @@ from a very old client, is stored unchecked. - `src/retry.ts`: `withRetry`, `isRetryable`, `isSafeToReplay`, `RetryOptions` - `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileType`, `CollectionType`, `RawCollection`, `RawEnteFile` -- `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult` + +`src/index.ts` lists what the package exports. `MLDataAPI` and `SimilarResult` +are the types of `lib.mldata` and of its results, and `MLData` is the type of a +stored payload. None of the three is exported by name. + +## Examples + +`examples/download-albums.ts` downloads every album's photos and their metadata +into a directory, `photos` in the working directory unless you name another. It +uses only the library's public API. The build compiles it; run it after +`yarn install`: + +```bash +yarn build +QUAK_EMAIL=… QUAK_PASSWORD=… node dist/examples/download-albums.js [dir] +``` + +It opens the library with `precacheThumbnails` and `precacheOriginals` off, as +`quak backup` does, so the only file content it fetches is the originals it +saves. It asks on the terminal for a second-factor code or an email code when +the account requires one. It writes: + +- each photo's original at its save path under `dir`, as `photo.download()` + writes it, and for a live photo its image, its video and the `.livephoto.json` + file naming them; +- beside each original, a JSON file named after it with `.json` added, such as + `2026-03-01.12345.jpg.json`. It holds the photo's record without its cache + paths, and under `exif` every EXIF tag of the photo; +- `albums/<collectionID>.json` for each album, holding its `collectionID`, its + `name` and, under `savePaths`, the save paths of its photos relative to `dir`, + newest first. + +A photo in several albums is downloaded once. A second run downloads nothing and +rewrites only the JSON files whose content changed. A failed download stops the +run; running it again carries on, since every photo already saved is skipped. + +## Rationale + +Ente encrypts photos end to end, so a backup of an Ente account cannot be made +from the server's side. Only a client that holds the account's keys can read the +photos. Ente's own clients are apps and a command-line tool; none is a library a +program can use. quak is the client for scripts and for cron. + +quak is a library first, so that a program of your own can read an account the +way quak's command line does. The backup it writes is plain files in a tree a +photo viewer can open, with no quak needed to read it. One failed file never +ends a run; it is recorded and tried again next time. + +## Design + +quak is a TypeScript library with a thin command-line wrapper. The library does +the work; the command line is for people and cron. Login uses SRP, a protocol +that proves the password to the server without sending it. + +### Layout + +``` +quak/ + src/ + crypto/ libsodium primitives: boxes, streams, key derivation, + hashing + api/ the HTTP client (ApiClient) + auth/ the login flow (SRP, email code, TOTP) and key unwrap + model/ the decrypted Collection, EnteFile and FileMetadata + types, and the functions that decrypt them + download/ streaming download and decryption of files and + thumbnails + library/ the Library over the cache: metadata store, reads, + records, content cache, background downloads, + machine-learning data and search, request pools + backup.ts the full-account backup + metadata-backup.ts + backup-metadata: the metadata quak keeps, as JSON + exif.ts EXIF read from an image's bytes with exifreader + mldata-fetch.ts fetch and decrypt per-file machine-learning data + clip-keywords.ts + a file's keywords from its CLIP embedding + clip-keyword-vectors.ts + Ente's Discover categories with their text + embeddings, written by bin/clip-keyword-vectors + filename.ts safe file names from server metadata + errors.ts error types shared across layers, and how an error + is described + retry.ts the retry classifier and the backoff loop + thumbnails.ts find and regenerate missing thumbnails + client.ts the Client class assembled from the above + cli-commands.ts the command line's commands as functions returning + exit codes + cli-log.ts the command line's log level and the one function + its commands write stderr lines through + cli-output.ts how the command line prints a file's title and time + cli-read.ts how the read commands get current data + cli-run.ts run a command, print its error, exit with its code + cli-session.ts read the saved session file back into a Client + index.ts the package's exports + bin/ + quak.ts the command-line entrypoint (commander.js) + clip-keyword-vectors, clip-keyword-vectors.py + write src/clip-keyword-vectors.ts and a test + fixture (see "Source attribution") + examples/ + download-albums.ts + download every album's photos and metadata + test/ unit and integration tests (vitest) + script/ the entrypoints (see "Entrypoints") + Makefile + Dockerfile lint phase, test phase, compile + package.json + tsconfig.json +``` + +`make build` compiles `src/`, `bin/` and `examples/` into `dist/`, keeping their +shape: the library lands in `dist/src/`, the examples in `dist/examples/` and +the command line in `dist/bin/quak.js`. `package.json` points `main` and `types` +at `dist/src/index.js` and `dist/src/index.d.ts`, and `bin` at +`dist/bin/quak.js`. The compiler's `rootDir` is the repository root rather than +`src/`, because `bin/` and `examples/` are compiled too. + +### Cryptography + +All cryptography is done by `libsodium-wrappers-sumo`, with two exceptions. The +SRP handshake uses `fast-srp-hap`, and the MD5 checksum sent with a thumbnail +upload uses Node's `node:crypto`. The "sumo" build is required for +`crypto_pwhash`, the Argon2id function. No cryptography is written by hand. + +Login derives the keys in this order: + +1. The user enters the password. +2. Argon2id (`crypto_pwhash`) derives a 32-byte key encryption key, the KEK, + from the password. It uses the server-issued `kekSalt`, `memLimit` and + `opsLimit`. +3. For SRP login, `crypto_kdf_derive_from_key` (BLAKE2b) derives a 32-byte + subkey from the KEK with subkey ID 1 and context `loginctx`. Its first 16 + bytes are the SRP password. +4. When SRP completes, the server returns the key attributes and an encrypted + token, or first asks for a second factor. quak answers a TOTP request with + the code (`POST /users/two-factor/verify`); the server then returns the key + attributes and the token. quak cannot answer a passkey request. When the + account has email verification on (`isEmailMFAEnabled`), an email code + replaces SRP. quak then expects the key attributes and the token in the + answer, and fails the login when the server asks for a second factor instead. +5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK + yields the 32-byte master key. +6. `crypto_secretbox_open_easy` over the encrypted secret key with the master + key yields the user's X25519 private key. The matching public key arrives in + cleartext. +7. `crypto_box_seal_open` over the encrypted token with the user's key pair + yields the token's bytes. Encoded as URL-safe base64 with padding, they are + the `X-Auth-Token` value for every later call. + +A collection the account owns carries its key as a secretbox under the master +key. A collection shared with the account carries it as a sealed box to the +account's public key. Each file's key is a secretbox under its collection's key. +File metadata is a secretstream blob, a single chunk with the final tag, under +the file key. File content is a chunked `crypto_secretstream_xchacha20poly1305` +stream under the file key, with 4 MiB plaintext chunks and 17 bytes of +authentication overhead per chunk. A thumbnail is such a stream too. For a +thumbnail upload, `encryptBlob` does the push side: one chunk with the final +tag, returning the header and the ciphertext. + +### HTTP API + +The production endpoints are: + +- API: `https://api.ente.io` +- File downloads: `https://files.ente.io/?fileID=<id>` +- Thumbnail downloads: `https://thumbnails.ente.io/?fileID=<id>` + +A self-hosted server is set through the `apiOrigin` option. File downloads then +go through `<apiOrigin>/files/download/<id>` and thumbnail downloads through +`<apiOrigin>/files/preview/<id>`. + +Every authenticated call carries two headers: `X-Auth-Token`, the token from +login, and `X-Client-Package`, which names the client as `berlin.sneak.quak`. + +Endpoints used: + +- `GET /users/srp/attributes?email=<email>`: fetch the SRP and key-derivation + parameters. +- `POST /users/srp/create-session`: begin the SRP handshake. +- `POST /users/srp/verify-session`: complete SRP; receive the second-factor + challenge or the encrypted token with the key attributes. +- `POST /users/ott` and `POST /users/verify-email`: the email code, used instead + of SRP when the SRP attributes have `isEmailMFAEnabled` set. +- `POST /users/two-factor/verify`: the TOTP second factor. +- `POST /users/logout`: end the calling token's session (`quak logout`). +- `GET /collections/v2?sinceTime=<usec>`: list the collections changed since a + microsecond timestamp; `0` lists them all. +- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list the files + changed in a collection; quak pages while `hasMore` is true. +- `GET https://files.ente.io/?fileID=<id>`: download a file's encrypted bytes. +- `GET https://thumbnails.ente.io/?fileID=<id>`: download a thumbnail's + encrypted bytes. +- `POST /files/data/fetch`: fetch the encrypted machine-learning data of a batch + of files. +- `POST /files/upload-url`: get a presigned upload URL, for thumbnail repair. +- `PUT /files/thumbnail`: register an uploaded thumbnail's object key. ## Source attribution -The cryptographic protocol and wire format implemented here are Ente's, taken -from the Ente open source clients at <https://github.com/ente-io/ente>. No code -is imported or vendored from those projects; any reference code that is copied -is rewritten in TypeScript in this repository. Protocol fidelity is verified -against the upstream implementations in `web/packages/base/`, -`mobile/apps/photos/lib/`, and `cli/`. +The cryptographic protocol and wire format are Ente's, read from the Ente +clients at <https://github.com/ente-io/ente>. No code from those projects is +included; quak's implementation is written in TypeScript from reading them. -## For LLMs +The keywords come from Ente too. `src/clip-keyword-vectors.ts` holds the +Discover categories of Ente's photos app, from +`mobile/apps/photos/assets/discover.json`, each with the text embedding of its +query from Ente's CLIP text model. `bin/clip-keyword-vectors` writes that file, +and the CLIP embedding of a public-domain photo that the tests use. It runs +Ente's models in Docker, with every download pinned by SHA-256 hash. It is run +by hand only, followed by `make fmt`. -If you are an LLM agent working on this repository, read and follow these -documents: +## Entrypoints -- **`REPO_POLICIES.md`** in the repo root. It is copied from +This repository adheres to the +[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) +standard. Normalized scripts in `script/` are the entrypoints for the +development workflow, and most Makefile targets are thin shims that call them. +The scripts are POSIX `sh`, not bash, so they run in minimal containers such as +Alpine. Of these, only `bootstrap`, `setup`, `test` and `cibuild` come from the +standard. The others are additions to it. + +- `script/bootstrap`: install all dependencies. It installs `make`, `git`, Node + and `yarn` when they are missing, then runs `yarn install --frozen-lockfile`. +- `script/setup`: set up the repo for development after a fresh clone. It runs + `script/bootstrap`, then `script/install-precommit`. +- `script/projectname`: print the project name. `script/docker` uses it for the + image tag. +- `script/build`: compile the TypeScript sources into `dist/` and stamp the + version into `dist/package.json`. Then check that the entrypoints + `package.json` declares (`main`, `types`, `bin`) were written, and make the + command line executable. +- `script/version`: print the version `script/build` stamps (see "Version"). +- `script/test`: run the test suite, by building the `test` phase of the + `Dockerfile`. Requires Docker. +- `script/lint`: run eslint and a prettier check, by building the `lint` phase + of the `Dockerfile`. Requires Docker. +- `script/fmt`: format all files with prettier. It rewrites files in place. +- `script/fmt-check`: check formatting on the host. It is read-only. +- `script/check`: run all checks: `test`, `lint`, `fmt-check`. +- `script/docker`: build the image, tagged with `script/projectname`. +- `script/cibuild`: what CI runs: `script/bootstrap`, then `script/check`, then + the image build. +- `script/precommit`: run by the git pre-commit hook. It runs `script/lint` and + `script/fmt-check`, and not the tests, so that a commit of failing tests can + land (see "Development workflow"). +- `script/install-precommit`: install the git pre-commit hook. `make hooks` + calls it. + +The Makefile targets are: + +- `make bootstrap`: runs `script/bootstrap`. +- `make setup`: runs `script/setup`. +- `make test`: runs `script/test`. +- `make lint`: runs `script/lint`. +- `make fmt`: runs `script/fmt`. +- `make fmt-check`: runs `script/fmt-check`. +- `make check`: runs `script/check`. +- `make build`: runs `script/build`. +- `make docker`: runs `script/docker`. +- `make hooks`: runs `script/install-precommit`. +- `make build-bin`: bundles the built `dist/` into one binary, `bin/quak`. It + runs bun through `nix-shell -p bun`. +- `make install`: runs `make build-bin` and copies the binary to `~/bin/quak`. +- `make dev`: runs the TypeScript compiler in watch mode. +- `make clean`: removes `dist/`, `coverage/`, `.vitest-cache/` and the + compiler's `*.tsbuildinfo` files. + +### Linting and testing + +Linting and testing are phases of the `Dockerfile`. The `lint` phase copies the +repo into a node image pinned by digest and runs eslint and +`prettier --check .`. The `test` phase runs the test suite the same way, as the +image's non-root `node` user. A 90-second `timeout` stops a hung test. A suite +that fails is run again with verbose output. `script/lint` and `script/test` +each build one phase with `docker build --no-cache --target <phase>`. Neither +script runs the tools on the host, so Docker is required. The scripts also work +where the Docker daemon is remote, because they copy the repo into the image +instead of sharing a directory with the container. + +The last stage of the `Dockerfile` compiles the package. It copies a file from +each phase, so it cannot be built unless lint and the tests pass. The image +build in `script/cibuild` therefore runs lint and the tests a second time, after +`script/check` has run them. + +Every `docker build` in `script/` passes `--no-cache`. On an unchanged tree +Docker would otherwise serve the lint and test steps from its cache, nothing +would run, and the build would still exit 0. + +`script/fmt-check` runs prettier on the host. Its verdict matches the `lint` +phase's, because prettier is pinned to one version and installed from +`yarn.lock` in both places. Prettier reads `.gitignore` as its default ignore +file, which is why `.dockerignore` keeps `.gitignore` in the build context. + +### Version + +`quak --version` reports the `version` in `dist/package.json`, which +`script/build` writes after compiling. The repo's own `package.json` keeps +`0.0.0`. The tests run from source, so they report `0.0.0`. `script/version` +decides what is written: + +- the `VERSION` environment variable, or the `Dockerfile`'s `VERSION` build + argument (`--build-arg VERSION=...`), when one is given and not empty; +- otherwise, in a checkout with `.git`, what `git describe --tags --always` + prints. That is the tag on a tagged commit. On a later commit it is the tag, + the number of commits since it and the short commit, as in + `v1.2.3-4-gabc1234`. When no tag is reachable it is the short commit; +- otherwise, as in a source tarball, the version `package.json` declares. + +The build fails if the checkout has `.git` and the version still comes out +empty, `dev` or `unknown`, because such a build could not be traced to its +commit. + +`.dockerignore` therefore does not leave out `.git`, so any `docker build .` of +a clone stamps the commit it was built from. A clone of depth 1 stamps a tag +only when its commit carries one, and otherwise the short commit. +`.dockerignore` does leave out `.git/config`, which holds the clone's remote URL +and any credential in it; `git describe` does not need that file. +`script/docker` and `script/cibuild` work out the version on the host, with +`--dirty`, and pass it as the `VERSION` build argument. That argument takes +precedence. The image's `org.opencontainers.image.version` label carries that +build argument only, so a build given none leaves it empty. `make build-bin` +bundles the built `dist/`, so the single binary reports the stamped version too. + +## Development workflow + +All work on quak is test-driven. + +1. Every change starts on a feature branch off `next`, and its pull request + targets `next`. +2. The first commit on the branch is the tests for what is being added or + changed. Those tests fail at that commit; the branch is red until the + implementation lands. +3. Later commits add the implementation and any refactors needed to make the + tests pass. +4. A pull request is merged into `next` only when `make check` is green. Once it + has passed review, a maintainer squash-merges it into `next`. Only sneak + merges `next` into `main`. `main` and `next` are always green. CI runs + `script/cibuild`, which builds the `Dockerfile`: its `lint` and `test` + phases, then the compile. So neither a red branch nor one that does not + compile can pass CI. +5. Tests are the canonical API documentation for this library. Every test file + is commented well enough that a reader who has never seen quak can learn how + to use it from the tests alone. Comments explain why a behavior matters, not + just what the assertion checks. +6. Test fixtures, such as cryptographic vectors, recorded HTTP responses and + sample files, are committed beside their tests. Where possible they are + generated by deterministic helpers in `test/`, so a reviewer can reproduce + them by running the helper. +7. `git rebase -i` is allowed on a feature branch before merge, to clean the + tests-then-implementation sequence into reviewable commits. The final history + must still show tests landing before, or with, the matching implementation. +8. The pre-commit hook that `make hooks` installs runs `script/precommit`, which + runs `script/lint` and `script/fmt-check` and not the tests. This is + deliberate, so that the commit of failing tests can land. CI runs the tests, + so a red branch still cannot reach `next`. + +### Rules for contributors + +These apply to people and to coding agents alike. + +- Read and follow `REPO_POLICIES.md` in the repo root. It is copied from <https://git.eeqj.de/sneak/prompts> and covers repository structure, tooling, - Makefile targets, Dockerfile conventions, dependency pinning, and commit - hygiene. All external dependencies must be pinned by cryptographic hash in - `yarn.lock`. Never `git add -A`. Never force-push to main. + Makefile targets, Dockerfile conventions, dependency pinning and commit + hygiene. Never `git add -A`. Never force-push `main`. +- Dependencies are pinned: + - npm packages by hash, in `yarn.lock`; + - container images by digest; + - the nvm archive `script/bootstrap` downloads, and the Python packages + `bin/clip-keyword-vectors` installs, by SHA-256 hash; + - Node and yarn, when `script/bootstrap` installs them, by version. +- Not pinned: + - bun, which `make build-bin` takes from `nix-shell -p bun`; + - `make` and `git`, which `script/bootstrap` installs from the system + package manager when they are missing, as every `Dockerfile` stage does; + - `bash` and `curl`, which `script/bootstrap` installs the same way when it + has to install Node. +- Follow the workflow above: branches off `next`, tests first and committed + failing, and `next` merged into `main` by sneak alone. +- Before every commit, `make lint` and `make fmt-check` must pass. `make lint` + builds the `lint` phase of the `Dockerfile`, so it needs Docker. The + pre-commit hook enforces exactly that. `make check`, which also runs the + tests, must pass before a merge into `next`. Never invoke eslint or prettier + directly; linting runs in the container only. +- Format with `make fmt`: prettier with 4-space indents and `proseWrap: always` + for markdown. Use `yarn`, not `npm`. +- Tests use vitest and live in `test/`, mirroring `src/`. Use `mkdtempSync` for + temporary directories, never a path built from a timestamp. `make test` must + finish in under 60 seconds. The 90-second `timeout` in the `test` phase of the + `Dockerfile` only catches a hung test. +- Code style: `const` for everything, `let` only when a variable is reassigned, + never `var`. Avoid comments that restate the code. No cryptography written by + hand. `LLM_PROSE_TELLS.md` in the prompts repo applies to any prose written in + this repository: the README, comments and commit messages. -- **The "Development workflow" section above.** All changes go on feature - branches off `next`, and every pull request targets `next`; only sneak merges - `next` into `main`. Tests are written first and committed in a failing state - before the implementation. Tests are the canonical API documentation and must - be commented thoroughly. `main` and `next` are always green. +## TODO -- **Required checks before every commit:** `make lint` and `make fmt-check` must - pass. `make lint` is eslint plus the prettier check, and it builds the `lint` - phase of the `Dockerfile`, so it needs docker. The pre-commit hook enforces - exactly that. `make check` (which also runs the tests) must pass before - merging into `next`. Never invoke eslint or prettier directly; linting runs in - the container only. +`TODO.md` in the repo root is the work log: the next step, the steps done and +the steps planned. Open now: -- **Formatting:** prettier with 4-space indents and `proseWrap: always` for - markdown. Use `make fmt` to format. Use `yarn` not `npm`. +- [ ] Check quak against a live Ente account. This needs an account only sneak + has. -- **Testing:** vitest. Tests go in `test/` mirroring the `src/` structure. - `make test` must finish in under 60 seconds (the hard cap) and should finish - in under 20. The 90-second `timeout` in the `test` phase of the `Dockerfile` - is a backstop that catches a hung test, not the time limit. Use `mkdtempSync` - for temporary directories, never manual timestamp paths. +Planned for a desktop client, in a separate repository, on top of this library: -- **Code style:** `const` for everything, `let` if reassignment is needed, never - `var`. Avoid unnecessary comments. No hand-rolled crypto. The - `LLM_PROSE_TELLS.md` document in the prompts repo applies to any prose written - in this repository (README, comments, commit messages). +- [ ] An Electron app skeleton that uses this library. +- [ ] A local cache in SQLite keyed on `(collectionID, fileID, updationTime)`. +- [ ] A background sync worker that streams new files into the cache. +- [ ] A gallery UI: thumbnails, full-image view, basic search. +- [ ] Upload, delete and share operations in the library. + +Tagging and releases are decided by sneak alone, and happen only when he +declares one. ## License diff --git a/TODO.md b/TODO.md index f1de6ec..854ff37 100644 --- a/TODO.md +++ b/TODO.md @@ -26,6 +26,43 @@ declares one. # Completed Steps +- 2026-10-26: On an empty cache, `quak helper fix-missing-thumbnails --file` + stops fetching collections' files once it finds every file it names + (https://git.eeqj.de/sneak/quak/issues/223). It used to wait for a refresh, + which on an empty cache fetched the files of every collection. Now it reads + the collection list, then one collection's files at a time, the most recently + updated first. It stores none of the collections or file records it reads. It + fetches each original through `lib.originalOf()`, which keeps it in the cache. + Once the cache holds the account's collections, it still waits for a refresh. + +- 2026-10-26: On an empty cache, `quak get` and `quak get-thumb` stop fetching + collections' files once they find the file + (https://git.eeqj.de/sneak/quak/issues/222). They used to wait for a refresh, + which on an empty cache fetched the files of every collection. Now they read + the collection list, then one collection's files at a time. They store none of + the collections or file records they read, and fetch the original or thumbnail + through the new `lib.originalOf()` and `lib.thumbnailOf()`. Once the cache + holds the account's collections, they still wait for a refresh. + +- 2026-10-26: `quak files` with a collection ID fetches that collection's files + alone (https://git.eeqj.de/sneak/quak/issues/217). It used to wait for a + refresh, which on an empty cache fetched the files of every collection. It now + reads the collection list through `lib.freshCollections()`, then the one + collection's files through the new `lib.freshFiles()`. It stores nothing, and + the files print in the server's order. + +- 2026-10-22: The JSON `quak backup` and `quak backup-metadata` write for a file + holds `keywords` in place of the CLIP embedding in its ML data + (https://git.eeqj.de/sneak/quak/issues/218). The keywords are the titles of + the Discover categories of Ente's photos app that the file falls in, by the + app's rule: the cosine similarity of the file's CLIP embedding and the text + embedding of a category's query is at least the category's minimum score, and + videos stay out of the categories that leave them out. `src/clip-keywords.ts` + applies the rule. `src/clip-keyword-vectors.ts` holds the 16 categories with + their text embeddings. `bin/clip-keyword-vectors` writes it with Ente's CLIP + text model, and writes the CLIP embedding of a public-domain photo of a sunset + that the tests use. + - 2026-10-18: `-v`/`--verbose` raises the CLI's log level once for each repeat, and `-q`/`--quiet` prints only fatal errors (https://git.eeqj.de/sneak/quak/issues/212). The levels are fatal, error, diff --git a/bin/clip-keyword-vectors b/bin/clip-keyword-vectors new file mode 100755 index 0000000..df4a75f --- /dev/null +++ b/bin/clip-keyword-vectors @@ -0,0 +1,31 @@ +#!/bin/sh +# Writes src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json by +# running bin/clip-keyword-vectors.py in a Python image pinned by digest, with +# its packages pinned by hash. Run it by hand from a checkout, then `make fmt`; +# no gate runs it. Besides the Python image, it downloads about 265 MB, most of +# it Ente's two CLIP models. +set -eu + +ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" + +# python:3.13-slim (Python 3.13.17 on Debian 13.7), 2026-10-21. The wheels +# below are the ones for this image on linux/amd64. +IMAGE=python@sha256:57b29a7c050c4e679e4bea572e14b3965cf4b5839de2da49d55e49ffc4024f39 + +main() { + cd "$ROOT" + docker run --rm -i --platform linux/amd64 \ + --user "$(id -u):$(id -g)" -e HOME=/tmp \ + -v "$ROOT:/repo" -w /repo "$IMAGE" sh -c ' + python -m venv /tmp/venv && + /tmp/venv/bin/pip install --quiet --disable-pip-version-check \ + --no-cache-dir --no-deps --require-hashes -r /dev/stdin && + /tmp/venv/bin/python bin/clip-keyword-vectors.py' <<'EOF' +numpy==2.5.3 --hash=sha256:a5fa86b80fd24bcd1aff83ad23be44ea323de3f787be8f8b15d4a65621e25321 +onnxruntime==1.30.0 --hash=sha256:86f940afc801ea9681a4da8af84fbe95e1d9ea7d80903952cc1bfad54faad38f +pillow==12.3.0 --hash=sha256:0847a763afefb695bc912d7c131e7e0632d4edc1d8698f58ddabec8e46b8b6d3 +tokenizers==0.23.2 --hash=sha256:41c2f84d172449b4dadb9cdc508e3e364076613c35b16e76ecfe47a60d1e3305 +EOF +} + +main "$@" diff --git a/bin/clip-keyword-vectors.py b/bin/clip-keyword-vectors.py new file mode 100644 index 0000000..af888fa --- /dev/null +++ b/bin/clip-keyword-vectors.py @@ -0,0 +1,158 @@ +"""Write src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json. + +bin/clip-keyword-vectors runs this in a pinned Python image. It downloads +Ente's Discover list, Ente's CLIP text and image models, the CLIP tokenizer +and a public-domain photo of a sunset, checks each against its sha256, and +writes: + +- src/clip-keyword-vectors.ts: each Discover category of Ente's photos app, + with the text embedding Ente's CLIP text model gives its query; +- test/clip-keywords-sunset.json: the CLIP embedding Ente's image model gives + the photo, prepared as Ente's apps prepare a photo. + +Both embeddings are scaled to length 1, as Ente's apps scale them. +""" + +import hashlib +import io +import json +import urllib.request + +import numpy as np +import onnxruntime +from PIL import Image, ImageOps +from tokenizers import Tokenizer + +ENTE_COMMIT = "f5ad70a860ed25d0c8057b0efb4bd6eaefe4b0c5" +DISCOVER_PATH = "mobile/apps/photos/assets/discover.json" +DISCOVER_URL = ( + f"https://raw.githubusercontent.com/ente-io/ente/{ENTE_COMMIT}/{DISCOVER_PATH}" +) +DISCOVER_SHA256 = "cf70ac7116b592f7921ca4cf67cdbb421a8e6b287fbe44b2be678b8d198e5c24" + +TEXT_MODEL = "mobileclip_s2_text_opset18_quant.onnx" +TEXT_MODEL_SHA256 = "d92f33dfcff83077fc2e0d3414250710efbb51795dfd89767bdbefb5fdc47322" +IMAGE_MODEL = "mobileclip_s2_image_gelu_opset20.onnx" +IMAGE_MODEL_SHA256 = "205a430af825e501c5138e5bb9abea942482a7a4fd4a680e98e47cf0830dce7e" + +# The CLIP tokenizer Hugging Face publishes for OpenAI's CLIP. Its vocabulary +# and merges are those of bpe_simple_vocab_16e6.txt, the vocabulary Ente's +# apps tokenize with. +TOKENIZER_URL = ( + "https://huggingface.co/openai/clip-vit-base-patch32/resolve/" + "3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268/tokenizer.json" +) +TOKENIZER_SHA256 = "b556ac8c99757ffb677208af34bc8c6721572114111a6e0aaf5fa69ff0b8d842" + +# "Sunset off the Massachusetts coast", a NOAA photo in the public domain: +# https://commons.wikimedia.org/wiki/File:Sunset_off_the_Massachusetts_coast_-_NOAA.jpg +PHOTO_URL = ( + "https://upload.wikimedia.org/wikipedia/commons/d/db/" + "Sunset_off_the_Massachusetts_coast_-_NOAA.jpg" +) +PHOTO_SHA256 = "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3" + +# Ente's text model takes 77 token ids, padded with 0; its image model takes +# a 256x256 RGB image. +TOKEN_COUNT = 77 +IMAGE_SIZE = 256 + + +def download(url, sha256): + # Wikimedia refuses requests without a User-Agent. + request = urllib.request.Request(url, headers={"User-Agent": "quak"}) + with urllib.request.urlopen(request) as response: + data = response.read() + digest = hashlib.sha256(data).hexdigest() + if digest != sha256: + raise SystemExit(f"{url} has sha256 {digest}, not {sha256}") + return data + + +def ente_model(name, sha256): + return onnxruntime.InferenceSession( + download(f"https://models.ente.com/{name}", sha256), + providers=["CPUExecutionProvider"], + ) + + +def run(session, tensor): + (output,) = session.run(None, {session.get_inputs()[0].name: tensor}) + embedding = output[0] + return embedding / np.linalg.norm(embedding) + + +def text_embedding(session, tokenizer, query): + ids = tokenizer.encode(query).ids + ids += [0] * (TOKEN_COUNT - len(ids)) + return run(session, np.array([ids], dtype=np.int32)) + + +def image_embedding(session, photo): + image = ImageOps.exif_transpose(Image.open(io.BytesIO(photo))).convert("RGB") + # Scale to cover IMAGE_SIZE x IMAGE_SIZE and crop the centre. + square = ImageOps.fit(image, (IMAGE_SIZE, IMAGE_SIZE), Image.Resampling.BILINEAR) + pixels = np.asarray(square, dtype=np.float32) / 255 + return run(session, pixels.transpose(2, 0, 1)[np.newaxis]) + + +def numbers(embedding): + return ", ".join(str(value) for value in embedding) + + +def write_categories(prompts, embeddings): + lines = [ + "// Written by bin/clip-keyword-vectors; do not edit by hand.", + "//", + "// The Discover categories of Ente's photos app, in their order, from", + f"// {DISCOVER_PATH} in https://github.com/ente-io/ente", + f"// at commit {ENTE_COMMIT}. Each has the text", + f"// embedding Ente's CLIP text model ({TEXT_MODEL})", + "// gives its query, scaled to length 1. `showVideo` is false for a category", + "// that leaves out videos.", + "", + "export const DISCOVER_CATEGORIES = [", + ] + for prompt, embedding in zip(prompts, embeddings): + lines += [ + " {", + f" title: {json.dumps(prompt['title'])},", + f" query: {json.dumps(prompt['query'])},", + f" minScore: {json.dumps(prompt['minScore'])},", + f" showVideo: {json.dumps(prompt.get('showVideo', True))},", + f" textEmbedding: [{numbers(embedding)}],", + " },", + ] + lines.append("];") + with open("src/clip-keyword-vectors.ts", "w", encoding="utf-8") as file: + file.write("\n".join(lines) + "\n") + + +def write_fixture(embedding): + with open("test/clip-keywords-sunset.json", "w", encoding="utf-8") as file: + file.write( + "{\n" + f' "photo": {json.dumps(PHOTO_URL)},\n' + f' "sha256": {json.dumps(PHOTO_SHA256)},\n' + f' "embedding": [{numbers(embedding)}]\n' + "}\n" + ) + + +def main(): + prompts = json.loads(download(DISCOVER_URL, DISCOVER_SHA256))["prompts"] + tokenizer = Tokenizer.from_str( + download(TOKENIZER_URL, TOKENIZER_SHA256).decode("utf-8") + ) + text_model = ente_model(TEXT_MODEL, TEXT_MODEL_SHA256) + embeddings = [ + text_embedding(text_model, tokenizer, prompt["query"]) for prompt in prompts + ] + write_categories(prompts, embeddings) + + image_model = ente_model(IMAGE_MODEL, IMAGE_MODEL_SHA256) + write_fixture(image_embedding(image_model, download(PHOTO_URL, PHOTO_SHA256))) + + +if __name__ == "__main__": + main() diff --git a/src/backup.ts b/src/backup.ts index 37e5116..659275c 100644 --- a/src/backup.ts +++ b/src/backup.ts @@ -14,9 +14,9 @@ // YYYY/YYYY-MM/YYYY-MM-DD/ // YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path) // YYYY-MM-DD.<fileID>.json per-file metadata sidecar, with -// the file's ML data and its -// original's EXIF, XMP and -// dimensions +// the file's ML data, its +// keywords, and its original's +// EXIF, XMP and dimensions // collections/<name>/<title> symlink to the original // collections/<name>.json per-collection metadata // account.json the account's email and user ID @@ -77,6 +77,7 @@ import { readFile } from "node:fs/promises"; import { dirname, extname, join, relative, resolve } from "node:path"; import lockfile from "proper-lockfile"; +import { clipKeywords } from "./clip-keywords.js"; import { removeLeftoverTempFiles } from "./download/index.js"; import { describeError } from "./errors.js"; import { @@ -497,8 +498,9 @@ const imageMetadataFor = async ( } }; -// The file's JSON: its basic fields, its magic metadata, its ML data or the -// reason the ML data is missing, and its image metadata. +// The file's JSON: its basic fields, its magic metadata, its ML data without +// the CLIP embedding and the keywords that embedding gives, or the reason the +// ML data is missing, and its image metadata. const writeSidecar = ( path: string, file: EnteFile, @@ -514,7 +516,13 @@ const writeSidecar = ( }; if (file.magicMetadata) meta.magicMetadata = file.magicMetadata; if (file.pubMagicMetadata) meta.pubMagicMetadata = file.pubMagicMetadata; - if (ml.mlData) meta.mlData = ml.mlData; + if (ml.mlData) { + const { clip: _clip, ...mlDataWithoutClip } = ml.mlData; + meta.mlData = mlDataWithoutClip; + const isVideo = file.metadata.fileType === "video"; + const keywords = clipKeywords(ml.mlData, isVideo); + if (keywords) meta.keywords = keywords; + } if (ml.mlDataError) meta.mlDataError = ml.mlDataError; if (image.imageMetadata) meta.imageMetadata = image.imageMetadata; if (image.imageMetadataError) { diff --git a/src/cli-commands.ts b/src/cli-commands.ts index 52f1b05..6089b9b 100644 --- a/src/cli-commands.ts +++ b/src/cli-commands.ts @@ -37,9 +37,16 @@ import { thumbnailName, } from "./cli-output.js"; import { type Log, logRefresh } from "./cli-log.js"; -import { freshCollections, freshFiles, freshFile } from "./cli-read.js"; +import { + cachedFilesByID, + freshCollections, + freshFiles, + freshFile, + freshFilesByID, +} from "./cli-read.js"; import { withExtension } from "./filename.js"; import { runMetadataBackup } from "./metadata-backup.js"; +import type { Collection, EnteFile } from "./model/types.js"; import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js"; export interface CliContext { @@ -128,12 +135,15 @@ const readLibraryClient = (client: Client): LibraryClient => ({ contentSource: () => client.contentSource(), }); -// Open a library for a single point command: the aggressive background precache -// (issue #48) is off — a one-shot `collections` or `get` must not start -// downloading the whole account — and no refresh starts as it opens or on a -// timer: each command asks for the one it needs, and `collections` needs none -// (issue #210). While a refresh runs, each collection whose files it has -// fetched is logged, so a large account shows progress rather than nothing. +// Open a library for a single point command. The aggressive background +// precache (issue #48) is off, because a one-shot `collections` or `get` must +// not start downloading the whole account. No refresh starts as the library +// opens or on a timer. Each command asks for the refresh it needs. +// `collections` and `files` with a collection ID need none (issues #210 and +// #217). Nor do `get` and `get-thumb` on an empty cache (issue #222). Nor does +// `helper fix-missing-thumbnails --file` on an empty cache (issue #223). While +// a refresh runs, each collection whose files it has fetched is logged, so a +// large account shows progress rather than nothing. const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> => Library.open({ client: readLibraryClient(client), @@ -282,11 +292,11 @@ export const filesCommand = async ( const lib = await openReadLibrary(ctx, client); try { - // Force a server round-trip and list in enumeration order (issue #36 - // amendment, issue #52). Each file prints from its own decrypted - // metadata (raw title, microsecond creationTime) via cli-output, and in - // the pre-library CLI's enumeration order, not the projection's - // newest-first order. + // Ask the server for current state (issue #36 amendment, issue #52). + // Each file prints from its own decrypted metadata (raw title, + // microsecond creationTime) via cli-output. The files print in the + // order `freshFiles` returns them, not the projection's newest-first + // order. const files = await freshFiles(lib, collectionID); if (!files) { ctx.log("fatal", `Collection ${collectionID} not found`); @@ -323,14 +333,13 @@ export const getCommand = async ( try { // Force a server round-trip so the file resolves against current state // (issue #36 amendment, issue #52). - const resolved = await freshFile(lib, fileID); - if (!resolved) { + const file = await freshFile(lib, fileID); + if (!file) { ctx.log("fatal", `File ${fileID} not found`); return 1; } - const { photo, file } = resolved; - const result = await photo.original(); + const result = await lib.originalOf(file); // Default name is the file's own title, as the pre-library CLI used // (not the editedName-preferring projection title) (issue #52). if (result.videoPath === undefined) { @@ -381,14 +390,13 @@ export const getThumbCommand = async ( try { // Force a server round-trip so the file resolves against current state // (issue #36 amendment, issue #52). - const resolved = await freshFile(lib, fileID); - if (!resolved) { + const file = await freshFile(lib, fileID); + if (!file) { ctx.log("fatal", `File ${fileID} not found`); return 1; } - const { photo, file } = resolved; - const result = await photo.thumbnail(); + const result = await lib.thumbnailOf(file); // Default name is thumb_<file's own title>, as the pre-library CLI // used (not the projection title) (issue #52). const outPath = opts.out ?? thumbnailName(file); @@ -565,10 +573,13 @@ export const fixMissingThumbnailsCommand = async ( } const lib = await openReadLibrary(ctx, client); try { - // Refresh first so files added since the cache was written are found; - // a failed refresh throws. - await lib.fresh(); - if (fileIDs.length === 0) { + let found: Map<number, { file: EnteFile; collection: Collection }>; + if (fileIDs.length > 0) { + found = await freshFilesByID(lib, fileIDs); + } else { + // Refresh first so files added since the cache was written are + // checked; a failed refresh throws. + await lib.fresh(); ctx.log("info", "Scanning for missing thumbnails..."); const missing = await listMissingThumbnails( lib, @@ -581,12 +592,14 @@ export const fixMissingThumbnailsCommand = async ( return 0; } ctx.log("info", `Found ${fileIDs.length} file(s) to fix.`); + found = cachedFilesByID(lib, fileIDs); } const results = await fixMissingThumbnails( lib, client, fileIDs, + found, (msg, level) => ctx.log(level ?? "info", msg), ); diff --git a/src/cli-log.ts b/src/cli-log.ts index 88916c2..a790c6b 100644 --- a/src/cli-log.ts +++ b/src/cli-log.ts @@ -1,15 +1,18 @@ // The CLI's log level, and the one function the commands write stderr lines // through. // -// Every line quak prints on stderr is a log line at one of six levels, from -// the quietest: fatal, error, warning, info, debug, trace. A run prints the -// lines at its own level and at every quieter one. Its level is info unless -// `-v` or `-q` asks for another (see "CLI surface" in the README). Every -// command that exits non-zero logs a fatal line saying why, so even `-q`, which -// prints only fatal lines, never leaves a non-zero exit unexplained. An error +// Every line quak logs goes to stderr, at one of six levels, from the +// quietest: fatal, error, warning, info, debug, trace. A run prints the lines +// at its own level and at every quieter one. Its level is info unless `-v` or +// `-q` asks for another (see "What quak prints" in the README). A command that +// runs and exits non-zero logs a fatal line saying why, so even `-q`, which +// prints only fatal lines, never leaves such an exit unexplained. An error // line reports something that failed, such as a file or an album, and a // warning something that did not fail but needs the user's attention. stdout -// carries only a command's output and is not a log. +// carries a command's output and the prompts of `login`, and is not a log. +// Two things on stderr are not log lines: the `error:` message for a +// command line that quak cannot parse, and the usage printed when no command +// is given. // // The library itself writes no lines: its layers report through callbacks, and // the CLI logs what they report. `logRequests` gives the API client options diff --git a/src/cli-read.ts b/src/cli-read.ts index 70b1f21..55b2d99 100644 --- a/src/cli-read.ts +++ b/src/cli-read.ts @@ -2,38 +2,40 @@ // // `collections`, `files`, `get`, and `get-thumb` must answer for // the account's state at the moment the command runs, not for whatever the -// local cache last happened to hold (owner amendment, issue #36). `files`, -// `get` and `get-thumb` therefore force a server round-trip through -// `Library.fresh()` and only then read — so a collection, file, or metadata -// change made elsewhere is visible. `collections` asks the library for the -// collection list alone through `Library.freshCollections()` (issue #210): a -// refresh would also fetch the files of every collection, which on a large -// account takes long and which `collections` does not print. +// local cache last happened to hold (owner amendment, issue #36). `get`, +// `get-thumb` and `files` without a collection ID therefore force a server +// round-trip through `Library.fresh()` and only then read, so a change made +// elsewhere is visible. `collections` asks the library for the collection list +// alone through `Library.freshCollections()` (issue #210): a refresh would also +// fetch the files of every collection, which on a large account takes long and +// which `collections` does not print. `files` with a collection ID reads that +// list too, then that one collection's files through `Library.freshFiles()` +// (issue #217). On an empty cache, `get` and `get-thumb` read that list, then +// one collection's files at a time, until they find the file (issue #222). +// `helper fix-missing-thumbnails --file` does the same until it finds every +// file it names (issue #223). // -// `collections` lists in the server's order and `files` in the library's own -// enumeration order (`listFiles()`), the orders the pre-library CLI printed, -// rather than the `albums`/`photos` projection's newest-first order, which -// re-sorts the rows. The field values still come from each record's raw -// metadata via `cli-output.ts`. `files` without a collection ID lists every -// file in the account once (issue #211): the library keeps a record of a file -// for each collection that holds it, and only the first collection's record is -// listed. +// `collections` and `files` with a collection ID list in the server's order. +// `files` without one lists in the library's own enumeration order +// (`listFiles()`). Neither command uses the `albums`/`photos` projection's +// newest-first order, which re-sorts the rows. The field values still come +// from each record's raw metadata via `cli-output.ts`. `files` without a +// collection ID lists every file in the account once (issue #211): the library +// keeps a record of a file for each collection that holds it, and only the +// first collection's record is listed. import type { Collection, EnteFile } from "./model/types.js"; -import type { Photo, PhotosAPI } from "./library/index.js"; // The slice of `Library` these helpers read. `Library` satisfies it -// structurally; a test can drive them with a stand-in that records the -// `fresh()` and `freshCollections()` calls and serves records in a known -// enumeration order. +// structurally; a test can drive them with a stand-in that records each fresh +// read and serves records in a known enumeration order. export interface FreshReadLibrary { fresh(): Promise<unknown>; freshCollections(): Promise<Collection[]>; - getCollection(id: number): Collection | undefined; + freshFiles(collection: Collection): Promise<EnteFile[]>; listCollections(): Collection[]; listFiles(collectionID: number): EnteFile[]; getFileByID(fileID: number): EnteFile | undefined; - photos: Pick<PhotosAPI, "byID">; } // Every live collection, current as of one collections request, in the @@ -42,21 +44,21 @@ export const freshCollections = ( lib: FreshReadLibrary, ): Promise<Collection[]> => lib.freshCollections(); -// The files of one collection, current as of a forced refresh, in enumeration +// The files of one collection, as the server holds them now, in the server's // order. `undefined` (not an empty list) when the collection does not exist, so // the caller can tell "no such collection" from "an empty collection". Without -// a collection ID, every file in the account, each once, as held by the first -// collection that holds it. +// a collection ID, every file in the account as of a forced refresh, each once, +// as held by the first collection that holds it. export const freshFiles = async ( lib: FreshReadLibrary, collectionID?: number, ): Promise<EnteFile[] | undefined> => { - await lib.fresh(); if (collectionID !== undefined) { - return lib.getCollection(collectionID) - ? lib.listFiles(collectionID) - : undefined; + const collections = await lib.freshCollections(); + const collection = collections.find((c) => c.id === collectionID); + return collection ? lib.freshFiles(collection) : undefined; } + await lib.fresh(); const listed = new Set<number>(); const files: EnteFile[] = []; for (const collection of lib.listCollections()) { @@ -69,16 +71,80 @@ export const freshFiles = async ( return files; }; -// One file, current as of a forced refresh, resolved to both its content -// handle (`Photo`, for fetching bytes) and its raw record (`EnteFile`, for the -// default output name and field values). `undefined` when the file is unknown. +// One file as the server holds it now, or `undefined` when no collection holds +// it. When the cache holds the account's collections, it refreshes and looks +// the file up in the cache. On an empty cache, it reads the collection list, +// then one collection's files at a time, and stops at the first collection +// that holds the file. That search stores nothing. export const freshFile = async ( lib: FreshReadLibrary, fileID: number, -): Promise<{ photo: Photo; file: EnteFile } | undefined> => { - await lib.fresh(); - const photo = lib.photos.byID({ fileID }); - const file = lib.getFileByID(fileID); - if (!photo || !file) return undefined; - return { photo, file }; +): Promise<EnteFile | undefined> => { + // Only a refresh stores collections, so a cache holding any holds them all. + if (lib.listCollections().length > 0) { + await lib.fresh(); + return lib.getFileByID(fileID); + } + for (const collection of await lib.freshCollections()) { + const files = await lib.freshFiles(collection); + const file = files.find((f) => f.id === fileID); + if (file) return file; + } + return undefined; +}; + +// The most recently updated collection first, and the higher ID first on a +// tie. `Library.albums.list()` lists albums in this order. +const byNewestCollection = (a: Collection, b: Collection): number => + b.updationTime - a.updationTime || b.id - a.id; + +// Each file named by `fileIDs` that the cache holds, mapped to its record and +// the collection it was found in. Collections are searched in +// `byNewestCollection` order, and a file is found in the first that holds it. +// So `helper fix-missing-thumbnails` names a file after the same album as +// `helper list-missing-thumbnails`. It reads the cache alone, so the caller +// refreshes first. +export const cachedFilesByID = ( + lib: FreshReadLibrary, + fileIDs: number[], +): Map<number, { file: EnteFile; collection: Collection }> => { + const wanted = new Set(fileIDs); + const found = new Map<number, { file: EnteFile; collection: Collection }>(); + const collections = [...lib.listCollections()].sort(byNewestCollection); + for (const collection of collections) { + for (const file of lib.listFiles(collection.id)) { + if (wanted.has(file.id) && !found.has(file.id)) { + found.set(file.id, { file, collection }); + } + } + } + return found; +}; + +// Each file named by `fileIDs` as the server holds it now, mapped to its record +// and collection as `cachedFilesByID` maps it. When the cache holds the +// account's collections, it refreshes and looks the files up in the cache. On +// an empty cache, it reads the collection list, then one collection's files at +// a time, and stops once it has found every file. That search stores nothing. +export const freshFilesByID = async ( + lib: FreshReadLibrary, + fileIDs: number[], +): Promise<Map<number, { file: EnteFile; collection: Collection }>> => { + // Only a refresh stores collections, so a cache holding any holds them all. + if (lib.listCollections().length > 0) { + await lib.fresh(); + return cachedFilesByID(lib, fileIDs); + } + const wanted = new Set(fileIDs); + const found = new Map<number, { file: EnteFile; collection: Collection }>(); + const collections = await lib.freshCollections(); + for (const collection of [...collections].sort(byNewestCollection)) { + if (found.size === wanted.size) break; + for (const file of await lib.freshFiles(collection)) { + if (wanted.has(file.id) && !found.has(file.id)) { + found.set(file.id, { file, collection }); + } + } + } + return found; }; diff --git a/src/clip-keyword-vectors.ts b/src/clip-keyword-vectors.ts new file mode 100644 index 0000000..9dd3543 --- /dev/null +++ b/src/clip-keyword-vectors.ts @@ -0,0 +1,1836 @@ +// Written by bin/clip-keyword-vectors; do not edit by hand. +// +// The Discover categories of Ente's photos app, in their order, from +// mobile/apps/photos/assets/discover.json in https://github.com/ente-io/ente +// at commit f5ad70a860ed25d0c8057b0efb4bd6eaefe4b0c5. Each has the text +// embedding Ente's CLIP text model (mobileclip_s2_text_opset18_quant.onnx) +// gives its query, scaled to length 1. `showVideo` is false for a category +// that leaves out videos. + +export const DISCOVER_CATEGORIES = [ + { + title: "Identity", + query: "identity documents and cards", + minScore: 0.2, + showVideo: false, + textEmbedding: [ + 0.03607934, -0.015425337, -0.0043316674, 0.009292002, -0.025645005, + 0.014152668, -0.0076436675, 0.014114335, -0.041070342, 0.007275668, + 0.015916003, 0.017763669, 0.033603005, 0.009023668, -0.03352634, + 0.5298818, 0.0017940004, -0.0076360013, -0.042412005, -0.0015103335, + 0.025008671, -0.023552004, -0.024357004, 0.008088335, -0.013907337, + -0.0021313336, -0.019741671, 0.017365003, 0.037566673, -0.018898336, + 0.023245338, -0.008395001, 0.018032003, 0.041430674, -0.007728001, + 0.019948669, -0.010189001, 0.03670034, -0.022923337, -0.04807001, + -0.060191013, 0.01716567, -0.013171336, -0.011308336, 0.011837335, + -0.015893003, -0.018377002, 0.026296671, 0.027684337, 0.043914672, + -0.038801007, -0.024265004, 0.010127668, -0.043102007, + 0.00044466674, -0.02084567, -0.003664667, -0.008786001, + -0.020301336, -0.026036004, 0.019933337, -0.033258006, 0.014275335, + -0.039866675, -0.11761435, 0.02468667, -0.007950335, -0.024817005, + -0.0035036672, 0.0030590005, -0.038517337, 0.035282005, + -0.059524007, -0.029716006, 0.0070456676, -0.024610003, + -0.042941008, -0.015663004, 0.016935669, -0.00097366684, + -0.00071300013, 0.016859002, 0.02664167, -0.0052976673, + -0.005075334, 0.013930335, -0.0065626674, -0.031096006, 0.019120669, + 0.010434335, 0.008287668, -0.006401668, 0.006647001, 0.006340334, + 0.0077510015, 0.02337567, 0.099183686, 0.011914002, -0.011776002, + -0.0072296676, 0.00088166684, 0.029363338, 0.0118986685, + -0.030551674, -0.04252701, -0.012105669, -0.05200301, 0.007666668, + 0.0038180007, -0.011132002, -0.009713668, 0.0006516668, 0.2377357, + -0.031441007, -0.028305337, -0.010825335, 0.02919467, -0.012190002, + 0.050508007, -0.019389004, -0.05011701, 0.0058113346, -0.017020002, + 0.013363002, -0.0026526672, 0.038640004, 0.01987967, -0.013577669, + -0.012542669, -0.008701668, -0.03423934, -0.034431007, 0.013570002, + -0.0051903343, -0.043056007, -0.0021773337, -0.022777671, + 0.021497337, -0.066209346, 0.004262667, 0.010388334, -0.009644669, + -0.015778001, 0.038494337, 0.026105003, 0.010741002, -0.048790675, + 0.002913334, 0.030812338, -0.016299335, 0.0048990007, -0.02036267, + -0.023306672, 0.0006823334, -0.029478338, 0.03545067, -0.01394567, + -0.0044083344, 0.04194434, 0.020025337, 0.012987335, 0.006969001, + 0.019941004, -0.0055966675, 0.009253668, -0.06916101, -0.006210001, + -0.53356177, -0.023207005, -0.020102004, 0.003204667, 0.057454012, + 0.013715669, 0.0029670005, -0.016238004, 0.01852267, 0.0058573345, + -0.0033580007, 0.0023613337, -0.01596967, -0.00085866684, + -0.016138336, 0.02236367, 0.00042166674, -0.02270867, 0.013301668, + -0.009000667, 0.024832336, 0.013447335, -0.033971004, -0.06357968, + -0.030689672, 0.0049220007, 0.012259002, 0.019059336, 0.005826668, + 0.008195668, -0.031111337, 0.0090620015, 0.0037413337, -0.017035335, + 0.00093533343, -0.054249346, 0.0041246675, 0.016912669, 0.020493003, + 0.025361337, -0.014306002, 0.011392668, 0.04227401, -0.014283002, + -0.0013953336, -0.059715677, -0.045777675, -0.012481336, + 0.022141337, 0.062138345, 0.031763006, -0.0031740007, -0.0062483344, + -0.043431673, 0.07113135, 0.016092336, 0.008218668, 0.016284002, + 0.03701467, 0.0007283335, -0.007620668, -0.0062866677, -0.039897338, + -0.006095001, 0.036884338, -0.011630335, 0.003626334, -0.0059416676, + -0.053567007, -0.007275668, 0.005704001, 0.0066623343, -0.026289005, + 0.017664002, 0.014482335, 0.01930467, 0.0073523344, -0.009805668, + 0.005397334, -0.04258067, -0.008149668, 0.025744673, 0.0076820017, + 0.019818338, 0.007291001, -0.040802006, -0.047065675, 0.0024150002, + -0.014950003, -0.0072680013, 0.04578534, 0.00057500013, -0.01806267, + 0.015410002, -0.013830668, 0.0106413355, -0.0013646669, 0.019910336, + -0.0023076672, 0.018262004, 0.0036876672, 0.020953003, + -0.0038793338, -0.011055335, 0.005650334, 0.07518701, -0.022931004, + -0.006210001, -0.016077003, 0.007291001, -0.010449668, -0.03227667, + -0.014957669, -0.021850005, -0.010871335, 0.02323767, -0.046237674, + 0.026925337, -0.010357669, 0.01642967, -0.020002337, 0.032614004, + -0.0039483337, -0.004937334, 0.03311234, -0.06581835, -0.01863767, + 0.007344668, 0.028650336, 0.006041334, -0.07347734, 0.033534005, + 0.029815672, 0.008272334, 0.063426346, 0.0032506671, -0.026818004, + -0.03571134, -0.017472336, -0.01576267, -0.006447668, 0.030712672, + 0.003626334, 0.012696003, 0.026404003, -0.025790673, 0.0023230005, + -0.025016338, -0.11878735, -0.04511834, 0.03575734, -0.041292675, + -0.014681669, -0.038709007, -0.042205006, -0.0078276675, + -0.028888004, -0.070548676, 0.038923673, 0.014037669, 0.0005366667, + -0.007981001, -0.03969034, 0.026818004, -0.010526335, -0.0039483337, + 0.02360567, -0.01921267, -0.02351367, -0.009069668, -0.0029363339, + 0.0018553336, 0.013508668, 0.003626334, -0.0011653336, -0.07035702, + 0.024924338, 0.025392005, -0.0013570002, 0.008441001, 0.021750337, + -0.07450468, -0.0063480013, 0.027906671, 0.0018630002, 0.007038001, + 0.026810339, 0.031625006, -0.01576267, -0.027707336, 0.011684002, + -0.021413004, -0.0032736673, -0.0057193344, -0.018055003, + -0.012857001, -0.04456634, 0.025085337, 0.03508267, -0.02153567, + -0.015172335, 0.016376004, 0.005152001, 0.0011040001, -0.0027983338, + -0.025085337, -0.05013234, 0.012711335, 0.095787354, -0.034423336, + -0.020723004, -0.065412015, 0.012427669, 0.0026833336, 0.0066546677, + -0.008134334, 0.014313668, 0.007919668, -0.009813335, -0.011607335, + -0.05103701, 0.025568336, -0.008379668, -0.0008740002, -0.064837016, + -0.018346336, -0.0062330007, -0.015954336, 0.005443334, + -0.007620668, 0.030766338, -0.004055667, -0.10353068, -0.009568001, + -0.010802335, -0.004692001, 0.0068386677, -0.018599337, 0.02758467, + 0.037727673, -0.07813101, 0.027822338, -0.019642003, -0.02181167, + -0.0015256669, -0.02726267, -0.03633234, -0.03617134, -0.010656669, + -0.008617334, 0.013501002, -0.003166334, -0.033219673, -0.015026669, + -0.0046996675, -0.038003676, 0.020439336, 0.012596336, -0.019864338, + -0.010488002, -0.021098671, 0.01631467, 0.041844673, -0.0133706685, + -0.04556301, 0.009491335, -0.007981001, 0.016935669, -0.03246067, + -0.029693006, 0.015272003, 0.010618336, 0.0025760005, -0.040863343, + 0.006164001, 0.010327002, 0.019642003, 0.023069004, -0.01665967, + -0.01769467, 0.009575669, -0.028198004, 0.011645668, -0.018921336, + 0.016537003, -0.02751567, 0.007544001, -0.017572002, 0.021045003, + -0.011553668, -0.029601006, -0.0057423343, -0.0029286672, + -0.03575734, -0.02243267, 0.020263003, -0.01442867, -0.0030436672, + 0.00042166674, 0.02266267, 0.04010434, -0.009460668, -0.022103004, + -0.0059186677, 0.022256337, -0.043914672, 0.0064706677, + -0.055591006, 0.010871335, -0.008878002, -0.009468335, 0.011024669, + -0.0070456676, -0.005282334, 0.0034730004, 0.010150668, + -0.013508668, 0.0069536683, -0.0011730002, -0.02929434, 0.024717337, + 0.0071683344, 0.04610734, -0.020378003, 0.04194434, -0.03824901, + -0.008195668, -0.02967767, -0.019381337, 0.01836167, 0.034898672, + -0.03818001, -0.010679669, 0.016874336, -0.008870334, 0.004339334, + 0.010986335, 0.008203334, 0.013117669, 0.044420674, -0.013501002, + -0.011960002, -0.013930335, -0.0070533347, -0.015456002, + ], + }, + { + title: "Screenshots", + query: "screenshots", + minScore: 0.18, + showVideo: false, + textEmbedding: [ + 0.043836724, 0.016966958, 0.017219765, -0.005612314, -0.024269465, + -0.010343416, 0.00104012, -0.02295487, 0.017357003, 0.03262654, + 0.013608237, -0.005966244, -0.020267894, -0.021791957, 0.0, + 0.5200383, -0.015457339, -0.009093828, 0.019870626, 0.010646784, + 0.025995778, 0.0034237283, -0.0064574117, -0.02659529, 0.0030481294, + 0.0038571118, -0.025750192, -0.013528784, -0.0042038183, + -0.020137878, -0.0064357426, -0.03201258, 0.049723517, + -0.0018418792, 0.0118096955, 0.012430879, -0.02231924, + -0.0026436383, 0.011788026, 0.017530356, 0.020939637, -0.007649216, + 0.007865908, 0.02413223, 0.043872837, -0.0033081593, -0.008003145, + 0.012336979, 0.030264601, -0.013962167, -0.0075553157, -0.008855466, + -0.031925905, -0.001170135, 0.011564112, -0.0023980546, + -0.031940352, 0.029282266, 0.034937922, -0.009548879, -0.023229348, + 0.028581629, -0.026768643, 0.0015168416, -0.05953965, 0.04403897, + 0.0075769853, 0.015334547, 0.005836229, 0.020585708, -0.006471858, + 0.015544015, 0.01524787, 0.003936565, 0.020672385, -0.01849102, + -0.028112132, 0.014525565, -0.09122719, -0.010733461, 0.02137302, + 0.027808763, -0.0100183785, 0.0661054, -0.034764566, -0.023178786, + -0.020462917, 0.0028314379, -0.026891435, -0.00867489, 0.011224628, + -0.00978724, -0.021019092, 0.0150095085, 0.0060529206, -0.008176499, + 0.14191137, 0.045209102, 0.048683394, -0.014366657, 0.040774148, + -0.013796036, 0.0137021355, -0.040933058, 0.01283537, 0.020773508, + -0.076333255, -0.0111451745, -0.01945891, 0.026855322, + -0.0023041547, -0.057965018, 0.2242542, -0.031131368, -0.007526424, + 0.07724336, 0.010574553, 0.023301577, -0.0034309514, -0.005735106, + -0.019921187, 0.030084027, -0.020390686, -0.0037704348, 0.04054301, + -0.008718228, 0.030315163, -0.0071291556, -0.034779012, 0.017450903, + -0.008624328, -0.034461197, 0.023344915, -0.016779158, -0.03203425, + -0.0041315877, 0.0064501883, 0.028740536, -0.021625828, -0.03971958, + 0.009158835, -0.030437957, -0.006941356, 0.030582415, -0.008494314, + -0.004572194, 0.0015674031, 0.010162839, 0.019516697, 0.002224701, + -0.02555517, -0.0058145598, -0.018230991, -0.0025497386, + 0.027750978, 0.011210182, 0.0017335333, 0.0007295286, -0.02125023, + 0.016598582, -0.0069991406, 0.015710145, 0.023186008, 0.020556815, + 0.019227773, -0.01526954, -0.014706141, -0.52540505, 0.018700492, + -0.023055993, -0.0026941996, 0.076246575, 0.009548879, 0.014496672, + -0.0005778444, 0.0048538935, -0.01636022, -0.018498246, + -0.048134442, -0.09872472, 0.0043916176, -0.006731888, 0.019588927, + -0.008891582, 0.025894655, -0.010668453, 0.05787834, -0.023012655, + 0.035783015, -0.0048972317, 0.0073747393, -0.011101836, + -0.015999068, 0.039437883, -0.03089301, -0.030452402, 0.035544656, + -0.052388825, -0.013897159, 0.03860001, -0.03905506, -0.006731888, + -0.025757415, -0.0015962953, 0.03382557, -0.03923564, 0.017096972, + 0.049774073, -0.011578558, 0.0129364915, -0.021329682, + -0.0021958088, -0.0033876128, -0.0137021355, 0.024197236, + 0.02167639, -0.030452402, 0.0023330469, -0.0100978315, -0.028747762, + -0.015832938, -0.038419433, 0.007865908, 0.0039510117, -0.005453407, + 0.04182149, 0.016598582, -0.0048611164, 7.2230555e-6, -0.03695315, + 0.032308728, 0.02316434, -0.016511904, -0.052749973, 0.0002961453, + -0.048567824, -0.032308728, 0.021069653, 0.005077808, -0.009462202, + -0.0321426, 0.013015946, 0.011997496, -0.011072944, -0.014843379, + 0.07510533, 0.028242148, -0.04544024, -0.05197711, -0.0107840225, + -0.010942929, -0.023539938, 0.020051202, 0.035494093, -0.04436401, + -0.0033081593, 0.013550451, -0.0076564387, -0.0065657576, + 0.0059445747, 0.014482226, -0.021900306, -0.013796036, -0.030669093, + 0.005467853, 0.0077142236, -0.007909246, -0.06104204, 0.02491232, + -0.0065802033, 0.0052439384, 0.010105055, 0.015565684, -0.0129076, + -0.034222838, 7.945361e-5, 0.01978395, -0.010906814, 0.010256739, + 2.8892222e-5, 0.034143385, 0.013391544, 0.005467853, -0.01212751, + 0.031405848, -0.010935706, -0.008552098, 0.0047094324, -0.012445324, + -0.02347493, -0.017060857, 0.027606517, -0.048524488, 0.010523992, + -0.0066163186, -0.006370735, -0.00048394472, 0.03725652, + -0.005937352, -0.0125753395, 0.010704569, 0.007887577, 0.0014518342, + 0.032055918, -0.012979831, 0.017956516, 0.008335406, -0.06319451, + 0.01430165, -0.019978972, 0.012806477, -0.0154140005, -0.006529642, + -0.012185294, 0.011535219, 0.07436857, -0.06877793, 0.009823356, + -0.021221336, 0.0028819991, 0.0064790808, -0.017927622, -0.01604963, + 0.0046516475, 0.043266103, 0.01014117, 0.0035176282, -0.05184709, + 0.02659529, -0.045895297, 0.022853747, -0.031875346, 0.0060673663, + -0.046632048, -0.0391923, -0.015406777, 0.015486231, 0.017523132, + -0.036209177, 0.0011629119, -0.0064285193, 0.00910105, -0.04517299, + -0.0026653074, -0.03222205, 0.0006717442, -0.025143456, + -0.005489522, 0.0025641848, 0.0030770216, 0.0035320742, + -0.012098618, 0.018541582, 0.007244725, 0.010386754, 0.010495099, + -0.1167968, 0.019581703, -0.007056925, 0.023597723, 0.043193873, + -0.028581629, -0.009216619, -0.013998281, -0.014908387, + -0.020217333, 0.03558077, -0.039488442, 0.022283128, 0.050575834, + 0.011751911, -0.031521413, -0.0065585347, 0.034872912, -0.00685468, + 0.0025280693, 0.014893941, -0.015298432, -0.102076225, 0.008328184, + -0.0020152326, 0.012676462, 0.016757488, 0.02319323, 0.004247157, + 0.0012640347, 0.0010545661, 0.01158578, -0.0022391472, 0.061634332, + -0.00026003, -0.089724794, 0.0062046046, -0.015442892, + -0.0103000775, -0.015038402, 0.046913747, 0.019372234, + -0.0041965954, -0.12853427, 0.0047599934, 0.007909246, -0.03179589, + -0.020571262, -0.0013868266, -0.006349066, -0.042016514, + -0.019632265, 0.022023097, -0.012228633, 0.05901236, -0.025287917, + -0.0015601801, 0.010942929, -0.062046047, -0.021365799, + -0.001170135, 0.016519127, -0.045656934, 0.007295286, -0.0009028819, + 0.03857834, -0.0066596568, -0.004536079, -0.004955016, -0.020224554, + -0.016634697, -0.017609809, 0.0028025454, 0.006002359, -0.024377814, + -0.011029606, 0.019379457, -0.007403632, 0.03589136, -0.028307155, + 0.002231924, -0.017602587, 0.018230991, -0.0048900084, -0.025562393, + 0.0012857039, -0.0021669166, -0.04036966, -0.0126259, -0.031564754, + -0.012770362, -0.019733388, -0.010372308, -0.023344915, + -0.0019718942, -0.0020224554, -0.027953224, 0.005113923, + 0.058268387, 0.046054203, -0.0065946495, -0.019791171, -0.0103073, + -0.006341843, -0.0074903085, 0.020470139, -0.010206178, + -0.0061395974, -0.009036043, 0.039026167, 0.0077575613, + 0.0034526207, 0.0100978315, -0.021864189, 0.0487484, -0.007085818, + 0.020686831, 0.009346633, -0.022817632, -0.0030481294, 0.0046516475, + -0.0049911314, 0.006211828, 0.006768003, 0.0051211463, -0.055826996, + -0.031694766, 0.018324891, 0.050951436, -0.017689263, -0.13221803, + 0.0153128775, 0.021777513, 0.0024991771, -0.010755129, -0.01959615, + 0.0067896727, -0.006529642, 0.0033081593, 0.023446038, -0.021329682, + 0.012929269, -0.0098955855, -0.02793878, -0.007461416, + -0.0048900084, -0.02316434, -0.0043771714, 0.036324747, + -0.0049622394, 0.0009462203, -0.0017624255, -0.06352678, + -0.029015014, 0.022694841, 0.0683301, + ], + }, + { + title: "QR Codes", + query: "image containing a black and white QR code", + minScore: 0.2, + showVideo: false, + textEmbedding: [ + 0.015853409, -0.00093012175, -0.0027214673, 0.019863265, + 0.0101142125, -0.007943928, -0.012732333, 0.009018735, 0.025230411, + -0.011078783, -0.028310148, -0.055221666, 0.06398548, -0.017796328, + -0.008426214, 0.47144082, -0.039871216, -0.013400643, 0.035992265, + -0.029688105, -0.043867294, 0.028875113, -0.039981455, 0.031603467, + -0.024672339, -0.025271751, 0.017314043, -0.0022254023, 0.007020696, + 0.03579935, -0.027214672, -0.010982326, 0.016197896, -0.014565017, + -0.007034476, -0.03582002, 0.018443968, 0.029054245, -0.015557148, + -0.028992238, -0.059217747, -0.0077923527, -0.013297294, + 0.009218539, 0.009370115, 0.0027972548, -0.019890824, 0.01249119, + 0.019387871, -0.011002995, 0.0040374175, -0.029881021, 0.026498135, + 0.00985929, -0.01610144, -0.012883908, 0.08034873, -0.0068829004, + -0.021764847, -0.021716619, -0.004740176, -0.036322974, + -0.0058425423, -0.07806133, 0.07493336, 0.05982405, -0.011912447, + 0.033835758, -0.024203833, -0.010782522, 0.0034724541, 0.008109283, + -0.0010265788, 0.04795983, -0.008253969, -0.03578557, + -0.00039960784, 0.00953547, -0.0060216766, 0.035020802, -0.01247052, + -0.003582691, 0.03843125, 0.023742218, -0.009721494, 0.05814983, + 0.007020696, 0.008095504, 0.0031072954, 0.0007096484, 0.05289292, + 0.004567931, 0.024210723, -0.0012263827, 0.011065003, -0.012945916, + 0.24611709, -0.014220527, 0.0026112306, 0.041834805, 0.015357343, + -0.0015708721, 0.014875057, 0.07415482, 0.023390837, 0.023452846, + -0.06579749, 0.036260966, 0.007068925, 0.017072901, -0.014089622, + -0.02133768, 0.18148397, 0.020228425, 0.028255029, -0.02333572, + 0.017837668, -0.0213239, 0.007999047, -0.007117153, 0.009122083, + -0.025554232, 0.010506931, -0.004230331, 0.0016259905, -0.01247052, + 0.008054165, -0.029915469, -0.016129, -0.02722845, 0.035020802, + 0.012566977, 0.015212657, 0.01280812, 0.010562048, -0.04532104, + 0.00394096, 0.041262954, 0.017169356, -0.032285556, -0.015522697, + 0.011030554, 0.017582744, 0.008570899, -0.00592522, 0.003720487, + -0.022467606, 0.0020049291, 0.03905133, 0.034035567, -0.013834699, + 0.00054429343, 0.054091744, -0.011953786, -0.0096939355, + -0.015667383, -0.019491218, 0.025333758, -0.018003022, 0.022508945, + 0.016935105, 0.03550309, -0.0041821026, 0.021916423, 0.03122453, + 0.067340806, -0.014172299, -0.47525772, -0.012787451, -0.025450885, + 0.0055187223, 0.063461855, 0.0062903785, 0.008315977, -0.02627077, + -0.019532556, 0.0018464639, 0.020359332, 0.012305166, 0.09251611, + 0.015880967, 0.018561095, -0.010024644, 0.010713624, -0.024017809, + -0.0059321094, -0.037976522, 0.026436124, -0.023707768, 0.012711663, + -0.009501021, 0.019126058, -0.019794367, 0.007937038, -0.018223496, + -0.046127148, -0.02594006, -0.075181395, 0.0066555375, 0.038582824, + 0.033153668, -0.00790259, -0.042551342, 0.006986248, -0.017651644, + 0.023129027, 0.015949864, 0.006676207, -0.03054244, 0.015701832, + -0.015908526, -0.0379903, 0.04526592, -0.03584069, 0.014861278, + -0.03122453, 0.055035647, -0.0049744286, -0.053016935, -0.007585659, + -0.020269763, -0.037673377, -0.013118161, 0.031658586, -0.032671385, + 0.039499167, -0.026877074, -3.444895e-5, 0.0027145774, -0.022033548, + -0.012732333, -0.03127965, 0.0053326976, -0.015667383, -0.037349552, + -0.036205847, -0.0006683096, 0.021744179, 0.009749053, 0.010768742, + 0.006049236, -0.019291412, 0.011857329, -0.0180857, 0.10372579, + -0.017858338, 0.021640832, -0.04204839, -0.048180304, -0.0007372076, + -5.5118322e-5, -0.0051328936, -0.029274719, 0.010754962, + 0.057075027, 0.010989215, -0.02365265, 0.018692, 0.038610384, + -0.031438112, 0.01745873, -0.009721494, 0.0022529615, -0.011402603, + -0.025437107, 0.014516789, -0.0117953215, 0.0030108383, + 0.0052362406, -0.050825983, -0.0103277955, 0.0013297294, + 0.022901662, 0.017679201, 0.028530622, 0.017768769, -0.025044387, + 0.018871136, 0.023852453, -0.027455814, -0.007937038, -0.048421446, + -0.0052362406, 0.020889845, 0.016066993, -0.004071866, 0.0017775659, + 0.024121156, 0.014716592, 0.0048504123, -0.014689034, 0.013931155, + -0.022453828, 0.02431407, -0.025416438, -0.0015639823, 0.015343564, + 0.07087527, -0.043185204, 0.0003651589, 0.008267748, 0.032058194, + -0.00024114265, 0.020545354, 0.017947905, -0.0086535765, 0.01478549, + 0.03147945, -0.008171291, 0.0098041715, -0.008839601, -0.005560061, + -0.00086811354, -0.025120176, 0.020869175, -0.09649841, -0.06394414, + 0.022784537, -0.059093732, 0.0018533536, -0.032602485, -0.033319023, + -0.046147816, -0.051129136, 0.03454541, 0.008088614, -0.017982353, + 0.051762994, -0.014268757, -0.0011161461, 0.027131995, -0.022095557, + -0.029192042, -0.007082704, -0.016225455, 0.009735274, 0.032058194, + 0.023893794, 0.01021067, 0.037921406, 0.002659459, 0.022577843, + 0.028702868, -0.030094605, -0.019429209, -0.00035826908, + 0.009025626, -0.010472481, -0.13410288, 0.03019795, -0.008908499, + 0.034979466, 0.024410527, 0.022570953, 0.0057529747, 0.014957734, + -0.09229563, -0.041676342, -0.0055807303, -0.020758938, + -0.047456875, 0.005001988, 0.014854388, -0.0030935158, 0.03641254, + 0.03418025, 0.00918409, -0.0005374036, 0.0016811087, -0.003382887, + 0.009473462, 0.027007978, 0.028310148, -0.018829796, -0.06887034, + 0.046292502, -0.021606382, -0.025499115, -0.14208815, -0.018926254, + -0.0044370247, 0.0048779715, -0.0059321094, -0.0071516023, + 0.014957734, -0.013125051, 0.009039405, 0.025237301, 0.01836818, + 0.04567931, -0.0051742326, -0.03025996, 0.026401676, 0.015488249, + -0.019229405, -0.032995205, 0.006793333, 0.13470918, 0.019567003, + -0.01904338, -0.0060010073, 0.011292366, -0.010830751, 0.025740257, + 0.0016811087, 0.0153711215, -0.043936193, 0.080059364, 0.017941015, + -0.048021838, 0.062290594, 0.034648754, 0.008164401, 0.008288417, + -0.008508892, 0.029371176, -0.002328749, 0.020111298, 0.002459655, + -0.01804436, 0.0047884043, -0.051749215, 0.035344627, 0.041145828, + -0.001839574, -0.0282757, 0.011175239, -0.029757006, -0.020131966, + 0.02559557, -0.023728438, -0.010424253, 0.01380025, 0.017369162, + 0.025561122, 0.02195776, 0.01411029, -0.037707824, 0.016135888, + 0.024555212, 0.028075894, -0.0007992157, 0.016253015, 0.039003104, + 0.029701887, -0.02822747, 0.014055173, 0.008350426, 0.0015157539, + -0.016053211, 0.023101468, 0.031100513, -0.0139931645, 0.0089636175, + -0.0102244485, 0.00016535497, -0.025113286, 0.025416438, + 0.022943001, 0.028255029, -0.003968519, -0.016066993, 0.040263932, + 0.049496252, -0.01442722, 0.06689986, 0.025712699, -0.006448844, + -0.02135146, 0.019525666, -0.021110317, -0.042930283, -0.016204787, + -0.0072549493, 0.014599466, 0.025223523, 0.0012401623, + -0.00016535497, -0.013497099, 0.038775742, -0.017672313, + -0.04007102, 0.0064695133, -0.014399661, 0.02266052, -0.046650767, + -0.14170921, 0.04127673, -0.028503062, 0.0011574848, -0.0051328936, + 0.05357501, -0.005126004, -0.015067972, 0.032285556, -0.00075787696, + 0.03803164, 0.02565069, 0.0095148, 0.020173306, -0.008825822, + -0.03055622, 0.0050571063, 0.001743117, -0.008681135, 0.0226054, + 0.043839738, 0.006648647, -0.014716592, 0.0054222653, 0.00787503, + 0.029991256, + ], + }, + { + title: "Receipts", + query: "receipts, bills and invoices", + minScore: 0.2, + showVideo: true, + textEmbedding: [ + -0.012871421, 0.017175015, 0.03869955, 0.015771098, -0.05597953, + 0.014459028, -0.014563994, 0.036213174, -0.006153615, -0.007446006, + -0.03248689, -0.002643824, 0.0044151205, 0.0035229118, -0.027901199, + 0.5036518, -0.015305313, 0.03251969, -0.01211698, 0.0145049505, + -0.01817219, 0.051118307, -0.021583576, 0.015462763, 0.02681874, + 0.02679906, -0.045587923, 0.0022502027, 0.014570554, -0.014728002, + -0.013041991, -0.0065800385, 0.010509692, 0.0016860118, 0.046558857, + -0.00495307, 0.0046381727, 0.041638587, -0.016118798, 0.0161844, + -0.0025913413, 0.009007371, -0.01946458, -0.009079535, 0.0058321576, + 0.045601044, 0.013967001, -0.036475588, -0.008049558, 0.031883337, + -0.032250717, 0.018605174, -0.020625765, -0.031089535, 0.009565001, + -0.015941668, -0.023276148, 0.00030177642, -0.01926777, + 0.0124843605, 0.0060355286, 0.0040871026, 0.01542996, -0.029993955, + -0.11969373, 0.013429052, 0.004270793, -0.00055107, 0.013245362, + 0.00033457825, -0.0078658685, 0.008961448, -0.016604265, + -0.0012727093, -0.032631215, -0.042675126, -0.03104361, + -0.034113858, 0.020114055, -0.011854566, 0.07291837, -0.015764538, + 0.04847448, -0.0028603158, -0.016210644, -0.023413915, 0.014826409, + 0.03197518, 0.024798151, 0.0003280179, 0.04501061, 0.0044872845, + 0.024155235, 0.025408264, -0.010680262, -0.0029652815, 0.08523872, + 0.031686526, 0.031594682, -0.014649278, 0.031161698, -0.0121957045, + -0.0040477403, -0.03986729, -0.009374751, 0.017647361, -0.046735983, + 0.021360524, -0.013855475, -0.024988402, -0.026700657, -0.027573183, + 0.2742295, -0.00052482856, 0.022016559, -0.008567827, 0.031876776, + 0.002256763, 0.00934851, -0.01761456, -0.021524534, -0.039985377, + -0.017194696, -0.04042492, 0.03219823, 0.019517064, -0.046230838, + 0.024922797, -0.034061372, -0.027671589, -0.0075312904, + -0.0008594068, -0.015193787, 0.018998794, 0.004579129, 0.025368901, + -0.010188235, 0.031620923, -0.023105578, -0.018998794, 0.030125162, + -0.01835588, 0.0077281008, 0.010785228, 0.031470034, -0.016250005, + -0.0031292904, 0.013284723, 0.040031303, 0.019963168, -0.0216295, + 0.0061929775, -0.065537974, 0.020835694, -0.024443893, 0.0059568044, + -0.028334184, -0.043055624, -0.019930365, -0.0017319344, + -0.0075050485, 0.021852551, -0.017876973, 0.014596796, 0.021058748, + -0.09324892, 0.025926534, -0.50703037, 0.052213885, -0.016919162, + 0.013914518, 0.05725224, 0.021878792, -0.016768275, -0.014340941, + -0.0055500623, -0.015305313, -0.0289771, -0.016381213, -0.006678444, + -0.016873239, 0.010214476, 0.017706405, -0.016105678, -0.0027487897, + 0.019051278, 0.05134136, 0.013461853, -0.00051826827, -0.008948327, + -0.012057937, -0.018211551, -0.040497087, 0.026457923, + 0.00082660496, 0.0056812693, 0.000990614, -0.011815203, 0.038135357, + 0.020501116, -0.05576304, -0.03269026, -0.034015454, -0.024732547, + 0.021773826, 0.0017122533, 0.02718612, 0.017115973, -0.031542197, + 0.00606177, 0.0040411805, 0.03806319, -0.05327666, 0.031424113, + 0.0047628195, 0.01229411, -0.014373742, -0.004480724, 0.008416939, + 0.011959531, -0.048185825, 0.08870259, 0.013422491, -0.005963365, + -0.0063110637, 0.060663626, -0.04789061, -0.00017712965, + -0.007557532, -0.012543404, -0.020455195, 0.033877686, -0.042091254, + -0.019661391, 0.006363547, -0.03137163, -0.0044151205, 0.053355385, + 0.051282313, -0.011113245, -0.014406545, -0.025985574, 0.06363547, + -0.0034835497, 0.004454483, -0.026825303, -0.022239612, + 0.0050383545, -0.05691766, -0.003103049, -0.0040805424, + -0.030623749, -0.04771348, -0.0477594, 0.009604364, 0.00990614, + 0.009223863, 0.05252878, 0.016105678, 0.026549768, 0.021950955, + -0.020422393, -0.008147963, -0.01633529, 0.0117036775, 0.024496375, + -0.05052131, -0.06555765, -0.008830241, -0.023276148, -0.0106671415, + -0.0070392634, 0.03454684, -0.02954785, -0.030282611, 0.0022502027, + 0.0023551683, 0.001725374, -0.0024338926, -0.0019484262, + -0.024122434, -0.013101033, -0.04315403, -0.030545022, 0.033497185, + 0.029692179, -0.0019418658, 0.0065603578, 0.00862687, -0.014668958, + -0.01817875, 0.009118897, -0.046558857, -0.005812477, -0.018559251, + -0.03874547, 0.014045725, -0.056819256, 0.018250914, 0.011119806, + 0.06538052, 0.026333276, 0.030571265, -0.009256665, 0.022606991, + 0.012969826, 0.011342858, -0.016243445, 0.00028865572, -0.045089334, + 0.030945206, 0.021544214, -0.017234059, -0.023171183, -0.027251724, + -0.06330745, -0.039952576, 0.02607742, -0.06366171, -0.026897466, + 0.0010627779, -0.048487604, -0.049491335, -0.029626574, + -0.054824907, 0.0038640504, 0.040831663, -0.0034769892, -0.02408307, + -0.032421287, 0.031666845, 0.009755251, -0.05476586, 0.016682988, + -0.005969925, 0.013002628, 0.0056091053, -0.028196417, -0.026320154, + 0.015462763, -0.06549205, -0.01064746, -0.040333077, 0.0202059, + -0.041264646, -0.008922086, 0.018441165, -0.011683996, -0.012740213, + 0.008908966, -0.009984864, 0.0030243248, -0.035767067, 0.027993044, + 0.00642259, 0.0023158062, -0.033818644, 0.037125062, -0.011611832, + 0.013980121, -0.016190961, 0.00532045, -0.008226688, -0.029869309, + 0.033156045, 0.0003411386, 0.0054647774, -0.03877827, 0.0073935227, + 0.0488681, -0.04382319, 0.011815203, -0.007983955, -0.0040871026, + -0.047090247, 0.068214595, -0.04290474, -0.0006625961, -0.028524432, + 0.015994152, 0.00787899, -0.017161895, -0.0048874663, 0.027205803, + 0.013324087, -0.00972245, -0.028334184, -0.006809651, 0.018296836, + 0.012825498, -0.020291185, -0.06573478, -0.031522516, 0.045909382, + -0.04077262, -0.015731737, -0.008403818, 0.061024442, 0.021386765, + -0.04957662, -0.043094985, -0.0006494754, 0.0028012726, 0.051728416, + 0.0064094695, 0.015443081, 0.025467308, -0.0057403124, -0.007091746, + -0.018808546, 0.017129093, -0.037886064, -0.010929555, -0.020579841, + -0.034973267, -0.01928745, 0.0031292904, -0.02075041, -0.0057140714, + -0.035865474, -0.0935179, -0.0006888375, 0.024686623, -0.023991227, + 0.051485684, 0.00825293, -0.006088012, 0.018447725, 0.0111854095, + 0.004677535, -0.015206908, -0.050042406, 0.025834689, -0.027481338, + -0.0031686525, 0.028471952, -0.012615567, -0.007446006, 0.013232241, + -0.009079535, -0.0003542593, 0.016112236, 0.0076165753, + -0.0070851855, 0.007931472, -0.027061474, -0.010745865, -0.04331148, + -0.010345683, 0.053033933, -0.00072819967, 0.023413915, + -0.020868497, 0.04771348, -0.023702571, -0.022534829, -0.04109408, + -0.020848814, 0.0028603158, -0.027999604, -0.050750926, + -0.005313889, 0.017647361, 0.0139538795, 0.018237794, 0.042124055, + 0.019562986, 0.032381922, 0.001928745, -0.016663307, 0.023735372, + 0.040155947, -0.04385599, -0.030577825, -0.01928745, 0.029462563, + -0.020507677, 0.017686725, 0.011598712, 0.024115874, -0.026825303, + -0.015521805, -0.025946213, -0.0032342563, 0.015475882, 0.037505563, + -0.009965182, 0.0316078, 0.0035294723, -0.035124153, -0.021891912, + -0.038469933, -0.04850072, -0.049661905, -0.040818542, -0.02206248, + 0.021084988, 0.030223567, -0.028242338, -0.049242042, 0.018605174, + -0.038141917, -0.0006691565, 0.07440758, -0.0012071057, + -0.0008594068, 0.0381616, -0.0003542593, 0.0026372636, -0.042458635, + 0.045384552, 0.0015351236, + ], + }, + { + title: "Notes", + query: "screenshots of digital notes, photos of notebook and whiteboards", + minScore: 0.2, + showVideo: true, + textEmbedding: [ + 0.03148959, -0.012310127, -0.008262599, -0.040193092, 0.008668235, + 0.006904606, -0.014038484, 0.012133764, -0.027979966, -0.048931874, + -0.010167318, -0.01888846, -0.0037388923, 0.0039328914, + -0.040907364, 0.5144681, 0.017318832, 0.024770161, -0.01295385, + 0.03631311, -0.005193886, 0.004259163, 0.0110932235, 0.014303027, + 0.01637529, 0.04230063, -0.04334117, 0.03342076, 0.020925451, + -0.0005026341, -0.002733624, -0.03218622, 0.0015960837, + 0.0084389625, 0.021622084, -0.0022662627, -0.0011463584, + -0.008262599, -0.04159518, -0.016692745, -0.0064107897, + 0.0019047188, 0.023059443, 0.03879101, -0.014497027, -0.023570895, + -0.023950074, -0.004276799, -0.012468853, -0.0035184387, + -0.022521535, 0.009955683, -0.0011992673, -0.008139146, 0.016622199, + 0.00522034, 0.00021163542, -0.0035272571, 0.025502069, + -0.0014990842, -0.008103873, -0.038773373, -0.0061903363, + 0.00384471, -0.072079495, 0.038641103, 0.012133764, -0.01362403, + -7.054514e-5, -0.048490964, -0.0067899697, -0.0031128044, + -0.0281916, 0.00028218055, -0.010176136, -0.031507224, -0.048755508, + -0.0140473, -0.02521107, -0.017265923, 0.031480767, 0.002036991, + -0.012645217, 0.038076736, -0.008659416, 0.06813779, -0.013500575, + 0.013826846, -0.04963732, -0.03164831, 0.023553258, 0.00029981683, + -0.049143508, 0.00820969, -0.029875867, 0.013756301, 0.1874296, + 0.016683925, -0.013333031, 0.00892396, -0.005088068, -0.017724466, + -0.026366245, 0.008694689, -0.016366472, -0.0023368078, -0.05606575, + 0.0061727, -0.012310127, -0.01442648, -0.02644561, -0.011084405, + 0.24448301, -0.019726183, -0.012856851, -0.0027600785, -0.017486377, + 0.036242567, 0.0301845, -0.047600333, 0.024055893, 0.032979853, + 0.02568725, -0.028764782, 0.027803604, -0.011878038, 0.0064107897, + -0.0057317927, -0.030660681, -0.052591402, 0.019391095, + -0.018200645, -0.03531666, 0.010176136, -0.053773034, -0.014735116, + -0.014117845, 1.7636285e-5, 0.022486264, 0.013456484, -0.014673389, + -0.011825128, -0.034152664, 0.012072037, -0.011375403, + -0.0012345399, 0.015669838, -0.0007142695, 0.0281916, 0.010043864, + 0.013641666, 0.012504126, -0.010193773, -0.023412168, 0.023473896, + 0.014303027, -0.0066488795, -0.028076967, 0.0065077893, 0.027591966, + 0.07500712, -0.014400026, -0.0022750807, -0.019302914, 0.015008478, + -0.012327763, -0.0049381596, -0.5187449, -0.007724693, -0.05559839, + 0.013571121, 0.061083272, -0.049161144, 0.001798901, -0.002354444, + -0.0065077893, 0.0091708675, -0.028500237, -0.03745947, + -0.025934156, -0.004814706, 0.008835779, -0.012380673, 0.007468967, + 0.01637529, -0.0018429917, 0.0284826, -0.002407353, 0.014408845, + 0.00768942, -0.017257104, -0.00033508943, 0.0071779676, + -0.027256878, -0.0019399914, -0.040810365, -0.015352386, + -0.05730029, 0.00308635, 0.043949623, -0.030131591, -0.010528862, + -0.0077335113, 0.025202252, 0.04715061, -0.058305558, 0.038896825, + -0.015026114, 0.016675107, -0.015590476, -0.0016666289, + 0.0130508505, -0.046771426, -0.03849119, -0.0051850677, 0.073922485, + -0.016754469, 0.0037565285, -0.015934383, -0.065183714, 0.04718588, + -0.0029011688, 0.0042856173, 0.013377122, -0.006825242, 0.055474933, + 0.021401633, -0.02235399, -0.016410563, -0.0142765725, 0.004267981, + -0.0039064367, -0.055748295, 0.012363035, 0.01167522, -0.021860175, + 0.0043032533, 0.024514437, -0.0027071696, -0.017398193, + 0.0043120715, -0.009047414, 0.0065430617, -0.039293643, + -0.011472403, 0.030978136, -0.031401407, -0.04954914, 0.027759513, + -0.0468067, 0.014893843, -0.0483146, -0.024435071, 0.02340335, + -0.00043208894, 0.00882696, -0.01979673, 0.025325704, -0.044461075, + 0.006886969, -0.008818143, 0.0049117054, -0.024099983, -0.005167431, + -0.0066576973, -0.011075586, 0.018130101, 0.0008377235, 0.01533475, + 0.0070897867, -0.063702255, -0.03418794, -0.009267868, -0.021613266, + -0.00061726995, -0.00322744, -0.025369795, -0.02207181, 0.013236032, + 0.017098378, 0.027574332, 0.0009964501, 0.022036538, -0.030484319, + -0.016886743, -0.018200645, -0.0067546973, 0.030299138, + 0.0009611775, -0.009770501, 0.014567572, 0.023526805, -0.053676035, + 0.04506071, 0.0074777845, 0.030501954, -0.009682321, -0.02372962, + -0.012424762, 0.035766385, -0.013553485, -0.005925792, 0.008359599, + 0.018835552, 0.0036859834, 0.016728016, -0.001172813, 0.027406786, + 0.029214505, -0.016287109, 0.0064107897, -0.0056612473, -0.0385441, + 0.024029437, -0.040713366, -0.031886403, -0.045254707, 0.005714156, + -0.0349463, 0.0052291583, 0.0066665155, 0.016736835, 0.004356162, + -0.002345626, 0.03151604, 0.019338185, 0.015775656, -0.09950392, + -0.007001605, -0.011833947, 0.042803265, 0.008562416, -0.039425917, + -0.08338436, -0.037062652, -0.033068035, 0.018121284, 0.015176023, + 0.02525516, 0.0064107897, -0.013818029, -0.009073868, -0.026163427, + -0.0151054775, -0.035202023, 0.023764893, 0.020475727, -0.027988784, + 0.00892396, -0.016551653, 0.028597236, 0.04570443, 0.0148056615, + 0.07217649, 0.013923847, -0.019867275, -0.14952925, 0.058367286, + 0.012424762, 0.06264409, 0.010811042, 0.0054760664, 0.026983516, + -0.016181292, 0.005767065, -0.04343817, -0.031551313, -0.021692632, + 0.013950301, 0.008703507, 0.015493477, -0.012971488, -0.01855337, + 0.0066224253, -0.027882965, 0.05995455, -0.0007936328, 0.05747665, + -0.03848237, -0.039796278, -0.03830601, 0.03682456, -0.025193432, + -0.00768942, -0.022159992, 0.018738551, 0.0032538946, -0.005828792, + 0.031683587, 0.08285526, -0.043085445, -0.048199967, 0.003597802, + 0.024981797, 0.017618649, 0.0036859834, 0.017239468, 0.035898656, + -0.022459809, -0.05572184, 0.014814478, -0.0070456956, -0.007830511, + -0.023182897, 0.029752413, -0.008253781, -0.0042062537, -0.0182271, + 0.018068373, -0.016287109, 0.0030069866, -0.0033244397, + -0.033050396, 0.037247833, -0.028447328, -0.015687475, + -0.0066665155, 0.00223099, -0.019920183, 0.001543175, 0.04078391, + 0.012380673, 0.016595745, -0.01828001, 0.03284758, -0.007839329, + 0.007407239, -0.02316526, -0.0013491758, -0.015087841, 0.023844257, + -0.049143508, -0.021771993, -0.013994392, 0.0029893504, + -0.021868993, -0.033782303, 0.0069663324, 0.019338185, 0.037380103, + 0.003641893, 0.013086123, -0.018271191, 0.018712098, -0.03128677, + 0.021348724, -0.0063226083, -0.01865919, -0.0118868565, + -0.022371627, -0.00044972525, 0.03457594, -0.04105727, 0.010149682, + 0.015255386, 0.008236145, -0.021181177, -0.011128495, 0.0011375403, + -0.024576163, 0.00731024, -0.04319126, -0.008527143, -0.028835325, + -0.00028218055, 0.031110404, 0.03949646, 0.044452257, -0.013712212, + -0.041797996, 0.02093427, 0.028403237, -0.015255386, 0.00731024, + -0.047547426, -0.0005643611, 0.016163655, -0.0033244397, + 0.012027946, -0.011710493, -0.0025131707, -0.004144527, 0.010714043, + 0.003976982, -0.010528862, -0.03949646, -0.08931015, 0.026401518, + -0.009647048, 0.035969205, 0.0021692629, 0.014770388, -0.0054319757, + 0.0042062537, -0.00721324, 0.01518484, -0.010846316, -0.0065430617, + 0.015087841, 0.0039858003, -0.0071603316, -0.019205915, + -0.008244963, 0.038985007, -0.026983516, -0.011428312, 0.006128609, + 0.013271305, -0.030546045, -0.020193547, 0.008941596, 0.05889637, + ], + }, + { + title: "Memes", + query: "memes", + minScore: 0.18, + showVideo: true, + textEmbedding: [ + 0.024733832, 0.02873802, -0.004280583, -0.048865262, 0.0050743334, + 0.007377628, 0.007221712, 0.0043231053, 0.05938246, -0.04826995, + 0.034428928, 0.002204075, 0.0025371667, 0.003791576, 0.0009284046, + 0.52122474, -0.025187403, -0.031905934, -0.0012969316, + -0.0030332608, 0.057624865, 0.010906982, -0.0064137876, 0.006725618, + -0.009695095, -0.015981317, -0.011906258, 0.01809326, -0.011303858, + -0.011211725, 0.017916081, -0.007306757, -0.014471772, 0.016895546, + -0.007604413, -0.04029701, 0.00023387291, -0.0073634526, + -0.025541758, 0.001346541, 0.010573891, -0.0027143431, 0.026902473, + 0.032359507, 0.026491422, -0.029255377, -0.014464686, 0.027859226, + 0.04314601, 0.004401063, 0.0140182, -0.0009921881, 0.03883708, + 0.05593815, 0.028865587, 0.02632842, -0.06859563, 0.026725296, + 0.012905533, -0.010609326, 0.008135943, 0.0043231053, -0.023004591, + 0.032324072, -0.061728273, 0.034372233, -0.00027639527, 0.009284046, + 0.033408392, 0.016023839, 0.011133768, -0.009113957, 0.0074343234, + -0.004642023, 0.023203027, -0.022359667, 0.015988404, 0.009503745, + -0.048666827, -0.0018851574, 0.016796328, -0.06535684, 0.01110542, + 0.021360392, -0.046115484, 0.0014953692, 0.031388577, 0.023465248, + 0.0019843762, -0.04834082, 0.009305307, -0.038496897, 0.018674398, + 0.014195377, -0.011870822, -0.010580977, 0.13177675, -0.039184343, + 0.04032536, -0.06712861, -0.0038766204, -0.030892486, -0.0062720464, + 0.027200127, 0.013266971, 0.033642262, -0.047660466, -0.01650576, + 0.01250157, 0.0125582665, 0.00073705404, -0.044598855, 0.14651075, + -0.01353628, 0.009964403, 0.053946685, -0.0039687525, -0.017916081, + -0.01284175, -0.020651687, 0.017278247, -0.03042474, -0.018532656, + -0.012480309, 0.0067823143, 0.018086173, 0.05493887, -0.015017476, + 0.009220262, 0.0036214867, -0.010432149, 0.0007654023, 0.024875574, + -0.013699283, -0.019468147, 0.012374003, 0.008887171, 0.014648949, + -0.042047516, -0.026753645, -0.0068177497, -0.0016371105, + 0.008306031, -0.01579705, -0.019446887, 0.0078028506, -0.0058397357, + -0.014415076, 0.013387453, -0.03543529, 0.0050955946, 0.032508336, + -0.044322457, -0.027214302, -0.03252251, 0.015549005, -0.010751067, + -0.0049892887, -0.0067327046, 0.025803978, 0.059467502, 0.023543207, + 0.009624224, 0.028220663, -0.016895546, 0.02160844, -0.031806715, + -0.5259943, -0.0060665216, -0.0310484, -0.020935168, 0.06222437, + -0.02355738, 0.039900135, 0.0019347668, -0.014450511, 0.0020410726, + -0.0069594905, -0.018801965, 0.000106305866, 0.011048723, + 0.0074910205, 0.020949343, 0.021700572, 0.063492954, -0.03522268, + 0.014974954, -0.025889022, 0.0240393, -0.000616574, 0.03145945, + -0.041756947, -0.02143835, -0.03777402, -0.009525006, 0.029347507, + 0.006520093, -0.008206814, 0.014656036, 0.012600789, 0.019787066, + -0.0164278, -0.022869937, 0.015145043, -0.055392448, -0.0055987756, + -0.019021664, 0.011310944, -0.013266971, -0.0017150679, + 0.0066618347, 0.0062791333, -0.023160506, 0.009078521, + -0.0023316422, -0.023940083, -0.09124587, -0.0089367805, + -0.0024946444, 0.010347105, 0.06452766, -0.07113989, 0.020113071, + -0.0065838764, 0.008001288, -0.0124661345, -0.04612966, + 0.0021828138, 0.04277748, -0.025577193, 0.007937505, -0.014634774, + -0.013047273, -0.048631392, -0.010481758, -0.014025288, + -0.020155592, 0.0029127807, 0.0016512844, -0.0065130065, + -0.022246275, -0.016052186, -0.028000964, 0.0070091, -0.02741274, + 0.015641136, 0.017164854, -0.0039829263, -0.055392448, -0.008695819, + -0.012898446, -0.050835468, -0.01534348, 0.10378288, -0.023188854, + 0.01031167, 0.020587903, -0.015003301, 0.008674559, -0.014776516, + -0.006236611, -0.028915197, -0.015371829, -0.014174115, 0.0448469, + 0.04323814, 0.0011339292, -0.042359345, 0.017816864, 0.021282436, + 0.016094709, 0.021672223, 0.021211565, -0.014450511, 0.018752355, + 0.027426915, 0.020807602, 0.0008929693, 0.018568091, -0.0074201496, + 0.02531497, 0.014797776, -0.0051948135, -0.004415237, -0.0017930256, + -0.039318997, 0.013642587, 0.014266248, 0.0065271803, 0.024613352, + 0.010942417, 0.019978417, -0.04790851, -0.01700185, 0.032458726, + 0.014174115, 0.027589917, 0.027575742, 0.012990577, 0.008894257, + -0.015471048, -0.009872272, 0.011084158, 0.030488525, -0.0033734394, + -0.02495353, 0.010396714, 0.024124345, 0.042097125, -0.03175002, + -0.026292983, -0.008178465, -0.010134493, -0.039106388, 0.04943223, + 0.019609889, -0.06208971, 0.03557703, -0.08570379, -0.0124661345, + 0.05873045, -0.033592656, -0.01963115, 0.039928485, -0.002941129, + -0.0027568655, -0.043004267, 0.038433116, 0.044811465, -0.008894257, + -0.027667874, -0.01206926, 0.0035789642, 0.01387646, -0.048191994, + -0.010906982, -0.0030332608, 0.04902118, -0.022083273, -0.010389627, + 0.0079658525, -0.034393493, -0.018057823, 0.03714327, -0.02440074, + 0.024939355, -0.017930256, 0.01201965, -0.024514133, 0.0072996696, + 0.0053436416, 0.0141315935, 0.003068696, -0.036703873, -0.03748345, + -0.03376274, -0.04908496, -0.009404526, 0.0015449787, 0.054350648, + -0.010155753, -0.0018709833, 0.0033805268, -0.020821776, + 0.0054712086, -0.0085328175, 0.013578803, 0.040176533, 0.007710719, + 0.010772328, 0.020389466, -0.026144156, -0.014025288, 0.035612464, + -0.02476218, -0.047596682, 0.0051877266, -0.035725858, -0.090976566, + -0.029021503, -0.012785052, -0.0106235, 0.002955303, 0.0332383, + -0.023940083, 0.0064988323, -0.0044790204, 0.024003865, 0.038893774, + 0.0543223, 0.011403076, -0.022097446, 0.007398888, 0.020807602, + -0.06272755, -0.0075902385, -0.021948619, 0.056760244, 0.010729806, + -0.19647451, 0.03880164, 0.00046774582, -0.003791576, 0.017646775, + 0.03071531, 0.022295883, -0.01589627, -0.0041388418, 0.016193926, + 0.030332608, 0.022657324, -0.04556978, 0.0057972134, -0.00049609406, + 0.0049255053, -0.015818313, -0.02384795, 0.027526133, 0.011615688, + -0.02782379, 0.06167158, 0.04608714, 0.013968592, -0.011133768, + -0.050204717, 0.004053797, -0.019517757, 0.006123218, -0.005308206, + 0.03252251, -0.047724247, 0.01818539, 0.0067823143, -0.016633324, + 9.9218814e-5, -0.0116794715, -0.043911412, -0.027561568, + -0.024173955, 0.028362405, -0.008802126, -0.0065271803, 0.00725006, + -0.043394055, 0.022175403, -0.02570476, 0.010446324, -0.014330031, + 0.0042238864, -0.02873802, -0.07274156, -0.04075058, 0.030963358, + -0.013621326, 0.008015462, -0.034216315, -0.021289522, -0.033330433, + -0.008702907, -0.024351131, 0.0047058063, -0.011360554, 0.008880083, + -0.019914633, 0.028121445, 0.008901345, -0.038128372, -0.027115082, + 0.004259322, -0.03988596, 0.03615108, 0.0054641217, -0.025931545, + 0.018369654, 0.020991866, 0.019213013, -0.0070445356, 0.008950954, + -0.006590964, 0.024939355, 0.0034443103, -0.019964242, -0.009602964, + 0.030438915, 0.0110203745, 0.029999517, -0.07240139, 0.0074484977, + -0.0033380042, 0.014266248, 0.023982603, -0.033139084, 0.014244986, + 0.00029765643, -0.013798501, 0.02211162, -0.013422888, 0.025874848, + -0.009595877, 0.041650638, 0.08546283, -0.0069736647, 0.006569702, + -0.013337842, 0.006243698, -0.014755256, 0.012005476, 0.03403205, + -0.024173955, 0.024492873, -0.011183377, 0.024960618, + ], + }, + { + title: "Visiting Cards", + query: "business card", + minScore: 0.2, + showVideo: false, + textEmbedding: [ + 0.0029489065, -0.009914337, 0.008136701, 0.013697645, 0.027322732, + -0.019538447, -0.019248221, 0.020445405, -0.0040579853, 0.020186275, + 0.02225932, 0.016019454, 0.013091278, 0.008665328, -0.021046588, + 0.50537205, -0.03424152, -0.05618988, -0.007680631, 0.007110544, + 0.017449856, 0.014583871, -0.029193653, 0.0013008356, 0.029888125, + -0.011790443, -0.011386199, 0.010857572, 0.0028400715, 0.039740272, + 0.033521138, 0.009370162, -0.022176398, -0.037345905, -0.0037211154, + 0.033816542, -0.007317848, -0.005400282, 0.004275655, 0.019217126, + -0.046778258, -0.0075717964, 0.0063901604, -0.0028037932, + -0.00079812226, -0.012811417, -0.02258064, 0.0021404189, + 0.047353525, 0.014521679, -0.04745718, -0.016491072, -0.03712305, + 0.030546315, -0.07583198, -0.0063331523, 0.048996415, -0.010888668, + -0.0035760025, -0.016869402, -0.012583383, -0.0021248711, + 0.03339675, -0.03584813, -0.07071674, 0.037713867, -0.0074525964, + -0.032723013, 0.0317435, 0.016993785, -0.025710938, -0.04251297, + -0.044834778, 0.013568078, -0.009007379, 0.01095086, -0.052111164, + 0.011831904, 0.0012230964, 0.002337358, 0.03213738, 0.014526862, + 0.028245235, 0.024783252, 0.008421745, -0.012438269, 0.0185693, + -0.0464673, 0.027270906, -0.012407173, -0.021792883, 0.0036692894, + -0.017708985, 0.011847451, 0.006089569, -0.0054987515, 0.19703254, + -0.015724046, 0.012915069, 0.008732702, 0.008768979, -0.019232674, + -0.020512778, -0.012707765, -0.0034671675, -0.022891598, + -0.057112385, -0.022456259, -0.013692461, 0.021202065, 0.01960064, + -0.04331627, 0.24108475, 0.030681064, -0.037097137, 0.062424563, + 0.024897268, -0.038081832, 0.004410403, 0.039486323, -0.008805258, + 0.015822515, 0.026752643, -0.018579664, -0.0065404563, 0.015656672, + -0.03855345, -0.034184508, 0.06779375, -0.010121642, -0.0024513756, + -0.0041357246, -0.0024358276, -0.001803549, -0.013998236, + -0.016019454, 0.010588077, 0.023725998, -0.05693618, -0.02987776, + -0.0074525964, 0.022513267, -0.013490339, -0.020087805, 0.00878971, + 0.035640825, -2.073045e-5, 0.01727883, 0.0146823395, -0.022269685, + 0.016796846, 0.0018709231, -0.014905193, 0.00049753074, + -0.029851845, 0.007359309, -0.03926347, 0.015718862, 0.002710506, + 0.016807212, 0.0074992394, 0.0026534975, 0.00048198292, + -0.0057941605, 0.0066337436, 0.0014200357, -0.005970369, + -0.50913984, 0.004182368, -0.022435527, -0.015553019, 0.0639638, + 0.042305663, 0.04190142, -0.007561431, -0.0012179138, -0.019792397, + 0.0012386444, 0.0001762088, 0.0035863677, 0.043466568, 0.016143836, + 0.003529359, 0.004425951, -0.0038973242, 0.011417294, -0.0047835507, + 0.027213896, 0.04382935, 0.019372603, -0.011868183, -0.009453085, + 0.012764773, -0.031831603, -0.0061154827, -0.03019908, -0.021637406, + 0.019408882, 0.0019590273, -0.021870622, 0.031753864, -0.054510716, + -0.0465865, -0.00498049, -0.044928063, 0.030240541, 0.031095672, + 0.015148776, 0.026286209, -0.00472136, -0.008877815, 0.01420554, + -0.076547176, -0.0077946484, 0.034101587, -0.04702184, 0.043451022, + 0.028162314, -0.028462905, 0.0050685946, 0.0014511314, 0.018626308, + -0.006975796, -0.032650456, -0.010251207, 0.009732946, 0.021855075, + 0.005586856, 0.011013051, -0.06129994, -0.018439734, 0.0056438646, + -0.021476744, 0.013080913, 0.013459245, 0.02295379, 0.008297361, + -0.0020004883, -0.04693892, -0.007551066, -0.00392842, 0.01938297, + 0.0010261572, -0.011318824, 0.039512236, -0.028841237, -0.007152005, + -0.002430645, 0.14335105, 0.008515031, -0.018097682, 0.02194318, + -0.026265478, -0.03372844, 0.031458456, -0.0007566614, 0.023554971, + 0.0071001784, 0.0072349263, 0.01585361, -0.0015340532, -0.008281815, + 0.00416682, -0.03297178, 0.009292424, -0.004052803, 0.0068047694, + -0.07608075, -0.006887692, 0.009670755, 0.004612525, -0.045669176, + 0.0019538447, -0.031935256, -0.039097626, -0.010588077, 0.031795327, + 0.02345132, -0.048913494, -0.0077687358, -0.013516253, 0.026654173, + -0.017900743, -0.023466866, 0.022611737, -0.0030421934, + -0.013593991, -0.026120367, 0.11547897, -0.00031095673, + 0.0024461928, 0.04492288, -0.053588208, -0.02704287, -0.034345172, + -0.016019454, 0.0131431045, -0.085704856, 0.014925922, 0.015915802, + 0.014837818, 0.03104903, 0.018579664, 0.011691973, -0.019942692, + -0.0064627174, 0.007613257, 0.011868183, 0.01936224, 0.03600361, + 0.025643567, 0.023104087, -0.006296874, -0.007950127, 0.020004883, + -0.046498396, -0.063222684, -0.017107802, -0.014713435, + 0.0045399684, -0.0034049763, -0.08453877, -0.02008262, -0.005306995, + 0.006866961, 0.04562253, -0.06647218, -0.03928938, 0.023777824, + 0.022145301, 0.004944212, 0.015267976, 0.004296385, -0.010183833, + -0.015464915, 0.014101888, 0.020631978, -0.024736607, 0.014863731, + 0.021528572, 0.023586068, -0.01144839, 0.000114017465, + -0.0072608395, -0.04225902, 0.026654173, -0.04146608, 0.011225538, + -0.020979214, 0.014936289, -0.013982688, -0.061455414, 0.008069327, + 0.013360774, -0.017973298, 0.018781785, -0.10951895, -0.010468877, + 0.016755385, -0.0023425405, 0.021062136, 0.009748493, -0.008919275, + -0.040642045, -0.02446193, 0.03081063, -0.0010261572, -0.005213708, + -0.0103341285, 0.0024617407, 0.013075731, 0.010645086, 0.0020212189, + -0.0071416395, 0.0070898137, 0.045524064, -0.035189938, + -0.012821783, -0.0982675, 0.0015236881, -0.010137189, -0.026125548, + 0.0019020187, -0.0017154446, -0.012930617, 0.012427905, 0.030644784, + -0.015775872, 0.015604846, 0.014817089, 0.020549057, -0.03503446, + 0.049872275, 0.019331142, -0.02659198, -0.014713435, -0.0070898137, + 0.05996282, 0.00022285232, -0.02797574, -0.01950217, -0.014252184, + 0.02234224, -0.005260351, -0.01075392, 0.006815135, -0.023565337, + -0.040538393, 0.0065663694, 0.0056231343, -0.019346692, + -0.033624787, -0.015806967, -0.0064523523, -0.037418462, + -0.006400526, -0.020181092, -0.023078172, 0.013127556, -0.06507288, + -0.069566205, 0.054458886, -0.020409126, 0.027965376, 0.025788678, + -0.024632957, 0.0109560415, 0.02715689, 3.6278285e-5, 0.005379551, + -0.042507786, 0.011598687, 0.00924578, 0.011780077, 0.00484056, + -0.02635358, 0.0052085253, -0.018009577, 0.013060183, 0.032292854, + 0.015765507, 0.01236053, 0.01107006, -0.047187682, 0.014122617, + -0.026607532, -0.00565423, 0.05966223, -0.017071525, 0.056734055, + 0.01607128, 0.02081337, 0.026970314, -0.018087316, 0.03444882, + 0.0071001784, -0.011438025, -0.0012490095, 0.054132383, + -0.021741059, 0.013246756, -0.012044391, 0.01835163, -0.0059185433, + -0.008126336, -0.0072193784, -0.024114693, -0.002632767, + -0.00064782647, -0.0048198295, -0.004006159, -0.024285719, + 0.052007515, -0.012287973, 0.0088259885, 0.025482906, -0.0002954089, + 0.011142616, -0.0033738804, -0.0059548216, 0.0020523143, + -0.014760079, -0.027198348, -0.008758615, 0.038138844, -0.035837762, + -0.10033019, 0.03762058, -0.0065715523, -0.0008706788, -0.024788434, + 0.013070548, 0.010930129, -0.019232674, 0.022653198, 0.025902696, + 0.017366933, -0.0015651488, 0.0071571874, -0.0034775329, + -0.0015444184, 0.020901475, 0.019061647, 0.010556981, 0.0018501924, + 0.016693193, 0.018346447, -0.006561187, -0.017138898, -0.025902696, + 0.025799042, 0.02654534, + ], + }, + { + title: "Babies", + query: "photo of toddler", + minScore: 0.2, + showVideo: true, + textEmbedding: [ + -0.037668705, 0.00064175244, -0.023759263, 0.025042767, + 0.0014709831, 0.015027101, -0.018747823, 0.018531503, -0.021221096, + 0.014623303, 0.040452033, 0.011789497, 0.06196877, -0.01336864, + 0.025980156, 0.55503654, -0.00842931, 0.0072323335, -0.006705953, + -0.07496245, 0.0052782334, -0.024631757, -0.04707146, -0.016930727, + -0.01767343, -0.02683823, -0.015690487, -0.0075856578, 0.021934954, + 0.014457457, -0.012885524, -0.03559923, -0.014536774, -0.010960266, + 0.01573375, 0.021466259, 0.010058929, 0.0024949026, 0.009402756, + 0.0049104877, -0.003208762, -0.0012186086, 0.0063814707, + -0.009265752, 0.017680641, -0.026355116, -0.027436718, -0.027804464, + 0.015329951, 0.0061939927, 0.00616515, 0.04787906, -0.0113496445, + -0.010246406, -0.007008802, 0.012777363, -0.016916307, 0.0013916654, + 0.017622955, 0.026124371, 0.0368539, -0.01377244, -0.0018243075, + 0.03783455, 0.001521458, 0.03125118, 0.0034395044, -0.019815007, + -0.015762594, 0.089218006, 0.0068213237, -0.02516535, -0.007787558, + 0.025792679, 0.014089711, -0.021963798, -0.066093296, 0.0018964144, + 0.05002064, -0.070527874, 0.010131036, -0.039413694, 0.009474862, + 0.018675717, -0.009186434, 0.0127917845, -0.025331194, 0.011674126, + -0.016635088, -0.0004326421, -0.007845243, 0.02827316, 0.007080909, + 0.038959417, -0.0016007758, 0.021862846, 0.034106616, 0.011313591, + 0.007621712, 0.02106246, 0.0036125614, 0.0018531504, -0.00077875576, + -0.03751728, -0.029729724, -0.0143420845, -0.07822169, 0.006979959, + 0.0038721466, 0.013498433, -0.015265056, -0.0040740464, 0.056019936, + -0.057426028, 0.002610274, -0.06806181, 0.003403451, 0.011890447, + 0.0072323335, 0.0034683477, -0.03414988, 0.015366005, 0.01470983, + -0.005847879, -0.06406708, 0.0052061267, -0.036046296, -0.07235218, + -0.0017810432, 0.012013029, 0.06053384, 0.017868118, 0.00915038, + -0.05428937, -0.030883435, -0.016195236, 0.0074919187, -0.007419812, + -0.046905614, 0.039031528, -0.028201055, 0.033673976, 0.0070881196, + 0.00023074244, 0.012964842, 0.011609229, 0.019613108, -0.0147170415, + 0.0021415784, 0.009532548, -0.011666915, -0.04653787, -0.08192078, + 0.03661594, 0.018675717, -0.04123079, 0.02106967, 0.007448655, + 0.03256353, -0.051859368, -0.042269133, 0.0046509025, -0.010505992, + 0.0009950768, 0.04112984, 0.018625243, -0.017868118, -0.5596875, + -0.030256104, -0.039925657, -0.016606247, 0.09181386, -0.03318365, + -0.019180465, 0.0014060867, -0.008551892, 0.015070367, 0.003042916, + 0.006208414, 0.029974885, 0.00031006016, 0.03846909, 0.0027400665, + 0.0059055644, -0.023283355, 0.013628227, 0.014810781, -0.022893978, + 0.059697397, 0.028114527, -0.0072828084, -0.0042903675, 0.060483366, + -0.07522925, -0.024141429, -0.034236412, 0.019115571, -0.04477846, + 0.057909146, 0.0089412695, 0.037337013, -0.03711348, -0.002098314, + -0.030147944, -0.0020694714, -0.010830474, 0.026549803, 0.03926948, + 0.0018387288, 0.04306952, -0.009309015, 0.015596747, 0.015755383, + -0.03440226, 0.008645631, 0.021134567, 0.026412798, 0.0041245213, + 0.007996668, -0.026066687, 0.01881272, 0.01953379, -0.002350689, + 0.05210453, 0.003958675, 0.010087771, 0.013887811, -0.02608832, + -0.00090854836, -0.044403497, 0.023283355, -0.0044850563, + -0.016880251, -0.11022999, 0.011111692, -0.04955194, -0.0036197721, + 0.029664826, -0.009323437, 0.0034683477, 0.015380426, -0.023103088, + 0.0016873041, 0.027205976, -0.02619648, -0.01953379, 0.009705604, + 0.066735044, -0.07096773, 0.004636481, -0.003331344, 0.02949177, + 0.004492267, 0.06757148, -0.02508603, -0.0012907155, -0.038368143, + -0.019461684, 0.019129992, -0.020939877, -0.031099755, -0.021776319, + 0.023687156, -0.014096921, -0.008256254, -0.014428614, -0.010008453, + -0.027941467, 0.004636481, 0.0039082007, 0.008277886, 0.0074342336, + 0.012510567, 0.006186782, 0.027169922, 0.016808145, 0.0158347, + 0.0071962797, 0.008724948, -0.01120543, 0.043119997, 0.03249863, + 0.001788254, 0.043949224, 0.002538167, 0.02918892, -0.03585161, + 0.0010094981, 0.030688744, -0.041266844, -0.031070912, -0.015243424, + 0.0005263812, 0.009705604, -0.0072828084, -0.001694515, 0.004506688, + 0.042002335, 0.023391517, 0.010909791, -0.024754338, 0.003475558, + 0.014089711, -0.009575811, -0.012373564, -0.00015142473, + 0.004326421, -0.01438535, 0.040401563, 0.017449897, -0.04408623, + -0.0035548757, 0.004492267, -0.038288828, -0.021855637, + -0.039392065, -0.066655725, -0.017211946, -0.009272962, 0.028785119, + -0.019613108, -0.0060137254, 0.021055248, 0.044569347, + -0.0054657115, 0.0063309963, 0.015697697, 0.01951216, 0.008458153, + 0.00063454174, 0.002026207, -0.02291561, -0.032743797, 0.008623999, + -0.041879755, -0.00057685614, 0.015611168, 0.002653538, 0.003136655, + 0.006410314, -0.05578199, -0.011573176, 0.05498881, 0.020132279, + -0.021906111, -0.013700333, 0.013008106, -0.024213536, -0.020427918, + 0.011017952, 0.01490452, 0.0023290566, 0.022901189, -0.017594112, + 0.0047879056, 0.0012258192, -0.027090607, -0.048181906, + 0.0030357053, 0.03383982, -0.08334129, -0.01972127, -0.0069150627, + -0.025158137, -0.025230244, 0.029859515, 0.03844746, -0.036630366, + 0.010902581, 0.004809538, 0.021797951, 0.005919986, -0.009035009, + -0.03577229, 0.024588494, -0.061795715, 0.01254662, 0.02598737, + -0.05224153, -0.002804963, -0.013080212, 0.01294321, -0.04575911, + -0.015236213, 0.029816251, 0.0014205083, 0.00034611367, 0.003893779, + 0.029044706, -0.02569894, -0.013628227, -0.010924213, 0.008458153, + -0.033529762, 0.024213536, 0.019598687, -0.027595354, 0.002487692, + -0.031597294, -0.022771396, -0.029044706, 0.005775772, 0.017226366, + 0.0022569494, 0.04748968, -0.004362474, 0.014024814, 0.0047518522, + 0.0046653235, 0.027595354, -0.0006778059, 0.028713014, 0.012049083, + -0.014190662, -0.013145109, -0.015654434, 0.03022726, -0.022634393, + -0.020637028, -0.006179571, 0.052169424, 0.031215128, 0.044028543, + 0.0073621264, -0.013202795, 0.008400467, -0.0041100997, + -0.013736386, -0.021148989, -0.032513052, 0.012690835, 0.02477597, + -0.029037496, 0.0057685613, 0.0036558257, 0.017723905, 0.013260481, + 0.034596946, 0.020737978, -0.0069655376, -0.025504252, 0.028352479, + 0.02076682, 0.024653388, -0.0044345814, -0.0012835049, + 0.00035332437, 0.016180813, -0.01181834, -0.021091303, -0.02538167, + 0.011313591, -0.0062156245, 0.015452534, -0.02642001, 0.024516385, + -0.0024732705, -0.03708464, -0.028496692, -0.00493212, -0.041591328, + 0.036745735, 0.0012690835, -0.00585509, 0.034899794, -0.0050835446, + 0.013296533, 0.017781591, 0.023413148, 0.016887464, -0.0015286687, + -0.01572654, 0.011248694, 0.014017604, -0.0136498585, 0.002538167, + 0.027516035, 0.008133672, -0.00493212, 0.054000944, -0.033241335, + 0.01574096, -0.032556318, -0.030133521, -0.009842608, -0.050943606, + -0.053589933, -0.012236561, -0.0029491768, 0.009777712, + -0.028244318, 0.0030357053, 0.014515142, 0.07838033, 0.019093938, + -0.018344024, -0.058197573, -0.009777712, 0.00585509, 0.029852306, + 0.019461684, 0.022576706, 0.0042831567, -0.020824507, 0.011068427, + 0.0042110495, 0.018228654, -0.03414988, -0.013339797, -0.02361505, + 0.028648118, 0.00873937, + ], + }, + { + title: "Pets", + query: "photos of pets", + minScore: 0.2, + showVideo: true, + textEmbedding: [ + 0.015499058, 0.038108576, 0.0033366026, -0.024849616, 0.011684836, + -0.0008677857, -0.036285553, -0.00020853766, -0.00016144852, + 0.0013319502, 0.03972306, 0.0017893877, -0.0593996, -0.04062448, + -0.005630517, 0.5492949, -0.022737332, -0.0070768264, 0.023954922, + -0.06362417, 0.015277065, 0.04124337, 0.01919219, -0.03480561, + 0.0033635108, -0.04083302, -0.0294509, 0.016461022, 0.041976616, + 0.0049174526, 0.029154912, -0.0050990824, -0.0110794045, + 0.032767322, -0.0073997234, -0.03424054, -0.0033029674, 0.009142023, + 0.017564254, 0.0022737333, 0.020423237, -0.017362442, -0.019118194, + -0.035148688, -0.011671382, -0.04937634, -0.021701371, -0.013097511, + 0.03356111, -0.014732176, 0.04385345, 0.02055105, -0.010541243, + 0.013588583, 0.0069759213, 0.0060677733, -0.05872017, -0.020786496, + -0.0064848484, -0.015162706, -0.0145034585, 0.037805863, + 0.002166101, 0.056722246, -0.04638954, 0.029928518, -0.007749529, + 0.021593738, 0.035700303, 0.010837232, 0.005529612, 0.013938388, + 0.0011906828, -0.03349384, -0.004379291, -0.007971521, -0.11617566, + 0.014160381, -0.0051058093, -0.012249906, 0.03877455, 0.004769458, + 0.02775569, -0.00030944298, -0.02714353, 0.019246008, + -0.00057852385, 0.019071106, -0.008758582, -0.0080858795, + 0.037038982, 0.0074199047, -0.039379984, 0.032464605, -0.01099868, + 0.00442638, -0.031879354, 0.019905256, 0.032464605, -0.027681693, + -0.035337046, 0.026928267, 0.0038142211, 0.047533132, 0.0017153904, + 0.01219609, -0.061458066, 0.0035585945, 0.019461274, -0.011906828, + 0.0022602794, -0.005583428, 0.09441374, -0.0012983151, 0.010070351, + -0.008227147, -0.019521816, 0.021075757, 0.0072920914, 0.019044198, + -0.027681693, -0.0016212121, 0.06122935, -0.0110794045, + 0.0096869115, -0.014597637, 0.0067673833, -0.004708915, 0.011637747, + 0.029444173, 0.024291275, 0.049255252, 0.015129071, 0.019992707, + -0.02283151, -0.03871401, -0.0042582043, -0.010319252, -0.036124103, + 0.02380693, -0.02461417, -0.008543317, 0.016931914, 0.0041438453, + 0.029417265, 0.002805168, -0.0067808377, -0.00090142083, + -0.0064310324, -0.016676286, 0.006659751, -0.02201754, -0.036413368, + -0.020584686, -0.007305545, -0.01131485, -0.005085628, 0.00833478, + -0.0059197787, 0.024789074, -0.04277713, 0.016790645, 0.0060543194, + 0.032108072, 0.01810914, 0.028058408, -0.007581353, -0.5537482, + 0.016582107, -0.0077293473, 0.014732176, 0.073230356, -0.026168115, + 0.043483466, -0.010581604, 0.04939652, 0.010722872, 0.030251415, + -0.02236062, -0.051972967, -0.014658179, 0.010278889, 0.005771784, + -0.010278889, 0.002832076, 0.005381617, 0.007023011, -0.02176864, + 0.014416007, -0.030446498, 0.0064377594, -0.045400668, 0.0179746, + -0.020470327, -0.018687665, -0.023995286, 0.0030137056, + -0.030688671, 0.025266692, -0.004749277, 0.04636936, -0.046470266, + 0.01176556, -0.0087384, -0.003087703, 0.0022535522, -0.02700899, + 0.026417013, -0.029471083, 0.0011974098, 0.010029988, -0.024264365, + -0.027204074, -0.0325386, 0.0016750284, 0.06352999, -0.04159317, + -0.03649409, -0.012317176, -0.012216271, 0.023248585, 0.041061737, + 0.019595813, -0.014247832, -0.004439834, 0.00085433177, 0.004332202, + 0.01865403, -0.0039554886, -0.08877651, -0.006040865, 0.032464605, + -0.028132405, -0.08509682, -0.026201747, -0.025555953, -0.032646235, + 0.019703444, 0.041902617, -0.024721805, -0.03310367, -0.011570477, + -0.015801774, -0.010796869, -0.032733686, -0.030009242, 0.014981077, + -0.040241044, -0.080683894, -0.022905508, 0.010036716, + -0.0045541935, -0.0073728156, 0.054576326, -0.03471816, + 0.0070297373, -0.01978417, -0.035855025, 0.0007466994, 0.014133472, + -0.042541683, -6.7270216e-6, 0.022703698, -0.002865711, 0.037651137, + 0.056163903, 0.03125374, -0.050311394, 0.00961964, 0.010783415, + -0.010238527, 0.0041842074, -0.003107884, 0.020154156, + -0.0052807117, 0.019420912, 0.035686847, 0.030856848, -0.0030406138, + -0.01662247, 0.014920533, 0.039359804, 0.023786748, -0.0474255, + 0.017510436, 0.008240601, -0.010420156, 0.0060745003, -0.0180486, + 0.004029486, -0.032330066, 0.017100088, 0.012841884, 0.017833333, + 0.016461022, -0.011052496, 0.007366088, 0.053822897, -0.023558028, + -0.01187992, -0.021580286, -0.009175657, 0.00978109, -0.00096869114, + 0.02488998, -0.0069893748, 0.045434304, 0.0036594996, 0.035350498, + 0.008254056, 0.032215703, 0.027230984, 0.021943543, -0.006269584, + -0.008045517, -0.042212058, -0.057394948, -0.0020181064, + -0.018344587, 0.002172828, 0.020261789, 0.0054623415, -0.006040865, + 0.035646487, -0.06451887, -0.014691815, 0.006249403, -0.034543257, + 0.010265434, -0.016575381, -0.012061549, 0.01477254, -0.02380693, + 0.06783528, -0.022024268, 0.02039633, -0.012754433, -0.012835157, + 0.017086634, 0.00713737, -0.012969697, 0.014785993, -0.025825035, + 0.005448887, -0.012492078, 0.0017153904, 0.0024217279, -0.021674464, + 0.0074468125, -0.0071306424, -0.02226644, 0.015633598, -0.01878857, + -0.01104577, 0.02123048, -0.009673457, 0.04532667, -0.04788294, + -0.028737836, 0.05532975, 0.049181253, -0.002858984, -0.0058726897, + -0.0058054198, 0.039992142, 0.021297751, -0.029282724, -0.004419653, + -0.010581604, 0.028885832, 0.020181064, -0.0029195272, 0.05319056, + -0.026376652, 0.016595563, -0.03611738, -0.0125862565, -0.04194298, + -0.08883704, -0.011489753, -0.0068682893, 0.0015337608, + -0.022340437, -0.02640356, 0.024183642, 0.0024015466, -0.015546147, + -0.022730606, 0.04638954, -0.0015001257, -0.04510468, -0.04361801, + -0.0037200428, 0.011489753, -0.002542814, -0.0074199047, + 0.029598894, 0.00905457, -0.042595502, -0.058309823, 0.028724382, + -0.006424306, -0.027506793, -0.015418333, 0.04352383, 0.027109895, + 0.038444925, -0.041001197, 0.05113882, 0.034617253, -0.0030137056, + -0.022844965, 0.029894883, 0.01176556, -0.023255313, -0.0006121589, + -0.029767068, -0.012498806, -0.0061888597, -0.03656809, 0.058121465, + -0.088890865, 0.05002886, 0.02850239, 0.015875772, -0.023820383, + -0.0067404755, 0.017079907, 0.011920282, -0.023571484, -0.03588866, + 0.03321803, 0.017712247, -0.04509795, -0.024150006, -0.0108708665, + -0.020409781, 0.038788006, -0.0015270339, 0.017853515, -0.07197913, + -0.0029397083, 0.000941783, 0.004688734, 0.008872941, -0.009599459, + 0.0090411175, -0.03679681, 0.042299513, -0.011671382, -0.062742926, + 0.050264303, -0.004567648, 0.004930907, 0.020490507, -0.04079266, + -0.02689463, -0.018122597, 0.0011099585, -0.034697976, + 0.00029598895, -0.029262545, -0.017766064, 0.02682736, 0.005852509, + 0.0021593738, 0.049147617, -0.011644474, -0.007406451, 0.037294604, + -0.030910663, 9.41783e-5, -0.01976399, -0.0072920914, 0.016198669, + 0.0049510878, 0.0189702, 0.007480448, 0.006209041, -0.03825657, + -0.023833837, -0.0065992083, -0.0002488998, 0.006249403, + -0.042138062, -0.025206149, -0.07841689, -0.002811895, 0.0017221175, + 0.021102667, 0.020490507, 0.032377154, 0.014537094, -0.019999435, + 0.020786496, 0.012404627, 0.015485603, -0.010776688, -0.006760657, + -0.0032087893, 0.025152333, -0.015821954, 0.0032760596, -0.01903747, + -0.018990383, 0.0089402115, -0.015014712, -0.0043120207, 0.03105193, + 0.00023544575, 0.018963473, -0.03123356, + ], + }, + { + title: "Selfies", + query: "selfie", + minScore: 0.18, + showVideo: true, + textEmbedding: [ + -0.0011263192, 0.040301856, -0.02074704, -0.11898843, 0.03114752, + -0.038252916, 0.0115447715, -0.0012581225, 0.029248353, 0.06059357, + 0.04173372, -0.017290197, -0.024018157, 0.010490346, -0.013917232, + 0.4287502, 0.05462049, -0.034514494, 0.07627218, -0.03058436, + 0.02120236, 0.03208212, 0.033400156, -0.028265819, 0.010819853, + 0.017481912, -0.017937232, -0.03074612, 0.061078854, -0.02775658, + 0.0045052767, -0.040799115, 0.0027618785, 0.0077644135, 0.008447395, + -0.038318817, 0.018470436, -0.015582745, 0.025066594, -0.0488511, + -0.058676437, 0.032261856, -0.041805614, 0.075205766, 0.040613394, + -0.011484861, -0.008195769, -0.002935619, 0.036275864, 0.007171299, + 0.005332043, 0.04541223, 0.0035227428, 0.026061108, -0.062486753, + -0.044771187, 0.03306466, 0.0012760957, -0.008657081, -0.01927923, + 0.0028517444, -0.00859717, -0.038079172, -0.0409429, -0.086433016, + 0.030069128, 0.006236693, 0.05291903, 0.034460574, 0.072869256, + 0.01470206, 0.031824507, -0.033382185, 0.014372553, 0.011161344, + 0.003025485, -0.06787271, -0.011029541, 0.086672656, 0.0046191067, + 0.034748144, -0.021633716, -0.018116964, 0.031518962, 0.02063321, + -0.0005451864, -0.037054703, 0.030308772, 0.022011153, -0.020004148, + 0.023838425, -0.004810821, -0.0072551733, 0.020111987, + -0.0012641135, 0.0024443523, 0.01323425, 0.021118484, -0.050055303, + -0.03371768, -0.013461911, 0.021328172, 0.009340062, -0.018021107, + -0.004379465, 0.032759115, -0.060647495, 0.015151391, -0.034400664, + 0.02549795, -0.022035118, -0.015391031, 0.12818472, -0.08661874, + -0.024036132, 0.071635105, -0.02102862, -0.087325685, -0.0015876308, + -0.018626204, -0.035227433, -0.021406054, 0.037348267, -0.009933176, + 0.02047145, 0.0014797917, 0.039013777, -0.04082308, -0.024868889, + -0.0091004195, 0.0050864094, 0.023161437, -0.018338634, -0.01685285, + 0.0063684965, -0.008279645, 0.025414074, 0.004307572, 0.015492881, + -0.020657172, -0.024437532, -0.007752431, 0.01153878, -0.02080695, + -0.008495322, -0.036785107, -0.021519886, -0.029014701, 0.00661413, + -0.047509104, 0.024563344, 0.027708651, -0.035485048, 0.0050684363, + 0.03207613, -0.019375086, -0.009999079, -0.002120835, -0.07661966, + -0.016739022, 0.027522927, 0.055093784, -0.016181853, 0.008165814, + 0.0416139, -0.075697035, -0.019261258, -0.43341124, 0.052601505, + 0.035790592, -0.010694042, 0.07215632, -0.009094428, -0.01707452, + 0.022250796, -0.045370292, 0.013072493, 0.0041877506, 0.0365874, + 0.04126642, -0.01696668, -0.015115443, 0.027061615, -0.028930828, + 0.04953408, 0.05953915, 0.044375777, -0.04473524, 0.058460757, + -0.060240105, 0.014426471, -0.0213761, 0.009124383, -0.03601226, + -0.04032582, -0.009963132, 0.003217199, 0.02758284, -0.03369372, + -0.009795383, 0.077338584, 0.021975206, -0.01600212, -0.02601318, + -0.045885526, 0.0046071247, 0.025827456, 0.061444305, 0.049929492, + 0.044603437, 0.007171299, -0.004331536, -0.027798515, 0.01712245, + 0.0054458734, -0.008069957, -0.06565003, 0.008788884, 0.048713304, + 0.0040679295, 0.009519793, -0.015187336, 0.026091065, -0.03308862, + 0.001815291, 0.024293747, -0.034975808, 0.02951196, 0.057034887, + -0.01295267, -0.033507995, -0.0050085257, 0.02380248, -0.09486843, + 0.00876492, 0.012766948, 0.029493988, -0.03177658, 0.019225309, + -0.025875386, 0.01176045, -0.023149453, -0.031932347, 0.015486889, + -0.025743583, 0.030470528, 0.043944422, -0.04116457, -0.075020045, + 0.0061468272, 0.0092741605, -0.009376008, 0.022640213, 0.04531038, + -0.022580303, 0.019992165, -0.017889302, -0.050444722, -0.009987096, + 0.0007488825, -0.050366838, 0.012677082, -0.014228767, 0.0014797917, + 0.051708832, -0.024185907, -0.011604683, -0.054290984, 0.058394857, + -0.028601317, 0.004427393, 0.016367575, 0.056339923, -0.043602932, + -0.007818333, 0.002324531, -0.0070275124, 0.042081203, -0.01714042, + 0.005308079, 0.04495691, -0.0101069175, -0.025258306, -0.043680813, + 0.030488504, 0.0003414904, -0.026384626, 0.0055776765, 0.015175354, + 0.041913453, -0.0587663, 0.008477349, -0.04929444, 0.04883912, + -0.043507077, -0.013995115, -0.0037983318, 0.09000968, + -0.0077883773, -0.024335682, 0.0062187198, -0.040194016, -0.0264685, + 0.03414904, -0.05403337, -0.018051064, 0.00040739204, 0.036281858, + -0.020117978, -0.050085258, -0.0228499, -0.04105074, -0.028199919, + 0.021957234, 0.029877415, 0.027277295, -0.059275545, 0.084264256, + -0.03755196, 0.009250196, 0.00046730263, -0.026606295, -0.029032674, + 0.011736486, -0.0040799114, -0.03198627, -0.011808378, 0.013282179, + 0.008471359, 0.007105396, 0.025360154, -0.01599613, 0.010058989, + 0.06759114, -0.020070048, -0.05330845, -0.018506382, 0.008681046, + -0.075097926, -0.016810913, -0.048587494, -0.0036845016, + -0.04048758, -0.040301856, 0.046892025, -0.0206392, -0.015696576, + -0.01752984, -0.11967141, 0.01940504, -0.012060003, 0.007542744, + 0.024389602, 0.012617171, 0.0044873036, 0.00024563345, -0.06508687, + -0.0065002996, -0.05510577, 0.043351308, -0.073037006, + -0.0032531454, -0.0022286742, -0.015558782, 0.032717176, + -0.01012489, 0.07116181, 0.06544633, 0.0058592563, -0.013881286, + 0.013995115, -0.03325637, 0.02165768, 0.009813355, 0.014252732, + -0.042201024, 0.0027379142, -0.059413336, 0.005302088, 0.014468409, + -0.038768146, 0.0030674224, 0.02618692, -0.05269736, 0.028092079, + -0.043327343, -0.00565556, -0.013449929, 0.012413475, -0.087115996, + 0.03048251, -0.086552836, -0.007500807, 0.024808979, -0.013575741, + 0.058406837, -0.010214756, -0.015211299, -0.025743583, -0.11988111, + 0.0047029816, -0.06276234, 0.02550394, 0.013431956, 0.031015715, + 0.020651182, 0.077284664, -0.0051942486, 0.009603669, 0.015001614, + -0.014294668, 0.0037084657, 0.007590672, 0.024802987, -0.016373565, + 0.06445181, -0.036209963, 0.005206231, -0.03969676, -0.031524956, + -0.009747454, -0.056807227, -0.02108853, 0.022700125, 0.01832665, + 0.0025941287, 0.012407484, 0.032213926, 0.045268446, 0.027774552, + 0.020375593, -0.0032291813, -0.030308772, 0.007452878, -0.012245725, + -0.05944329, -0.008387484, -0.024581317, 0.027936311, -0.019482926, + -0.01814692, -0.062564634, -0.048108205, -0.057358403, -0.023323195, + -0.049623948, -0.05750219, -0.01786534, 0.0009705517, -0.012754966, + -0.050294947, 0.02753491, 0.09078852, -0.019434998, -0.008974607, + -0.03320245, 0.059245586, -0.048689343, -0.0057094796, 0.007291119, + -0.0055716857, 0.025456011, -0.0036964838, 0.03136919, -0.023419052, + 0.057238586, -0.0349818, -0.01328817, -0.032765105, -0.032363705, + -0.0046610446, -0.018093, 0.016774967, 0.003930135, 0.010718006, + -0.0008806857, 0.044243973, 0.04494493, 0.0025641737, 0.010975622, + 0.00605097, -0.053146686, -0.043033782, 0.028775059, 0.050420757, + -0.0063325497, -0.00938799, 0.034634314, -0.019177383, 0.023928292, + 0.040739205, -0.047119685, -0.024785014, 0.026708145, 0.0089566335, + 0.0398705, 0.025605788, -0.014947694, -0.0062906127, -0.014342598, + 0.0042776163, -0.0048707314, 0.015846353, -0.022646205, + -0.014324623, -0.0026121018, 0.01780543, 0.037641827, 0.027127517, + -0.009939168, -0.050019357, -0.013030554, + ], + }, + { + title: "Wallpapers", + query: "high resolution hd wallpaper", + minScore: 0.18, + showVideo: false, + textEmbedding: [ + 0.032093976, -0.022385579, -0.00020051774, 0.005241311, -0.05675209, + 0.012348551, 0.039089818, -0.040042277, 0.020720165, 0.101189055, + -0.02948168, -0.02221291, -0.04773436, 0.0058929934, -0.014816034, + 0.49386406, 0.009563583, -0.0051299124, 0.03526327, 0.04364046, + -0.015127949, 0.013312151, 0.02672456, -0.014086371, 4.45595e-5, + -0.005029653, 0.005230171, -0.0012810855, 0.012348551, -0.032238796, + 0.034138143, -0.016425746, -0.034266256, -0.029414838, 0.004812426, + 0.011462931, -0.03292947, -0.050123867, 0.0028295282, 0.018514471, + 0.03807609, 0.014337019, 0.031481285, 0.037268452, -0.009312935, + -0.012209302, -0.033285942, 0.01945579, 0.0030244759, 0.020174311, + -0.007558405, 0.0059876824, 0.009658271, 0.007614104, -0.0013757745, + -0.0177681, 0.02490319, -0.018787399, -0.047478143, 0.0024452023, + 0.022107081, -0.019316543, -0.022942571, 0.064511016, -0.03565874, + 0.04229253, -0.022859022, -0.027036475, -0.0076363836, 0.05596116, + 0.03557519, 0.05645131, -0.013735466, 0.0093407845, -0.0032249936, + -0.004556209, -0.072086126, -0.043874394, 0.0070292605, 0.068293, + -0.007491566, -0.049021017, -0.0035703299, -0.006043382, + -0.023365887, 0.062132653, 0.044966105, -0.0012532359, -0.008750372, + -0.006082372, -0.05675209, -0.0057815947, -0.033274807, + 0.0031525844, -0.024752801, -0.0045116493, -0.003096885, + -0.030021962, -0.019355532, -0.027827406, 0.014743623, 0.021789595, + -0.002049737, 0.08173326, 0.009351925, -0.042838387, -0.06335803, + -0.0041551734, 0.031230638, 0.013223032, 0.041451473, 0.0005569937, + 0.12252748, -0.04196391, -0.0048291357, 0.1443115, -0.011964225, + 0.0019884675, -0.014242329, -0.028829996, 0.035369102, 0.030551108, + 0.037920132, 0.0424652, 0.015478856, 0.006862163, -0.07678716, + -0.027147874, 0.020631047, -0.02044724, 0.020057343, 0.027465362, + -0.002216835, -0.017244525, -0.017996468, -0.0032138538, + 0.011368242, 0.03188789, -0.041234244, -0.014426137, 0.005625637, + -0.019299833, -0.019344391, -0.010265394, -0.008143248, + 0.0090010185, 0.026362512, -0.0127774365, -0.011869536, + -0.019349962, 0.00793716, 0.0035536198, 0.05486945, 0.0034477913, + -0.023722362, -0.034010038, 0.0047678663, 0.017055148, -0.021388559, + -0.015840901, -0.025120417, -0.015378597, 0.013451398, 0.014877303, + 0.018330663, -0.06190985, 0.012226013, -0.49835342, 0.033302654, + -0.040287357, -0.010204125, 0.0561951, -0.024117827, -0.009056718, + 0.0024674824, 0.012365261, 0.0026958496, 0.005920843, 0.010744409, + 0.058796257, 0.018341804, -0.014559816, -0.0038766763, 0.015373027, + 0.026323525, 0.0065836655, 0.0025176115, 0.021388559, 0.007820192, + 0.009530162, 0.01413093, -0.017038438, -0.044648618, 0.011490781, + -0.015573544, -0.045528665, 0.028206162, 0.049700547, 0.047884747, + 0.005085353, 0.057281237, -0.052491087, 0.00707382, 0.0379257, + 0.024262648, -0.016414605, -0.046776332, -0.010967206, -0.033831798, + -0.01248223, -0.038616374, 0.030734912, -0.016726522, 0.030902011, + -0.003013336, 0.04066611, -0.046553534, -0.0009134697, 0.027309401, + 0.009947908, -0.024635833, -0.02480293, 0.017523022, -0.029392557, + -0.026100725, -0.05679665, 0.056423463, 0.006962422, -0.014119791, + -0.029024944, 0.034477912, 0.031642813, -0.038772333, -0.0042164424, + 0.010989486, -0.008934179, -0.03391535, -0.022831174, -0.0058874236, + 0.00013367849, -0.028022354, -0.013612926, -0.07437537, + -0.016481444, -0.012866555, -0.026407074, -0.011563189, + -0.047622964, -0.06853808, 0.01962289, -0.00980866, 0.017110847, + -0.0231041, 0.005408409, -0.032127395, 0.00531372, -0.0134124085, + -0.032461595, -0.002851808, 0.011680159, -6.126931e-5, 0.020023923, + 0.014905152, -0.018965635, 0.009936769, -0.023482855, -0.039886322, + -0.028880125, -0.007591825, 0.017127557, 0.0039379457, -0.016726522, + -0.020848276, 0.0020107473, 0.0014871733, 0.0025398915, + -0.018319523, -0.0135182375, -0.06772486, -0.043985795, -0.03196587, + 0.009106847, 0.031213928, -0.014699064, 0.05031881, -0.0011084175, + 0.010354513, 0.026340235, 0.02676355, 0.019968225, -0.014905152, + 0.0074302964, -0.050497055, -0.0018157995, -0.018792968, + -0.020068483, -0.01665968, -0.008299206, 0.06564171, -0.0021444259, + 0.021020943, -0.014437278, 0.024986738, 0.025009017, -0.004327841, + 0.024652543, -0.031169368, 0.009530162, 0.024524432, 0.00013924843, + 0.08978739, 0.022574956, -0.0054418286, -0.0062717493, -0.004506079, + 0.013445828, -0.044553928, 0.020113043, -0.06200454, 0.0068510226, + -0.04795159, -0.06583666, -0.031770922, 0.0006294029, -0.050491482, + -0.0017601001, 0.053048085, -0.07610762, 0.0155122755, -0.008254647, + -0.027320541, 0.028551497, -0.009491173, 0.0056479163, + -0.0071016704, 0.016425746, 0.0070571103, -0.008388326, 0.012810856, + 0.035318974, 0.018397503, -0.0025788809, 0.059659597, -0.0021945552, + -0.02670228, -0.023644384, 0.005831724, -0.025471324, 0.023700083, + -0.025070287, -0.0097640995, -0.007302188, -0.022491407, + -0.014648935, 0.010387933, -0.00892304, 0.06877758, 0.010527181, + -0.008187808, -0.0020998663, 0.0022056953, -0.0030801753, + -0.006800893, 0.0020218873, 0.04506636, 0.042843957, -0.0009970188, + -0.043228284, -0.038315598, -0.009608141, 0.02392845, 0.03625472, + -0.02561614, 0.012120184, -0.0014927431, -0.02485306, 0.0248085, + -0.008182238, -0.11733073, 0.04351235, 0.011930806, 0.0059876824, + 0.024413036, -0.01686577, 0.0051466217, 0.019628458, -0.019155014, + 0.009591431, -0.01784608, 0.036978815, -0.023388166, -0.07350647, + 0.03270667, 0.005096493, 0.011746998, 0.015679374, 0.03409359, + -0.04006456, 0.0019996073, -0.10097182, -0.060623195, + -0.00049572444, 0.016442455, -0.06061763, -0.02830085, -0.007190789, + -0.054824892, 0.04801843, -0.011992075, -0.07789, 0.03106911, + -0.0056367763, 0.003737428, 0.008544284, 0.016392326, -0.013150621, + -0.05674095, 0.013702045, 0.00797615, -0.027977794, -0.032316778, + 0.057698984, -0.02484192, -0.007903741, 0.015712794, -0.052897695, + 0.008327057, -0.0178795, -0.014743623, -0.004701027, -0.013757746, + 0.06597034, 0.03799811, 0.060962964, 0.019004626, -0.024385184, + 0.006087941, -0.01336228, -0.033982188, 0.03980277, 0.03016121, + -0.0328125, 0.032311205, 0.013835724, 0.012593628, -0.016564993, + -0.015412017, -0.0017601001, -0.06927331, -0.0041050436, + 0.065608285, 0.00060155324, -0.010120575, 0.009658271, -0.029370278, + 0.0031470144, -0.00444481, -0.025866788, 0.017378204, 0.008087549, + -0.0195839, -0.002049737, 0.016208518, -0.012103474, -0.017606571, + 0.06707318, -0.012576918, 0.020202164, 0.021717185, 0.0134959575, + 0.03835459, 0.023315758, -0.006750764, 0.009847648, -0.00021722756, + 0.010794538, -0.012844276, 0.012604767, 0.00971954, -0.05551557, + -0.0033642421, -0.07514402, 0.03486781, 0.013401269, 0.08324828, + -0.042598877, -0.10107208, 0.04008684, -0.0031637242, 0.0026234405, + -0.009212676, -0.031731933, -0.028601626, -0.0345169, 0.036728166, + -0.00064054277, 0.014816034, -0.045924135, 0.0036037494, + -0.011674589, 0.004272142, -0.036165603, 0.041791238, -0.032366905, + 0.023571974, -0.022095941, -0.027716009, 0.03546936, -0.0002840668, + 0.0011418371, 0.0044949395, -0.0117804175, + ], + }, + { + title: "Food", + query: "photo of food", + minScore: 0.2, + showVideo: true, + textEmbedding: [ + -0.012083651, 0.016745469, -0.007044219, -0.033847954, -0.020761909, + 0.01960847, 0.0015722477, -0.0116099175, -0.018303985, -0.039670076, + 0.027504038, 0.026577167, 0.017823385, -0.006968696, -0.00054925686, + 0.5646292, 0.018441299, -0.0029522558, -0.016690543, -0.043597262, + 0.015530238, -0.021654453, 0.007071682, -0.021935945, -0.016676812, + -0.0141914245, -0.040198736, 0.023013864, 0.011094988, -0.006103617, + -0.009577666, -0.0047854003, 0.016072629, -0.014088438, -0.01623054, + 0.03081331, 0.0015035907, -0.013195896, 0.033078995, -0.003844798, + 0.046000265, -0.0061997366, -0.013662764, -0.0039065895, + -0.024757752, -0.052172538, -0.02633, 0.00061791396, -0.0044009206, + -0.0244282, 0.012056189, 0.0056161513, 0.016347257, -0.006831382, + -0.030909428, 0.01824906, -0.046158176, -0.015420386, -0.010133789, + 0.042663526, -0.0068451134, -0.008781244, 0.0029385241, -0.0399859, + -0.054431356, 0.04163367, -0.00893229, -0.0073463107, 0.04817669, + 0.049240876, 0.032866158, -0.008389899, -0.02981778, -0.019670261, + 0.017260397, 0.002279416, -0.07170548, 0.017596817, 0.012275891, + 0.0028904642, -0.009742443, -0.009357964, 0.011129317, 0.026790004, + -0.006069288, 0.013786348, 0.0063713794, -0.0044215177, + 0.0052110744, -0.028774194, 0.054905087, 0.029206732, -0.006851979, + 0.0040233065, -0.010978271, -0.0033436012, -0.052337315, + 0.028478969, -0.029364644, -0.017280994, -0.020720715, -0.00521794, + 0.03899037, 0.035736024, -0.0045245034, 0.0014417992, -0.07982762, + -0.011671708, 0.023109982, 0.0014143364, 0.0009886624, -0.019471155, + 0.12529236, -0.04319905, 0.0256091, -0.03793305, 0.013271418, + -0.0105457315, -0.03239929, 0.012028726, -0.03534468, -0.009200052, + 0.0021215046, -0.0411668, -0.025355069, 0.011836486, 0.034452137, + 0.0034191238, -0.022368485, 0.006790188, -0.001139708, -0.021702511, + 0.0074630277, 0.0008650796, -0.013031119, -0.017157411, 0.02173684, + 0.023295356, -0.005286597, 0.014143364, -0.013978587, 0.0003844798, + 0.009941549, -0.015001577, -0.0030209126, -0.003130764, + -0.0036182296, 0.01395799, 0.00801915, -0.0167592, -0.023432672, + 0.024853874, 0.03587334, -0.027600158, 0.002162699, 0.013443061, + 0.02809449, 0.009893489, 0.006618545, -0.010943944, 0.026549704, + -0.016608154, 0.025739549, 0.021427885, 0.043020543, -0.054191053, + -0.030023754, -0.5698471, -0.008657661, -0.0045931605, 0.016052032, + 0.08418048, 0.00749049, -0.01946429, -0.017603682, 0.0021695646, + 0.009763041, 0.01592845, -0.009701249, -0.0036250954, 0.0007964224, + 0.011637379, 0.005471972, 0.00083761674, -0.011760962, 0.019951755, + -0.027016573, -0.030566145, 0.045526527, -0.0017988163, + -0.005135552, -0.050092224, -0.02023325, 0.0066460078, -0.015180087, + -0.03463751, -0.018812047, -0.00787497, -0.0068725767, 0.040164407, + 0.031596, 0.00275315, 0.0007620939, 0.009783638, 0.003412258, + 0.020013547, 0.050600287, -0.0040988293, 0.021448482, -0.03798111, + 0.0060280943, -0.0066666054, -0.0037418124, -0.0048334603, + -0.0017095619, 0.027050901, -0.06111169, 0.0013319479, -0.013855004, + 0.021805497, -0.007497356, 0.061125424, -0.012399473, 0.014417993, + 0.0022382217, -0.009632592, 0.013930527, -0.04257427, -0.007085413, + -0.06352842, -0.020864895, 0.030854503, -0.014589635, -0.10236088, + -0.014170826, -0.01000334, -0.02037743, 0.004867789, 0.007984822, + -0.0056916745, -0.016360989, -0.0009886624, -0.0090078125, + 0.026247611, 0.016601289, 0.0041949493, 0.028760463, -0.009344232, + -0.059395265, -0.0018949362, -0.008369301, -0.004908983, + -0.017562488, 0.07001652, -0.01661502, -0.025162831, -0.004016441, + 0.02028131, 0.0069549647, 0.009714981, -0.0013250822, -0.015646955, + -0.009522741, -0.000116717085, 0.0065155593, 0.0032406154, + 0.007222728, -0.05040118, -0.0046412204, -0.017795922, -0.009234381, + 0.0075866105, 0.024174167, 0.02633, 0.0009955281, -0.00046000263, + 0.008204524, 0.016944574, 0.0029453898, -0.02254013, 0.029440168, + 0.014301275, 0.025444325, -0.020356832, 0.029755991, 0.03331243, + -0.031190922, -0.010875286, 0.020658923, 0.002300013, -0.03776141, + -0.00080328813, -0.040535156, -0.03139003, -0.0040027094, + 0.0072295936, -0.006577351, 0.0057260026, 0.029913902, 0.0082731815, + 0.025739549, 0.0049295803, 0.03047689, -0.016381586, 0.009817966, + -0.0132920155, 0.016827857, -0.0010710509, 0.02703717, + -0.0020459818, -0.013518585, -0.010037669, 0.022581322, + -0.0118227545, -1.3731422e-5, -0.02024698, -0.070950255, + 0.0020253847, -0.031781375, 0.0048334603, -0.026240747, 0.032859292, + -0.043803234, 0.045354884, -0.0061173486, 0.0026295672, + 0.0123926075, -0.036422595, 0.023878943, -0.0005904511, 0.011802156, + -0.003844798, -0.019581007, 0.07515894, -0.020713849, + -0.00033641982, -0.00565048, 0.015386058, -0.031753913, 0.02733926, + -0.010477074, 0.018880704, 0.044565327, 0.010971406, 0.026288806, + -0.022066394, 0.023652375, -0.016409049, -0.080521055, 0.020775642, + 0.005643614, 0.011156781, -0.022306696, 0.017583085, -0.027085228, + 0.009426621, 0.017823385, -0.0046343547, 0.011033197, -0.05370359, + 0.033641983, -0.00083761674, -0.021908483, -0.004085098, + 0.014163961, 0.0060418253, -0.008657661, 0.011115585, 0.005266, + -0.004270472, 0.002732553, -0.033113323, 0.007003025, 0.0036182296, + 0.035811547, -0.041606206, 0.008300644, -0.009584532, -0.115179166, + -0.017054426, -0.009872892, -0.012557385, 0.0027600157, + -0.024084914, 0.022045797, 0.019629067, -0.014562172, 0.0021901617, + 0.023103116, 0.034761094, -0.023830881, 0.04333637, 0.0147612775, + -0.0057809283, 0.012447534, -0.022203708, 0.013374405, 0.011136183, + -0.00029522556, 0.048327737, 0.037747677, 0.0027874787, 0.008410496, + 0.0029522558, 0.009344232, 0.007284519, 0.023597447, 0.042491883, + 0.0014761279, -0.010133789, -0.013182165, -0.01261231, -0.011335288, + 0.003748678, -0.02024698, -0.023995658, -0.030167934, -0.025217757, + -0.023501327, 0.0018880705, 0.023782821, -0.0058701825, 0.010662449, + -0.012866342, 0.013731421, 0.012481863, -0.008554676, -0.0046137576, + 0.013024253, 0.019443693, -0.007318848, 0.011706037, -0.006440037, + -0.034040194, -0.008589004, 0.029460765, 0.011383349, 0.035179902, + 0.009103932, -0.01989683, -0.04038411, -0.028039563, -0.017514428, + 0.004936446, -0.0047579375, 0.0049433117, 0.019965487, 0.014012916, + 0.013868735, 0.01487113, -0.07837895, 0.017610548, -0.0445104, + 0.020658923, -0.0017919504, 0.0121797705, -0.02048728, + -0.0024304616, -0.008197659, -0.02671448, -0.010683046, + -0.009783638, -0.019972352, -0.009591398, 0.009790503, 0.058392867, + 0.0005767197, -0.0031925554, -0.019574141, -0.010133789, 0.01763801, + 0.055049267, 0.0038173355, -0.024078049, 0.014315006, 0.013092911, + -0.024325214, -0.010751703, 0.02333655, -0.011912008, -0.009145127, + -0.006611679, 0.00260897, -0.05136238, -0.13511032, -0.01999295, + -0.1386805, 0.017802788, 0.004442115, 0.0072089965, -0.0012701565, + 0.025018651, -0.009996475, 0.0027668814, -0.00061791396, + -0.007044219, -0.010696777, -0.013312614, 0.029604943, -0.014424859, + 0.0148779955, -0.041283518, 0.02476462, -0.005197343, 0.00067283964, + -0.010030803, -0.024922531, -0.00048059976, -0.029584348, + 0.014397396, 0.007689596, -0.040693067, + ], + }, + { + title: "Celebrations", + query: "celebration with friends", + minScore: 0.2, + showVideo: true, + textEmbedding: [ + 0.026342282, 0.01797094, -0.003183855, -0.0350773, 0.01633098, + -0.010189708, -0.041939057, -0.044670034, -0.029567307, + -0.034555804, -0.013538245, -0.00022643796, -0.00014409689, + 0.016029064, -0.008419375, 0.4912194, 0.031111203, 0.007348941, + 0.039407067, -0.010114229, 0.025381636, -0.0009126136, -0.007877297, + 0.0039386484, -0.0057089813, 0.001605651, -0.0016674068, + 0.017922906, -0.014828255, -0.018464986, 0.006614733, -0.05048194, + -0.026932394, -0.025264986, 0.020564685, -0.05590959, 0.017476894, + -0.032106157, 0.081654906, 0.0328335, 0.032806057, -0.01785429, + -0.037245613, 0.016962262, -0.023384865, -0.027028458, -0.02778325, + -0.018533604, 0.0028064582, 0.02000202, 0.0020173562, -0.002312412, + 0.016193746, 0.104593754, 0.01206983, -0.0011870839, 0.024791526, + 0.017168114, 0.0046042386, -0.05118184, -0.03469304, 0.0052217967, + 0.009215339, -0.036737844, -0.10656994, 0.012268821, 0.015946722, + 0.009229062, -0.017607268, 0.014917458, 0.0107729575, -0.018101314, + -0.019288396, 0.04593946, 0.013133402, 0.008151767, -0.029025229, + -0.010169122, 0.034761656, 0.075623415, -0.0019556005, 0.015946722, + 0.009078104, 0.038041577, -0.0042611505, 0.0004871847, 0.022925127, + 0.030610293, -0.029450657, 0.028538045, -0.034500908, 0.016454492, + 0.006504945, -0.006511807, 0.023062363, 0.03228456, -0.109788105, + -0.0023673058, -0.013895056, -0.062325332, 0.020530375, 0.000535217, + -0.007842988, -0.003801413, 0.006786277, 0.025621798, -0.042364486, + -0.012241374, 0.0052355197, 0.0058256313, -0.023179013, 0.010498487, + 0.11106439, -0.041486178, -0.033855908, 0.005715843, 0.0059560044, + -0.04880081, -0.027344098, 0.042289004, 0.027186278, -0.027028458, + 0.052567914, -0.034500908, 0.022726137, -0.02632856, 0.03277861, + -0.07358547, -0.008240969, -0.035084162, -0.024647428, -0.003609284, + -0.005180626, 0.027947934, -0.024503332, -0.0032318872, + -0.027323514, -0.012543291, -0.017819982, -0.019665794, 0.004274874, + -0.025964886, -0.016674068, -0.047373563, -0.0012694249, 0.0296222, + -0.01195318, 0.004590515, -0.017785672, -0.0040415744, 0.012783452, + -0.010697478, -0.088667616, 0.01750434, 0.020448033, -0.015493846, + -0.0060726544, 0.013414733, -0.035845812, 0.023858327, -0.029066399, + -0.010539657, 0.022568317, 0.020667609, 0.021758629, -0.040388297, + 0.018005248, -0.49567956, -0.019370737, -0.028901717, 0.008920283, + 0.05122987, -0.04781272, 0.058935624, 0.045706157, 0.008879113, + 0.024777802, 0.0050159437, 0.000102926344, -0.07427851, + -0.021093039, -0.014004845, 0.014327347, -0.022925127, 0.010011302, + -0.030335825, 0.018355198, 0.009002624, 0.016159438, -0.030877903, + 0.031934615, 0.0019693242, -0.044217158, 0.012255097, -0.009606459, + -0.015384058, 0.0067725535, 0.00030877904, -0.032092433, + 0.036586884, -0.014533199, 0.024510194, -0.011829668, -0.0058462163, + -0.04714713, 0.016694652, -0.04055298, 0.033505954, -0.004020989, + -0.0053658937, -0.015363472, 0.00406216, 0.03762301, -0.063533, + 0.0048238146, -0.022355603, -0.0031975785, -0.015987892, + -0.009771141, -0.0056746723, -0.04106075, 0.016118266, 0.025765896, + 0.035084162, 0.01633098, 0.03136509, -0.05513421, 0.039324727, + -0.012900102, 0.0027858731, -0.010663169, 0.018663976, -0.005633502, + 0.004638547, -0.020331383, -0.032476693, 0.009359436, -0.032408074, + -0.01578204, 0.005715843, -0.032593343, 0.028757619, -0.049871244, + 0.046824623, -0.07769567, 0.029601617, 0.008899698, 0.040038347, + -0.05852392, 0.04053239, -0.04418971, -0.02055096, -0.021971343, + 0.029423213, 0.007040162, 0.037986685, 0.014601817, -0.013133402, + -0.03594188, 0.03199637, -0.033931382, -0.028421395, -0.04235076, + 0.008227246, 0.0550107, -0.031241575, -0.0038219984, -0.047455907, + 0.035056714, 0.01930212, -0.02812634, 0.0036024218, 0.055353787, + -0.010670031, 0.040882345, 0.018787488, 0.029594755, -0.0191443, + -0.034713626, -0.019583452, 0.004508174, 0.019748135, 0.043084968, + -0.0004871847, 0.002332997, 0.037163273, -0.053816754, 0.0052149347, + 0.031145513, 0.039805047, -0.027851868, -0.017030878, -0.063210495, + -0.012364885, 0.02750192, -0.019164884, 0.013421595, 0.027460746, + -0.0044326945, -0.030376995, 0.018547326, -0.022787893, 0.00906438, + 0.004199395, 0.02437982, 0.02605409, 0.023233907, 0.006882341, + 0.031166095, -0.02895661, -0.0039386484, 0.013565692, -0.018190516, + 0.016042786, -0.031474873, -0.020873463, -0.03955116, 0.029196773, + 0.044409286, 0.016831888, 0.0009194754, -0.0063814335, -0.06200283, + 0.041740064, -0.04781958, -0.01933643, 0.011781636, 0.013764682, + -0.014903734, -0.028146923, 0.06463775, -0.0031358227, 0.038501315, + 0.059381638, -0.025999194, -0.023508377, 0.049452677, -0.029018367, + 0.018471848, -0.00203108, -0.02977316, -0.030829871, 0.07470394, + 0.01781312, 0.014862564, 0.026534412, -0.040470637, 0.025985472, + -0.06216065, -0.026589306, -0.02934087, 0.019357014, -0.022863371, + -0.025930578, -0.084303536, 0.038309183, -0.09027327, 0.0066764886, + 0.016749546, -0.033396166, 0.01418325, 0.0047414736, -0.044553384, + -0.031769935, 0.043942686, -0.013798991, 0.02378971, 0.031241575, + 0.012913825, 0.029231083, 0.025223818, -0.022609487, 0.006312816, + 0.049795765, 0.0027035319, 0.020221597, 0.026616752, -0.03675157, + -0.073194355, -0.009407467, 0.0020653887, -0.006779415, + -0.0150684165, -0.061556816, -0.011507166, -0.023096673, + -0.025367914, 0.0039592334, -0.0063265394, 0.025539456, 0.027097076, + -0.09998951, 0.017689608, 0.03093966, 0.005969728, -0.0029162464, + 0.016619174, 0.008405651, 0.028421395, -0.1283766, 0.04167831, + -0.018238548, -0.0108552985, -0.03868658, -0.044937644, 0.022046823, + 0.009098689, -0.0033553988, -0.02437982, -0.018705148, + -0.0015576186, 0.018684562, -0.023796571, 0.01890414, 0.043544706, + -0.030075077, -0.030720085, -0.022142887, -0.0152468225, + 0.006902927, 0.0066559035, 0.010361252, 0.017133806, -0.011136631, + 0.011479719, 0.015617357, 0.027323514, 0.010656307, -0.017964078, + -0.012687388, -0.03907084, 0.004624824, 0.023062363, 0.0024222, + -0.045157216, 0.0034171545, -0.0027035319, -0.041520488, + -0.02848315, -0.030322101, 0.010368113, -0.034720488, -0.014725328, + 0.034816552, 0.047256913, 0.03532432, -0.009640767, -0.030150557, + 0.017600406, -0.019631485, -0.00691665, 0.006079516, 0.05363149, + 0.01277659, 0.008954592, 0.007822402, 0.02562866, -0.017524926, + -0.0154183665, -0.018389506, -0.025361052, -0.043819174, + 0.0038631689, -0.00929768, -0.026472656, 0.007486176, -0.045610093, + 0.02891544, -0.046941273, -0.0001578204, 0.033341274, -0.047188297, + -0.016186884, -0.007897881, 0.014766499, 0.014979214, -0.015672252, + 0.066078715, 0.031022001, 0.009798588, -0.028297883, -0.06480929, + -0.007122503, -0.018272856, 0.003924925, -0.02610212, -0.102102935, + 0.075760655, 0.012275682, 0.006779415, 0.008590919, 0.0196109, + 0.02055096, 0.0050433907, -0.007211706, 0.0054207873, 0.021168519, + -0.016536832, -0.02660303, 0.040422607, -0.006902927, -0.0059354194, + -0.06325853, -0.0034994958, -0.019528558, -0.021223411, + -0.003520081, 0.017998386, 0.06544057, -0.04656388, -0.021017559, + -0.10281656, + ], + }, + { + title: "Sunset", + query: "photo of setting sun", + minScore: 0.2, + showVideo: true, + textEmbedding: [ + 0.0338652, 0.003039766, 0.02503756, -0.016871026, 0.031680975, + -0.0032925399, -0.04358078, 0.026450502, 0.039121594, 0.020092271, + 0.01947654, -0.027597705, 0.010940565, -0.031583752, 0.02112281, + 0.522237, 0.003124024, -0.024194982, 0.032102264, -0.012509059, + 0.005405469, 0.014570137, -0.02804492, 0.035323508, 0.0032990212, + -0.028531026, 0.02337833, 0.008775786, -0.027306044, -0.053276923, + 0.032380965, -0.0024434794, -0.038914192, -0.013954407, + -0.039717883, 0.027856963, -0.0070063695, 0.007142478, -0.004958254, + 0.014233105, 0.033670757, 0.018335817, -0.02804492, -0.015600677, + 0.0031499495, -0.024065355, 0.0024629235, 0.016721955, + -0.0022879262, 0.0064878594, 0.006604524, 0.04352245, -0.050684374, + 0.0024629235, -0.025199596, 0.02790233, 0.012787758, -0.005509171, + -0.014719209, 0.02385147, -0.004932328, 0.011847959, -0.0054702824, + -0.042926166, 0.018873772, 0.036341086, 0.047871456, -0.013157196, + 0.013228492, 0.0053989873, 0.01867933, 0.028414361, -0.037818838, + -0.046212222, 0.016611772, 0.010324835, -0.039542884, 0.03268559, + 0.025588479, -0.035498504, -0.03513555, -0.0043165977, 0.0059499047, + 0.005424913, -0.019178396, -0.06280455, -0.010506313, -0.039763253, + -0.0074341404, 0.0057295375, 0.042854868, 0.018880254, -0.007310994, + 0.0327504, -0.023067223, -0.015600677, 0.014894205, -0.015983077, + 0.027131047, 0.0049647354, -0.011958141, -0.013183122, 0.006960999, + -0.058941647, 0.038071614, 0.015516418, -0.06952573, 0.054495424, + -0.003778643, -0.015179386, 0.0267292, -0.01641733, 0.04549927, + 0.027228266, -0.013422933, 0.0068767415, 0.0048351074, 0.0075637675, + -0.022503344, 0.0068184095, 0.015224756, -0.032717995, -0.027351415, + -0.01595715, 0.0047314055, 0.02521256, 0.033087432, -0.016683066, + -0.00031110612, 0.020785777, 0.059492562, -0.020403376, + -0.038486417, 0.033625387, -0.042226177, -0.024032949, -0.044416882, + -0.0011796107, 0.020766335, -0.0061897156, 0.023592213, + -0.0055804662, 0.0041416003, -0.031830046, 0.060205515, -0.05368525, + -0.0046277037, -0.0109146405, -0.040631756, 0.014090515, + -0.03293836, -0.016533993, 0.0020481155, -0.0076156184, + -0.0033249469, -0.011050749, 0.02821992, -0.052797303, -0.026411613, + -0.022030203, -0.0120423995, -0.024305167, 0.032316152, + 0.0032147632, -0.018808957, -0.057327785, -0.016339552, -0.5266703, + 0.0148164295, 0.013397008, -0.013947925, 0.068773896, 0.02880324, + -0.056407433, -0.049057547, 0.028336585, -0.012722944, -0.026469946, + -0.0066823, -0.025847733, 0.00931374, -0.0016981208, 0.042621538, + 0.05301767, -0.023339441, 0.0072785867, -0.011958141, -0.05072326, + 0.02134966, 0.027980108, 0.005800833, -0.014284956, 0.020882998, + -0.037818838, -0.0012573872, 0.0034740183, -0.018925622, + -0.08952727, -0.020545967, -0.012411838, 0.029768968, -0.03866142, + 0.033281874, 0.007498954, 0.0067471145, 0.04041139, 0.040566944, + -0.0043360414, -0.026366243, -0.011614629, 0.025387555, 0.018510815, + -0.029328233, 0.007356364, 0.015775673, -0.018329335, 0.0134294145, + -0.020818185, 0.0025536628, -0.023352403, -0.042271543, 0.034850366, + 0.031402275, -0.008944301, 0.007712839, -0.0109081585, -0.032030966, + 0.013072939, 0.039322518, 0.003311984, 0.01735713, 0.036768854, + -0.031784676, -0.0821385, 0.0012833127, 0.022898708, -0.00058332394, + 0.040949345, 0.06251288, -0.0040573427, 0.030339329, 0.016734917, + -0.02187465, 0.028893983, -0.0639712, -0.004932328, -0.05363988, + -0.013202567, -0.08604029, -0.013422933, 0.020857073, -0.02982082, + -0.002780511, 0.0313245, -0.031972636, 0.017823787, 0.018335817, + -0.013649781, 0.0057619447, 0.008976708, -0.01735065, -0.025076449, + 0.0066369306, -0.022282977, 0.03600405, 0.05933053, -0.0054638013, + -0.0038953077, 0.0067730392, 0.030352293, 0.04864922, -0.029866187, + 0.008555418, -0.015542344, 0.00014259032, 0.03542721, 0.029269902, + -0.020740408, -0.005321211, -0.028531026, 0.0011472038, -0.03497999, + 0.00058980536, 0.0020481155, 0.022483898, 0.034467965, -0.018620998, + 0.041565076, 0.000978688, 0.037241995, -0.030015262, 0.013584967, + 0.021414472, 0.019048769, -0.008043389, 0.043652076, -0.016864544, + 0.106294595, -0.013986813, 0.020708002, -0.014615507, -0.028686577, + -0.042880796, -0.024966266, -0.013085901, 0.022010759, 0.017551571, + 0.0266255, -0.042407654, 0.015451604, 0.072863646, -0.014388657, + 0.0135460785, 0.0051267697, 0.020345045, 0.0078035784, -0.05250564, + 0.034111492, -0.07254606, 0.013338676, -0.0032666142, -0.016871026, + 0.012483133, 0.039212335, -0.08336996, 0.01086927, 0.029982852, + 0.0541195, -0.0033573536, 0.061786972, -0.008354496, 0.016948802, + 0.0044915946, 0.07134052, -0.002061078, 0.056990754, -0.01693584, + 0.020526523, 0.079727426, 0.026269022, -0.0011536853, 0.02305426, + 0.015419197, -0.003124024, -0.0075767306, 0.02381258, 0.01952839, + 0.020008013, -0.023397774, 0.02135614, 0.010947047, 0.026094027, + 0.013850704, -0.010616496, 0.006202678, 0.006591561, -0.02983378, + -0.024188502, -0.005885091, -0.006131383, -0.009475774, 0.018549703, + -0.004977698, -0.03251707, 0.035978127, 0.028654171, 0.041053046, + 0.038123462, 0.02432461, -0.015730303, -1.9444133e-5, 0.009799843, + -0.0071554407, -0.015892338, 0.041416004, 0.004925847, 0.017344166, + -0.01985246, -0.07217662, 0.0030138404, 0.039400294, 0.0063063805, + -0.03166153, -0.0071943286, -0.0023916285, 0.031830046, + 0.0074082147, -0.01495902, 0.004809182, -0.012794239, 0.093824424, + -0.07693396, 0.043347456, -0.05726945, -0.020014493, -0.029243976, + 0.02413017, -0.02530978, 0.030462474, -0.01547753, 0.04019102, + -0.010979454, -0.014952539, -0.013844223, -0.005710094, 0.01820619, + -0.04529835, 0.00936559, -0.02991804, 0.00564528, -0.040385462, + -0.031253204, 0.0035064255, -0.03284762, -0.0063323057, 0.019521909, + 0.051118627, 0.007129515, -0.009825769, 0.013254417, 0.0068637785, + -0.011685924, -0.02597736, -0.0013481266, 0.02465516, 0.008367458, + -0.01900988, 0.035731833, 0.005431394, 0.02028023, -0.00093331834, + 0.0024953305, -0.030637473, -0.016812693, -0.07578675, 0.046607587, + 0.015885856, 0.037831802, -0.018964512, -0.033599462, -0.005632317, + -0.002262001, -0.036548488, -0.05151399, -0.018867292, 0.008717453, + -0.0008879488, 0.06401008, -0.00020740408, -0.013260898, + -0.00014907168, 0.027215306, -0.009702622, 0.015399754, 0.011109081, + -0.017804345, -0.0013934962, 0.046918694, 0.039154, 0.014498841, + -0.034610555, 0.0033573536, -0.043891888, 0.017700642, 0.012729425, + -0.0057943515, -0.04294561, 0.0020999664, -0.019599685, -0.01068131, + -0.022976482, 0.013643299, 0.006060088, 0.0070452574, 0.00073887705, + 0.027824553, -0.033100393, 0.034429077, 0.069435, 0.002061078, + -0.010979454, -0.028621763, -0.003227726, -0.014790503, 0.010571127, + -0.052797303, -0.12194064, 0.024557939, -0.028381951, 0.0019508946, + -0.042362284, 0.047404796, -0.020980218, -0.0071619223, 0.010733161, + 0.00837394, -0.033787422, -0.014647913, -0.017940454, -0.038486417, + -0.016916396, 0.0007129515, -0.008944301, -0.022004277, + -0.010849826, -0.05809259, -0.004277709, -0.012321099, -0.06377028, + 0.03492166, 0.021304287, 0.040677123, + ], + }, + { + title: "Hills", + query: "photo of mountains or hills", + minScore: 0.2, + showVideo: true, + textEmbedding: [ + 0.020085756, 0.022995783, 0.01930537, 0.011414018, 0.013222759, + -0.004777113, 0.0046385406, 0.001684753, 0.053211935, 0.024593016, + 0.0017430993, -0.03710103, 0.030733977, -0.022033067, 0.017905058, + 0.55763865, -0.01830619, 7.2933026e-6, 0.015527441, -0.013266518, + -0.02212788, 0.0019473118, -0.03455567, -0.009794906, 0.0030996536, + -0.022310212, -0.022601943, 0.0072568366, -0.016847529, 0.0481358, + 0.04608638, -0.0040915427, -0.036094554, 0.025074374, -0.01258824, + 0.033009488, -0.011523418, -0.0044051544, -0.008576924, + -0.022405026, 0.039989177, 0.018824015, -0.015753534, -0.034015965, + -0.0068848776, -0.012814333, -0.035693422, -0.024636777, + 0.029836902, -0.025563026, 0.0007147436, 0.020836966, -0.03551838, + 0.03173316, -0.005083432, -0.00560855, -0.012960199, -0.008905123, + 0.0091312155, 0.037925176, 0.00010210624, 0.014484499, 0.0044416212, + -0.013259225, 0.027882297, 0.040229857, -0.03148519, -0.015410749, + 0.048332717, 0.018036338, 0.019932596, 0.06964374, -0.0011231686, + -0.020501474, 0.013499903, 0.00014586606, -0.06148254, 0.015425335, + 0.041608293, 0.039551582, 0.0029100277, 0.0031288266, 0.02645281, + -0.00644728, -0.011166046, -0.049368367, -0.013084185, -0.01258824, + -0.016191132, 0.008350831, 0.046932403, 0.03791788, -0.008146619, + 0.0033913855, -0.0022755102, 0.022003895, 0.018539574, 0.052592, + -0.020012822, 0.001976485, -0.0065420927, -0.002625589, 0.0251546, + -0.06844765, 0.018597921, 0.012004776, -0.07617125, 0.026715368, + 0.025511973, -0.005061552, 0.001662873, -0.045335166, 0.012391321, + -0.0109253675, 0.021755923, -0.015717067, 0.007774661, 0.0122746285, + -0.011311912, -0.0040186094, -0.00472606, -0.04054347, 0.024461737, + -0.0031652933, -0.013069599, 0.00013127945, 0.102376096, + -0.055509325, 0.009174975, -0.0010575289, 0.0133759165, + -0.021894494, -0.010801381, -0.0005105312, 0.013178998, + -0.0062211873, -0.03951511, 0.009488587, -0.050068524, -0.008810309, + 0.009306254, 0.0073881154, 0.0036758247, 0.009145801, 0.009109335, + -0.055655196, 0.002049418, -0.018933414, -0.01413442, + -0.00037195845, -0.04357019, 0.00238491, 0.022536304, 0.02457843, + 0.009853251, -0.04352643, 0.014520965, 0.016526623, -0.023484435, + -0.009226028, -0.003515372, 0.031507067, 0.028604334, -0.02230292, + -0.039471354, -0.042738754, -0.037881415, -0.5623866, 0.022383146, + -0.026489275, -0.005688776, 0.08086814, -0.0054408037, -0.021668402, + -0.005645016, -0.015177363, -0.013893741, -0.01534511, -0.030551644, + -0.040069405, -0.0073224762, -0.0053168177, -0.034774467, + 0.021063058, -0.026007917, 0.02034102, -0.017664379, -0.006162841, + 0.02097554, 0.016417224, -0.038238786, -0.006316, -0.02627777, + -0.0813495, -0.005783589, -0.02592769, 0.0005032379, -0.0567346, + 0.012500721, 0.017153848, -0.01035649, -0.0148054045, 0.039289024, + 0.0024870161, 0.009182268, 0.019385599, 0.01903552, 0.011187926, + -0.032710463, 0.008204965, 0.028808545, -0.0016993396, 0.030281793, + -0.008540457, -0.016657902, -0.012092296, 0.020742152, -0.014499086, + -0.0016993396, -0.019378304, 0.025234828, 0.014338633, 0.034701534, + 0.0016920463, -0.010283557, 0.029253436, 0.03559861, -0.015016911, + -0.042585593, -0.10697816, -0.04332951, 0.00068557047, 0.018845895, + -0.070176154, -0.0068994644, -0.09547663, -0.0364811, -0.011516125, + 0.035861168, 0.0062576532, 0.021836147, 0.010261677, -0.05277434, + -0.012019362, -0.01183703, 0.046830297, -0.03769179, -0.03413995, + -0.09999117, 0.012777866, -0.0005178245, -0.0024943093, + -0.034417093, 0.007789247, -0.02926073, -0.0102325035, 0.018656267, + 0.0035883049, 0.022740517, 0.008234139, -0.026423635, -0.021427725, + 0.009692799, 0.029975474, -0.016074438, 0.017365353, -0.019123038, + -0.006439986, 0.015403455, -0.013222759, 0.011114993, 0.027437406, + 0.012427788, 0.016067147, 0.0063451733, 0.063641354, 0.03654674, + 0.01879484, 0.03968286, -0.026263181, 0.010268969, -0.018029043, + 0.021901788, -0.028093802, 0.010575289, 0.0008970762, -0.0022098708, + 0.015257589, 0.026372582, 0.00781842, 0.0033840926, -0.012216282, + 0.010983714, -0.018670855, -0.012697641, -0.019159507, 0.0055574966, + 0.041972958, 0.0055210297, 0.002961081, -0.02345526, 0.019961769, + 0.0208005, 0.0055210297, 0.011676578, 0.009401067, -0.01316441, + -0.018225962, -0.00242867, -0.009153095, 0.10668643, -0.0030923602, + -0.00940836, 0.0028298015, 0.033133473, 0.038246077, -0.0537954, + -0.0039237966, -0.016687077, 0.017037155, -0.0012106881, + -0.018780256, -0.0005907575, 0.046757363, -0.05332863, 0.021194337, + -0.044284936, 0.040047526, 0.015476389, -0.013930209, -0.019086573, + 0.024877455, -0.023156237, 0.047800303, -0.020567114, 0.018138442, + -0.035671543, -0.015841054, 0.014878337, 0.007249543, 0.013966675, + 0.0070161573, -0.009379188, 0.0036831177, -0.006782771, 0.01821867, + 0.02239044, 0.030945482, -0.020902606, -0.0072203698, 0.0013638476, + -0.024024138, 0.0011596351, -0.035277706, 0.014623072, -0.02981502, + -0.025511973, 0.05313171, 0.0173143, 0.028451173, 0.050491538, + -0.004711474, -0.01806551, -0.031040296, -0.03389927, -0.019801317, + -0.010743035, -0.00065639726, -0.021653816, 0.030879842, + -0.017868591, 0.0014149008, -0.011078526, -0.016008798, 0.03063187, + -0.012325682, 0.015666014, 0.008832189, -0.090925604, 0.0021734042, + -0.015126309, 0.011705751, -0.017321594, 0.0017285127, + 0.00019691917, -0.014396979, 0.0023703235, -0.026576793, + -0.0042228224, 0.0009189562, -0.07509914, 0.008380005, -0.0139375, + 0.0012909146, 0.0020567114, -0.01942936, -0.010268969, + -0.0022025772, 0.0010648222, -0.013499903, 0.0117422175, + -0.011034767, 7.2933026e-6, 0.026343409, 0.026846647, 0.020027408, + -0.049470473, 0.019990943, -0.0018597922, 0.008671737, -0.04440892, + -0.064574905, -0.013543664, -0.020691099, -0.004244702, + -0.012318389, 0.015841054, 0.0017868591, -0.0004959446, 0.011844324, + 0.013507197, -0.0019400185, -0.07307889, 0.020253502, -0.03182797, + -0.030515177, -0.008066393, 0.01636617, -0.028451173, 0.038895182, + 0.020122223, -0.0075996215, -0.010808675, 0.012566361, + -0.0017868591, 0.017438287, -0.020836966, 0.033439793, -0.009860545, + -0.02696334, -0.0075266883, 0.004361395, 0.021799682, 0.013354036, + 0.036772832, 0.006323293, 0.020370195, -0.07023451, 0.027065445, + -0.006264947, -0.049601752, 0.0049959123, -0.013076892, -0.02799899, + -0.004959446, -0.0074756355, 0.009889718, 0.07115346, 0.010152278, + -0.013798929, -0.004594781, -0.009379188, 0.012296508, 0.014010434, + 0.007125557, 0.059010115, 0.011567178, 0.0019473118, 0.0128435055, + -0.013003958, -0.0022828039, 0.05829537, -0.042366795, 0.007935113, + -0.053824574, 0.021901788, -0.0014076075, -0.019108452, + -0.004047783, 0.010837848, -0.014520965, -0.0011669284, + -0.0054335105, 0.018700028, 0.01035649, 0.016971515, -0.0676235, + -0.013018546, 0.008423764, -0.011465072, -0.015272176, + -0.0016701663, -0.011348379, 0.0039821435, -0.007876767, + -0.009328133, -0.035831995, -0.022930143, -0.0023776167, + -0.0005907575, -0.012704934, 0.008606098, -0.021763215, -0.02034102, + 0.017875886, -0.005995095, 0.016220305, -0.023907445, -0.0023922033, + -0.013550956, 0.016074438, 0.036313355, + ], + }, + { + title: "Greenery", + query: "outdoor photo of trees, leaves, flowers", + minScore: 0.2, + showVideo: true, + textEmbedding: [ + -0.007156132, -0.01483226, 0.0016425262, -0.0041434583, + 0.0028806115, 0.0021790299, -0.0032767986, 0.014353533, 0.02069253, + -0.05503701, -0.0010482455, 0.011753554, 0.06092204, 0.004613931, + 0.012496406, 0.49475536, -0.032041643, -0.0028558497, 0.031637203, + -0.016020821, -0.01912429, -0.027213112, -0.0074285106, + -0.043093614, -0.0060253474, -0.014312264, -0.025884233, + -0.008567549, -0.011307844, 0.018959211, -0.001411417, + -0.0012380852, 0.0058602695, -0.008121839, 0.0022120455, + 0.0027898184, 0.03533495, 0.0071808943, 0.0017993505, 0.019751586, + 0.021129988, -0.012323074, 0.0057529686, -0.044612333, 0.022607435, + -0.0024761704, -0.0050101182, 0.00935167, -0.008303424, + -0.0023110923, 0.035681613, 0.026585815, 0.005719953, -0.0041104425, + -0.019603014, 0.006751691, -0.040006656, -0.0006768199, + -0.024448056, 0.04070824, 0.02441504, -0.0032767986, -0.008749135, + -0.039148252, 0.019685555, 0.007255179, -0.020717291, -0.007445019, + 0.0030126737, 0.017696364, 0.05585415, -0.0048202784, 0.0039536185, + -0.030663243, 0.024134407, 0.0136602055, -0.1504686, 0.010086267, + 0.00266601, -0.04824405, -0.040988874, 0.0027320413, 0.020981416, + 0.0074202567, -0.0038463178, 0.02020555, -0.015789712, 0.0027733108, + -0.021105224, 0.018397946, 0.05896587, 0.012801801, -0.01747351, + 0.020238565, 0.009624049, 0.021633474, -0.14422867, 0.0062151873, + -0.041211728, 0.01747351, -0.012983386, 0.007981523, -0.0022202993, + -0.08149902, -0.012174504, 0.0051586884, -0.04341552, -0.0069580385, + -0.0065453434, -0.0015764951, 0.008798659, -0.040931094, -0.1643847, + 0.013907823, 0.003598701, 0.034435276, -0.008773897, -0.007898984, + 0.020956654, 0.016070344, -0.009739603, -0.0008006284, 0.00897199, + -0.01637574, -0.037992705, 0.018785879, 0.08574152, 0.007015816, + 0.019008733, 0.0040196497, -0.0010647532, 0.03245434, -0.04447202, + -0.016029077, 0.046386924, 0.027097557, -0.0012628469, 0.031397842, + 0.011118005, 0.011959903, 0.012091965, 0.001411417, 0.024084883, + 0.030423878, 0.0011142766, -0.00014031632, -0.0067104213, + -0.0011060227, -0.03654002, 0.049374834, 0.012281804, 0.007197402, + 0.052734174, -0.014749722, 0.0006768199, 0.0041434583, 0.025083605, + -0.029548965, -0.013635444, 0.009491987, 0.07705017, 0.005885031, + 0.0018158582, 0.021534426, -0.07587811, 0.077892065, -0.020898879, + -0.49878326, 0.02576868, -0.025182651, -0.0033015604, 0.06256457, + -0.005340274, 0.030168008, -0.012686246, -0.023655681, -0.022549657, + -0.008584057, 0.02725438, 0.028327389, 0.020552212, 0.040122215, + 0.0161859, 0.011687524, -0.0055053523, 0.002996166, -0.008798659, + 0.007981523, 0.03664732, -0.004878056, 0.0059758243, -0.026263913, + 0.068342306, -0.012248789, -0.01070531, -0.020618245, 0.030729273, + -0.0508853, 0.052527826, 0.017993504, 0.03138133, -0.048929125, + 0.041640934, -0.019099526, -0.022062676, -0.005076149, 0.012554184, + 0.009525002, -0.021237286, 0.017490016, 0.048103735, 0.0060253474, + 0.03127403, 0.0103338845, -0.016986528, 0.05153736, 0.10163853, + -0.03116673, 0.00072634325, -0.005530114, -0.008707866, 0.031389583, + 0.016532563, 0.01961127, -0.02354838, -0.002344108, -0.009624049, + -0.046543747, -0.0030291816, -0.11197242, -0.0019644285, 0.03652351, + -0.033188935, 0.03323846, 0.0069910544, -0.012851324, -0.014658929, + 0.011687524, -0.018934447, 0.015682412, -0.0070570856, -0.017712872, + -0.0020304597, -0.03505432, 0.04561931, -0.0029879122, -0.02274775, + -0.079873, -0.14792642, -0.05039832, 0.0027402951, -0.013354812, + 0.015682412, 0.020527452, -0.0041847276, -0.031554665, 0.011737047, + -0.086418346, 0.0014774483, 0.014386549, 0.026239151, -0.025537571, + 0.009888174, -0.008064061, 0.004498376, -0.021806806, -0.010267853, + 0.0031282285, 0.016053837, -0.01802652, -0.009962458, 0.010573247, + 0.023820758, -0.016062092, -0.0025834711, 0.042441558, 0.011604984, + -0.014898291, 0.028046755, -0.053410996, -0.014039885, -0.025034081, + -0.0015930028, 0.019371906, -0.014782737, 0.008064061, -0.03370068, + -0.008245647, 0.052808456, -0.010573247, -0.03406385, -0.031117208, + 0.109141335, -0.006759945, -0.012091965, -0.0025504555, -0.0355413, + -0.023350285, 0.05615954, 0.0042837746, 0.014328773, -0.029301347, + -0.016425263, -0.0052329735, 0.012892594, 0.007403749, 0.004671708, + 0.008039299, 0.00878215, -0.018084297, 0.066237554, -0.0022037916, + 0.011770062, -0.002344108, 0.017754141, 0.010465947, -0.028005486, + 0.0054640826, 0.0233833, -0.012438629, 0.081771396, 0.022186486, + -0.04242505, 0.03782763, -0.07913015, 0.007081847, 0.028682305, + -0.023507109, 0.034988288, -0.022136962, -0.005579637, 0.004646946, + -0.0242252, 0.015500827, -0.0072799404, 0.016590342, 0.027559776, + 0.016499547, -0.011976411, -0.0848501, -0.018686833, -0.010862134, + 0.002979658, -0.019338889, -0.0014361787, -0.023895044, + -0.027163588, 0.016400501, 0.06977848, -0.008352947, 0.0015682412, + 0.010482455, -0.019107781, 0.053138614, -0.005736461, -0.038199052, + 0.06096331, 0.053221155, -0.008740881, -0.0403038, -0.0022368073, + -0.006900261, -0.040254276, 0.01100245, 0.010688801, -0.0033263222, + -0.0025999788, -0.02467091, -0.023556633, 0.008130093, 0.052882746, + -0.019726822, 0.012752278, 0.007445019, 0.0031199746, 0.023564888, + -0.005356782, 0.035739392, 0.020453166, -0.036168594, 0.006446297, + 0.016565578, -0.01453512, -0.038669527, 0.013264019, -0.0073789875, + -0.0015187177, 0.004192982, 0.014741467, -0.010408169, -0.030919112, + -0.053881466, -4.9523405e-5, -0.011753554, 0.007395495, + -0.023663934, -0.020477928, -0.045487247, 0.020230312, 0.009665318, + 0.00039618724, 0.05012594, -0.029152777, -0.01074658, -0.06847437, + 0.010845627, -0.011547208, -0.0008914213, -0.002509186, -0.03701049, + 0.035962246, -0.0059345546, -0.0015022099, -0.01889318, + -0.008410726, -0.0030374355, -0.016499547, -0.011737047, + 0.009104053, -0.00052824966, 0.06893658, -0.05145482, 0.026263913, + 0.04291203, 0.07261782, -0.00014857022, 0.0032933066, -0.008641834, + 0.0005695192, -0.0022285532, 0.008097077, 0.03805874, -0.0139986165, + 0.011794825, -0.015872251, 0.017052561, -0.041319028, 0.057645246, + -0.025908995, -0.028236596, -0.043242186, -0.013808776, -0.0138583, + -0.008641834, -0.031686727, 0.008468502, 0.008518025, 0.022632197, + 0.031554665, -0.004267267, -0.03546701, -0.045751374, -0.054417968, + 0.010185314, 0.009450716, -0.008220885, -0.00070983544, -0.06394297, + 0.0075935894, -0.013594175, -0.005406305, -0.016152885, 0.012422121, + 0.05346877, 0.02166649, 0.005431067, 0.004770755, 0.02445631, + 0.0012133234, 0.003937111, -0.00054475747, 0.010251345, 0.020172535, + 0.0040279035, -0.04656851, -0.012224028, 0.01836493, -0.029623251, + -0.00047872626, -0.027411206, -0.00184062, 0.0011307844, 0.03762128, + 0.00024761702, 0.01772938, 0.02786517, -0.12921482, 0.037456203, + -0.009830396, 0.012644976, -0.0031447362, 0.0035821933, + 0.0043745674, 0.0148652755, -0.010994197, -0.016342724, + -0.021889346, -0.0096818255, 0.023069654, 0.0021872837, 0.019495714, + -0.027328666, 0.000998722, 0.016458279, 0.008229139, -0.027535014, + 0.011695778, 8.2539016e-5, -0.0024431548, 0.020832846, 0.02932611, + 0.10528676, + ], + }, +]; diff --git a/src/clip-keywords.ts b/src/clip-keywords.ts new file mode 100644 index 0000000..f8c3a37 --- /dev/null +++ b/src/clip-keywords.ts @@ -0,0 +1,57 @@ +// Keywords for a file from its CLIP embedding. +// +// Ente stores no keywords. A file's ML data holds a CLIP embedding, the 512 +// numbers Ente's image model gave the photo, and Ente's photos app sorts files +// into the categories of its Discover section with it: a file falls in a +// category when the cosine similarity of its CLIP embedding and the text +// embedding of the category's query is at least the category's `minScore`, +// unless the file is a video and the category leaves out videos. A file's +// keywords are the titles of the categories it falls in. The categories and +// their text embeddings are in `clip-keyword-vectors.ts`. + +import { DISCOVER_CATEGORIES } from "./clip-keyword-vectors.js"; +import { clipEmbedding } from "./library/mldata.js"; +import type { MLData } from "./mldata-fetch.js"; + +// How many numbers a CLIP embedding from Ente's image model has. +const EMBEDDING_LENGTH = 512; + +const norm = (vector: number[]): number => Math.hypot(...vector); + +const cosineSimilarity = (a: number[], b: number[]): number => { + let dot = 0; + a.forEach((value, i) => { + dot += value * (b[i] ?? 0); + }); + return dot / (norm(a) * norm(b)); +}; + +// The titles of the Discover categories a file falls in, highest score first, +// or undefined when its ML data has no CLIP embedding of 512 finite numbers +// that are not all zero. Tests pass their own `categories`. +export const clipKeywords = ( + mlData: MLData, + isVideo: boolean, + categories = DISCOVER_CATEGORIES, +): string[] | undefined => { + const embedding = clipEmbedding(mlData); + if ( + !embedding || + embedding.length !== EMBEDDING_LENGTH || + norm(embedding) === 0 + ) { + return undefined; + } + + const matches: { title: string; score: number }[] = []; + for (const category of categories) { + if (isVideo && !category.showVideo) continue; + const score = cosineSimilarity(embedding, category.textEmbedding); + if (score >= category.minScore) { + matches.push({ title: category.title, score }); + } + } + return matches + .sort((a, b) => b.score - a.score) + .map((match) => match.title); +}; diff --git a/src/library/content.ts b/src/library/content.ts index 9f2d6e6..906c0d6 100644 --- a/src/library/content.ts +++ b/src/library/content.ts @@ -581,14 +581,34 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI { fileID: number, opts?: ContentOptions, ): Promise<ContentResult> { - return this.get(fileID, "original", "on-demand", opts?.onProgress); + const file = this.fileOrThrow(fileID); + return this.get(file, "original", "on-demand", opts?.onProgress); } async thumbnail( fileID: number, opts?: ContentOptions, ): Promise<ContentResult> { - return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress); + const file = this.fileOrThrow(fileID); + return this.get(file, "thumbnail", "on-demand", opts?.onProgress); + } + + // The original of `file`, fetched and cached as `original` does. `file` is + // used as given, so `getFile` need not know it. + async originalOf( + file: EnteFile, + opts?: ContentOptions, + ): Promise<ContentResult> { + return this.get(file, "original", "on-demand", opts?.onProgress); + } + + // The thumbnail of `file`, fetched and cached as `thumbnail` does. `file` + // is used as given, so `getFile` need not know it. + async thumbnailOf( + file: EnteFile, + opts?: ContentOptions, + ): Promise<ContentResult> { + return this.get(file, "thumbnail", "on-demand", opts?.onProgress); } // Put the original at the save path of `file` under the download directory @@ -615,7 +635,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI { destination: string, ): Promise<ContentResult> { const result = await this.acquire( - fileID, + this.fileOrThrow(fileID), "original", "on-demand", undefined, @@ -638,8 +658,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI { fileID: number, destination: string, ): Promise<{ path: string; videoPath?: string }> { - const file = this.getFile(fileID); - if (!file) throw new Error(`content cache: unknown file ${fileID}`); + const file = this.fileOrThrow(fileID); return this.pools.content.run( async () => { const fetched = await this.fetchInto( @@ -721,7 +740,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI { onProgress: ((event: EnsureEvent) => void) | undefined, ): Promise<EnsureResult> { try { - const result = await this.acquire(fileID, kind, priority, signal); + const file = this.fileOrThrow(fileID); + const result = await this.acquire(file, kind, priority, signal); const status = result.cached ? "skipped" : "done"; onProgress?.({ fileID, status, path: result.path }); return { fileID, path: result.path }; @@ -736,8 +756,16 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI { } } + // The record `getFile` gives for `fileID`. It throws for a file `getFile` + // does not know. + private fileOrThrow(fileID: number): EnteFile { + const file = this.getFile(fileID); + if (!file) throw new Error(`content cache: unknown file ${fileID}`); + return file; + } + private async get( - fileID: number, + file: EnteFile, kind: Kind, priority: Priority, onProgress: ((event: ContentEvent) => void) | undefined, @@ -745,7 +773,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI { const onByte: ProgressCallback | undefined = onProgress ? (bytesDone) => onProgress({ status: "downloading", bytesDone }) : undefined; - const result = await this.acquire(fileID, kind, priority, undefined, { + const result = await this.acquire(file, kind, priority, undefined, { onByte, }); onProgress?.( @@ -767,7 +795,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI { // photo's original is present only with its video, and is returned with // it. private async acquire( - fileID: number, + file: EnteFile, kind: Kind, priority: Priority, signal: AbortSignal | undefined, @@ -778,8 +806,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI { videoPath?: string; cached: boolean; }> { - const file = this.getFile(fileID); - if (!file) throw new Error(`content cache: unknown file ${fileID}`); + const fileID = file.id; const isLivePhoto = kind === "original" && file.metadata.fileType === "livePhoto"; diff --git a/src/library/index.ts b/src/library/index.ts index a3bed63..23239e9 100644 --- a/src/library/index.ts +++ b/src/library/index.ts @@ -20,9 +20,10 @@ // // `freshCollections()` is the fresh read of the collection list alone (issue // #210): one collections request, no collection's files, and nothing stored, -// for a caller such as `quak collections` that needs no more. A caller that -// asks for each refresh it needs can also open the library with -// `refreshOnOpen: false`, so that `open()` starts none. +// for a caller such as `quak collections` that needs no more. `freshFiles()` +// is the fresh read of one collection's files, and stores nothing either +// (issue #217). A caller that asks for each refresh it needs can also open the +// library with `refreshOnOpen: false`, so that `open()` starts none. // // A refresh reports through `onProgress` each collection whose files it has // fetched, so a large account, whose first refresh fetches the files of every @@ -65,6 +66,8 @@ import { ContentCache, savePath, storedAtSavePath, + type ContentOptions, + type ContentResult, type ContentSource, type ThumbnailsAPI, type EnsureOptions, @@ -625,6 +628,54 @@ export class Library { return collections; } + // The fresh read of one collection's files (issue #217): every live file + // in it that decrypts, in the server's order. `collection` is a record + // `freshCollections()` returned, whose key decrypts the files. It fetches + // no other collection's files. Like `freshCollections()`, it stores + // nothing. It returns each file once, as the latest copy the server sent, + // where the first copy was. + async freshFiles(collection: Collection): Promise<EnteFile[]> { + if (this.closed) throw new Error("the library is closed"); + const { files } = await this.client.filesSince({ + collectionID: collection.id, + collectionKey: collection.key, + sinceTime: 0, + }); + const byID = new Map<number, EnteFile>(); + for (const file of files) byID.set(file.id, file); + return [...byID.values()]; + } + + // The original of `file`, fetched and cached as `Photo.original()` does. + // `file` is used as given, so the store need not hold it. On an empty + // cache, `quak get` passes a file that `freshFiles()` returned. + async originalOf( + file: EnteFile, + opts?: ContentOptions, + ): Promise<ContentResult> { + if (!this.cache) { + throw new Error( + "originalOf requires a library opened with a content cache", + ); + } + return this.cache.originalOf(file, opts); + } + + // The thumbnail of `file`, fetched and cached as `Photo.thumbnail()` does. + // `file` is used as given, so the store need not hold it. On an empty + // cache, `quak get-thumb` passes a file that `freshFiles()` returned. + async thumbnailOf( + file: EnteFile, + opts?: ContentOptions, + ): Promise<ContentResult> { + if (!this.cache) { + throw new Error( + "thumbnailOf requires a library opened with a content cache", + ); + } + return this.cache.thumbnailOf(file, opts); + } + // Back up every in-scope file to `opts.downloadDirectory`, or else the // library's, each original at its save path, with a durable failure // ledger (issue #51). Takes the lock in that directory first, unless the diff --git a/src/library/mldata.ts b/src/library/mldata.ts index 06307d8..fcc624a 100644 --- a/src/library/mldata.ts +++ b/src/library/mldata.ts @@ -61,7 +61,7 @@ export interface MLIndex { // Pull the CLIP embedding out of a payload, or undefined when it is absent or // misshapen. Kept strict so a bad payload is skipped rather than corrupting the // packed index. -const clipEmbedding = (payload: MLData): number[] | undefined => { +export const clipEmbedding = (payload: MLData): number[] | undefined => { const clip = payload.clip; if (typeof clip !== "object" || clip === null) return undefined; const embedding = (clip as { embedding?: unknown }).embedding; diff --git a/src/metadata-backup.ts b/src/metadata-backup.ts index 35e13e2..fb07840 100644 --- a/src/metadata-backup.ts +++ b/src/metadata-backup.ts @@ -1,6 +1,7 @@ import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; import { join } from "node:path"; import * as jpeg from "jpeg-js"; +import { clipKeywords } from "./clip-keywords.js"; import type { Client } from "./client.js"; import { describeError } from "./errors.js"; import { readExifTags } from "./exif.js"; @@ -97,7 +98,8 @@ const extractExif = async ( // Dump every decrypted metadata layer the account holds into a directory tree // of plain JSON: account, per-collection, and per-file records including the -// private and public magic metadata and (by default) the ML data. Collections +// private and public magic metadata and the ML data, with the keywords of its +// CLIP embedding in place of the embedding (see clip-keywords.ts). Collections // and files are enumerated from the library's cache, which the caller refreshes // first. Returns how many files have no ML data because their ML data request // failed or their ML data did not decrypt, gunzip or parse; they are still @@ -230,8 +232,16 @@ export const runMetadataBackup = async ( if (file.pubMagicMetadata) fileMeta.pubMagicMetadata = file.pubMagicMetadata; + // The ML data without the CLIP embedding, and the keywords that + // embedding gives. const ml = mlDataMap.get(file.id); - if (ml) fileMeta.mlData = ml; + if (ml) { + const { clip: _clip, ...mlDataWithoutClip } = ml; + fileMeta.mlData = mlDataWithoutClip; + const isVideo = file.metadata.fileType === "video"; + const keywords = clipKeywords(ml, isVideo); + if (keywords) fileMeta.keywords = keywords; + } const mlError = mlDataErrors.get(file.id); if (mlError) fileMeta.mlDataError = mlError; diff --git a/src/thumbnails.ts b/src/thumbnails.ts index 0205296..471866b 100644 --- a/src/thumbnails.ts +++ b/src/thumbnails.ts @@ -5,7 +5,7 @@ import type { Client } from "./client.js"; import type { Library } from "./library/index.js"; import { ApiError } from "./api/client.js"; import { encryptBlob, toBase64 } from "./crypto/index.js"; -import type { EnteFile } from "./model/types.js"; +import type { Collection, EnteFile } from "./model/types.js"; // The server refuses a thumbnail larger than the one it already records for the // file (`thumbnail.size`, the encrypted size), so these encodings are tried @@ -226,16 +226,18 @@ const encryptThumbnailWithin = ( return undefined; }; -// Regenerate and upload a thumbnail for each requested file. Originals are read -// through the library's content cache (`photo.original()`); the generated -// thumbnail is JPEG-encoded, encrypted under the file's own key, and registered -// with the server — the encrypt-and-upload path is unchanged. Each file is +// Regenerate and upload a thumbnail for each requested file. `found` holds the +// record of each requested file that a collection holds, and the collection it +// was found in. Each original is read through the library's content cache, by +// `lib.originalOf(file)`. The new thumbnail is a JPEG, encrypted under the +// file's own key. It is uploaded, then registered with the server. Each file is // resolved to one outcome (fixed / skipped / failed) and a failure on one file // never stops the others. export const fixMissingThumbnails = async ( lib: Library, client: Client, fileIDs: number[], + found: Map<number, { file: EnteFile; collection: Collection }>, onProgress?: ProgressCallback, ): Promise<ThumbnailFixResult[]> => { const log = onProgress ?? (() => {}); @@ -243,31 +245,8 @@ export const fixMissingThumbnails = async ( const api = client.getApiClient(); const { userID } = client.whoami(); - // Resolve each requested fileID to its file record and owning album by - // enumerating the library, each file taken from the first album that holds - // it. The raw `EnteFile` carries the per-file key the thumbnail is - // encrypted under, which the projected records deliberately do not. - const wanted = new Set(fileIDs); - const fileMap = new Map< - number, - { file: EnteFile; collectionName: string } - >(); - for (const album of lib.albums.list()) { - for (const photo of album.photos.list()) { - if (!wanted.has(photo.fileID) || fileMap.has(photo.fileID)) - continue; - const file = lib.getFile(album.collectionID, photo.fileID); - if (file) { - fileMap.set(photo.fileID, { - file, - collectionName: album.name, - }); - } - } - } - for (const fileID of fileIDs) { - const entry = fileMap.get(fileID); + const entry = found.get(fileID); if (!entry) { results.push({ fileID, @@ -279,7 +258,8 @@ export const fixMissingThumbnails = async ( continue; } - const { file, collectionName } = entry; + const { file } = entry; + const collectionName = entry.collection.name; const title = file.metadata.title; const skipReason = reasonToSkip(file, userID); @@ -297,15 +277,10 @@ export const fixMissingThumbnails = async ( const maxSize = file.thumbnail.size!; try { - const photo = lib.photos.byID({ fileID }); - if (!photo) { - throw new Error("file not present in the library cache"); - } - log( `[${collectionName}] Downloading ${title} for thumbnail generation...`, ); - const { path } = await photo.original(); + const { path } = await lib.originalOf(file); const fileBytes = new Uint8Array(readFileSync(path)); if (!isJpeg(fileBytes)) { diff --git a/test/cli/backup-end-to-end.test.ts b/test/cli/backup-end-to-end.test.ts index 46895e3..6b2e432 100644 --- a/test/cli/backup-end-to-end.test.ts +++ b/test/cli/backup-end-to-end.test.ts @@ -86,10 +86,17 @@ const DAY = "2024/2024-03/2024-03-01"; const TOAST = new TextEncoder().encode("a few seconds of the wedding toast"); // Ente's ML data for beach.jpg: a face it found, and the photo's CLIP -// embedding. +// embedding, here that of a photo of a sunset (see clip-keywords.test.ts). const ML_DATA = { face: { faces: [{ faceID: "100_1", score: 0.98 }] }, - clip: { embedding: [0.5, 0.6, 0.7] }, + clip: { + embedding: JSON.parse( + readFileSync( + new URL("../clip-keywords-sunset.json", import.meta.url), + "utf-8", + ), + ).embedding, + }, }; // Each file's ID, its owner, its original as uploaded, and the metadata its @@ -411,7 +418,7 @@ describe("the first backup of the account", () => { expect(downloads.sort()).toEqual([100, 200, 300]); }); - // The tree "Backup layout" in the README describes. Each original is at + // The tree under "The backup directory" in the README. Each original is at // its save path, `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, with // the extension of its title; the live photo is its image and its video, // each with the extension it had inside the ZIP, and a `.livephoto.json` @@ -486,7 +493,10 @@ describe("the first backup of the account", () => { }, updationTime: TAKEN, }); - expect(beach.mlData).toEqual(ML_DATA); + // The JSON holds the ML data without its CLIP embedding, and the + // keywords the embedding gives in its place. + expect(beach.mlData).toEqual({ face: ML_DATA.face }); + expect(beach.keywords).toEqual(["Sunset"]); expect(beach.imageMetadata.exif.Make.description).toBe("Canon"); expect(beach.imageMetadata.exif.Model.description).toBe("EOS R5"); @@ -497,6 +507,7 @@ describe("the first backup of the account", () => { metadata: { title: "toast.mp4", fileType: "video" }, }); expect(toast).not.toHaveProperty("mlData"); + expect(toast).not.toHaveProperty("keywords"); expect(toast).not.toHaveProperty("imageMetadata"); expect(toast).not.toHaveProperty("imageMetadataError"); diff --git a/test/cli/backup.test.ts b/test/cli/backup.test.ts index c6258dd..df203c7 100644 --- a/test/cli/backup.test.ts +++ b/test/cli/backup.test.ts @@ -52,6 +52,7 @@ import { tmpdir } from "node:os"; import { describe, it, expect, beforeEach, afterEach, vi } from "vitest"; import { runBackup, type BackupLibrary } from "../../src/backup.js"; +import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js"; import { Library } from "../../src/library/index.js"; import type { ContentSource } from "../../src/library/content.js"; import type { CollectionsPage, FilesPage } from "../../src/client.js"; @@ -946,9 +947,19 @@ describe("the backup lock", () => { }); }); -// Also serves ML data: file 100 has `ML_PAYLOAD`, the other two have none. -// While `mlError` is set, every ML data request fails with it. -const ML_PAYLOAD = { face: { faces: [] }, clip: { embedding: [0.5, 0.25] } }; +// Also serves ML data: file 100 has `ML_PAYLOAD`, whose CLIP embedding is that +// of a photo of a sunset (see clip-keywords.test.ts), and the other two have +// none. While `mlError` is set, every ML data request fails with it. +const SUNSET_EMBEDDING: number[] = JSON.parse( + readFileSync( + new URL("../clip-keywords-sunset.json", import.meta.url), + "utf-8", + ), +).embedding; +const ML_PAYLOAD = { + face: { faces: [] }, + clip: { embedding: SUNSET_EMBEDDING }, +}; class MLClient extends MockClient { mlError?: string; @@ -968,16 +979,99 @@ const fileJSON = (outDir: string, fileID: number): Record<string, unknown> => JSON.parse(readFileSync(saved(outDir, `${fileID}.json`), "utf-8")); describe("ML data in each file's JSON", () => { - it("writes a file's ML data, and no ML field for a file that has none", async () => { + // The JSON holds the ML data without its CLIP embedding, and in its place + // the keywords the embedding gives. + it("writes a file's ML data and keywords, and no ML field for a file that has none", async () => { const lib = await openLibrary(stubSource(), new MLClient()); const outDir = join(root, "backup"); const result = await lib.backup({ downloadDirectory: outDir }); expect(result.failed).toBe(0); - expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD); + expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } }); + expect(fileJSON(outDir, 100).keywords).toEqual(["Sunset"]); expect(fileJSON(outDir, 101)).not.toHaveProperty("mlData"); expect(fileJSON(outDir, 101)).not.toHaveProperty("mlDataError"); + expect(fileJSON(outDir, 101)).not.toHaveProperty("keywords"); + await lib.close(); + }); + + // A file's ML data can hold faces and no CLIP embedding. Its JSON then has + // the ML data and no keywords. + it("writes no keywords for ML data without a CLIP embedding", async () => { + const faces = { face: { faces: [{ faceID: "101_1", score: 0.9 }] } }; + + // Gives file 101 ML data holding only faces. + class FacesClient extends MockClient { + async fetchMLData(args: { fileIDs: number[] }): Promise<{ + payloads: Map<number, MLData>; + errors: Map<number, unknown>; + }> { + const payloads = new Map<number, MLData>(); + if (args.fileIDs.includes(101)) payloads.set(101, faces); + return { payloads, errors: new Map() }; + } + } + const lib = await openLibrary(stubSource(), new FacesClient()); + const outDir = join(root, "backup"); + + const result = await lib.backup({ downloadDirectory: outDir }); + + expect(result.failed).toBe(0); + expect(fileJSON(outDir, 101).mlData).toEqual(faces); + expect(fileJSON(outDir, 101)).not.toHaveProperty("keywords"); + await lib.close(); + }); + + // Ente's photos app keeps videos out of some Discover categories, Identity + // among them. Here a photo and a video share one CLIP embedding, Identity's + // own text embedding, so the photo gets the keyword Identity and the video + // does not. + it("keeps a video out of a category that leaves out videos", async () => { + const identity = DISCOVER_CATEGORIES.find( + (c) => c.title === "Identity", + )!; + expect(identity.showVideo).toBe(false); + const toast = file(102, 1, "toast.mp4"); + const video = { + ...toast, + metadata: { ...toast.metadata, fileType: "video" as const }, + }; + + // Serves beach.jpg and the video in the Vacation album, and gives + // each ML data holding only that embedding. + class VideoClient extends MockClient { + override async filesSince(args: { + collectionID: number; + }): Promise<FilesPage> { + const files = + args.collectionID === 1 + ? [file(100, 1, "beach.jpg"), video] + : []; + return { files, deleted: [], cursor: 1 }; + } + async fetchMLData(args: { fileIDs: number[] }): Promise<{ + payloads: Map<number, MLData>; + errors: Map<number, unknown>; + }> { + const payloads = new Map<number, MLData>(); + for (const id of args.fileIDs) { + payloads.set(id, { + clip: { embedding: identity.textEmbedding }, + }); + } + return { payloads, errors: new Map() }; + } + } + const lib = await openLibrary(stubSource(), new VideoClient()); + const outDir = join(root, "backup"); + + const result = await lib.backup({ downloadDirectory: outDir }); + + expect(result.failed).toBe(0); + expect(fileJSON(outDir, 100).keywords).toContain("Identity"); + expect(fileJSON(outDir, 102)).toHaveProperty("keywords"); + expect(fileJSON(outDir, 102).keywords).not.toContain("Identity"); await lib.close(); }); @@ -993,7 +1087,7 @@ describe("ML data in each file's JSON", () => { const result = await lib.backup({ downloadDirectory: outDir }); // File 100's ML data was cached by the first run and is kept. - expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD); + expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } }); expect(fileJSON(outDir, 100)).not.toHaveProperty("mlDataError"); for (const fileID of [101, 200]) { expect(fileJSON(outDir, fileID).mlDataError).toBe( @@ -1026,7 +1120,7 @@ describe("ML data in each file's JSON", () => { expect(result.failed).toBe(0); expect(result.skipped).toBe(3); - expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD); + expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } }); for (const fileID of [100, 101, 200]) { expect(fileJSON(outDir, fileID)).not.toHaveProperty("mlDataError"); } diff --git a/test/cli/commands.test.ts b/test/cli/commands.test.ts index 3269fdb..2bd5e53 100644 --- a/test/cli/commands.test.ts +++ b/test/cli/commands.test.ts @@ -194,9 +194,6 @@ const FETCHED_BOTH_ALBUMS = "Fetched the files of 1 of 2 collections\n" + "Fetched the files of 2 of 2 collections\n"; -// The same for an account with one album. -const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n"; - // Collects everything written to it. class Output { text = ""; @@ -535,9 +532,7 @@ describe("files", () => { it("exits 1 for an unknown collection", async () => { expect(await filesCommand(context(), "9", {})).toBe(1); - expect(stderr.text).toBe( - FETCHED_BOTH_ALBUMS + "Collection 9 not found\n", - ); + expect(stderr.text).toBe("Collection 9 not found\n"); }); // Without a collection ID it lists every file in the account once (issue @@ -578,27 +573,27 @@ describe("get and get-thumb", () => { const out = join(root, "diagram.png"); expect(await getCommand(context(), "200", { out })).toBe(0); expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200)); - expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + `7 bytes -> ${out}\n`); + expect(stderr.text).toBe(`7 bytes -> ${out}\n`); }); it("get-thumb finds a file in any album without --collection", async () => { const out = join(root, "thumb.jpg"); expect(await getThumbCommand(context(), "200", { out })).toBe(0); expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200)); - expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + `3 bytes -> ${out}\n`); + expect(stderr.text).toBe(`3 bytes -> ${out}\n`); }); it("get exits 1 when no album has the file", async () => { const out = join(root, "x"); expect(await getCommand(context(), "999", { out })).toBe(1); - expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + "File 999 not found\n"); + expect(stderr.text).toBe("File 999 not found\n"); expect(existsSync(out)).toBe(false); }); it("get-thumb exits 1 when no album has the file", async () => { const out = join(root, "x"); expect(await getThumbCommand(context(), "999", { out })).toBe(1); - expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + "File 999 not found\n"); + expect(stderr.text).toBe("File 999 not found\n"); expect(existsSync(out)).toBe(false); }); }); @@ -646,7 +641,7 @@ describe("names from an album name or a title over 255 bytes", () => { expect(Buffer.byteLength(name)).toBe(255); expect(code).toBe(0); expect(readdirSync(dir)).toEqual([name]); - expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `7 bytes -> ${name}\n`); + expect(stderr.text).toBe(`7 bytes -> ${name}\n`); }); it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => { @@ -658,7 +653,7 @@ describe("names from an album name or a title over 255 bytes", () => { expect(Buffer.byteLength(name)).toBe(255); expect(code).toBe(0); expect(readdirSync(dir)).toEqual([name]); - expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `3 bytes -> ${name}\n`); + expect(stderr.text).toBe(`3 bytes -> ${name}\n`); }); it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => { @@ -747,8 +742,7 @@ describe("a live photo", () => { Buffer.from(VIDEO), ); expect(stderr.text).toBe( - FETCHED_ONE_ALBUM + - `${IMAGE.length} bytes -> IMG_0300.heic\n` + + `${IMAGE.length} bytes -> IMG_0300.heic\n` + `${VIDEO.length} bytes -> IMG_0300.mov\n`, ); }); @@ -802,8 +796,7 @@ describe("a live photo", () => { expect(readFileSync(out)).toEqual(Buffer.from(IMAGE)); expect(readFileSync(video)).toEqual(Buffer.from(VIDEO)); expect(stderr.text).toBe( - FETCHED_ONE_ALBUM + - `${IMAGE.length} bytes -> ${out}\n${VIDEO.length} bytes -> ${video}\n`, + `${IMAGE.length} bytes -> ${out}\n${VIDEO.length} bytes -> ${video}\n`, ); }); @@ -817,8 +810,7 @@ describe("a live photo", () => { expect(existsSync(out)).toBe(false); expect(existsSync(join(root, "photo.mov"))).toBe(false); expect(stderr.text).toBe( - FETCHED_ONE_ALBUM + - `File 300 is a live photo, and its video would also be written to ${out}\n`, + `File 300 is a live photo, and its video would also be written to ${out}\n`, ); }); @@ -1361,6 +1353,23 @@ describe("helper fix-missing-thumbnails", () => { ).toBe(1); expect(stderr.text).toBe(last); }); + + // beach.jpg (100) is in Vacation (1) and Work (2), both updated at the same + // time. The fixer names a file after the first album holding it in the + // order of `Library.albums.list()`. On a tie, the album with the higher ID + // comes first, so beach.jpg is named after Work. + it("names a photo in two albums after the same album on an empty cache as on a cache that holds the account", async () => { + const ctx = context(fakeClient({ beachInWork: true })); + const opts = { file: ["100"], json: true }; + const inWork = [{ fileID: 100, collection: "Work", status: "skipped" }]; + expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0); + expect(JSON.parse(stdout.text)).toMatchObject(inWork); + + expect(await filesCommand(ctx, undefined, {})).toBe(0); + stdout.text = ""; + expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0); + expect(JSON.parse(stdout.text)).toMatchObject(inWork); + }); }); describe("backup-metadata --exif", () => { @@ -1390,11 +1399,11 @@ describe("backup-metadata --exif", () => { }); // Each test first runs `files` so the cache holds the account as it was, then -// changes the server under it. (`collections` would not do: it reads the -// collection list alone and stores nothing.) +// changes the server under it. (`collections` and `files` with a collection ID +// would not do: they store nothing.) describe("backup-metadata and the thumbnail helpers refresh first", () => { beforeEach(async () => { - expect(await filesCommand(context(), "1", {})).toBe(0); + expect(await filesCommand(context(), undefined, {})).toBe(0); stdout.text = ""; stderr.text = ""; }); @@ -1577,20 +1586,83 @@ describe("an ID given on the command line", () => { ); }); -// An account with many albums (issue #210). On an empty cache `collections` -// used to wait for a refresh that fetched and decrypted the files of every -// album, one album after another, before it printed anything, so on such an -// account it seemed to hang. The account here has 40 albums of one photo each, -// encrypted as Ente's clients encrypt them, on the fake Ente server of -// `test/ente-server.ts`, which also records the path of each request quak -// sends. The session is the one logging in to that server saves. +// An account with many albums (issues #210, #217 and #222). On an empty cache +// `collections`, `files` with a collection ID and `get` used to wait for a +// refresh that fetched and decrypted the files of every album, one album after +// another. So did `helper fix-missing-thumbnails --file` (issue #223). On such +// an account they seemed to hang. The account here has 40 albums of one photo +// each, encrypted as Ente's clients encrypt them, on the fake Ente server of +// `test/ente-server.ts`. The tests record the path and query of each request +// quak sends. The session is the one logging in to that server saves. describe("an account with many albums", () => { const ALBUMS = 40; const PASSWORD = "many-albums"; - const paths: string[] = []; + const requests: string[] = []; let server: typeof globalThis.fetch; let snapshot: ClientSnapshot; + // What `files` without a collection ID prints on an empty cache: every + // album's photo on stdout, and on stderr each album whose files its + // refresh fetched. + let everyPhoto = ""; + let progress = ""; + for (let n = 1; n <= ALBUMS; n++) { + everyPhoto += `${n * 100}\timage\tphoto-${n}.jpg\n`; + progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`; + } + + // The key of each album, by album ID. + const albumKeys = new Map<number, Uint8Array>(); + + // A photo in album `collectionID` as the server sends it, and its original + // encrypted as the server stores it. The original's bytes are the title. + // These tests fetch no thumbnail. The server records a thumbnail size, so + // `helper fix-missing-thumbnails` downloads the original before it skips + // the photo as not a JPEG. + const photoRow = (photo: { + id: number; + collectionID: number; + title: string; + updationTime: number; + }): { row: Record<string, unknown>; original: Uint8Array } => { + const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen(); + const photoKeyBox = encryptSecretbox( + photoKey, + albumKeys.get(photo.collectionID)!, + ); + const metadata = encryptStreamBlob( + new TextEncoder().encode( + JSON.stringify({ + title: photo.title, + fileType: 0, + creationTime: 1000, + modificationTime: 1000, + }), + ), + photoKey, + ); + const original = encryptStreamBlob( + new TextEncoder().encode(photo.title), + photoKey, + ); + const row = { + id: photo.id, + collectionID: photo.collectionID, + ownerID: USER_ID, + encryptedKey: toBase64(photoKeyBox.ciphertext), + keyDecryptionNonce: toBase64(photoKeyBox.nonce), + metadata: { + encryptedData: toBase64(metadata.ciphertext), + decryptionHeader: toBase64(metadata.header), + }, + file: { decryptionHeader: toBase64(original.header) }, + thumbnail: { decryptionHeader: "" }, + info: { thumbSize: 1000 }, + updationTime: photo.updationTime, + }; + return { row, original: original.ciphertext }; + }; + // Album n is named `Album n` and holds one photo, n * 100, titled // `photo-n.jpg`. beforeAll(async () => { @@ -1598,8 +1670,10 @@ describe("an account with many albums", () => { const account = await buildAccount(PASSWORD); const collections: Record<string, unknown>[] = []; const filesByCollection: Record<number, Record<string, unknown>[]> = {}; + const fileCiphertexts: Record<number, Uint8Array> = {}; for (let id = 1; id <= ALBUMS; id++) { const albumKey = sodium.crypto_secretbox_keygen(); + albumKeys.set(id, albumKey); const albumKeyBox = encryptSecretbox(albumKey, account.masterKey); const name = encryptSecretbox( new TextEncoder().encode(`Album ${id}`), @@ -1615,37 +1689,14 @@ describe("an account with many albums", () => { type: "album", updationTime: 1, }); - const photoKey = - sodium.crypto_secretstream_xchacha20poly1305_keygen(); - const photoKeyBox = encryptSecretbox(photoKey, albumKey); - const metadata = encryptStreamBlob( - new TextEncoder().encode( - JSON.stringify({ - title: `photo-${id}.jpg`, - fileType: 0, - creationTime: 1000, - modificationTime: 1000, - }), - ), - photoKey, - ); - filesByCollection[id] = [ - { - id: id * 100, - collectionID: id, - ownerID: USER_ID, - encryptedKey: toBase64(photoKeyBox.ciphertext), - keyDecryptionNonce: toBase64(photoKeyBox.nonce), - metadata: { - encryptedData: toBase64(metadata.ciphertext), - decryptionHeader: toBase64(metadata.header), - }, - // These tests fetch no original and no thumbnail. - file: { decryptionHeader: "" }, - thumbnail: { decryptionHeader: "" }, - updationTime: 1, - }, - ]; + const { row, original } = photoRow({ + id: id * 100, + collectionID: id, + title: `photo-${id}.jpg`, + updationTime: 1, + }); + filesByCollection[id] = [row]; + fileCiphertexts[id * 100] = original; } const ente = buildMetaFetch({ verifier: account.verifier, @@ -1655,7 +1706,7 @@ describe("an account with many albums", () => { collections, filesByCollection, encryptedMLData: {}, - fileCiphertexts: {}, + fileCiphertexts, }); server = (async (input: RequestInfo | URL, init?: RequestInit) => { const url = @@ -1664,7 +1715,8 @@ describe("an account with many albums", () => { : input instanceof URL ? input.href : input.url; - paths.push(new URL(url).pathname); + const { pathname, search } = new URL(url); + requests.push(pathname + search); return ente(input, init); }) as typeof globalThis.fetch; const client = await Client.login({ @@ -1676,16 +1728,17 @@ describe("an account with many albums", () => { }); beforeEach(() => { - paths.length = 0; + requests.length = 0; }); - // A context whose saved session sends its requests to the fake server. - const accountContext = (): CliContext => { + // A context whose saved session sends its requests to `fetch`, by default + // the fake server. + const accountContext = (fetch = server): CliContext => { const ctx = context(); saveSession(ctx.sessionDir, snapshot); return { ...ctx, - loadSession: (path) => loadSession(path, { fetch: server }), + loadSession: (path) => loadSession(path, { fetch }), }; }; @@ -1698,46 +1751,219 @@ describe("an account with many albums", () => { albums += `${id}\talbum\tAlbum ${id}\n`; expect(stdout.text).toBe(albums); expect(stderr.text).toBe(""); - expect(paths).toEqual(["/collections/v2"]); + expect(requests).toEqual(["/collections/v2?sinceTime=0"]); // It stored nothing that would make a refresh skip an album's files: - // `files` after it still finds the album's photo. + // `files` after it still lists every album's photo. stdout.text = ""; - expect(await filesCommand(ctx, "7", {})).toBe(0); - expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n"); + expect(await filesCommand(ctx, undefined, {})).toBe(0); + expect(stdout.text).toBe(everyPhoto); }); - it("files counts on stderr each album whose files its refresh fetched", async () => { + it("files with a collection ID asks for the album list and that album's files alone", async () => { expect(await filesCommand(accountContext(), "7", {})).toBe(0); expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n"); - let progress = ""; - for (let n = 1; n <= ALBUMS; n++) { - progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`; - } + expect(stderr.text).toBe(""); + expect(requests).toEqual([ + "/collections/v2?sinceTime=0", + "/collections/v2/diff?collectionID=7&sinceTime=0", + ]); + }); + + // Here the server sends album 7's files in two pages. Photo 700 changed + // between them, so both pages hold a copy of it. + it("files with a collection ID lists only the later copy of a photo sent in two pages", async () => { + const firstPage = [ + photoRow({ + id: 700, + collectionID: 7, + title: "photo-7.jpg", + updationTime: 1, + }).row, + photoRow({ + id: 701, + collectionID: 7, + title: "photo-701.jpg", + updationTime: 1, + }).row, + ]; + const secondPage = [ + photoRow({ + id: 700, + collectionID: 7, + title: "photo-7-edited.jpg", + updationTime: 2, + }).row, + ]; + const paged = (async (input: RequestInfo | URL, init?: RequestInit) => { + const url = new URL(input instanceof Request ? input.url : input); + if ( + url.pathname !== "/collections/v2/diff" || + url.searchParams.get("collectionID") !== "7" + ) { + return server(input, init); + } + const first = url.searchParams.get("sinceTime") === "0"; + return new Response( + JSON.stringify({ + diff: first ? firstPage : secondPage, + hasMore: first, + }), + ); + }) as typeof globalThis.fetch; + + expect(await filesCommand(accountContext(paged), "7", {})).toBe(0); + + // Photo 700 is listed once, with the later title, before photo 701. + expect(stdout.text).toBe( + "700\timage\tphoto-7-edited.jpg\n701\timage\tphoto-701.jpg\n", + ); + }); + + // The cache stays empty, so a later `files` without a collection ID still + // fetches the files of all 40 albums. + it("files with a collection ID stores nothing in the cache", async () => { + const ctx = accountContext(); + expect(await filesCommand(ctx, "7", {})).toBe(0); + expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false); + + stdout.text = ""; + expect(await filesCommand(ctx, undefined, {})).toBe(0); + expect(stdout.text).toBe(everyPhoto); + expect(stderr.text).toBe(progress); + }); + + // Photo 200 is in album 2, so `get` asks for no file of album 3 or later. + // The last request downloads the photo's original. + it("get on an empty cache asks for the albums' files only until one holds the photo", async () => { + const out = join(root, "photo.jpg"); + expect(await getCommand(accountContext(), "200", { out })).toBe(0); + + expect(readFileSync(out, "utf-8")).toBe("photo-2.jpg"); + expect(stderr.text).toBe(`11 bytes -> ${out}\n`); + expect(requests).toEqual([ + "/collections/v2?sinceTime=0", + "/collections/v2/diff?collectionID=1&sinceTime=0", + "/collections/v2/diff?collectionID=2&sinceTime=0", + "/?fileID=200", + ]); + expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false); + }); + + // `files` first fills the cache, so `get` waits for a refresh. No album has + // changed since, so the refresh asks for no album's files. + it("get on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => { + const ctx = accountContext(); + expect(await filesCommand(ctx, undefined, {})).toBe(0); + requests.length = 0; + stderr.text = ""; + + const out = join(root, "photo.jpg"); + expect(await getCommand(ctx, "4000", { out })).toBe(0); + expect(readFileSync(out, "utf-8")).toBe("photo-40.jpg"); + expect(stderr.text).toBe(`12 bytes -> ${out}\n`); + expect(requests).toEqual([ + "/collections/v2?sinceTime=1", + "/?fileID=4000", + ]); + + requests.length = 0; + stderr.text = ""; + expect(await getCommand(ctx, "999", { out })).toBe(1); + expect(stderr.text).toBe("File 999 not found\n"); + expect(requests).toEqual(["/collections/v2?sinceTime=1"]); + }); + + // The helper searches the most recently updated album first. All 40 were + // updated at the same time, and on a tie the higher ID comes first. Photo + // 3900 is in album 39, so the helper asks for no file of album 38 or + // earlier. The last request downloads the photo's original. + it("fix-missing-thumbnails --file on an empty cache asks for the albums' files only until one holds the photo", async () => { + const opts = { file: ["3900"], json: true }; + expect(await fixMissingThumbnailsCommand(accountContext(), opts)).toBe( + 0, + ); + + const reason = + "unsupported image format (only baseline JPEG can be regenerated)"; + expect(JSON.parse(stdout.text)).toEqual([ + { + fileID: 3900, + title: "photo-39.jpg", + collection: "Album 39", + status: "skipped", + reason, + }, + ]); + expect(stderr.text).toBe( + "[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" + + `[Album 39] Skipping photo-39.jpg: ${reason}\n`, + ); + expect(requests).toEqual([ + "/collections/v2?sinceTime=0", + "/collections/v2/diff?collectionID=40&sinceTime=0", + "/collections/v2/diff?collectionID=39&sinceTime=0", + "/?fileID=3900", + ]); + expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false); + }); + + // `files` first fills the cache, so the helper waits for a refresh. No + // album has changed since, so the refresh asks for no album's files. + it("fix-missing-thumbnails --file on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => { + const ctx = accountContext(); + expect(await filesCommand(ctx, undefined, {})).toBe(0); + requests.length = 0; + stdout.text = ""; + stderr.text = ""; + + const opts = { file: ["3900"], json: true }; + expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0); + const reason = + "unsupported image format (only baseline JPEG can be regenerated)"; + expect(JSON.parse(stdout.text)).toEqual([ + { + fileID: 3900, + title: "photo-39.jpg", + collection: "Album 39", + status: "skipped", + reason, + }, + ]); + expect(stderr.text).toBe( + "[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" + + `[Album 39] Skipping photo-39.jpg: ${reason}\n`, + ); + expect(requests).toEqual([ + "/collections/v2?sinceTime=1", + "/?fileID=3900", + ]); + + requests.length = 0; + stdout.text = ""; + const notFound = { file: ["999"], json: true }; + expect(await fixMissingThumbnailsCommand(ctx, notFound)).toBe(1); + expect(JSON.parse(stdout.text)).toMatchObject([ + { fileID: 999, reason: "file not found in any collection" }, + ]); + expect(requests).toEqual(["/collections/v2?sinceTime=1"]); + }); + + it("files counts on stderr each album whose files its refresh fetched", async () => { + expect(await filesCommand(accountContext(), undefined, {})).toBe(0); + + expect(stdout.text).toBe(everyPhoto); expect(stderr.text).toBe(progress); }); // `--json` shapes stdout only, so the refresh's progress is still counted // on stderr. it("files --json counts on stderr each album whose files its refresh fetched", async () => { - expect(await filesCommand(accountContext(), "7", { json: true })).toBe( - 0, - ); + const opts = { json: true }; + expect(await filesCommand(accountContext(), undefined, opts)).toBe(0); - expect(JSON.parse(stdout.text)).toEqual([ - { - id: 700, - title: "photo-7.jpg", - fileType: "image", - creationTime: 1000, - collectionID: 7, - }, - ]); - let progress = ""; - for (let n = 1; n <= ALBUMS; n++) { - progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`; - } + expect(JSON.parse(stdout.text)).toHaveLength(ALBUMS); expect(stderr.text).toBe(progress); }); }); diff --git a/test/cli/log.test.ts b/test/cli/log.test.ts index 137c983..a407293 100644 --- a/test/cli/log.test.ts +++ b/test/cli/log.test.ts @@ -234,13 +234,15 @@ describe("-q", () => { }); }); - // The refresh's progress is left out; the line saying why the command - // failed is the fatal one, and it is printed. + // No album holds file 999, so fixing it fails. The summary from + // `--- Done ---` to the end of the `Failed files:` list is left out. The + // line saying why the command failed is the fatal one, and it is printed. it("prints only the line a failing command exits with", async () => { - expect(await quak("-q", "files", "9")).toEqual({ + const fix = ["helper", "fix-missing-thumbnails", "--file", "999"]; + expect(await quak("-q", ...fix)).toEqual({ code: 1, stdout: "", - stderr: "Collection 9 not found\n", + stderr: `quak: 1 file(s) failed; see the "Failed files:" list, or the JSON with --json\n`, }); }); @@ -326,7 +328,7 @@ describe("what the library reports", () => { ]); }); - // Each operation is named in plain words, as the README names it. + // Each operation is named in plain words. it("logs a refresh's progress at info, and its start and end at debug", () => { const { log, lines } = collect(); const onProgress = logRefresh(log); diff --git a/test/cli/metadata-backup.test.ts b/test/cli/metadata-backup.test.ts index 0b75804..c66c799 100644 --- a/test/cli/metadata-backup.test.ts +++ b/test/cli/metadata-backup.test.ts @@ -41,6 +41,7 @@ import { beforeAll, afterAll, describe, expect, it, vi } from "vitest"; import { init, toBase64, encryptBlob } from "../../src/crypto/index.js"; import * as jpegJs from "jpeg-js"; import { Client } from "../../src/client.js"; +import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js"; import { Library } from "../../src/library/index.js"; import { runMetadataBackup, @@ -56,8 +57,8 @@ import { type MetaMockState, } from "../ente-server.js"; -// One file per ML data request, so the two files of the mock account are -// fetched in two requests and one of them can fail on its own. +// One file per ML data request, so each file of the mock account is fetched in +// a request of its own and one of them can fail on its own. vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({ ...(await importOriginal<typeof import("../../src/mldata-fetch.js")>()), MLDATA_BATCH_SIZE: 1, @@ -66,6 +67,21 @@ vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({ const TEST_EMAIL = "metabackup@example.com"; const TEST_PASSWORD = "metapass"; +// The CLIP embedding of a photo of a sunset (see clip-keywords.test.ts), which +// the ML data of beach.jpg holds. +const SUNSET_EMBEDDING: number[] = JSON.parse( + readFileSync( + new URL("../clip-keywords-sunset.json", import.meta.url), + "utf-8", + ), +).embedding; + +// Identity, a Discover category of Ente's photos app that leaves out videos. +const IDENTITY = DISCOVER_CATEGORIES.find((c) => c.title === "Identity")!; + +// ML data holding faces and no CLIP embedding, which file 302 has. +const FACES_ONLY = { face: { faces: [{ faceID: "face-def", score: 0.9 }] } }; + let mock: MetaMockState; let testDir: string; @@ -204,6 +220,65 @@ const buildMetaMock = async (): Promise<MetaMockState> => { updationTime: 1710000000000000, }; + // A file in coll 10 with the ML data given. `fileType` is Ente's number: 0 + // an image, 1 a video. + const vacationFile = ( + id: number, + title: string, + fileType: number, + mlData: object, + ) => { + const fileKey = sodium.crypto_secretstream_xchacha20poly1305_keygen(); + const { ciphertext: encKey, nonce } = encryptSecretbox(fileKey, ck1); + const meta = encryptStreamBlob( + new TextEncoder().encode( + JSON.stringify({ + title, + fileType, + creationTime: 1700000000000000, + modificationTime: 1700000000000000, + }), + ), + fileKey, + ); + const ml = encryptBlob( + new Uint8Array(gzipSync(Buffer.from(JSON.stringify(mlData)))), + fileKey, + ); + return { + raw: { + id, + collectionID: 10, + ownerID: 42, + encryptedKey: toBase64(encKey), + keyDecryptionNonce: toBase64(nonce), + metadata: { + encryptedData: toBase64(meta.ciphertext), + decryptionHeader: toBase64(meta.header), + }, + file: { + decryptionHeader: toBase64(sodium.randombytes_buf(24)), + }, + thumbnail: { + decryptionHeader: toBase64(sodium.randombytes_buf(24)), + }, + updationTime: 1700000000000000, + }, + mlData: { + encryptedData: toBase64(ml.ciphertext), + decryptionHeader: toBase64(ml.header), + }, + }; + }; + // Files 300 and 301 in coll 10: a photo and a video whose ML data holds + // only a CLIP embedding, Identity's own text embedding. + const identityML = { clip: { embedding: IDENTITY.textEmbedding } }; + const identityPhoto = vacationFile(300, "passport.jpg", 0, identityML); + const identityVideo = vacationFile(301, "passport.mp4", 1, identityML); + // File 302 in coll 10: a photo whose ML data holds faces and no CLIP + // embedding. + const facesPhoto = vacationFile(302, "party.jpg", 0, FACES_ONLY); + // Encrypt ML data for file 100 (gzipped JSON, encrypted with file key) const mlPayload = JSON.stringify({ face: { @@ -230,7 +305,7 @@ const buildMetaMock = async (): Promise<MetaMockState> => { clip: { version: 1, client: "test", - embedding: [0.5, 0.6, 0.7], + embedding: SUNSET_EMBEDDING, }, }); const gzipped = gzipSync(Buffer.from(mlPayload)); @@ -270,12 +345,23 @@ const buildMetaMock = async (): Promise<MetaMockState> => { keyAttributes: account.keyAttributes, encryptedToken: account.encryptedToken, collections: [rawColl1, rawColl2], - filesByCollection: { 10: [rawFile1], 20: [rawFile2] }, + filesByCollection: { + 10: [ + rawFile1, + identityPhoto.raw, + identityVideo.raw, + facesPhoto.raw, + ], + 20: [rawFile2], + }, encryptedMLData: { 100: { encryptedData: toBase64(mlCiphertext), decryptionHeader: toBase64(mlHeader), }, + 300: identityPhoto.mlData, + 301: identityVideo.mlData, + 302: facesPhoto.mlData, }, fileCiphertexts: { 100: encFileBody1 }, }; @@ -432,7 +518,9 @@ describe("quak backup-metadata", () => { expect(account.email).toBe(TEST_EMAIL); }); - it("fetches and decrypts ML data by default", async () => { + // The CLIP embedding is left out of the ML data, and the keywords it gives + // are written in its place. + it("fetches and decrypts ML data by default, with keywords for the CLIP embedding", async () => { const outDir = join(testDir, "ml-data"); await runBackup(outDir); @@ -452,8 +540,41 @@ describe("quak backup-metadata", () => { expect(fileMeta.mlData.face.faces[0].faceID).toBe("face-abc"); expect(fileMeta.mlData.face.faces[0].score).toBeCloseTo(0.98); expect(fileMeta.mlData.face.faces[0].detection.box.x).toBeCloseTo(0.1); - expect(fileMeta.mlData.clip).toBeDefined(); - expect(fileMeta.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]); + expect(fileMeta.mlData).not.toHaveProperty("clip"); + expect(fileMeta.keywords).toEqual(["Sunset"]); + }); + + // Ente's photos app keeps videos out of some Discover categories, Identity + // among them. Files 300 and 301, a photo and a video, share one CLIP + // embedding, Identity's own text embedding, so the photo gets the keyword + // Identity and the video does not. + it("keeps a video out of a category that leaves out videos", async () => { + expect(IDENTITY.showVideo).toBe(false); + const outDir = join(testDir, "video-keywords"); + await runBackup(outDir); + + const vacation = join(outDir, "collections", "10-Vacation"); + const fileJSON = (fileID: number) => + JSON.parse(readFileSync(join(vacation, `${fileID}.json`), "utf-8")); + expect(fileJSON(300).keywords).toContain("Identity"); + expect(fileJSON(301)).toHaveProperty("keywords"); + expect(fileJSON(301).keywords).not.toContain("Identity"); + }); + + // A file's ML data can hold faces and no CLIP embedding. File 302's JSON + // then has the ML data and no keywords. + it("writes no keywords for ML data without a CLIP embedding", async () => { + const outDir = join(testDir, "no-clip"); + await runBackup(outDir); + + const fileMeta = JSON.parse( + readFileSync( + join(outDir, "collections", "10-Vacation", "302.json"), + "utf-8", + ), + ); + expect(fileMeta.mlData).toEqual(FACES_ONLY); + expect(fileMeta).not.toHaveProperty("keywords"); }); it("extracts EXIF from downloaded files when --exif is set", async () => { @@ -541,7 +662,7 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => { "utf-8", ), ); - expect(ok.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]); + expect(ok.keywords).toEqual(["Sunset"]); expect(ok.mlDataError).toBeUndefined(); const failed = JSON.parse( @@ -599,7 +720,7 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => { "utf-8", ), ); - expect(ok.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]); + expect(ok.keywords).toEqual(["Sunset"]); }); it("exits 0 when every ML data request succeeds", async () => { diff --git a/test/cli/read.test.ts b/test/cli/read.test.ts index 66ff369..d292041 100644 --- a/test/cli/read.test.ts +++ b/test/cli/read.test.ts @@ -3,20 +3,25 @@ * issue #36, issue #52). * * The `collections`, `files`, `get`, and `get-thumb` commands must answer for - * current server state, not the local cache. `files`, `get` and `get-thumb` - * force a `Library.fresh()` round-trip before they read; the stand-in library - * below serves nothing until `fresh()` has been awaited, so a helper that read - * without refreshing would come back empty and fail here. `collections` asks - * the library for the collection list alone (`freshCollections()`, issue - * #210), and must not wait for a refresh, which would fetch the files of every - * collection. + * current server state, not the local cache. `get`, `get-thumb` and `files` + * without a collection ID force a `Library.fresh()` round-trip before they + * read. The stand-in library below serves no cached record until `fresh()` has + * been awaited, so a helper that read without refreshing would come back empty + * and fail here. `collections` asks the library for the collection list alone + * (`freshCollections()`, issue #210). `files` with a collection ID asks for + * that list, then for that one collection's files (`freshFiles()`, issue + * #217). Neither may wait for a refresh, which would fetch the files of every + * collection. On an empty cache, `get` and `get-thumb` ask for that list, then + * for one collection's files at a time, until they find the file (issue #222). + * `helper fix-missing-thumbnails --file` does the same until it finds every + * file it names, reading the most recently updated collection first (issue + * #223). * - * `collections` and `files` also list in the library's enumeration order - * (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not + * `collections` and `files` also list in the order the library gives them, not * the albums/photos projection's newest-first order. The fixtures are seeded in - * an enumeration order that a newest-first sort would rearrange, so a - * regression to the projection order would fail here too. Field values still - * come from the raw metadata via `cli-output.ts`. + * an order that a newest-first sort would rearrange, so a regression to the + * projection order would fail here too. Field values still come from the raw + * metadata via `cli-output.ts`. */ import { describe, it, expect } from "vitest"; @@ -25,10 +30,10 @@ import { freshCollections, freshFiles, freshFile, + freshFilesByID, type FreshReadLibrary, } from "../../src/cli-read.js"; import { fileListRow } from "../../src/cli-output.js"; -import type { Photo } from "../../src/library/index.js"; import type { Collection, EnteFile } from "../../src/model/types.js"; const collection = (id: number, updationTime: number): Collection => ({ @@ -62,14 +67,15 @@ const file = ( updationTime: creationTime, }); -// A library that reveals its records only after `fresh()` has been awaited, and -// serves them in the enumeration order it was given. `freshCollections()` -// answers the collection list in that order without a refresh. `photos.byID` -// returns a stand-in `Photo` carrying just the fileID the helper passes -// through. +// A library that reveals its cached records only after `fresh()` has been +// awaited, and serves them in the enumeration order it was given. +// `freshCollections()` and `freshFiles()` answer in that order without a +// refresh. class FakeLibrary implements FreshReadLibrary { freshCalls = 0; freshCollectionsCalls = 0; + // The ID of each collection `freshFiles()` was asked for. + freshFilesCalls: number[] = []; private refreshed = false; constructor( @@ -88,9 +94,9 @@ class FakeLibrary implements FreshReadLibrary { return this.collections; } - getCollection(id: number): Collection | undefined { - if (!this.refreshed) return undefined; - return this.collections.find((c) => c.id === id); + async freshFiles(collection: Collection): Promise<EnteFile[]> { + this.freshFilesCalls.push(collection.id); + return this.files.filter((f) => f.collectionID === collection.id); } listCollections(): Collection[] { @@ -107,14 +113,6 @@ class FakeLibrary implements FreshReadLibrary { if (!this.refreshed) return undefined; return this.files.find((f) => f.id === fileID); } - - photos = { - byID: ({ fileID }: { fileID: number }): Photo | undefined => { - if (!this.refreshed) return undefined; - if (!this.files.some((f) => f.id === fileID)) return undefined; - return { fileID } as unknown as Photo; - }, - }; } describe("CLI read helpers (issue #36 amendment, issue #52)", () => { @@ -138,18 +136,24 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => { expect(rows.map((c) => c.id)).not.toEqual(newestFirst); }); - it("freshFiles refreshes first, lists in enumeration order, keeps raw fields", async () => { - // Enumeration order by id 10, 11, 12; creationTimes ascending, so a + it("freshFiles with a collection ID reads that collection's files alone and keeps their order and raw fields", async () => { + // Given in the order 10, 11, 12; creationTimes ascending, so a // newest-first sort would reverse them. const files = [ file(10, 1, 1_700_000_000_000_000), file(11, 1, 1_700_000_000_000_001), file(12, 1, 1_700_000_000_000_002), + file(20, 2, 1_700_000_000_000_003), ]; - const lib = new FakeLibrary([collection(1, 100)], files); + const lib = new FakeLibrary( + [collection(1, 100), collection(2, 200)], + files, + ); const rows = await freshFiles(lib, 1); - expect(lib.freshCalls).toBe(1); + expect(lib.freshCalls).toBe(0); + expect(lib.freshCollectionsCalls).toBe(1); + expect(lib.freshFilesCalls).toEqual([1]); expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]); // Field values come from raw metadata: microsecond creationTime and the @@ -206,27 +210,117 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => { it("freshFiles returns undefined for an unknown collection", async () => { const lib = new FakeLibrary([collection(1, 100)], []); const rows = await freshFiles(lib, 999); - expect(lib.freshCalls).toBe(1); + expect(lib.freshCalls).toBe(0); + expect(lib.freshFilesCalls).toEqual([]); expect(rows).toBeUndefined(); }); - it("freshFile refreshes first, then resolves the photo and its raw record", async () => { - const f = file(10, 1, 1_700_000_000_000_000); - const lib = new FakeLibrary([collection(1, 100)], [f]); - - const resolved = await freshFile(lib, 10); - expect(lib.freshCalls).toBe(1); - expect(resolved?.photo.fileID).toBe(10); - expect(resolved?.file.metadata.title).toBe("file-10.jpg"); - expect(resolved?.file.metadata.creationTime).toBe( - 1_700_000_000_000_000, + // Three collections, each holding one file. + const threeCollections = (): FakeLibrary => + new FakeLibrary( + [collection(1, 100), collection(2, 200), collection(3, 300)], + [ + file(10, 1, 1_700_000_000_000_000), + file(20, 2, 1_700_000_000_000_001), + file(30, 3, 1_700_000_000_000_002), + ], ); + + it("freshFile on an empty cache reads each collection's files in the server's order until one holds the file", async () => { + const lib = threeCollections(); + + const found = await freshFile(lib, 20); + expect(lib.freshCalls).toBe(0); + expect(lib.freshCollectionsCalls).toBe(1); + expect(lib.freshFilesCalls).toEqual([1, 2]); + expect(found?.metadata.title).toBe("file-20.jpg"); + expect(found?.metadata.creationTime).toBe(1_700_000_000_000_001); }); - it("freshFile returns undefined for an unknown file", async () => { - const lib = new FakeLibrary([collection(1, 100)], []); - const resolved = await freshFile(lib, 404); - expect(lib.freshCalls).toBe(1); - expect(resolved).toBeUndefined(); + // The stand-in's cache holds what the server holds once `fresh()` has run. + it("freshFile on a cache that holds the collections refreshes, then looks the file up in the cache", async () => { + const lib = threeCollections(); + await lib.fresh(); + + const found = await freshFile(lib, 30); + expect(lib.freshCalls).toBe(2); + expect(lib.freshCollectionsCalls).toBe(0); + expect(lib.freshFilesCalls).toEqual([]); + expect(found?.collectionID).toBe(3); + }); + + it("freshFile on an empty cache returns undefined for an unknown file, after reading every collection's files", async () => { + const lib = threeCollections(); + + const found = await freshFile(lib, 404); + expect(lib.freshCalls).toBe(0); + expect(lib.freshFilesCalls).toEqual([1, 2, 3]); + expect(found).toBeUndefined(); + }); + + // The server lists collections 1, 2 and 3, and collection 3 was updated + // most recently. File 11 is in collections 1 and 2. + const fileInTwoCollections = (): FakeLibrary => + new FakeLibrary( + [collection(1, 100), collection(2, 200), collection(3, 300)], + [ + file(10, 1, 1_700_000_000_000_000), + file(11, 1, 1_700_000_000_000_001), + file(11, 2, 1_700_000_000_000_001), + file(20, 2, 1_700_000_000_000_002), + file(30, 3, 1_700_000_000_000_003), + ], + ); + + it("freshFilesByID on an empty cache reads the most recently updated collection's files first, until it has found every file", async () => { + const lib = fileInTwoCollections(); + + const found = await freshFilesByID(lib, [11, 20]); + expect(lib.freshCalls).toBe(0); + expect(lib.freshCollectionsCalls).toBe(1); + expect(lib.freshFilesCalls).toEqual([3, 2]); + expect(found.get(11)?.collection.id).toBe(2); + expect(found.get(11)?.file.collectionID).toBe(2); + expect(found.get(20)?.collection.id).toBe(2); + }); + + // Collection 1 was updated last, though collections 2 and 3 have higher + // IDs. File 11 is in collections 1 and 2. + it("freshFilesByID on an empty cache reads the most recently updated collection first, though its ID is the lowest", async () => { + const lib = new FakeLibrary( + [collection(1, 400), collection(2, 200), collection(3, 300)], + [ + file(11, 1, 1_700_000_000_000_001), + file(11, 2, 1_700_000_000_000_001), + ], + ); + + const found = await freshFilesByID(lib, [11]); + expect(lib.freshFilesCalls).toEqual([1]); + expect(found.get(11)?.collection.id).toBe(1); + expect(found.get(11)?.file.collectionID).toBe(1); + }); + + // The stand-in's cache holds what the server holds once `fresh()` has run. + it("freshFilesByID on a cache that holds the collections refreshes, then finds each file in the most recently updated collection that holds it", async () => { + const lib = fileInTwoCollections(); + await lib.fresh(); + + const found = await freshFilesByID(lib, [11, 404]); + expect(lib.freshCalls).toBe(2); + expect(lib.freshCollectionsCalls).toBe(0); + expect(lib.freshFilesCalls).toEqual([]); + expect(found.get(11)?.collection.id).toBe(2); + expect(found.get(11)?.file.collectionID).toBe(2); + expect(found.has(404)).toBe(false); + }); + + it("freshFilesByID on an empty cache reads every collection's files when a file is in none of them", async () => { + const lib = fileInTwoCollections(); + + const found = await freshFilesByID(lib, [10, 404]); + expect(lib.freshFilesCalls).toEqual([3, 2, 1]); + expect(found.get(10)?.collection.id).toBe(1); + expect(found.has(404)).toBe(false); }); }); diff --git a/test/clip-keywords-sunset.json b/test/clip-keywords-sunset.json new file mode 100644 index 0000000..499071f --- /dev/null +++ b/test/clip-keywords-sunset.json @@ -0,0 +1,109 @@ +{ + "photo": "https://upload.wikimedia.org/wikipedia/commons/d/db/Sunset_off_the_Massachusetts_coast_-_NOAA.jpg", + "sha256": "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3", + "embedding": [ + -0.0021123015, -0.04437615, -0.062354308, 0.006249992, 0.022141185, + 0.011159019, 0.01227392, -0.00030532264, 0.03186437, 0.09196922, + 0.039111234, -0.037084445, -0.032962628, -0.019820206, -0.013141825, + 0.0047183544, 0.017137334, 0.046581306, 0.0047622235, 0.04982521, + 0.023128154, 0.032080866, 0.037709303, 0.019241406, 0.0032100384, + -0.014275843, 0.049337693, 0.025953151, 0.014263995, -0.0062328726, + -0.040504098, -0.015083303, -0.008123875, -0.006233639, -0.05352683, + -0.0048083956, 0.0763486, -0.013155067, 0.025114346, 0.022016674, + 0.0001588023, -0.010341572, -0.0037518279, -0.019181008, 0.04549337, + -0.0010725886, 0.004636866, 0.017177915, -0.0117317885, 0.0026097358, + 0.030738998, 0.013158572, -0.08856403, -0.0121391015, 0.011477229, + -0.0033104012, 0.035524756, -0.018607879, -0.0071739843, -0.031906743, + 0.0056713386, -0.0044195866, 0.00292471, 0.00061686034, -0.043075223, + 0.09040427, 0.001522537, -0.011445652, 0.009609886, -0.0053201183, + -0.0057452377, 0.00972853, -0.0024397348, -0.021255083, 0.015893811, + -0.006020548, -0.09428023, 0.014277583, -0.017975299, -0.011878315, + 0.011825521, 0.00014203975, -0.0020437292, -0.05305343, -0.024840152, + -0.011336872, -0.002796541, -0.00036171873, 0.004530499, 0.044486605, + 0.021602718, -0.01292857, 0.024530916, 0.019559948, -0.039928272, + -0.0681174, 0.049101, 0.02461532, 0.0365531, -0.04070358, -0.011011939, + -0.0027188526, -0.020370876, 0.012904856, 0.043157287, -0.051658224, + -0.3726012, 0.019549074, -0.0499252, -0.04800559, -0.011493859, + 0.02893256, 0.060341954, 0.0013929688, -0.051283866, -0.024286803, + 0.044124287, -0.005995806, -0.062817656, 0.007369614, -0.021470634, + -0.028147545, 0.00411659, 0.04234773, 0.03787647, 0.008579682, + -0.038568925, -0.027498025, -0.028416378, 0.0022012168, 0.010906191, + -0.013745787, 0.018267838, 0.013531097, -0.061559845, 0.004682517, + -0.005993303, 0.0028113858, -0.0028286583, 0.0026029332, -0.06693403, + -0.0072381413, -0.02419584, 0.0037009937, -0.0011296495, -0.030310456, + 0.0005999609, 0.009945354, 0.024915686, 0.038672045, 0.023009432, + -0.05093051, -0.09101409, -0.06686167, -0.06211481, -0.022600474, + 0.0024902367, -0.0067171026, -0.016769778, -0.0032803628, 0.008623083, + -0.0016110836, 0.014925073, 0.033811048, 0.012829449, -0.052575592, + 0.03728281, -0.0051201633, 0.021238064, -0.031086605, -0.0043402985, + 0.25054494, -0.028396713, -0.010725523, -0.023445267, 0.021751683, + 0.003130971, 0.012270548, -0.014836475, 0.014466885, 0.0021478906, + -0.088153616, 0.015356315, 0.039639276, -0.040283255, 0.014923924, + -0.029303031, -0.004475201, 0.047299225, -0.06726505, -0.012491676, + -0.002080582, 0.03506341, 0.04146718, 0.0074114986, -0.009582633, + 0.006623749, -0.05454898, -0.0034387247, -0.014780821, 0.03698328, + -0.034441486, 0.22032379, -0.021792699, 0.008001907, 0.0341019, + 0.0025784625, -0.0071689896, -0.014151225, -0.050547987, -0.030372301, + -0.03288388, -0.042987447, -0.0033656165, 0.021502906, -0.0030374376, + -0.024413, -0.03506603, 0.030891743, 0.0069994032, 0.004887366, + 0.030707076, 0.0065348223, 0.0008240555, 0.016809713, -0.024151647, + -0.0040458064, 0.044567548, 0.03908212, -0.061782435, 0.0026002831, + 0.02003242, -0.0115451, -0.045135964, -0.0663542, 0.005240975, + 0.0010777109, -0.009276002, -0.011919081, 0.027654119, -0.013634551, + 0.0129154315, -0.013195512, 0.040571272, -0.099290505, -0.04656693, + -0.20979854, 0.04181004, -0.06769166, -0.04121946, 0.018260444, + 0.005617506, -0.15694222, 0.032472793, 0.014189598, 0.008345103, + -0.041118417, 0.0074527003, -0.011494104, 0.004906474, -0.0027100458, + 0.0009994594, -0.041061398, -0.045274835, 0.015187793, 0.034275036, + -0.02813289, 0.02098019, -0.015043198, 0.03716667, 0.013542642, + -0.008104835, 0.014965486, -0.027978558, 0.021163663, -0.006942737, + -0.034969814, -0.038961593, 0.016802894, -0.012318175, -0.047821593, + -0.01933216, 0.0064432374, 0.0322358, -0.020437784, 0.02659447, + -0.03022022, -0.019443465, -0.094178624, -0.017009107, 0.03860671, + -0.00023695534, -0.033089854, -0.014677462, 0.012356399, 0.018040499, + -0.19360113, 0.011710817, -0.019871691, 0.020124774, 0.024948105, + -0.024573864, -0.028349072, -0.041731454, 0.032646254, 0.015098165, + 0.0018447012, -0.01129152, -0.02481687, 0.036048, 0.20797811, + 0.04317757, -0.017239755, 0.033431605, -0.011384422, 0.0099174455, + -0.19739223, -0.008901062, -0.056426458, 0.014591864, 0.0044474266, + 0.008209326, -0.012241819, 0.042553645, -0.021552453, 0.008611734, + 0.048944127, -0.013872049, 0.019179989, 0.008183264, -0.053223096, + -0.027327595, -0.0033542586, -0.012353858, -0.010941209, 0.010212546, + 0.029714134, -0.011385593, -0.0013529479, 0.014505385, 0.01148663, + 0.012613636, -0.0074430504, -0.014008101, -0.028146038, -0.042778917, + 0.02143165, 0.016812252, -0.06286215, 0.02085532, -0.035128277, + 0.0241282, 0.01397908, -0.034992814, 0.013153766, 0.0068236147, + 0.013594184, -0.06859393, 0.051849406, 0.029675119, -0.0037699796, + -0.00013216575, 0.02916647, 0.031859048, -0.015006051, 0.0001439966, + 0.0026584554, -0.0046948595, 0.039754897, 0.034873154, 0.0012870439, + -0.011516082, -0.074089095, 0.019241294, 0.00076195505, 0.0027334464, + -0.036206458, -0.0011109262, -0.048838835, -0.00204685, 0.01179991, + 0.0068136845, 0.009852495, 0.017308988, 0.03159545, 0.058803353, + 0.038116634, 0.008836143, -0.010460637, -0.010994099, -0.0012926942, + -0.048370842, 0.037696566, -0.08093239, -0.0041242186, -0.005866373, + 0.02048607, -0.12549639, 0.011705739, -0.0021425558, 0.018078592, + -0.038228087, 0.004245971, -0.11560507, 0.01381203, -0.02734893, + 0.029595908, 0.101082794, -0.042842254, -0.08397301, -0.1546237, + -0.037624564, 0.0069606556, -0.008020479, 0.0043268185, 0.016504597, + 0.044554908, -0.05240209, 0.020519335, -0.050994243, -0.019861093, + 0.022066997, -0.0188911, 0.012919696, -0.022872692, -0.012961374, + -0.020071112, 0.0074553066, 0.06402014, 0.04065211, 0.014139608, + 0.06221546, -0.02574805, 0.010114545, 0.020619567, 0.032264773, + -0.009148828, 0.022849955, -0.016593235, -0.032456633, -0.006737303, + -0.0037493357, -0.065912865, -0.0024181998, -0.0037048983, -0.019304488, + 0.009085037, 0.057529982, -0.0378338, 0.01792042, 0.01885012, + -0.0043600523, 0.017382368, -0.027792772, 0.006749118, -0.0053297766, + -0.011628587, 0.015736576, 0.024801968, 0.011543913, -0.01159873, + -0.000759976, 0.042183932, 0.05764386, -0.06422416, 0.010577374, + -0.02261249, 0.02282793, -0.05141075, -0.11287078, -0.020670144, + -0.020953743, 0.030885734, -0.012597732, -0.121472664, 0.028447151, + -0.0415664, -0.010061708, 0.06441338, 0.0271207, -0.0016693748, + -0.0045694243, 0.020342957, 0.030528478, 0.007129517, -0.0029600926, + -0.007061397, 0.06036486, -0.004736018, -0.0090786135, -0.07601116, + -0.039556272, -0.051269546, -0.019473685, 0.029858151, 0.028464556, + 0.012424476, 0.013742254, 0.008742927, 0.006065751, -0.009892663, + 0.07699999, -0.003443197, 0.0027863425, 0.029197995, 0.0211013, + 0.01098379, 0.016684106, -0.028917195, 0.008122037, 0.0265241, + -0.04461223 + ] +} diff --git a/test/clip-keywords.test.ts b/test/clip-keywords.test.ts new file mode 100644 index 0000000..13a6a5d --- /dev/null +++ b/test/clip-keywords.test.ts @@ -0,0 +1,129 @@ +// `clipKeywords`: the keywords quak writes for a file in place of its CLIP +// embedding. +// +// Ente's ML data for a file holds a CLIP embedding, 512 numbers its image +// model gave the photo. Ente's photos app puts a file in a category of its +// Discover section when the cosine similarity of that embedding and the text +// embedding of the category's query is at least the category's minimum score, +// and keeps videos out of some categories. `clipKeywords` applies that rule +// and returns the titles of the categories a file falls in, highest score +// first; `quak backup` and `quak backup-metadata` write them as `keywords`. + +import { readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; + +import { DISCOVER_CATEGORIES } from "../src/clip-keyword-vectors.js"; +import { clipKeywords } from "../src/clip-keywords.js"; + +// 512 numbers: the ones given, then zeros. +const vector = (...start: number[]): number[] => [ + ...start, + ...new Array<number>(512 - start.length).fill(0), +]; + +const withEmbedding = (embedding: unknown) => ({ + face: { faces: [] }, + clip: { embedding }, +}); + +// Two made-up categories, each with a text embedding along one axis. The +// embedding [3, 4] has a cosine similarity of 3/5 with "Across" and 4/5 with +// "Up". Videos are kept out of "Across". +const ACROSS = { + title: "Across", + query: "across", + minScore: 0.6, + showVideo: false, + textEmbedding: vector(1), +}; +const UP = { + title: "Up", + query: "up", + minScore: 0.5, + showVideo: true, + textEmbedding: vector(0, 1), +}; + +describe("clipKeywords", () => { + it("gives each category whose minimum score the similarity reaches, highest score first", () => { + expect( + clipKeywords(withEmbedding(vector(3, 4)), false, [ACROSS, UP]), + ).toEqual(["Up", "Across"]); + }); + + it("leaves out a category when the similarity is just below its minimum score", () => { + const stricter = { ...ACROSS, minScore: 0.6 + 1e-9 }; + + expect( + clipKeywords(withEmbedding(vector(3, 4)), false, [stricter, UP]), + ).toEqual(["Up"]); + }); + + it("keeps a video out of a category that leaves out videos", () => { + expect( + clipKeywords(withEmbedding(vector(3, 4)), true, [ACROSS, UP]), + ).toEqual(["Up"]); + }); + + it("gives an empty list for an embedding that falls in no category", () => { + expect( + clipKeywords(withEmbedding(vector(0, 0, 1)), false, [ACROSS, UP]), + ).toEqual([]); + }); + + it.each([ + ["no CLIP embedding", { face: { faces: [] } }], + ["a CLIP part that is not an object", { clip: null }], + ["a CLIP part without an embedding", { clip: {} }], + ["an embedding of 511 numbers", withEmbedding(vector(3, 4).slice(1))], + ["an embedding with NaN in it", withEmbedding(vector(3, NaN))], + ["an embedding with Infinity in it", withEmbedding(vector(Infinity))], + ["an embedding of strings", withEmbedding(vector(3, 4).map(String))], + ["an embedding of zeros", withEmbedding(vector())], + ])("gives no keywords for ML data with %s", (_, mlData) => { + expect(clipKeywords(mlData, false, [ACROSS, UP])).toBeUndefined(); + }); +}); + +// The CLIP embedding Ente's image model gives a public-domain NOAA photo of a +// sunset, made by bin/clip-keyword-vectors from the photo at `photo`. +const SUNSET: { photo: string; sha256: string; embedding: number[] } = + JSON.parse( + readFileSync( + new URL("clip-keywords-sunset.json", import.meta.url), + "utf-8", + ), + ); + +describe("Ente's Discover categories", () => { + it("are the 16 of Ente's photos app, each with a text embedding of norm 1", () => { + expect(DISCOVER_CATEGORIES.map((c) => c.title)).toEqual([ + "Identity", + "Screenshots", + "QR Codes", + "Receipts", + "Notes", + "Memes", + "Visiting Cards", + "Babies", + "Pets", + "Selfies", + "Wallpapers", + "Food", + "Celebrations", + "Sunset", + "Hills", + "Greenery", + ]); + for (const category of DISCOVER_CATEGORIES) { + expect(category.textEmbedding).toHaveLength(512); + expect(Math.hypot(...category.textEmbedding)).toBeCloseTo(1, 5); + } + }); + + it("give a photo of a sunset the keyword Sunset and no other", () => { + expect( + clipKeywords({ clip: { embedding: SUNSET.embedding } }, false), + ).toEqual(["Sunset"]); + }); +}); diff --git a/test/library/content-library.test.ts b/test/library/content-library.test.ts index 48a1094..92e0985 100644 --- a/test/library/content-library.test.ts +++ b/test/library/content-library.test.ts @@ -253,6 +253,12 @@ describe("Library content wiring", () => { await expect( lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }), ).rejects.toThrow(/content cache/i); + await expect(lib.originalOf(file(1, 1))).rejects.toThrow( + /content cache/i, + ); + await expect(lib.thumbnailOf(file(1, 1))).rejects.toThrow( + /content cache/i, + ); await lib.close(); }); }); diff --git a/test/library/content.test.ts b/test/library/content.test.ts index b461742..1809116 100644 --- a/test/library/content.test.ts +++ b/test/library/content.test.ts @@ -375,6 +375,25 @@ describe("ContentCache.original / thumbnail", () => { await cache.open(); await expect(cache.original(999)).rejects.toThrow(/unknown file/i); }); + + // `getFile` knows no file here, so the cache can fetch only the file it is + // given. + it("originalOf and thumbnailOf fetch and cache the file they are given", async () => { + const { cache, source } = buildCache({ files: [] }); + await cache.open(); + + const original = await cache.originalOf(file(5)); + const thumbnail = await cache.thumbnailOf(file(5)); + + expect(source.originalCalls).toEqual([5]); + expect(source.thumbnailCalls).toEqual([5]); + expect(original.path).toBe(join(cacheDir, "originals", "5.jpg")); + expect(thumbnail.path).toBe(join(cacheDir, "thumbnails", "5.jpg")); + expect(cache.pathsFor(5)).toEqual({ + originalPath: original.path, + thumbnailPath: thumbnail.path, + }); + }); }); describe("ContentCache.ensureThumbnails", () => { diff --git a/test/thumbnails/thumbnails.test.ts b/test/thumbnails/thumbnails.test.ts index bf2be5e..9db0e48 100644 --- a/test/thumbnails/thumbnails.test.ts +++ b/test/thumbnails/thumbnails.test.ts @@ -6,10 +6,12 @@ * working thumbnails, others return 404 or empty bodies. The tests * verify that the detection and repair logic handles each case correctly. * - * As of issue #52 both helpers take an open `Library` for enumeration and the - * `Client` for the API operations that stay unchanged (the thumbnail existence - * check, and the encrypt-and-upload path). `fixMissingThumbnails` reads each - * original through the library's content cache (`photo.original()`). + * `listMissingThumbnails` takes an open `Library` to enumerate the files. Both + * helpers take the `Client` for the API operations: the thumbnail existence + * check, and the encrypt-and-upload path. `fixMissingThumbnails` takes the + * record of each file, which these tests look up with `cachedFilesByID`. It + * reads each original through the library's content cache, by + * `lib.originalOf(file)`. * * `fixMissingThumbnails` is the most complex function in quak: it * downloads the original file, generates a JPEG thumbnail with jpeg-js, @@ -41,6 +43,7 @@ import { import { SRP, SrpServer } from "fast-srp-hap"; import { Client } from "../../src/client.js"; import { Library } from "../../src/library/index.js"; +import { cachedFilesByID } from "../../src/cli-read.js"; import { listMissingThumbnails, fixMissingThumbnails, @@ -656,7 +659,12 @@ describe("fixMissingThumbnails", () => { const client = await login(buildThumbFetch(fixMock)); const lib = await openLib(client); - const results = await fixMissingThumbnails(lib, client, [101]); + const results = await fixMissingThumbnails( + lib, + client, + [101], + cachedFilesByID(lib, [101]), + ); lib.close(); expect(results.length).toBe(1); @@ -693,7 +701,12 @@ describe("fixMissingThumbnails", () => { const client = await login(buildThumbFetch(fixMock)); const lib = await openLib(client); - const results = await fixMissingThumbnails(lib, client, [999]); + const results = await fixMissingThumbnails( + lib, + client, + [999], + cachedFilesByID(lib, [999]), + ); lib.close(); expect(results.length).toBe(1); @@ -715,6 +728,7 @@ describe("fixMissingThumbnails", () => { lib, client, [101, 102], + cachedFilesByID(lib, [101, 102]), (msg, level) => { if (level === "error") errors.push(msg); }, @@ -745,6 +759,7 @@ describe("fixMissingThumbnails", () => { lib, client, [101, 103, 104], + cachedFilesByID(lib, [101, 103, 104]), ); lib.close(); @@ -779,7 +794,12 @@ describe("fixMissingThumbnails", () => { const client = await login(counted.fetch); const lib = await openLib(client); - const results = await fixMissingThumbnails(lib, client, [101]); + const results = await fixMissingThumbnails( + lib, + client, + [101], + cachedFilesByID(lib, [101]), + ); lib.close(); expect(results[0]!.status).toBe("skipped"); @@ -800,7 +820,12 @@ describe("fixMissingThumbnails", () => { const client = await login(counted.fetch); const lib = await openLib(client); - const results = await fixMissingThumbnails(lib, client, [101]); + const results = await fixMissingThumbnails( + lib, + client, + [101], + cachedFilesByID(lib, [101]), + ); lib.close(); expect(results[0]!.status).toBe("skipped"); @@ -841,7 +866,12 @@ describe("fixMissingThumbnails", () => { const client = await login(buildThumbFetch(fixMock)); const lib = await openLib(client); - const results = await fixMissingThumbnails(lib, client, [101]); + const results = await fixMissingThumbnails( + lib, + client, + [101], + cachedFilesByID(lib, [101]), + ); lib.close(); expect(results[0]!.status).toBe("fixed");