diff --git a/README.md b/README.md index 926ba26..d595f91 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,37 @@ # quak -quak is a WTFPL-licensed TypeScript client library and CLI by -[@sneak](https://sneak.berlin) for the [Ente](https://ente.io) end-to-end -encrypted photo hosting service. It logs in, enumerates collections and files, -and downloads individual images while decrypting them on the way to disk. +quak is a WTFPL-licensed TypeScript library and command-line tool by +[@sneak](https://sneak.berlin) for reading and backing up an +[Ente](https://ente.io) photo account. -quak also includes a resilient backup command that downloads every file in the -account into a deduplicated local directory tree, skipping files that already -exist on disk and continuing past individual download failures instead of -crashing. For each file it persists the basic metadata fields quak keeps (title, -file type, creation and modification time, latitude, longitude, content hash), -its update time, and the private and public magic metadata in full. It keeps any -ML data Ente has for the file, except the CLIP embedding, and writes the file's -keywords in place of the embedding. For an image (for a live photo, its image), -it keeps its original's EXIF and XMP, with its dimensions for a JPEG only. For a -video it keeps none of these three. It runs unattended from cron (see "Running -the backup from cron"). A helper subcommand can detect and regenerate missing -thumbnails, encrypting and uploading them back to the server. +Ente is a photo service that encrypts photos end to end. The server never sees a +photo, its name or its date unencrypted. Only a client that holds the account's +keys can read them. quak is such a client, built for scripts and for jobs that +run unattended. It logs in with the account's email and password, derives the +keys, and keeps a decrypted copy of the account's album and file lists in a +local cache. + +With the command line you can: + +- list the account's albums and the files in them; +- download and decrypt one file, or its thumbnail; +- back up every file in the account into a directory tree that a photo viewer + can open, and keep that backup current from cron; +- write the metadata quak keeps about the account, its albums and its files as + JSON; +- find files whose thumbnails are missing on the server, and repair them. + +The library underneath lets a program of your own read and back up the account. +It keeps the account's album and file lists on disk and answers from that copy +without asking the server. + +A backup run never prompts and never stops on one bad file. It skips files that +are already on disk, records each failure in a file, and tries the failed files +again on the next run. Every download is decrypted on its way to disk. ## Getting Started -The commands below need git, node 22 or later, and yarn. `make install`, which -builds the single binary that "Running the backup from cron" uses, also needs -[nix](https://nixos.org/download). +The commands below need `git`, Node 22 or later and `yarn`. ```bash git clone https://git.eeqj.de/sneak/quak.git @@ -30,43 +39,64 @@ cd quak yarn install yarn build -# Log in (prompts for email, password, and OTP/TOTP if required). +# Log in. quak asks for the email and the password, and for a code when +# the account needs one. yarn quak login -# List the user's collections (albums). +# List the account's collections. Ente calls an album a collection. yarn quak collections # List every file in the account. yarn quak files -# List the files in a collection. +# List the files in one collection. yarn quak files 12345 -# Download and decrypt a single file. +# Download and decrypt one file. yarn quak get 67890 --out ./photo.jpg # Back up every file in the account. yarn quak backup ./my-backup ``` -For library use, the primary surface is the cache-backed `Library`: +`yarn quak` runs the command line that `yarn build` wrote to `dist/`. To run +quak from cron, build it as one binary instead; see "Running the backup from +cron". + +### Using the library + +The library keeps a local copy of the account's album and file lists, the cache. +It brings the cache up to date with the server in rounds called refreshes. + +quak is not published to a package registry: `package.json` marks it `private`, +and `dist/` is not committed. Clone and build it as above, then add the clone to +your own program's `package.json` by its path, for example +`yarn add file:../quak`. The package's `main`, `types` and `exports` fields +point into `dist/`, so the clone must be built first. The package is an ES +module only, so a CommonJS program cannot `require` it. ```ts import { Client, Library } from "quak"; -// Log in once; the client satisfies the library's client interface. +// Log in once. The client holds the account's keys in memory. const client = await Client.login({ email: "you@example.com", password: "your-password", }); -// Open a cache-backed library. On an empty cache this awaits one server -// refresh; on an existing cache it returns immediately and refreshes in the -// background. Later refreshes start `refreshIntervalSeconds` (default 3) after -// the previous one ends. -const lib = await Library.open({ client }); +// Open the library. On an empty cache this waits for one refresh. On an +// existing cache it returns at once and refreshes in the background. +const lib = await Library.open({ + client, + // Without these two, open() also starts downloading every thumbnail in + // the account, and the originals of the favorites album and the latest + // week, into the cache. It fetches machine-learning data after each + // refresh either way. + precacheThumbnails: false, + precacheOriginals: false, +}); -// Default reads answer synchronously from the local cache — no network. +// Default reads answer from the cache and never touch the network. for (const album of lib.albums.list()) { console.log(album.collectionID, album.name); for (const photo of album.photos.list()) { @@ -74,11 +104,11 @@ for (const album of lib.albums.list()) { } } -// Fresh reads await a server round-trip, joining one already running. +// A fresh read waits for a refresh first. const { albums } = await lib.fresh(); console.log(`${albums.list().length} albums as of now`); -// Fetch (and cache) one photo's full-resolution bytes. +// Fetch one photo's original. The bytes land in the cache. const photo = lib.photos.byID({ fileID: 12345 }); if (photo) { const { path } = await photo.original(); @@ -88,46 +118,59 @@ if (photo) { await lib.close(); ``` -The lower-level `Client` (login, session serialization, and the raw -enumeration/download calls) is exported too and documented under Design below. +`Library.open({ client })` with no other options does more than the example asks +for. It refreshes every 3 seconds until `close()`. It also downloads, in the +background, every thumbnail in the account and the originals of the favorites +album and of the latest week (see "Opening a library"). The example turns those +two downloads off. After each refresh that succeeds, the library also fetches +each file's machine-learning data into the cache. That is the faces and the +other data Ente's apps compute for a photo (see "The JSON beside each +original"). No option in `LibraryOptions` turns this off. A `client` without a +`fetchMLData` method fetches none, which is how quak's own read commands open +their library. + +The `Client` under the library handles login, saved sessions and the raw +requests. The "Library reference" below covers both. ## Running the backup from cron -`quak backup` never prompts, so cron can run it. `make install` builds quak as a -single binary and copies it to `~/bin/quak`. Besides what Getting Started needs, -it needs [nix](https://nixos.org/download), even where bun is already installed: -`make install` runs `make build-bin`, which bundles the built `dist/` with bun -run through `nix-shell -p bun`. Log in once with `~/bin/quak`, as the user the -cron job will run as; the session is saved in that user's data directory (see -"Session handling"): +`quak backup` never prompts, so cron can run it. `make install` builds quak as +one binary and copies it to `~/bin/quak`. Besides what Getting Started needs, +`make install` needs `make` and [nix](https://nixos.org/download). It uses bun, +a JavaScript runtime, to bundle the built `dist/` into one executable. It runs +bun through `nix-shell -p bun`. + +Log in once with `~/bin/quak`, as the user the cron job will run as: ```bash ~/bin/quak login ``` +The session is saved in that user's data directory (see "The session file"). Then add the backup to that user's crontab with `crontab -e`. These two lines -back up the account to `~/photos-backup` at 03:30 every night, with `--verify` -on Sundays, and append all output to `~/quak-backup.log`: +back up the account to `~/photos-backup` at 03:30 every night and append all +output to `~/quak-backup.log`. Sunday's run adds `--verify`: ``` 30 3 * * 1-6 $HOME/bin/quak backup $HOME/photos-backup >> $HOME/quak-backup.log 2>&1 30 3 * * 0 $HOME/bin/quak backup --verify $HOME/photos-backup >> $HOME/quak-backup.log 2>&1 ``` -A run with `--verify` does all a plain run does, and also checks each original -already in the backup against the content hash Ente records for it, and replaces -any that do not match (see "Backup layout"). Sunday's run is the `--verify` one, -not a second job that night, because a backup that starts while another backup -of the same directory is running exits 2 without backing anything up. +A run with `--verify` does all a plain run does. It also hashes each original +already in the backup and compares the result with the content hash. The content +hash is the hash of the file's bytes that Ente recorded at upload. It fetches a +new copy of each original that does not match, and replaces the original once +the copy matches. Sunday's run is the `--verify` run, rather than a second job +that night, because two backups of one directory cannot run at once. The second +one exits 2 without doing anything. Cron runs the job with a short `PATH`, usually `/usr/bin:/bin`, so the crontab -names quak by its full path. To find the saved session, quak needs the same -`HOME` as when you logged in, which cron sets from the password file, and on -Linux the same `XDG_DATA_HOME`: quak looks for the session in -`$XDG_DATA_HOME/quak`, or in `~/.local/share/quak` when that is not set. Cron -does not set `XDG_DATA_HOME`, so if your login shell does, set it at the top of -the crontab too. Cron does not expand variables in such a line, so give the full -path: +names quak by its full path. quak finds the saved session through `HOME`, which +cron sets from `/etc/passwd`. On Linux it also reads `XDG_DATA_HOME`: the +session is in `$XDG_DATA_HOME/quak`, or in `~/.local/share/quak` when that +variable is not set. Cron does not set `XDG_DATA_HOME`. If your login shell sets +it, set it at the top of the crontab too, as a full path, because cron does not +expand variables there: ``` XDG_DATA_HOME=/home/you/.data @@ -136,747 +179,554 @@ XDG_DATA_HOME=/home/you/.data On macOS the session is in `~/Library/Application Support/quak`, and only `HOME` matters. +A backup that ends with failures exits 1, and its last line says how many files +and albums failed. The next run fetches each missing original again and rebuilds +the album links. The exit codes are listed under "Exit codes", and the backup's +files under "The backup directory". + +## Command reference + +Every command has the form `quak [options] [arguments]`. The options +below belong to quak itself, and go before or after the command's name. + +### Global options + +- `--cache-dir `: where the local cache lives. Without it, each account + gets its own directory under the per-user cache directory (see "The cache"). +- `-v`, `--verbose`: log more on stderr. `-v` adds debug lines, and `-vv` adds + trace lines too (see "What quak prints"). +- `-q`, `--quiet`: print nothing on stderr but the reason a failing command + exits. `-q` with `-v` is refused with one line, + `error: option '-q, --quiet' cannot be used with option '-v, --verbose'`, and + exit status 1. +- `-V`, `--version`: print the version and exit (see "Version"). +- `-h`, `--help`: print the usage and exit. + +### The commands + +``` +quak login +quak whoami +quak logout +quak collections [--json] +quak files [collectionID] [--json] +quak get [--out ] +quak get-thumb [--out ] +quak backup [--json] [--verify] +quak backup-metadata [--exif] +quak helper list-missing-thumbnails [--json] +quak helper fix-missing-thumbnails [--file ...] [--json] +``` + +Every command but `login` and `logout` needs a saved session. Every command but +`login`, `whoami` and `logout` opens the library. How each one gets current data +from the server is described under "How a command gets current data". + +#### `quak login` + +Logs in to an account and saves the session. It reads the email from +`QUAK_EMAIL` and the password from `QUAK_PASSWORD`. For each one that is not +set, it asks on the terminal. The password shows as `*` characters while it is +typed. + +When the account has a second factor, quak asks for the code. It asks +`TOTP code: ` for a time-based code from an authenticator app. An account whose +only second factor is a passkey cannot log in with quak. When the account has +email verification on, quak asks `Email verification code: ` for the code sent +by email. The password is then used only to unlock the keys. After the email +code quak expects the login to complete. When the server asks for a second +factor at that point, login fails. + +On success it logs `Logged in as (user )` and +`Session saved to `. On failure it logs `Login failed: `, and the +exit status is 1. `login` is the only command that prompts, and its prompts go +to stdout. + +#### `quak whoami` + +Prints the saved session's account on stdout as one line of JSON: +`{"email":"you@example.com","userID":12345}`. It reads the session file only and +sends no request, so it succeeds even when the server no longer accepts the +session. + +#### `quak logout` + +Ends the session on the server and deletes the session file. Ending it on the +server makes the saved token stop working everywhere, even in a copy of the +file. With no session file, the command logs `No session found.` and exits 0. + +When the server call fails, or the session file is corrupt, the file is still +deleted. The command then logs +`Could not end the session on the server: ` and exits 1. + +`logout` does not delete the cache. It warns that the cache directory still +holds decrypted data. It knows the directory from `--cache-dir` or from a +session file it can read. Delete that directory yourself if you want the data +gone. + +#### `quak collections [--json]` + +Lists every collection in the account, in the server's order, after one request +to the server. + +Without `--json`, each line holds the collection's ID, its type and its name, +separated by tabs. A collection another account owns gets ` (shared)` after its +name. A collection's type is one of `album`, `folder`, `favorites`, +`uncategorized` and `unknown`. + +With `--json`, the output is a JSON array with one object per collection. Each +object holds `id`, `name`, `type`, `ownerID`, `isShared` and `updationTime`. +Update times are in microseconds, as Ente records them. + +#### `quak files [collectionID] [--json]` + +With a collection ID, lists the files in that collection, in the server's order. +Without one, lists every file in the account once, collection by collection. A +file that is in several collections is listed under the first collection that +holds it. + +Without `--json`, each line holds the file's ID, its type and its title, +separated by tabs. A file's type is one of `image`, `video`, `livePhoto` and +`unknown`. With `--json`, each object holds `id`, `title`, `fileType`, +`creationTime` in microseconds and `collectionID`. + +A collection ID that does not exist gives `Collection not found` and exit +status 1. + +#### `quak get [--out ]` + +Downloads and decrypts one file's original, the full-size file as uploaded. +Without `--out`, it writes the file to the working directory under the file's +title, with unsafe characters replaced (see "File names from the server"). With +`--out`, it writes to that path as given. It logs ` bytes -> `. The +original is kept in the cache as well. + +A file ID that no collection holds gives `File not found`, and the exit +status is 1. The option `--collection ` is accepted and ignored. + +A live photo is a still image and a short video that Ente stores as one file. +`get` writes both, each named after the title with its own extension, as Ente's +own clients name them: `IMG_0001.heic` and `IMG_0001.mov`. With `--out `, +the image goes to `` and the video beside it, with the same name and the +video's extension. A `` that already ends in the video's extension is +refused, because both files would land on it. + +#### `quak get-thumb [--out ]` + +Downloads and decrypts one file's thumbnail, the small JPEG Ente shows in a +grid. It works as `get` does. Without `--out`, the name is `thumb_` followed by +the title. + +#### `quak backup [--json] [--verify]` + +Backs up every file in the account to ``. The layout is described under +"The backup directory". It downloads only the originals that are not yet in the +backup, and it goes on past a file that fails. A file is downloaded once, +however many collections hold it. An original the cache already holds is copied +from there instead. + +Progress goes to stderr: one line per original fetched, and a `FAILED` line for +each original or album that failed. The run ends with a summary of counts and, +when any file failed, a `Failed files:` list of every failed file. With +`--json`, the summary and the list are replaced by one JSON object on stdout. +Its fields are: + +- `totalFiles`: how many distinct files the run covered, however many + collections hold each; +- `downloaded`: how many originals this run fetched or copied from the cache; +- `skipped`: how many originals were already in the backup and left alone; +- `verified`, `mismatched` and `unchecked`: the counts a `--verify` run makes + (see "`--verify`"), all 0 without it; +- `failed`: how many files are in `failures.json` after this run; +- `errors`: this run's failures, one object per file, with `fileID`, `title`, + `collection` and `error`; +- `undecryptableAlbums`: the IDs of the collections quak could not decrypt; +- `failedAlbums`: the IDs of the collections whose directory or JSON could not + be written. + +With `--verify`, the run also hashes each original already in the backup and +replaces any whose bytes do not match the content hash Ente records. "The backup +directory" describes what it checks and what it changes. + +A backup refreshes the library as it starts and not again while it runs. Each +request it sends retries for longer than other commands' requests do (see +"Retries and timeouts"). + +The exit code is 0 when every original is in the backup and nothing failed. It +is 1 when files or albums failed, 2 when another backup of `` is running, +and 3 when there is no usable session (see "Exit codes"). + +#### `quak backup-metadata [--exif]` + +Writes the metadata quak keeps about the account, its collections and its files +as JSON files under ``. The layout is described under "The metadata dump +directory". Photo content is not written. With `--exif`, or its alias `--all`, +the command also fetches each file's original and records the image's own +metadata from it (see "The metadata dump directory"). + +Ente's apps compute machine-learning data for each photo, such as the faces in +it (see "The JSON beside each original"). The command fetches it in requests of +up to 200 files. When a request fails, each of its files is written with the +reason in `mlDataError` instead of `mlData`, and the dump goes on. A file whose +data does not decrypt, decompress or parse also gets `mlDataError`. When any +file was written with `mlDataError`, the last line says how many and the exit +status is 1. + +A collection or file that quak cannot decrypt is left out of the dump without a +word, and does not change the exit status. + +#### `quak helper list-missing-thumbnails [--json]` + +Finds the files whose thumbnail is missing on the server. It downloads every +thumbnail in the account, one collection at a time, newest collection first, so +on a large account it takes a while. A thumbnail counts as missing when the +server answers with an empty body or with HTTP 404. Any other failure is logged +as `[] Could not check : <reason> (not reported as missing)`, +and the file is left out of the list. The command exits 0 even when some files +could not be checked. + +The server lets only a file's owner replace its thumbnail. A file another +account owns, in a collection shared with you, is skipped and not checked. + +Without `--json`, the command logs `<n> file(s) with missing thumbnails:` and +prints one line per file on stdout. Each line holds the file's ID, its title, +its collection and the reason, separated by tabs. With nothing missing it logs +`No missing thumbnails found.` With `--json`, the output is a JSON array; each +object holds `fileID`, `title`, `collection` and `reason`. + +#### `quak helper fix-missing-thumbnails [--file <id>...] [--json]` + +Makes a new thumbnail for each file that lacks one, and uploads it. Without +`--file`, it first finds the files as `list-missing-thumbnails` does. With +`--file`, it works on the files named and checks nothing else. + +For each file, the helper: + +1. downloads the original; +2. makes a JPEG thumbnail from it; +3. encrypts the thumbnail under the file's own key; +4. uploads it. + +The server refuses a thumbnail larger than the one it already records for the +file. The helper therefore tries five encodings in turn and uploads the first +that fits. They run from 720 pixels on the long side at JPEG quality 50 down to +160 pixels at quality 20. + +Each file ends in one of three states: + +- `fixed`: a thumbnail was uploaded. +- `skipped`: the helper cannot make a thumbnail for the file, or the server + would refuse it. This covers: + - a file another account owns; + - a file that is not an image; + - an image that is not a JPEG; + - a file whose recorded thumbnail size is 0 or unknown; + - a file for which even the smallest encoding is too large. +- `failed`: the download, the encoding or the upload failed, or an ID given with + `--file` is held by no collection. + +Without `--json`, the command ends with counts of fixed, skipped and failed +files. A `Skipped:` list and a `Failed files:` list follow when they have +entries. With `--json`, one JSON array on stdout replaces them; each object +holds `fileID`, `title`, `collection`, `status` and, for a skipped or failed +file, `reason`. When any file failed, the exit status is 1. A skipped file does +not change it. When nothing needs fixing, the command logs +`No missing thumbnails found.` and exits 0. With `--json` it then writes nothing +on stdout. + +quak refuses the whole command, before any request, when one `--file` ID is +invalid (see "IDs"). + +### How a command gets current data + +Every command that reads the account answers for the account as it is now, not +for whatever the cache last held. Each one gets current data this way: + +- `collections` asks for the collection list alone, in one request. It fetches + no collection's files and changes nothing in the cache. +- `files` with a collection ID asks for the same list, then for that one + collection's files. It changes nothing in the cache either. +- `get`, `get-thumb` and `fix-missing-thumbnails --file` look the file up in the + cache after a refresh. On an empty cache, they ask for the collection list + instead, then for one collection's files at a time, and stop at the collection + that holds the file. `fix-missing-thumbnails --file` stops once it has found + every file named. These searches store nothing in the cache. The original or + thumbnail fetched is kept in the cache. +- `files` without a collection ID, `backup`, `backup-metadata` and the two + thumbnail helpers without `--file` wait for a refresh. + +A refresh asks the server which collections changed since the last refresh and +fetches the files of each one, one collection after another. On an empty cache +that is every collection. On a large account this takes a while. Nothing is +stored until every collection is in. Meanwhile the command logs a line for each +collection fetched, such as `Fetched the files of 3 of 120 collections`. + +When a refresh or a search fails, the command logs the error on one line and +exits 1, or 3 when the server no longer accepts the saved session. A collection +or file that quak cannot decrypt does not stop a refresh or a search: the +command leaves it out and answers from what did decrypt (see "Default reads and +fresh reads"). `backup` counts it as failed (see "What quak cannot decrypt"). + +### IDs + +A collection ID or file ID given to `files`, `get`, `get-thumb` or +`fix-missing-thumbnails --file` is written in decimal digits and is at +most 9007199254740991. Anything else, such as `0x10`, `1e3` or `-1`, is refused +before any request is sent. The command logs `Invalid file ID: <arg>` or +`Invalid collection ID: <arg>` and exits 1. + +### What quak prints + +stdout holds only a command's output: the listing, the JSON, or nothing. The +prompts of `login` go to stdout too. Every other line goes to stderr as a log +line at one of six levels. From the quietest, the levels are fatal, error, +warning, info, debug and trace. A run prints the lines at its own level and at +every quieter one. The level is info unless `-v`, `-vv` or `-q` says otherwise. + +The one exception is a command line quak cannot parse, such as a missing +argument, an unknown option or `-q` with `-v`. quak refuses it with one +`error: …` line that is not a log line, and exits 1. `-q` does not hide that +line. `quak` with no command prints the usage on stderr and exits 1. + +`yarn quak` prints yarn's own lines on stdout as well. To pipe a command's +output, run `yarn --silent quak files --json` or +`node dist/bin/quak.js files --json` instead. + +- Fatal: why a command exits non-zero. A command that runs and exits non-zero + logs at least one fatal line. A command that exits 1 because files or albums + failed logs it last, saying how many failed and where to look. +- Error: something that failed without ending the run. These lines are: + - the `FAILED` lines of `backup` and `fix-missing-thumbnails`; + - their `Failed files:` lists; + - the `ML data` lines of `backup-metadata`; + - `[<collection>] Could not check <title>` from the thumbnail helpers. +- Warning: something that did not fail but needs your attention. The only + warning is the one `logout` prints: the cache directory still holds decrypted + data. +- Info: everything else quak prints by default, such as a refresh's progress, + each original a backup fetches, and the summaries. +- Debug, with `-v`: each retry of a request, with why the attempt failed and how + long quak waits. Also the start and end of each refresh and of each fetch of + machine-learning data, with the error of one that failed. +- Trace, with `-vv`: each request quak sends, as its method, its URL and the + HTTP status of the answer or the error it failed with. The URL of a thumbnail + upload is logged without its query, which holds the upload's signature. + +`-q` lowers the level to fatal: a run that exits 0 prints nothing on stderr, and +one that exits non-zero prints only its fatal lines. The prompts of `login` are +not log lines, and `-q` leaves them alone. + +With `--json`, a command's JSON on stdout takes the place of its summary on +stderr. For `backup`, that is the `--- Backup complete ---` line, the counts and +the `Failed files:` list. For `list-missing-thumbnails`, it is +`No missing thumbnails found.` or `<n> file(s) with missing thumbnails:`. For +`fix-missing-thumbnails`, it is the `--- Done ---` line, the counts, the +`Skipped:` list and the `Failed files:` list. Every other line is logged as +without `--json`. + ### Exit codes -| Code | Meaning | -| ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `0` | The backup is complete: every original is at its save path, and no file failed. | -| `1` | The run finished with files in `failures.json` or albums it could not decrypt or write (see "Backup layout"), and its last line says how many, or it stopped on another error, printed as one line `quak: <message>`. The next run tries again. | -| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. | -| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. | +- `0`: the command succeeded. For `backup`, every original is in the backup and + no file or album failed. +- `1`: the command failed. For `backup`, the run finished with files in + `failures.json` or with albums it could not decrypt or write, and its last + line says how many. The next backup run tries again. An unexpected error is + printed as one line, `quak: <message>` (see "Error messages"). +- `2`: another backup of the same directory is running. This run sent no request + and changed nothing. Only `backup` exits 2. +- `3`: there is no usable session. None is saved, the saved one is corrupt, or + the server no longer accepts it. Run `quak login` as the user the command runs + as. -`<message>` is the error's message, followed by the message of each error that -caused it. Under Node, as `yarn quak` runs quak, `fetch` reports every failure -below HTTP as just `fetch failed` and gives the reason as its cause, so a run -that could not reach the server prints, for example, +### Error messages + +An unexpected error is printed as one line, `quak: <message>`, with no stack +trace. `<message>` is the error's own message, followed by the message of each +error that caused it, separated by `: `. Some lines a command logs itself, such +as `File <id> not found` and `Login failed: <reason>`, have no prefix. + +`yarn quak` runs quak under Node. Node's `fetch` reports every network failure +as `fetch failed`, with the reason as its cause. A run that could not reach the +server therefore prints, for example, `quak: fetch failed: connect ECONNREFUSED 1.2.3.4:443`. When the server has -several addresses and none of them could be reached, the line gives the reason -for each, separated by `, `. A file whose download failed that way is in -`failures.json` with that error, as a transient failure. So is each file with no -ML data cached when the ML data fetch failed that way, its error starting -`ML data: `. The `~/bin/quak` that `make install` builds runs under bun, and -prints bun's own wording for such a failure, with no reason after it, such as -`quak: Unable to connect. Is the computer able to access the url?` for a refused -connection. It records such a failure in `failures.json` as transient too. +several addresses and none could be reached, the line gives the reason for each +one, separated by `, `. -After a `1`, the next run fetches each missing original again, fetches the ML -data that is not cached, and rebuilds the album links. An album quak could not -decrypt, or a file in `failures.json` because quak could not decrypt it, keeps -the exit code at `1` until the server sends a copy that decrypts, or deletes it -(see "Backup layout"). An original that a `--verify` run could not read, or -could not replace with a download that matches, stays at its save path as it -was. Only a later `--verify` run checks it again: a run without `--verify` -leaves it as it is and takes the file out of `failures.json`, so that run can -exit 0. +The `~/bin/quak` that `make install` builds runs under bun. Bun gives its own +wording for such a failure, with no reason after it. A refused connection, for +example, prints +`quak: Unable to connect. Is the computer able to access the url?`. -## Examples +A file whose download failed that way goes into the backup's `failures.json` as +a transient failure, with that error. So does each file whose machine-learning +data could not be fetched that way, with an error starting `ML data: `. -`examples/download-albums.ts` downloads every album's photos and their metadata -into a directory, `photos` in the working directory unless you name another. The -build compiles it; run it after `yarn install`: +## Files quak writes -```bash -yarn build -QUAK_EMAIL=… QUAK_PASSWORD=… node dist/examples/download-albums.js [dir] -``` +### The session file -It opens the library with `precacheThumbnails` and `precacheOriginals` off, as -`quak backup` does, so the only file content it fetches is the originals it -saves. It asks on the terminal for a two-factor or email code when the account -requires one, and writes: +`quak login` writes the session to `session.json` in quak's data directory. On +Linux that is `$XDG_DATA_HOME/quak/session.json`, or +`~/.local/share/quak/session.json` when the variable is not set. On macOS it is +`~/Library/Application Support/quak/session.json`. quak creates the file with +mode `0600`, in a directory it creates with mode `0700`. -- each photo's original at its save path under `dir`, as `photo.download()` - writes it: `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, and for a live - photo its image, its video and the `.livephoto.json` file naming them -- beside each original, a JSON file named after it with `.json` added, for - example `2026-03-01.12345.jpg.json`: the photo's record (`photo.record()`) - without its cache paths, and every EXIF tag of the photo (`photo.exif()`) - under `exif` -- `albums/<collectionID>.json` for each album: its `collectionID`, its `name`, - and under `savePaths` the save paths of its photos relative to `dir`, newest - first +The file holds the account's email, its user ID, the session token and the +account's three keys, in base64 and unencrypted. Treat it as you would treat the +password. A missing file means "not logged in". A file that cannot be read back +is reported as corrupt, with the reason. Run `quak logout`, then `quak login`. -A photo in several albums is downloaded once. A second run downloads nothing and -rewrites only the JSON files whose content changed. A failed download stops the -run; running it again carries on, since every photo already saved is skipped. +A command that needs a session exits 3 when the file is missing or corrupt. It +also exits 3 when the server no longer accepts the session, and prints +`quak: the saved session is no longer valid; run "quak login"`. `quak backup` +learns this at its first refresh. By then it has written only its lock. When the +session stops working partway through a backup, each remaining file goes into +`failures.json`. That run exits 1, and the next run stops at its refresh with +status 3. -## Entrypoints +### The cache -This repository adheres to the -[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) -standard: normalized scripts in `script/` are the entrypoints for the -development workflow, and most Makefile targets are thin shims that call them. -The scripts are POSIX sh (not bash) so they run in minimal containers such as -alpine. We provide: +Without `--cache-dir`, each account has its own cache directory: -- `script/bootstrap` — install all dependencies (node/yarn if missing, then - `yarn install --frozen-lockfile`) -- `script/setup` — set up the repo for development after a fresh clone: runs - `script/bootstrap`, then `script/install-precommit` -- `script/projectname` — output the project name (our own extension); used by - `script/docker` for the image tag -- `script/build` — compile the TypeScript sources into `dist/`, stamp the - version into `dist/package.json`, then verify that the entrypoints - `package.json` declares (`main`, `types`, `bin`) are among the files the - compiler wrote, and make the CLI executable (our own extension) -- `script/version` — print the version `script/build` stamps (our own - extension); see Version below -- `script/test` — run the test suite, by building the `test` phase of the - `Dockerfile` (vitest, 90s timeout, verbose rerun on failure); requires docker -- `script/lint` — run eslint and a prettier check, by building the `lint` phase - of the `Dockerfile`; requires docker (see Linting and testing below) -- `script/fmt` — format all files with prettier (writes) -- `script/fmt-check` — check formatting on the host (read-only) -- `script/check` — run all checks: `test`, `lint`, `fmt-check` (our own - extension) -- `script/docker` — build the image, tagged via `script/projectname` -- `script/cibuild` — what CI runs: `script/bootstrap`, then `script/check`, then - the image build -- `script/precommit` — run by the git pre-commit hook (our own extension); runs - `script/lint` and `script/fmt-check` but deliberately not the tests, so the - TDD red-phase commit can land -- `script/install-precommit` — installs the git pre-commit hook (our own - extension); `make hooks` shims to it +- on Linux, `$XDG_CACHE_HOME/quak/<userID>`, or `~/.cache/quak/<userID>` when + the variable is not set; +- on macOS, `~/Library/Caches/quak/<userID>`. -`make hooks` installs the pre-commit hook that runs `script/precommit`. - -### Linting and testing - -Linting and testing are phases of the `Dockerfile`. The `lint` phase copies the -repo into a digest-pinned node image and runs eslint and `prettier --check .`; -the `test` phase does the same with the suite. `script/lint` and `script/test` -each build one phase with `docker build --no-cache --target <phase>`. Neither -script runs the tools on the host: docker is required, and that also works where -the docker daemon is remote and bind mounts are impossible. - -The last stage of the `Dockerfile` compiles the package, and it copies a file -from each phase, so it cannot be built unless lint and the tests pass. The image -build in `script/cibuild` therefore runs lint and the tests a second time, after -`script/check` has run them. - -Every `docker build` in `script/` passes `--no-cache`. On an unchanged tree -Docker would otherwise serve the lint and test steps from cache, nothing would -run, and the build would still exit 0. - -`script/fmt-check` runs prettier on the host. Its verdict matches the `lint` -phase's: prettier is pinned to an exact version, installed from `yarn.lock` -under `--frozen-lockfile` in both places, and reads `.gitignore` as its default -ignore file — which is why `.dockerignore` keeps `.gitignore` in the build -context. - -### Version - -`quak --version` reports the `version` of `dist/package.json`, which -`script/build` writes after compiling; the repo's own `package.json` keeps -`0.0.0`, and that is what the tests, which run from source, report. -`script/version` decides what is written: - -- the `VERSION` environment variable, or the `Dockerfile`'s `VERSION` build arg - (`--build-arg VERSION=...`), when one is given and not empty; -- otherwise, in a checkout with `.git`, `git describe --tags --always`: the tag - on a tagged commit; the tag, the commits since it and the short commit on a - later commit (`v1.2.3-4-gabc1234`); the short commit when no tag is reachable; -- otherwise, as in a source tarball, the version `package.json` declares. - -The build fails if the checkout has `.git` and the version still comes out -empty, `dev` or `unknown`: such a build could not be traced back to its commit. - -`.dockerignore` therefore does not leave out `.git`, so any `docker build .` of -a clone stamps the commit it was built from; a shallow clone stamps a tag only -when the cloned commit itself carries one, and otherwise the short commit. It -leaves out `.git/config`, which holds the clone's remote URL and any credential -in it, so the image carries `.git` without its config; `git describe` does not -need that file. `script/docker` (and so `make docker`) and `script/cibuild` pass -the version they resolve on the host, with `--dirty`, as the build arg, which -takes precedence. The image's `org.opencontainers.image.version` label carries -that build arg only, so a build given none leaves it empty. `make build-bin` -bundles the built `dist/`, so the single binary reports the stamped version too. - -## Rationale - -Ente is one of very few photo services with a credible end-to-end encryption -story. The shipping clients (mobile Flutter, web React, desktop Electron, and Go -CLI) work, but they are slow, buggy, and difficult to script against. The -Flutter app fails to sync reliably. The web app is heavy. The desktop app is the -web app inside a slow Electron wrapper. The Go CLI is the closest thing to a -usable tool, but it is awkward to integrate from anything that is not a shell. -The Go CLI's backup mode crashes entirely when a single file download fails, -which makes it useless as an actual backup tool. - -quak fixes these problems. This repo ships a correct, well-tested implementation -of Ente's cryptographic protocol and API surface, plus a CLI that proves the -library is enough to do real work without a UI. The backup command is resilient -by design: per-file errors are logged and the run continues. - -The longer-term goal of this project is a simple desktop client for Ente, built -on this library in Electron (or a comparable runtime), with two priorities above -everything else: correctness and stability. Performance and simplicity follow -from those. Features will be added only after the protocol layer is correct, the -local cache is reliable, and the UI is responsive on a five-year-old laptop. - -## Development workflow - -All work on quak is test-driven. No exceptions. - -1. Every change starts on a feature branch off `next`, and its pull request - targets `next`. -2. The first commit on the branch is the test suite for what is being added or - changed. Those tests must fail at that commit; the branch is red until the - implementation lands. -3. Subsequent commits add the implementation and any refactors needed to make - the tests pass. -4. A pull request can only be merged into `next` when `make check` is green. - Once it has passed review, the repository manager squash-merges it into - `next`. Only sneak merges `next` into `main`. `main` and `next` are always - green. CI runs `script/cibuild`, which builds the `Dockerfile`: its `lint` - and `test` phases, then the compile, so neither a red branch nor one that - does not compile can pass CI. -5. Tests are the canonical API documentation for this library. Every test file - is commented thoroughly enough that a reader who has never seen quak can - learn how to use it from the tests alone. Comments explain why a behavior - matters, not just what the assertion checks. -6. Test fixtures (cryptographic vectors, recorded HTTP responses, sample files) - are committed alongside their tests. Where possible they are generated by - deterministic helpers in the `test/` tree so any reviewer can reproduce them - by running the helper. -7. `git rebase -i` is allowed on a feature branch before merge to clean up the - test-then-implementation sequence into reviewable commits, but the final - history must still show tests landing before (or with) the matching - implementation. -8. The pre-commit hook installed by `make hooks` runs `script/precommit`, which - runs `script/lint` and `script/fmt-check` but not the tests, and so not the - full `make check`. This is deliberate so the TDD red-phase commit (failing - tests, no implementation yet) can land. CI executes `script/cibuild`, which - runs the tests, so a red branch still cannot reach `next`. - -## Design - -quak is a TypeScript library with a thin CLI wrapper. The library does the work; -the CLI is for humans. - -### Layout +The directory is readable by its owner only (mode `0700`), because it holds +decrypted data, file keys included. ``` -quak/ - src/ - crypto/ libsodium primitives (boxes, secretstreams, KDF, hash) - api/ HTTP client (ApiClient class) - auth/ login flow (SRP + email OTP + TOTP), key unwrap - model/ decrypted Collection, File, Metadata types + decrypt fns - download/ streaming file/thumbnail download + decryption - library/ the cache-backed Library: metadata store, read - surface, records, content cache, precache, ML data - and search, request pools - backup.ts resilient full-account backup with dedup - metadata-backup.ts - backup-metadata: the metadata quak keeps, as JSON - exif.ts EXIF read from an image's bytes with exifreader - mldata-fetch.ts fetch + decrypt per-file ML data - clip-keywords.ts - a file's keywords from its CLIP embedding - clip-keyword-vectors.ts - Ente's Discover categories with their text - embeddings, written by bin/clip-keyword-vectors - filename.ts safe file names from server metadata - errors.ts error types shared across layers, and how an error - is described - retry.ts retry classifier + exponential backoff with jitter - thumbnails.ts detect + regenerate missing thumbnails - client.ts high-level Client class assembled from the above - cli-commands.ts the CLI's commands as functions returning exit codes - cli-log.ts the CLI's log level, and the one function the - commands write stderr lines through - cli-output.ts how the CLI prints a file's title and time - cli-read.ts fresh reads for the CLI's read commands - cli-run.ts run a command, print its error, exit with its code - cli-session.ts read the saved session file back into a Client - index.ts public library exports - bin/ - quak.ts CLI entrypoint (commander.js) - clip-keyword-vectors, clip-keyword-vectors.py - write src/clip-keyword-vectors.ts and a test - fixture (see Source attribution) - examples/ - download-albums.ts - download every album's photos and metadata - test/ unit + integration tests (vitest) - Makefile - Dockerfile lint phase, test phase, compile - package.json - tsconfig.json +<cacheDirectory>/ + metadata.json the decrypted collection and file records, + the refresh cursor, and the list of what + did not decrypt + originals/<fileID>.<ext> cached originals + originals/<fileID>.livephoto.json + which of a live photo's two files is which + thumbnails/<fileID>.jpg cached thumbnails + mldata/ + <fileID>.json one decrypted machine-learning payload per + file + clip.f32, clip.json the packed CLIP index and its list of IDs + fetched.json per-file fetch bookkeeping ``` -`make build` compiles that tree into `dist/`, preserving its shape: the library -lands in `dist/src/`, the examples in `dist/examples/`, and the CLI in -`dist/bin/quak.js`, which is what `package.json` points `main`, `types` and -`bin` at. The compiler's `rootDir` is the repository root rather than `src/`, -because `bin/` is compiled too and `rootDir` has to contain everything that is -compiled. +When `metadata.json` belongs to another account than the one logged in, opening +the library deletes it and `mldata/` and starts from an empty cache. Cached +originals and thumbnails are kept. They are reached only through the records of +the current account, so another account's files are never served. -### Cryptography +A file appears in the cache only once it is complete: each one is written to a +temporary file and renamed into place. Ente's clients record a hash of each +file's bytes at upload, the content hash. Every original whose metadata records +one is hashed as it is written. One that does not match is not stored: the +download fails with an error naming the file, so `quak get` exits 1. The hash is +Ente's: unkeyed BLAKE2b with a 64-byte output, in standard base64. A live photo +arrives as a ZIP of its image and its video and is unpacked as it is written. +Its two files are hashed separately and compared as `<imageHash>:<videoHash>`, +and neither is stored unless both match. quak refuses a live photo ZIP that +holds a second image or video. It also refuses one whose files unpack to more +than 20 times its size plus 16 MiB. An original with no recorded hash, uploaded +by a very old client, is stored unchecked. -All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is -required for `crypto_pwhash` / Argon2id), except the SRP handshake, which uses -`fast-srp-hap`, and the MD5 checksum sent with a thumbnail upload, which uses -Node's built-in `node:crypto`. No hand-rolled crypto. +The cache has a size limit for originals (see "Opening a library"). Thumbnails +have none. A live photo's image and video leave the cache together. -The key hierarchy, derived during login, is: +### The backup directory -1. The user enters their password. -2. Argon2id (`crypto_pwhash`) over the password and a server-issued `kekSalt`, - with server-issued `memLimit` and `opsLimit`, produces a 32-byte Key - Encryption Key (KEK). -3. SRP login: `crypto_kdf_derive_from_key` (BLAKE2b) derives a 32-byte subkey - from the KEK with subkey id 1 and context `loginctx`. Its first 16 bytes are - the SRP password. -4. When SRP completes, the server returns a blob of "key attributes" plus an - encrypted auth token, or first asks for a second factor. quak answers a TOTP - request with the code (`POST /users/two-factor/verify`), after which the - server returns them, and cannot answer a passkey request. When the account - has email MFA on (`isEmailMFAEnabled`), an email OTP replaces SRP and the - server returns them after it. -5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK - yields the 32-byte master key. -6. `crypto_secretbox_open_easy` over the encrypted secret key with the master - key yields the user's X25519 private key. The matching public key is - delivered in cleartext. -7. `crypto_box_seal_open` over the encrypted token with the user's keypair - yields the auth token's bytes. Encoded as URL-safe base64 with padding, they - are the `X-Auth-Token` value for all subsequent calls. - -Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the -master key (for owned collections). Per-file keys are decrypted with -`crypto_secretbox_open_easy` using the collection key. File metadata is a -secretstream blob (single chunk, TAG_FINAL) under the file key. File content is -a chunked `crypto_secretstream_xchacha20poly1305` stream under the file key, -with a 4 MiB plaintext chunk size and a 17-byte authentication overhead per -chunk. Thumbnails use the same secretstream blob format as metadata. - -For upload (thumbnail repair), `encryptBlob` performs the push side: a single -secretstream chunk with TAG_FINAL, returning the header and ciphertext. - -### HTTP API - -Production endpoints: - -- API: `https://api.ente.io` -- File download CDN: `https://files.ente.io/?fileID=<id>` -- Thumbnail CDN: `https://thumbnails.ente.io/?fileID=<id>` - -A custom API endpoint is configurable for self-hosted servers via the -constructor option `apiOrigin`. When set, file downloads route through -`<apiOrigin>/files/download/<id>` instead of the dedicated CDN host. - -The constructor option `onRequest` is called once for each request sent, a -retried request once for each attempt, with its method, its URL and the status -of the answer or the error the request failed with. The URL of a presigned -upload is given without its query, which holds the upload's signature. The CLI -logs each request with `-vv` (see "CLI surface"). - -Required request headers on every authenticated call: - -- `X-Auth-Token`: the decrypted auth token from login. -- `X-Client-Package`: identifies the client. quak uses `berlin.sneak.quak`. - -Endpoints used: - -- `GET /users/srp/attributes?email=<email>`: fetch SRP and KDF parameters. -- `POST /users/srp/create-session`: begin SRP handshake. -- `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or the - encrypted token plus key attributes. -- `POST /users/ott` and `POST /users/verify-email`: email OTP, used instead of - SRP when the SRP attributes have `isEmailMFAEnabled` set. -- `POST /users/two-factor/verify`: TOTP second factor. -- `POST /users/logout`: end the calling token's session (`quak logout`). -- `GET /collections/v2?sinceTime=<usec>`: list collections changed since - microsecond timestamp; pass 0 for a full enumeration. -- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a - collection; paginate while `hasMore` is true. -- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes. -- `POST /files/data/fetch`: fetch encrypted ML data for a batch of files. -- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair). -- `PUT /files/thumbnail`: register an uploaded thumbnail's object key. - -### Retries and timeouts - -Every request in the library goes through one policy, in `src/retry.ts`. A -request is repeated only when repeating it could produce a different answer: - -- `ApiError` with a 5xx status: retried. So are `408` and `429`, the two 4xx - codes that are statements about timing rather than about the request. -- Every other 4xx: not retried. A 404 in particular is an answer, and - `listMissingThumbnails` depends on getting it promptly and once. -- Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or - TLS failure — and deadline aborts: retried. Node's `fetch` rejects with a - plain `TypeError`, so every `TypeError` is retried. The errno is looked for in - the error's `cause` chain, because that is where Node's `fetch` puts it. -- A truncated download: retried. -- Anything else, including a secretstream authentication failure that is not - truncation: not retried. The default answer is no. For a backup tool, retrying - a permanent failure spends round trips and delays every remaining file, while - declining to retry a transient one costs a single file that the next run picks - up. - -The `~/bin/quak` that `make install` builds runs under bun, whose `fetch` -rejects with a plain `Error` that has no `cause` and carries a code of bun's -own. quak retries a refused connection, a name that does not resolve, a reset -connection and a deadline under bun as it does under Node. Bun reports a refused -connection, and a TLS handshake that failed, as `ConnectionRefused`. For a name -that does not resolve it gives `ConnectionRefused` on some requests and -`FailedToOpenSocket` on others. A connection reset or closed before the response -ended is `ECONNRESET`, as under Node, and a deadline is the same `TimeoutError`. -A certificate bun rejects, reported with a code such as -`DEPTH_ZERO_SELF_SIGNED_CERT`, is not retried. - -Backoff is exponential with full jitter: the delay before retry _n_ is -`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term -is the ceiling and the wait is drawn below it, so a client that lost many -parallel downloads to one CDN blip does not send them all again at the same -instant. The numbers, configurable through `ApiClientOptions.retry`: - -| Option | Default | `quak backup` | Meaning | -| ------------- | ------- | ------------- | ----------------------------------- | -| `attempts` | `4` | `10` | total calls, not retries | -| `baseDelayMs` | `500` | `1000` | ceiling for the first retry's delay | -| `maxDelayMs` | `10000` | `60000` | upper bound on that ceiling | - -With the defaults a file that is going to fail gives up after at most three and -a half seconds of waiting. `quak backup` usually runs from cron with nobody -watching, so every request it makes uses the `quak backup` column instead, -exported as `UNATTENDED_RETRY_OPTIONS`: a request that keeps failing gives up -after at most 243 seconds of waiting, and usually after about half that, since -each wait is drawn at random below its ceiling. Every other command uses the -defaults. A library user gets the same budget by passing -`UNATTENDED_RETRY_OPTIONS` as `ApiClientOptions.retry`. `sleep` and `random` are -injectable through the same option, which is how the test suite exercises the -whole policy without waiting. `onRetry`, given there too, is called before each -wait with the error the attempt failed with, the attempt's number and the wait; -the CLI logs it with `-v` (see "CLI surface"). - -Two deadlines, renewed for each attempt: - -| Option | Default | Applies to | Kind | -| ------------------- | ------- | ------------------------------------------- | ------------------------------------- | -| `requestTimeoutMs` | `30000` | `getJSON`, `postJSON`, `putJSON`, `putFile` | the whole request | -| `downloadTimeoutMs` | `60000` | file and thumbnail downloads | idle: no bytes received for this long | - -They are different kinds because a download's length depends on the file and the -link: a whole-transfer deadline short enough to catch a hung connection would -cancel a large video on a slow link that is still making progress. The download -deadline restarts every time bytes arrive, so a slow download runs as long as it -keeps moving, and one that stalls is aborted after 60 seconds of silence. It -covers the wait for the headers and the body — `getFileStream` returns as soon -as headers arrive, so a deadline that only guarded the initial request would -leave the same hang one layer down. There is no limit on the total length of a -download. - -**Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON` -send every `POST` and `PUT` in the endpoint list above; some of them change -server state, and `/users/two-factor/verify` consumes one of a small number of -second-factor attempts. They are retried only when every errno in the error's -`cause` chain is one of the three that establish no TCP connection to the server -ever existed, so no request byte can have been transmitted: `ENOTFOUND` and -`EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer -refused the connection). A 5xx, a mid-flight reset and a deadline are all left -to the caller, because each of them can happen after the server has already -acted. These two do not follow redirects either: a redirect means the server -already received the request, so it is reported as an error and not retried. The -routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the -same reason, despite looking like connect-time failures: on Linux an ICMP -unreachable arriving mid-flight, or a local interface going down after the -request was written, delivers them on an already-established socket. They stay -retryable for the idempotent calls. So do bun's `ConnectionRefused` and -`FailedToOpenSocket`, on which a `POST` or `PUT` is not replayed either. When a -connection bun kept open from an earlier request closes before any answer, bun -sends the request again on a new connection, and if that one fails it gives one -of these codes although the server may already have the request. `putFile` is -exempt: a presigned PUT stores one whole object at one key in one request, so -replaying it has no partial state to damage. - -A download is retried as a whole — request, stream consumption, and decryption — -because a socket reset after the response headers have arrived surfaces in the -download layer rather than in `ApiClient`, and that is the common failure for -multi-megabyte photos over a CDN. The secretstream pull state is not resumable -and these endpoints have no Range support, so a retry starts the file over. The -atomic write is part of the retried unit: each attempt writes its own temporary -files, one for most files and two for a live photo (its image and its video), -and removes them if it fails. Only the attempt that completes renames anything -into place. The retry sits below `runBackup` and `runMetadataBackup`. -`runBackup` logs, counts and steps over a file that still fails after its -retries; `runMetadataBackup`, which downloads only with `--exif`, records the -error in that file's JSON as `imageMetadataError` and goes on. - -One imprecision is deliberate and worth knowing about. When a body ends part-way -through a secretstream chunk, Poly1305 fails and carries no framing signal, so a -cut connection and genuinely corrupt bytes are indistinguishable. quak reports -that as truncation, which means it is retried. For a body of more than one chunk -the distinction is real — a chunk that failed while the stream carried on past -it stays an authentication failure and is not retried — but for a single-chunk -body, which is most thumbnails and every small file, a wrong key, server-side -corruption and a mid-chunk cutoff all present alike and all get retried. The -cost is bounded by the attempt count, and it buys never silently keeping a -truncated file. - -### Session handling - -The `Client` class holds the auth token, master key, secret key, and public key -in memory. There is no on-disk session store in the library; the consumer -decides how to persist sessions. - -`client.toJSON()` returns a `ClientSnapshot` (a plain serializable object with -base64-encoded keys) that the consumer can write to disk, a database, or -whatever else fits their use case. `Client.fromJSON(snapshot)` restores a -working client from that snapshot without re-authenticating; it checks every -field and each key's length first, and throws an error naming the bad field. -`client.logout()` clears the token and zeroes the key buffers in place; after -it, every other method on that client throws. It does not contact the server, so -the token stays valid there and in any saved snapshot; -`await client.logoutOnServer()` first ends the session on the server -(`POST /users/logout`). - -The CLI stores the snapshot at the platform-appropriate data directory via -`env-paths`: `~/Library/Application Support/quak/session.json` on macOS, -`$XDG_DATA_HOME/quak/session.json` on Linux. The file is written with mode -`0600`. The key material is stored in cleartext in the JSON; treat this file as -you would treat the password itself. A missing file is reported as "not logged -in"; a file that exists but is corrupt is reported as such, naming the bad -field. When the refresh a command starts with gets HTTP 401 from the server, -because it no longer accepts the saved session's token, the command prints one -line, `quak: the saved session is no longer valid; run "quak login"`, with no -stack trace. All three exit with status 3, which means the user must run -`quak login` again. `quak logout` is the exception: with no file it says there -is no session and exits 0, and it handles a corrupt file or a failed server call -as described below. `quak backup` meets an expired session on the refresh that -starts every run, before it touches any file but its lock (see "Backup layout"). -A session that stops working partway through a backup instead fails each -remaining file into `failures.json`, so that run exits 1 and the next one stops -at its refresh with status 3. No command but `quak login` ever prompts. - -`quak logout` ends the session on the server, so the token in `session.json` -stops working even in a copy of the file, and then deletes the file. If the -server call fails (or the file is corrupt), the file is still deleted, the -command says the server session could not be ended, and it exits with status 1. -It does not delete the cache. When it knows the cache directory, from -`--cache-dir` or from a session file it could read, it prints it and says it -still holds decrypted data (file keys in `metadata.json`, cached originals and -thumbnails), for the user to delete if they want it gone. - -### CLI surface - -``` -quak [--cache-dir <path>] <command> global: local metadata/content cache location -quak [-v | -vv | -q] <command> global: log more on stderr, or only fatal errors -quak login interactive or QUAK_EMAIL/QUAK_PASSWORD -quak whoami print logged-in account as JSON -quak logout end the session, delete it -quak collections [--json] list all collections -quak files [collectionID] [--json] list files in a collection, or every file -quak get <fileID> [--out path] [--collection] download and decrypt a file -quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail -quak backup <dir> [--json] [--verify] full incremental backup -quak backup-metadata <dir> [--exif] dump the metadata quak keeps as JSON -quak helper list-missing-thumbnails [--json] find files with missing thumbnails -quak helper fix-missing-thumbnails [--file ids] [--json] generate + upload missing thumbnails -``` - -Every command except `login`, `whoami` and `logout` runs on the cache-backed -library. The read commands — `collections`, `files`, `get`, `get-thumb`, -`backup-metadata`, `helper list-missing-thumbnails` and -`helper fix-missing-thumbnails` — force a fresh server round-trip before they -answer, so they report current account state rather than whatever the cache last -held. `collections` asks the server for the collection list alone, in one -request, and neither fetches any collection's files nor changes the cache. -`files` with a collection ID asks for the same list, then for that one -collection's files, and does not change the cache either. On an empty cache, -`get` and `get-thumb` ask for the same list, then for one collection's files at -a time, until they find the file. `helper fix-missing-thumbnails --file` does -the same until it finds every file it names. They store none of the collections -or file records they read. The original or thumbnail they fetch is kept in the -cache. Once the cache holds the account's collections, they wait for a refresh -instead. The others wait for a refresh, and so does `files` without a collection -ID. With an empty cache, as on a first run, that refresh fetches the files of -every collection, one collection after another, and stores nothing until it has -them all, which on a large account takes a while. Meanwhile the command logs a -line on stderr for each collection whose files it has fetched, such as -`Fetched the files of 3 of 120 collections`, with `--json` too; `backup` does -the same while the refresh it starts with runs. stdout holds only the command's -output. If that round-trip fails, the command prints the error on one line and -exits 1, or 3 when the server no longer accepts the saved session (see "Session -handling"). An album or file quak cannot decrypt does not fail it; the command -answers from what did decrypt (see "Default reads vs. fresh reads"). -`--cache-dir` overrides where the cache lives; without it each account gets its -own directory under the per-user cache path. - -`-v` (`--verbose`) and `-q` (`--quiet`) set how much quak logs on stderr. Like -`--cache-dir`, they go before or after any command's name. Every line quak -prints on stderr is a log line at one of six levels, from the quietest: fatal, -error, warning, info, debug and trace. A run prints the lines at its own level -and at every quieter one. stdout holds only the command's output, at every -level. - -- Every command that exits non-zero logs a fatal line saying why, such as - `quak: <message>`, `Not logged in`, an invalid ID or a collection that does - not exist. A command that exits 1 because files or albums failed logs it last, - saying how many failed and where to look: `backup` points to `failures.json` - and its `FAILED` lines, `backup-metadata` to the `mlDataError` in the JSON of - each such file, and `helper fix-missing-thumbnails` to its `Failed files:` - list, or to its JSON with `--json`. -- An error line reports something that failed: the `FAILED` lines that `backup` - and `helper fix-missing-thumbnails` log as they work, - `ML data of file <id> could not be read` and - `ML data request for <n> file(s) failed` from `backup-metadata`, - `Could not check <title>` from both thumbnail helpers, and the `Failed files:` - list at the end of `backup` and of `helper fix-missing-thumbnails`. -- A warning reports something that did not fail but needs your attention: after - `logout`, that the cache directory still holds decrypted data. -- Every other line quak logs by default is info. - -With `--json`, a command's result on stdout takes the place of its summary on -stderr: for `backup`, the lines from `--- Backup complete ---` to the end of its -`Failed files:` list; for `helper list-missing-thumbnails`, -`No missing thumbnails found.` or `<n> file(s) with missing thumbnails:`; and -for `helper fix-missing-thumbnails`, the lines from `--- Done ---` to the end of -its `Failed files:` list. Every other line is logged as it is without `--json`. - -- With neither option the level is info: the lines quak prints as it works, such - as a refresh's progress, the files a backup fetches, its summary and the files - that failed. -- `-v` raises the level to debug, which adds each retry of a request, with why - the attempt failed and how long quak waits before the next, and the start and - end of each refresh and each ML data fetch, with the error of one that failed. -- `-vv` raises it to trace, which adds each request quak sends: its method, its - URL and the status of the answer, or the error the request failed with. The - URL of an upload of a thumbnail is logged without its query, which holds the - upload's signature. More repeats stay at trace, and `--verbose --verbose` is - `-vv`. -- `-q` lowers it to fatal: a run that exits 0 prints nothing on stderr, and one - that exits non-zero prints only its fatal lines. The prompts of `login` are - not log lines, and `-q` leaves them as they are. - -`-q` with `-v` is refused with one line, -`error: option '-q, --quiet' cannot be used with option '-v, --verbose'`, and -exit status 1. - -`files <collectionID>` lists the files in that collection. `files` with no -argument lists every file in the account, collection by collection, each file -once: a file in several collections is listed with the first of them, which is -its `collectionID` in `--json`. - -An ID given to `files`, `get`, `get-thumb` or -`helper fix-missing-thumbnails --file` is written in decimal digits, no larger -than 9007199254740991. Anything else, such as `0x10`, `1e3` or `-1`, is refused -before any request with one line, `Invalid file ID: <arg>` or -`Invalid collection ID: <arg>`, and exit status 1. One bad `--file` ID refuses -the whole command, so nothing is fixed. - -Without `--out`, `get` writes the file to the working directory under its title, -and `get-thumb` writes the thumbnail there under `thumb_` followed by the title, -each with unsafe characters replaced. A name over 255 bytes, which most file -systems refuse, is cut short between two characters before its extension, so the -extension is kept whenever it fits. An extension too long to leave room for the -name's first character is cut short too, so the name never starts with a dot. - -`get` and `get-thumb` resolve the file by ID directly, so `--collection` is -accepted for backward compatibility but ignored. For a live photo, `get` writes -its image and its video, each named after the title with its own extension, as -Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`), and each name is -cut short as above once its own extension is in it. With `--out PATH`, the image -is written to `PATH` and the video beside it, with `PATH`'s name and the video's -extension; a `PATH` with the video's extension is refused. -`backup-metadata --exif` (alias `--all`) additionally fetches each file's -original through the cache and records, from it or a live photo's image, its XMP -metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is read with -[exifreader](https://github.com/mattiasw/ExifReader) from any image format it -reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's `exif` field -is exifreader's EXIF tag output: each tag by name, with its `value`, -`description` and `computed` value. An EXIF block exifreader finds but reads no -tag from is recorded, base64, as `exifRaw`, with the reason in `exifError`. -`collections`, `files`, `backup`, `helper list-missing-thumbnails` and -`helper fix-missing-thumbnails` take `--json` for machine-readable output. - -`backup --verify` also hashes the originals already in the backup and replaces -any that do not match the content hash Ente records; one it cannot replace stays -as it was and goes into `failures.json` (see "Backup layout"). - -`backup-metadata <dir>` writes `account.json` and, for each collection, a -directory `collections/<id>-<name>/` holding `_collection.json` and a -`<fileID>.json` for each of its files. Each `<fileID>.json` holds the file's ML -data and keywords as `quak backup` writes them (see "Backup layout"). The -collection's name has unsafe characters replaced and is cut short between two -characters so that the directory's name fits in 255 bytes; the ID keeps the -directories apart. - -`backup-metadata` fetches ML data in requests of up to 200 files. When a request -fails, the error is logged, each of its files is written with the reason in an -`mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML -data does not decrypt, gunzip or parse is logged and written the same way, with -why in `mlDataError`. The exit code is non-zero if any file was written with -`mlDataError`, and the last line then says how many were. - -`helper fix-missing-thumbnails` regenerates thumbnails for JPEG images only, -because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG image -(PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept -distinct from a `failed` repair, and does not affect the exit code; a genuine -failure still exits non-zero. The server accepts a new thumbnail only from the -file's owner and only when it is no larger than the thumbnail size it records -for the file. So a file another account owns, in an album shared with you, is -skipped by both thumbnail helpers without being fetched, and the fixer skips a -file whose recorded thumbnail size is 0 or unknown. Otherwise the fixer lowers -the quality and size of the thumbnail until it fits, and skips the file if even -the smallest does not. - -### Backup layout - -`quak backup <dir>` produces: +`quak backup <dir>` produces this tree: ``` <dir>/ YYYY/YYYY-MM/YYYY-MM-DD/ - YYYY-MM-DD.<fileID>.<ext> actual file content, at its save path (one - per unique file, two for a live photo: see - below) - YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak - keeps, its update time, its private and - public magic metadata, its ML data and - keywords, and, for an image (for a live - photo, its image), its original's EXIF - and XMP, with dimensions for a JPEG - only; none of these three for a video + YYYY-MM-DD.<fileID>.<ext> the original, at its save path (one per + file, two for a live photo) + YYYY-MM-DD.<fileID>.json the metadata quak keeps about the file, + plus its machine-learning data, its + keywords and its image metadata YYYY-MM-DD.<fileID>.livephoto.json which of a live photo's two files is which collections/ <name>/ <title> -> ../../YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext> - (symlink) - <name>.json collection metadata + file list + a symlink per file + <name>.json the collection's metadata and file list account.json the account's email and user ID - backup.lock the lock a running backup holds (see below) - failures.json files that failed and have not yet succeeded + backup.lock the lock a running backup holds + failures.json the files that failed and have not yet + succeeded ``` -Each original is saved at its save path, the same path `photo.savePath` gives -and `photo.download()` writes (see Read surface below). The date is the photo's -`takenAt` (the date set in Ente if it was edited, else its creation time) in the -time zone of the machine running quak. The extension is the one in the file's -name as uploaded, case kept, or `.bin` when it has none, it holds anything but -letters and digits, or it is more than 16 of them long. When the date or the -time zone changes, the next run saves the original at its new path and leaves -the old copy where it is. +#### Save paths -`account.json` holds the account's `email` and `userID`, as `backup-metadata` -writes it. A collection's JSON holds its `id`, `name`, `type`, `ownerID`, -`isShared` and `updationTime`, its `magicMetadata`, `pubMagicMetadata` and -`sharedMagicMetadata` when it has them, as `backup-metadata`'s -`_collection.json` does, and `files`, each file's `id` and `metadata`. A file's -JSON holds its `id`, `collectionID`, `ownerID`, `metadata` and `updationTime`, -and its `magicMetadata` and `pubMagicMetadata` when it has them. Update times -are in microseconds, as Ente records them. +Each original is stored once, at its save path: +`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under `<dir>`. The date is +the day the photo was taken, in the time zone of the machine running quak. That +is the date set in Ente when the date was edited, and otherwise the file's +creation time. The extension is the one in the file's title, with its case kept. +It is `.bin` when the title has no extension, when the extension holds anything +but letters and digits, or when it is longer than 16 characters. -A file's JSON holds Ente's ML data for it as `mlData`, such as the faces Ente -found, without its CLIP embedding. It holds `keywords` in place of the -embedding. +When the date or the time zone changes, the next run stores the original at its +new path and leaves the old copy where it is. -The Discover section of Ente's photos app has 16 categories: +The library gives the same path as `photo.savePath` and writes it with +`photo.download()` (see "A photo's content"). + +#### The JSON beside each original + +The JSON file beside an original holds: + +- `id`, `collectionID`, `ownerID` and `updationTime`; +- `metadata`: the fields quak keeps from the file's own metadata. They are + `title`, `fileType`, `creationTime`, `modificationTime`, `latitude`, + `longitude` and `hash`. The last three are present only when the file has + them. +- `magicMetadata` and `pubMagicMetadata`, when the file has them. The magic + metadata is Ente's name for the fields its clients add later, such as an + edited name, an edited date or a caption. + +Update times are in microseconds, as Ente records them. + +The file also holds the machine-learning data Ente computed for the file, as +`mlData`. Ente's apps detect faces in each photo and compute a CLIP embedding +for it, a list of 512 numbers that describes what the photo shows. `mlData` +holds everything but that embedding. In its place the file holds `keywords`, +derived from the embedding as described under "Keywords". A file with no +machine-learning data has neither field. A file whose data holds no usable +embedding has no `keywords`. + +The backup waits for the library's fetch of machine-learning data to finish +before it writes the JSON files. When a file's data cannot be fetched, +decrypted, decompressed or parsed, the JSON holds the reason in `mlDataError` +instead of `mlData`. Such a file counts as failed and goes into `failures.json` +with an error starting `ML data: `. Nothing is cached for it, so the next run +asks for its data again. The JSON files are rewritten on every run, so data that +arrived since the last run appears. + +Every file but a video also holds `imageMetadata`: what the original itself +records. That is its `format`, `width` and `height` for a JPEG, its EXIF tags as +`exif`, and its XMP packet as `xmp`. EXIF and XMP are the two blocks of metadata +a camera or an editor writes inside an image file. For a live photo, these come +from its image. An original with none of them gets `{}`. When the original +cannot be read, the reason is in `imageMetadataError` instead, and neither the +file nor the run fails. + +EXIF is read with [exifreader](https://github.com/mattiasw/ExifReader) from any +image format it reads, such as JPEG, HEIC, AVIF, PNG and WebP. `exif` holds each +tag by name, with its `value`, `description` and `computed` value. An EXIF block +that exifreader finds but reads no tag from is kept in base64 as `exifRaw`, with +the reason in `exifError`. + +The original is read for this when the run stores it, or when the JSON beside it +has neither `imageMetadata` nor `imageMetadataError`. Otherwise the two fields +are copied from the JSON being rewritten, so a run does not read every stored +original again. + +#### Keywords + +The Discover section of Ente's photos app sorts photos into 16 categories. quak +derives a file's keywords the way the app sorts it. The categories are: - Identity (no videos) - Screenshots (no videos) @@ -895,507 +745,773 @@ The Discover section of Ente's photos app has 16 categories: - Hills - Greenery -A file's keywords are the titles of the categories it falls in. A file's score -for a category is the cosine similarity of its CLIP embedding and the text -embedding of the category's query. As in Ente's app, the file falls in the -category when that score is at least the category's minimum score. A video never -falls in a category marked "no videos". The keywords are listed highest score -first. +Each category has a query, and a text embedding of that query from Ente's CLIP +text model. A file's score for a category is the cosine similarity of the file's +embedding and the category's: a number from -1 to 1, higher the more alike the +two embeddings are. The file falls in the category when its score reaches the +category's minimum score. The keywords are the titles of the categories the file +falls in, highest score first. A video never falls in a category marked "no +videos", as in the app. -A file with no ML data has neither `mlData` nor `keywords`. A file whose ML data -holds no usable CLIP embedding has no `keywords`. `backup-metadata` writes both -fields the same way. +#### Collection directories -The backup waits for the library's ML data fetch to finish before it writes the -JSON files. If that fetch fails, each file whose ML data is not in the cache -gets the reason in `mlDataError` instead and counts as failed, and the next run -fetches it again. The same goes for a file with no ML data in the cache whose ML -data the server sent but quak could not decrypt, gunzip or parse. Its -`mlDataError` says why, and `failures.json` records it with an error starting -`ML data: `. quak keeps that reason in memory only and caches nothing for the -file, so the next run asks for its ML data again. An ML data fetch that fails, -such as one the library's refresh starts while the backup writes the JSON files, -leaves the reasons as the last fetch that succeeded left them. The JSON files -are rewritten on every run, so ML data that arrived since the last run appears. +Each collection gets a directory under `collections/`, named after the +collection, and a JSON file beside it, `<name>.json`. The directory holds one +symlink per file, named after the file's title, pointing at the original's save +path. A live photo gets two symlinks, one to its image and one to its video, +each named after the title with that file's extension. -A file's JSON holds, as `imageMetadata`, what `backup-metadata --exif` records -from its original, or from a live photo's image: `format`, `width` and `height` -for a JPEG, `exif` (or `exifRaw` and `exifError`), and `xmp`. An original with -none of these gets `{}`. A video gets no `imageMetadata`: reading a whole video -to look for tags is not worth it. When the original cannot be read, the reason -is in `imageMetadataError` instead; neither the file nor the run fails. The -original is read when the run stores it, or when the JSON beside it has neither -field, as one an earlier version wrote. Otherwise the field is taken from that -JSON when it is rewritten, so a run does not read every stored original again. +The JSON file holds: -`failures.json` records each failed file with the kind of failure, how many -times it has been tried and when it was last tried. A file leaves it once it -succeeds, or once it is no longer in the library or in the backup's scope. The -library's `lib.backup({ includeThumbnails: true })` also writes -`thumbnails/<fileID>.jpg` beside `collections/`; `quak backup` does not. +- `id`, `name`, `type`, `ownerID`, `isShared` and `updationTime`; +- `magicMetadata`, `pubMagicMetadata` and `sharedMagicMetadata`, when the + collection has them; +- `files`: each file's `id` and `metadata`. -A file in the backup's scope that the library could not decrypt (see "Default -reads vs. fresh reads") goes into `failures.json` with an error that starts -`could not be decrypted:` and gives the reason. If a copy of it in another album -decrypted, it does not: its original is stored from that copy, and the album -where it does not decrypt neither links nor lists it. An album the library could -not decrypt has no name quak can read and no files it can list. `quak backup` -prints one line for it on stderr, -`FAILED album <id>: could not be decrypted: <reason>`, and `--json` and the -`BackupResult` list its ID in `undecryptableAlbums`. Such a file in -`failures.json`, or such an album, keeps the exit code at 1, run after run, -until the server sends a copy that decrypts or deletes it; the next run then -drops it. +Names come from the server, so unsafe characters in them are replaced (see "File +names from the server"). When two collections would get the same directory name, +ignoring case, each gets its ID added: two albums named `Trip` become +`Trip (10)/` and `Trip (11)/`. Two files in one collection whose titles match, +ignoring case, get their IDs added before the extension: `IMG_0001 (12345).JPG` +and `IMG_0001 (12346).JPG`. A collection named like another collection's JSON +file, such as `Trip.json` beside `Trip`, also gets its ID added. A name too long +for a file name, which most file systems limit to 255 bytes, gets its ID added +too. It is then cut short before the ID, just far enough to fit. IDs never +change, so a name stays the same from run to run until such a clash appears or +goes away. + +Each run removes every symlink in a collection's directory that points into the +date folders and is not one of its current links, yours included. A file or +symlink of yours at the name of one of its links is replaced by the link. A +directory at that name stays, and the file goes into `failures.json`. Anything +else you put under `collections/` stays. A run removes the directory and JSON of +a collection that was deleted or renamed, once the directory holds nothing else. +One that still holds something of yours stays, with its JSON. + +A collection whose directory or JSON cannot be written does not stop the run. +For example, a file you put there may already have the directory's name. The run +logs `FAILED album <name>: <reason>`, lists the collection's ID in +`failedAlbums`, and exits 1. The other collections and `failures.json` are still +written. + +#### What quak cannot decrypt + +A collection or file that quak cannot decrypt is left out of the library (see +"Default reads and fresh reads"). The backup treats the two cases this way: + +- A file quak cannot decrypt goes into `failures.json` with an error starting + `could not be decrypted:` and the reason. It stays out of `failures.json` when + its copy in another collection decrypts. The original is then stored from that + copy, and the collection where it does not decrypt neither links nor lists it. +- A collection quak cannot decrypt has no name quak can read and no files it can + list. The run logs `FAILED album <id>: could not be decrypted: <reason>` and + lists the ID in `undecryptableAlbums`. + +Every run exits 1 until the server either sends a copy that decrypts or deletes +the record. The run after that drops the entry. A collection that stops +decrypting loses its directory and JSON as a deleted one does, and a file that +stops decrypting loses its symlink in that collection. The originals stay at +their save paths. + +#### Live photos + +Ente stores a live photo as one ZIP of its image and its video. The backup +stores the two files, which a photo viewer can open: each is +`YYYY-MM-DD.<fileID>.<ext>` with the extension it has inside the ZIP, such as +`2026-03-01.12345.heic` and `2026-03-01.12345.mov`. The file +`YYYY-MM-DD.<fileID>.livephoto.json` beside them names which is which. The live +photo counts as stored only when its `.livephoto.json` and both files it names +are present and not empty. + +#### `failures.json` + +`failures.json` records each file that failed and has not succeeded since. It +holds `version`, which is 1, and `files`, one entry per file keyed by its ID. +Each entry holds: + +- `fileID` and `title`; +- `classification`: the kind of failure; +- `attempts`: how many runs have tried the file; +- `lastTriedAt`: when it was last tried, as a Unix time in milliseconds; +- `error`: the error, as quak printed it. + +The kinds are: + +- `transient`: a network or server problem; +- `permanent`: a refusal, or a local file-system or decryption error; +- `unknown`: anything else. + +A file leaves the list once it succeeds, or once it is no longer in the library. +A library caller that backs up only some albums (`onlyAlbumNames`) also drops a +file outside them. `failures.json` is deleted when the list is empty. + +#### `backup.lock` `backup.lock` keeps two backups of the same directory from running at once, such as a cron run that starts while the previous one is still going. A backup -creates `<dir>` if it is missing and takes the lock before its refresh, and -removes the lock when it ends, whether it succeeds or fails, unless it lost the -lock (see below); `quak backup` takes it before it opens its library, so a -refused run sends no request. The lock is a directory that -[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates -and keeps touching while the backup runs. A second backup of the directory, from -another process or the same one, fails at once: `quak backup` prints -`quak: another backup of <dir> is running` and exits with status 2. A run -stopped with Ctrl-C or `kill` removes the lock as it exits. Only a run that -cannot, such as one killed with SIGKILL or cut off by a crash or power loss, -leaves it behind; once it has gone 10 seconds untouched, the next run takes it -over, so nobody has to remove it. The lock is outside the date folders and -`collections/`, so it is never taken for an original, and the removal of old -album directories never touches it. +creates `<dir>` if it is missing and takes the lock before its refresh, so a +refused run sends no request. A run that still holds the lock removes it when it +ends, whether it succeeded or failed. -A running backup loses its lock when someone removes `backup.lock`, or when -another backup takes it over because the running one left it untouched for 10 -seconds, as a stopped or stalled process does. The backup finds this out the -next time it goes to touch the lock, which it does every 5 seconds, and then -stops before it starts another file. The originals it is downloading at that -moment are still stored. It leaves `failures.json` and `backup.lock` as they -are, since another backup may be using them. `quak backup` prints -`quak: lost the lock on <dir>: <reason>` as one line and exits 1, and -`lib.backup()` rejects with `lost the lock on <dir>: <reason>`. +The lock is a directory that +[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates. +While the backup runs, the backup updates the directory's modification time +every 5 seconds. A second backup of the directory fails at once: it prints +`quak: another backup of <dir> is running` and exits 2. A run that is killed, or +cut off by a crash, can leave the lock behind. Once the lock has gone 10 seconds +without an update, the next run takes it over, so nobody has to remove it by +hand. -A collection's directory and JSON are named after the collection, and a symlink -after the file's title, both with unsafe characters replaced. When two -collections would get the same name, or two symlinks in one collection the same -name (ignoring case in both), each of them gets its ID added: two albums named -`Trip` become `Trip (10)/` and `Trip (11)/`, and two files titled `IMG_0001.JPG` -become `IMG_0001 (12345).JPG` and `IMG_0001 (12346).JPG`. A collection whose -directory would have the name of another collection's JSON, again ignoring case, -gets its ID added as well: beside an album `x`, whose JSON is `x.json`, an album -named `x.json` gets the directory `x.json (12)/`. A name too long for a file -name, which most file systems limit to 255 bytes, gets its ID added too. A name -with its ID added is cut short between two characters in the part before the ID, -just far enough that it fits. When a symlink's extension leaves no room for the -ID and the first character of the name, the extension is cut short, and the part -before the ID keeps that character and any bytes the cut leaves over. A -collection's directory name also leaves room for the `.json` its JSON adds. IDs -never change, so a name stays the same from run to run until such a clash -appears or goes away. +A running backup loses its lock when someone removes `backup.lock`. Another +backup takes the lock over when this one has not updated it for 10 seconds, for +example because this process was stopped. The backup notices at its next update +of the lock, and stops before it starts another file. The originals it is +downloading at that moment are still stored. It leaves `failures.json` and +`backup.lock` as they are, since another backup may be using them. It prints +`quak: lost the lock on <dir>: <reason>` and exits 1. -A collection whose directory or JSON cannot be written, such as one whose -directory name is taken by a file you put in `collections/`, does not stop the -run. `quak backup` prints `FAILED album <name>: <reason>` for it on stderr, -`--json` and the `BackupResult` list its ID in `failedAlbums`, and the exit code -is 1. The other collections and `failures.json` are still written. +#### `--verify` -A live photo, which Ente stores as one ZIP of its image and its video, is stored -as those two files, which a photo viewer can open: each is -`YYYY-MM-DD.<fileID>.<ext>` with the extension it has inside the ZIP (for -example `2026-03-01.12345.heic` and `2026-03-01.12345.mov`), and -`YYYY-MM-DD.<fileID>.livephoto.json` names the two. The live photo counts as -stored only when both files are present and not empty. Its album folder links -both, each named after the title with that file's extension (`IMG_0001.heic` and -`IMG_0001.mov`). +With `--verify`, a run hashes each original already at its save path the way a +download is checked. For a live photo it hashes the image and the video and +joins the two as `<imageHash>:<videoHash>`. An original that matches the content +hash in its metadata is left as it is. One that does not is logged on one line +naming the file, downloaded again in the same run, and replaced once the new +copy matches. The new copy is downloaded even when the cache holds the original, +because a copy from the cache is not checked as a download is. -Each run removes the symlinks into the date folders that no longer belong in -their collection's directory, and the directories (and JSON) of collections that -were deleted or renamed. An album quak can no longer decrypt loses its directory -and JSON the same way, and a file whose copy in an album quak can no longer -decrypt loses its symlink there and its entry in that album's JSON; the -originals stay at their save paths. Nothing else in `collections/` is touched: a -file or a symlink you put there stays, and a directory that still holds one -after its symlinks are removed stays too, with its JSON. +The new copy is written to a temporary file beside the original, hashed the same +way, and renamed over the original only when it matches. A live photo's image +and video are both hashed before either is renamed. Until then the original +stays as it was. When the new copy fails or does not match: -Each file is downloaded exactly once regardless of how many collections it -appears in, and written once: straight to its save path, with no copy left in -the cache. An original the cache already held is copied from there instead. On -subsequent runs, existing originals are skipped. If a download fails, the error -is logged and the backup continues with the next file. The exit code is non-zero -if any files failed. `quak backup` opens its library with the thumbnail and -originals precache off, so the only file content it fetches is the originals the -backup stores. It also opens it with no background timer, so it asks the server -for changes only as it starts, not again while it downloads. +- the temporary files are deleted; +- the original keeps its bytes and its album links; +- the file goes into `failures.json`. -With `--verify`, or `lib.backup({ verify: true })`, a run also hashes each -original already at its save path the way a download is checked (see "On-disk -cache layout" below): its bytes, read in chunks, or a live photo's image and -video, joined as `<imageHash>:<videoHash>`. An original that matches the content -hash its metadata records is left as it is. One that does not is logged on one -line naming the file, and replaced once a matching copy is downloaded, in the -same run. The copy is downloaded even when the cache holds the original, because -a copy from the cache is not checked as a download is. It is written to a -temporary file beside the original, a live photo's image and video each to its -own, hashed the same way, and renamed over the original only when it matches; a -live photo's image and video are both hashed before either is renamed. Until -then the original stays as it was: when the download fails, or what it brings -does not match either, the temporary files are deleted, the original keeps its -bytes and its album links, and the file goes into `failures.json`. A run that -stops between renaming a live photo's new image and its new video leaves the new -image beside the old video, which a later `--verify` run replaces if the video -was the part that did not match. A file whose metadata records no hash is left -as it is and counted as unchecked. A stored original that cannot be read is left -as it is and counts as failed. The summary and `--json` add the counts -`verified`, `mismatched` and `unchecked`, all of originals that were already -stored; one first downloaded in this run is in none of them. A mismatch that was -replaced with matching bytes does not make the exit code non-zero. Without -`--verify` nothing is hashed, the summary is unchanged, and the three counts are -0 in `--json`. +A run that stops between renaming a live photo's new image and its new video +leaves the new image beside the old video. A later `--verify` run then replaces +the video, if the video was the part that did not match. + +A file whose metadata records no hash is left as it is and counted as unchecked. +A stored original that cannot be read is left as it is and counted as failed. +The summary and the `--json` output add the counts `verified`, `mismatched` and +`unchecked`. Each counts only originals that were already stored. A mismatch +that was replaced with matching bytes does not make the exit code non-zero. + +A run without `--verify` drops two kinds of failure from `failures.json`: + +- a mismatched original whose new copy failed; +- a stored original that cannot be read. + +That run finds the original in place, counts it as skipped and drops the entry, +so it can exit 0. Only a `--verify` run hashes the original again. + +Without `--verify`, no stored original is hashed, the summary is unchanged, and +the three counts are 0 in the JSON. + +#### Temporary files and crash safety Each original is written to a temporary file in the same directory, synced to -disk, and renamed into place, so an original is either complete or absent, even -after a power cut. A downloaded original's temporary file is named -`.quak-<pid>-<random>.tmp`, one copied from the cache -`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A copy that -`--verify` downloads to replace an original is written to such a -`.quak-<pid>-<random>.tmp` too, a live photo's image and video to -`.quak-<pid>-<random>.<ext>`, each with its own extension. A run that is killed -can leave one of these temporary files behind; the next backup deletes those -whose process is no longer running. The content cache uses the same scheme, and -opening a library deletes the temporary files in the cache whose process is no -longer running, so a download another process has in progress in the same cache -is left alone. The rename replaces whatever was at the destination rather than -writing through it: a symlink there is replaced, not followed, and the new file -has the temporary file's permissions, not those of the file it replaced. +disk, and renamed into place. So an original is either complete or absent, even +after a power cut. The temporary files are named: -## TODO +- `.quak-<pid>-<random>.tmp` for a downloaded original, and for the copy that + `--verify` downloads; +- `.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp` for an original + copied from the cache; +- `.quak-<pid>-<random>.<ext>` for the image and the video of a live photo that + `--verify` downloads, each with its own extension. -- [x] Retry policy: no retry on 4xx (except `408` and `429`), exponential - backoff on 5xx and network errors -- [x] Update the API reference section below to match the current implementation -- [x] `make docker` green -- [x] Store live photos in a form a photo viewer can open - (https://git.eeqj.de/sneak/quak/issues/107): unpacked into the image and - the video +A run that is killed can leave one of these files behind. The next backup +deletes those whose process is no longer running. The cache uses the same +scheme. Opening a library deletes such leftovers in the cache's `originals/` and +`thumbnails/`. A download another process has in progress is left alone. -Tagging and releases are decided by sneak alone, and happen only when he -declares one. +The rename replaces whatever was at the destination rather than writing through +it. A symlink there is replaced, not followed, and the new file has the +temporary file's permissions, not those of the file it replaced. Each original +is written with mode `0600`, readable by its owner only. Everything else the +backup writes, its folders and JSON files included, gets the mode your umask +allows, which usually lets every user on the machine read it. The JSON beside an +original holds the file's location and faces. The `backup-metadata` tree is +written the same way. -Future (desktop client, separate repo): +### The metadata dump directory -- [ ] Electron app skeleton consuming this library -- [ ] Local cache (SQLite) keyed on `(collectionID, fileID, updationTime)` -- [ ] Background sync worker that streams new files into the cache -- [ ] Gallery UI: thumbnails, full-image view, basic search -- [ ] Upload, delete, and share operations in the library +`quak backup-metadata <dir>` produces this tree: -## API reference +``` +<dir>/ + account.json the account's email and user ID + collections/ + <id>-<name>/ + _collection.json the collection's metadata + <fileID>.json one file's metadata, one per file in + the collection +``` -The library's primary surface is the cache-backed `Library`; the lower-level -`Client` sits underneath it and is covered by the Design sections above. The -test suite is the canonical, executable documentation — `test/library/` and -`test/client/usage.test.ts` walk most operations, `test/cli/backup.test.ts` -walks `lib.backup()`, `test/cli/backup-end-to-end.test.ts` runs `quak backup` -against a fake of the Ente HTTP API, and `yarn test` verifies them. +Each run writes into `<dir>` and deletes nothing. `account.json` holds `email` +and `userID`. In `<id>-<name>`, the name has unsafe characters replaced. It is +cut short so the directory name fits in 255 bytes. The ID keeps two collections +of the same name apart. + +`_collection.json` holds the fields a backup's collection JSON holds, except +`files`. Each `<fileID>.json` holds the fields a backup's JSON beside an +original holds, with these differences: + +- `imageMetadata` is written only with `--exif`. The original of every file is + then downloaded, videos included, and read as a backup reads it (see "The JSON + beside each original"). +- A file whose original holds no image metadata gets no `imageMetadata` field. +- An original that cannot be downloaded or read gives `imageMetadataError` + instead. +- A file in several collections has its original read once; its image metadata + is written with the first of them. + +### File names from the server + +A file's title and a collection's name come from the server, so a name taken +from them is made safe before it is used on disk. Each path separator, each +character Windows forbids in a file name and each control character becomes `_`. +A leading run of dots becomes one `_`. A name Windows reserves for a device, +such as `CON`, gets a leading `_`. An empty title becomes `file-<id>`. An empty +collection name becomes `collection-<id>` in a backup and `unnamed` in the +metadata dump. + +A file name over 255 bytes is shortened before its extension, without splitting +a character, so the extension is kept when it fits. An extension too long to +leave room for the name's first character is cut short too, so the name never +starts with a dot. A collection's directory name is cut at its end. A path you +give with `--out` is used as it is. + +## Library reference + +The library's main class is `Library`. The `Client` sits underneath it and +handles login, saved sessions and the raw requests. The test suite is the +executable documentation: + +- `test/library/` shows most of what `Library` does; +- `test/client/usage.test.ts` shows the `Client`; +- `test/cli/backup.test.ts` shows `lib.backup()`; +- `test/cli/backup-end-to-end.test.ts` runs `quak backup` against a fake of the + Ente HTTP API. ### Opening a library -`Library.open(options)` loads the on-disk cache, starts the background refresh -loop, and resolves to a `Library`. On an empty cache it awaits the first -refresh, so it opens onto the account's data whenever the server is reachable; -if that refresh fails, it opens with no data and records the error in -`lib.status()`. On an existing cache it returns immediately and refreshes in the -background, so an unreachable server does not block opening. With -`refreshOnOpen: false` it starts no refresh and returns at once onto what the -cache holds, empty or not; a refresh then starts only when `fresh()` or -`lib.backup()` asks, or the background timer fires. The CLI's read commands open -it so (see "CLI surface"). +`Library.open(options)` loads the on-disk cache and resolves to a `Library`. On +an empty cache it waits for the first refresh, so it opens onto the account's +data whenever the server is reachable. If that refresh fails, it opens with no +data and records the error in `lib.status()`. On an existing cache it returns at +once and refreshes in the background, so an unreachable server does not block +opening. With `refreshOnOpen: false` it starts no refresh and returns at once +onto what the cache holds, empty or not. A refresh then starts only when +`fresh()` or `lib.backup()` asks, or when the background timer fires. `LibraryOptions`: -| Option | Default | Meaning | -| ------------------------ | -------------------------------------------------- | -------------------------------------------------------------------- | -| `client` | required | the account client (a `Client`, or any `LibraryClient`) | -| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live | -| `downloadDirectory` | `photos` in the working directory | root of the save paths; an original stored there counts as cached | -| `refreshIntervalSeconds` | `3` | background refresh cadence; `Infinity` for no background timer | -| `refreshOnOpen` | `true` | start a refresh as the library opens | -| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first | -| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals | -| `precacheOriginalsDays` | `7` | length in days of that latest window | -| `cacheOriginalsMaxBytes` | 100 GiB | size limit on the originals cache; pinned originals can exceed it | -| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down | -| `isOriginalPinned` | none | extra predicate for originals that must never be evicted | -| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) | -| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) | -| `contentSource` | the client's own | override the byte source (mainly for tests) | +- `client`, required: the account's `Client`, or anything that satisfies + `LibraryClient`. +- `cacheDirectory`: where `metadata.json` and the content cache live. Default: + `quak/<userID>` under the per-user cache directory (see "The cache"). +- `downloadDirectory`: the root of the save paths. Default: `photos` in the + working directory, resolved once when the library opens; + `lib.downloadDirectory` holds the result. An original stored at its save path + counts as cached. +- `refreshIntervalSeconds`: how long after one refresh ends the background timer + starts the next. Default `3`. `Infinity` starts no timer. +- `refreshOnOpen`: whether `open()` starts a refresh. Default `true`. +- `precacheThumbnails`: fetch every thumbnail in the account in the background, + newest first, until all are on disk. Default `true`. +- `precacheOriginals`: fetch, in the background, the originals of the favorites + album and of every file taken in the latest days (see the next option). These + originals are pinned: they are never evicted from the cache. Default `true`. +- `precacheOriginalsDays`: how many days count as the latest, counted back from + the newest file in the account. Default `7`. +- `cacheOriginalsMaxBytes`: the size limit on the originals cache. Default 100 + GiB. The limit in force is this, or less when the disk is fuller (see the next + option). Pinned originals can push the cache over it. +- `freeBelowBytes`: the free space to keep on the cache's volume. Default 50 + GiB. When the volume cannot hold the limit above and this much free space too, + the limit in force shrinks to what fits. `lib.status().originalsLimitBytes` + reports the limit in force. +- `isOriginalPinned`: a function that names more originals to pin. +- `pools`: the request pools, which set how many requests run at once (see + "Request pools"). Default: a fresh `RequestPools`. +- `onProgress`: a callback for the progress of refreshes, machine-learning + fetches and the background downloads, called with a `RefreshEvent`. +- `contentSource`: a replacement for the client's own byte source. Mainly for + tests. -The default `downloadDirectory` is resolved against the working directory once, -when the library opens; `lib.downloadDirectory` holds the result. +In the content and thumbnail pools, your own requests take the next free slot +ahead of the two background downloads. A file already in the cache is not +fetched again. When the originals cache grows past its limit, the least recently +used originals are evicted, but never a pinned one, and never one whose fetch is +still in flight. -`onProgress` receives a `RefreshEvent` with status `"started"` as a refresh, an -ML data fetch or a precache sweep starts, and `"done"` or `"failed"` as it ends. -In between, a refresh sends `"progress"` after each collection whose files it -fetched, with `fetched`, how many collections it has fetched the files of so -far, and `total`, how many it will. A refresh fetches the files of each -collection that is new or changed since the cache's copy, so on an empty cache -it fetches those of every collection. +`onProgress` receives a `RefreshEvent`. Its `operation` names the work: +`"refresh"`, `"fetchMLData"` for a machine-learning fetch, and +`"precacheThumbnails"` or `"precacheOriginals"` for the two background +downloads. Its `status` is `"started"` as the work starts and `"done"` as it +ends. A refresh or a machine-learning fetch that fails ends with `"failed"` and +the error message instead. In between, a refresh sends `"progress"` after each +collection whose files it fetched: `fetched` is the number of collections +fetched so far, and `total` the number it will fetch. A refresh fetches the +files of each collection that is new or changed since the cache's copy, so on an +empty cache it fetches those of every collection. The `"done"` event of a +machine-learning fetch, and of each background download, also carries `fetched`: +how many items it newly stored. -Concurrency is set through `pools`: construct -`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })` -and pass it. The three pools default to 10 / 5 / 25 (see Request pools below). +`lib.status()` returns a `LibraryStatus`. It holds the collection and file +counts. `lastRefreshAt` is the time of the last refresh that succeeded, and +`lastError` the message of the latest refresh, set only while that refresh +failed. `lastMLFetchAt` and `lastMLError` do the same for machine-learning +fetches. It also holds the originals cache's usage and limit in force, the +background downloads' progress and `closed`. `lib.status().mlDataErrors` lists +each file whose data arrived in the last successful fetch but could not be +decrypted, decompressed or parsed. Each entry is `{ fileID, error }`. The +`fetchMLData` event with status `"done"` carries the same list. Such data is not +cached, so the next fetch asks for it again. The reasons are kept in memory +only. -`lib.status()` returns a `LibraryStatus` (collection/file counts, last -refresh/ML times and errors, originals usage and effective limit, precache -progress, and `closed`). `lib.close()` stops the background timer; it is -idempotent, and an in-flight refresh is left to finish. The promise it returns -resolves once that refresh (including its cache write), the ML data fetch and -the precache fetches already running have all finished, so the cache directory -can then be removed. +`lib.close()` stops the background timer. It can be called more than once. A +refresh in flight is left to finish, with its write to the cache. The promise +`close()` returns resolves once that refresh, the machine-learning fetch and the +background downloads already running have all finished. The cache directory can +then be removed. After `close()`, `fresh()`, `freshCollections()`, +`freshFiles()` and `lib.backup()` reject with `the library is closed`. -`lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML -data the last ML data fetch that succeeded got but could not decrypt, gunzip or -parse; the `fetchMLData` event with status `"done"` carries the same list. A -fetch that is still running leaves the list as it is until it ends, and a fetch -that fails leaves it as it was. Such ML data is not cached, so the next ML data -fetch asks for it again. The reasons are kept in memory only, never in the -cache. +### Default reads and fresh reads -### Default reads vs. fresh reads +Default reads, through `lib.albums`, `lib.photos` and `lib.timeline`, answer +synchronously from the copy held in memory and never touch the network. A +refresh replaces that copy only once all its server requests have succeeded. So +a default read is immediate, but only as current as the last refresh whose +requests all succeeded. Right after opening an existing cache, it is the copy on +disk. -Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer -synchronously from the copy held in RAM and never touch the network. A refresh -changes that copy only once all its server requests have succeeded, and the -background timer starts the next refresh `refreshIntervalSeconds` after the -previous one ends. So a default read is immediate, but only as current as the -last refresh whose requests all succeeded; right after opening an existing -cache, it is the copy on disk. - -`await lib.fresh()` waits for a refresh to complete and persist, and returns the -same `{ albums, photos, timeline }` namespaces, which then reflect a completed -server round-trip. When a refresh is already running, background or not, -`fresh()` waits for that one, so its answer can come from requests made before -the call; only when none is running does it start one. A refresh that fails -rejects the caller (default reads stay silent and keep serving the last good -copy). The CLI's read commands use fresh reads (issue -https://git.eeqj.de/sneak/quak/issues/75). +`await lib.fresh()` waits for a refresh to complete and persist, then returns +the same `{ albums, photos, timeline }`, which then reflect a completed server +round-trip. When a refresh is already running, `fresh()` waits for that one, so +its answer can come from requests made before the call. Only when none is +running does it start one. When the refresh fails, `fresh()` rejects. Default +reads stay silent and keep serving the last good copy. `await lib.freshCollections()` is the fresh read of the collection list alone: -one collections request, answered with every live collection that decrypts, as -the `Collection` objects `listCollections()` returns, in the server's order. It -fetches no collection's files and stores nothing, not even the list: a refresh -fetches a collection's files only when the collection is new or has changed -since the stored copy, so a collection stored without its files would never get -them. `quak collections` uses it (issue -https://git.eeqj.de/sneak/quak/issues/210). +one request, answered with every collection that is not deleted and decrypts, as +`Collection` objects, in the server's order. It fetches no collection's files +and stores nothing, not even the list. It stores nothing because a refresh +fetches a collection's files only when the collection is new or changed. A +collection stored without its files would never get them. -`await lib.freshFiles(collection)` is the fresh read of one collection's files. -It answers with every live file in that collection that decrypts, in the -server's order. `collection` is a record that `freshCollections()` returned. The -method fetches no other collection's files and stores nothing. `quak files` with -a collection ID uses it, and so do `quak get` and `quak get-thumb` on an empty -cache. `quak helper fix-missing-thumbnails --file` uses it on an empty cache -too. +`await lib.freshFiles(collection)` is the fresh read of one collection's files, +as `EnteFile` objects, in the server's order, each file once. `collection` is an +object that `freshCollections()` returned. The method fetches no other +collection's files and stores nothing. -An album or file in the server's answer that quak cannot decrypt, or whose -metadata is not JSON, does not fail the refresh. The refresh leaves it out, an -album with all its files, and drops any older copy the cache held, so reads -answer from what did decrypt. The library keeps a list of what it left out, with -the reason, in `metadata.json`: the next refresh asks the server only for what -changed, so it would not see the record again. A record stays on that list until -a later refresh gets a copy that decrypts or learns that the server deleted it. -`lib.backup()` reports each such album, and each such file in its scope that has -no copy that decrypted (see "Backup layout"). +The refresh does not fail on a record it cannot decrypt, or whose metadata is +not JSON. It leaves the record out; a collection left out takes its files with +it. It also drops any older copy the cache held, so reads answer from what did +decrypt. The library keeps a list of what it left out, with the reason, in +`metadata.json`. It needs the list because the next refresh asks only for what +changed, and would not see the record again. A record stays on that list until a +later refresh gets a copy that decrypts or learns that the server deleted it. -### Read surface +### Albums, photos and the timeline -- `lib.albums.list()` → `Album[]`, newest-updated first. - `lib.albums.byID({ collectionID })` and `byName({ albumName })` → - `Album | undefined`. -- `lib.photos.byID({ fileID })` → `Photo | undefined`. - `lib.photos.records({ fileIDs })` → `PhotoRecord[]` in the requested order, - each id once, unknown ids dropped. -- `lib.timeline.groups({ groupBy, filter? })` → `TimelineGroup[]`, grouped by - `"day" | "week" | "month"` (keys `YYYY-MM-DD`, ISO `YYYY-Www`, `YYYY-MM`), - newest group first. A `PhotoFilter` combines `albumID`, `text` - (title/caption/album-name substring), `fileTypes`, `hasLocation`, and - `includeArchived`; hidden photos are always excluded. +- `lib.albums.list()` returns `Album[]`, newest updated first. + `lib.albums.byID({ collectionID })` and `lib.albums.byName({ albumName })` + return `Album | undefined`. Names are not unique in Ente, so `byName` picks + the newest updated album of that name. +- `lib.photos.byID({ fileID })` returns `Photo | undefined`. + `lib.photos.records({ fileIDs })` returns `PhotoRecord[]` in the order asked + for, each ID once, unknown IDs dropped. +- `lib.timeline.groups({ groupBy, filter? })` returns `TimelineGroup[]`, newest + group first. `groupBy` is `"day"`, `"week"` or `"month"`, and the group keys + are `YYYY-MM-DD`, `YYYY-Www` (the ISO week) and `YYYY-MM`. Each group holds + `key`, `startsAt` and `fileIDs`. `key` is that group key. `startsAt` is the + start of the period, in local-time milliseconds. `fileIDs` lists its files, + newest first. Weeks start on Monday. -An `Album` exposes its record fields and `album.photos.list()` → `Photo[]` -(newest first). A `Photo` exposes its record fields other than `thumbnailPath` -and `originalPath`, and `photo.year`, the local-time year of `takenAt`, all as -synchronous getters read from RAM; `photo.record()` → `PhotoRecord`. Two more -synchronous getters look at the disk and never touch the network: +A `PhotoFilter` narrows the timeline. Its fields combine with AND: -- `photo.savePath` → `string` — where `photo.download()` and `lib.backup()` put - the original, whether or not it is there yet: - `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's - `downloadDirectory` (see Backup layout above for the date and the extension). - For a live photo already stored, its image. For a live photo not yet stored, - it carries the title's extension, and the image may be stored under a - different one, found inside the live photo. It needs no content source. -- `photo.isLocal` → `boolean` — whether the whole original is at `savePath`. A - copy only in the cache does not count. +- `albumID`: only photos in that album. +- `text`: a case-insensitive substring of the title, the caption or the name of + an album the photo is in. +- `fileTypes`: only photos of one of these types. +- `hasLocation`: `true` keeps only photos with coordinates, `false` only those + without. +- `includeArchived`: archived photos are left out unless this is `true`. -These async methods may download: +Hidden photos are always left out. -- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the - full-resolution file. For a live photo, `path` and `bytes` are its image's and - `videoPath` is its video. -- `await photo.download()` → `{ path, bytes, videoPath? }`, as `original()` — - puts the original at `savePath`, creating its folders. When it is already - there, nothing is written. When the cache holds it, it is copied from the - cache; otherwise it is fetched straight to `savePath`, with no copy left in - the cache. Afterwards `isLocal` is true. -- `await photo.thumbnail(opts?)` → `{ path, bytes }`. -- `await photo.content(opts?)` → `Uint8Array` — the original's bytes, read - through `original()`; for a live photo, its image's. -- `await photo.exif(opts?)` → `ExifTags` — every EXIF tag in the file, keyed by - tag name, each as exifreader decodes it, with its `id`, `value`, `description` - and `computed` value: for example `Make` is - `{ id: 271, value: ["Canon"], description: "Canon", computed: "Canon" }`. A - tag exifreader has no name for is keyed `undefined-<tag number>`. The embedded - thumbnail's tags are under `Thumbnail`, so they cannot hide the main image's - tags of the same name; the thumbnail image itself is left out. EXIF is read - from any image format exifreader reads (such as JPEG, HEIC/HEIF, AVIF, PNG, - WebP and TIFF), a live photo's image included. Any other original gives `{}`, - and a video gives `{}` without being downloaded. -- `await photo.make(opts?)`, and likewise `model()`, `lensModel()`, - `dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`, - `iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()` - and `gpsAltitude()` → one common field each, picked from the tags `exif()` - returns and typed as in `PhotoExif`, or `undefined` when the file lacks it. - GPS values are signed decimal degrees and metres. `dateTimeOriginal()` is the - camera's clock reading held in the `Date`'s UTC fields; - `offsetTimeOriginal()`, when present, is that clock's offset from UTC. Each - calls `exif()` with its `opts`, so each call reads the original again. +An `Album` exposes its record's fields and `album.photos.list()`, which returns +`Photo[]`, newest first. A `Photo` exposes its record's fields except the cache +paths, plus `photo.year`, the local-time year of `takenAt`. All of these are +synchronous getters read from memory. `photo.record()` returns the +`PhotoRecord`. -They serve from the on-disk content cache when the bytes are present and -otherwise fetch through the pools; `original()`, `content()` and `exif()` also -serve an original already stored at its save path. `opts.onProgress` reports -per-file progress. They throw when the library was opened without a content -source. An original that `original()`, `content()` or `exif()` downloads lands -in the cache, which does not make `isLocal` true; only `download()` and -`lib.backup()` do. +Two more getters look at the disk and never touch the network: + +- `photo.savePath`: where `photo.download()` and `lib.backup()` put the + original, whether or not it is there yet (see "Save paths"). For a live photo + already stored, it is the image. For a live photo not yet stored, it carries + the title's extension, and the image may be stored under another one. +- `photo.isLocal`: whether the whole original is at `savePath`. A copy only in + the cache does not count. + +### A photo's content + +These methods may download. They serve from the cache when the bytes are present +and otherwise fetch through the pools. They throw when the library was opened +without a content source. `opts.onProgress`, where accepted, reports the +progress of that one file. + +- `await photo.original(opts?)` returns `{ path, bytes, videoPath? }`, the + full-size file. `bytes` is the file's size. For a live photo, `path` and + `bytes` are its image's and `videoPath` is its video. `original()` also serves + an original already stored at its save path. +- `await photo.download()` returns the same shape and puts the original at + `savePath`, creating its folders. When it is already there, nothing is + written. When the cache holds it, it is copied from the cache. Otherwise it is + fetched straight to `savePath`, with no copy left in the cache. Afterwards + `isLocal` is true. +- `await photo.thumbnail(opts?)` returns `{ path, bytes }`. +- `await photo.content(opts?)` returns the original's bytes as a `Uint8Array`, + read through `original()`. For a live photo, its image's. +- `await photo.exif(opts?)` returns `ExifTags`: every EXIF tag in the file, + keyed by tag name, each as exifreader decodes it with its `id`, `value`, + `description` and `computed` value. For example, `Make` is + `{ id: 271, value: ["Example"], description: "Example", computed: "Example" }`. + A tag exifreader has no name for is keyed `undefined-<tag number>`. The tags + of the thumbnail embedded in the file are under `Thumbnail`, so they cannot + hide the main image's tags of the same name. The thumbnail image itself is + left out. Any file exifreader cannot read gives `{}`, and a video gives `{}` + without being downloaded. +- `await photo.make(opts?)` returns one common EXIF field, picked from the tags + `exif()` returns and typed as in `PhotoExif`, or `undefined` when the file + lacks it. These methods do the same for their fields: + - `model()` and `lensModel()`; + - `dateTimeOriginal()` and `offsetTimeOriginal()`; + - `exposureTime()`, `fNumber()`, `iso()` and `focalLength()`; + - `orientation()`; + - `gpsLatitude()`, `gpsLongitude()` and `gpsAltitude()`, as signed decimal + degrees and metres. + +`dateTimeOriginal()` is the camera's clock reading, held in the `Date`'s UTC +fields. `offsetTimeOriginal()`, when present, is that clock's offset from UTC. +Each of these methods calls `exif()`, so each call reads the original again. + +An original that `original()`, `content()` or `exif()` downloads lands in the +cache, which does not make `isLocal` true. Only `download()` and `lib.backup()` +do. `await lib.originalOf(file, opts?)` and `await lib.thumbnailOf(file, opts?)` fetch and cache as `photo.original()` and `photo.thumbnail()` do. `file` is an -`EnteFile` the caller holds, and the library need not hold it. On an empty -cache, `quak get` and `quak get-thumb` pass one that `lib.freshFiles()` -returned. +`EnteFile` the caller holds, and the library need not hold it. `quak get` and +`quak get-thumb` use them; on an empty cache they pass a file that +`lib.freshFiles()` returned. -Lower-level accessors that return decrypted model objects (which hold key -material) are also available: `listCollections()`, `getCollection(id)`, -`listFiles(collectionID)`, `getFile(collectionID, fileID)`, and -`getFileByID(fileID)`. +Lower-level reads return the decrypted model objects, which hold key material: +`lib.listCollections()`, `lib.getCollection(id)`, `lib.listFiles(collectionID)`, +`lib.getFile(collectionID, fileID)` and `lib.getFileByID(fileID)`. ### Records and change notifications -The GUI-facing records hold no key material and no binary, so they survive -`structuredClone`/JSON across the Electron IPC boundary: +The records hold no key material and no binary, so they survive +`structuredClone` and JSON, for example between processes: -- `PhotoRecord`: `fileID`, `albumIDs`, `title`, `takenAt` and `modifiedAt` - (milliseconds), `fileType`, optional `caption` / `width` / `height` / - `latitude` / `longitude`, optional `hash` (the content hash recorded at - upload; very old files have none), `isArchived`, `isHidden`, and - `thumbnailPath` / `originalPath` once the bytes are cached (for a live photo, - `originalPath` is its image). -- `AlbumRecord`: `collectionID`, `name`, `type`, `isShared`, `updationTime`, and - `fileIDs` (newest first). -- `LibrarySnapshot`: `{ albums, photos, takenAt }`. +- `PhotoRecord` holds: + - `fileID` and `albumIDs`; + - `title`; + - `takenAt` and `modifiedAt`, in milliseconds; + - `fileType`, `isArchived` and `isHidden`; + - optional: `caption`, `width`, `height`, `latitude` and `longitude`; + - optional: `hash`, the content hash recorded at upload, which very old + files lack; + - optional: `thumbnailPath` and `originalPath`, once the bytes are cached. A + live photo's `originalPath` is its image. +- `AlbumRecord` holds `collectionID`, `name`, `type`, `isShared`, `updationTime` + and `fileIDs`, newest first. +- `LibrarySnapshot` is `{ albums, photos, takenAt }`. Here `takenAt` is when the + snapshot was taken, in milliseconds. -`lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first, -photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange` -(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`, -`refreshedAt`) whenever a refresh alters the projection, and returns -`{ unsubscribe }`; a refresh that changes nothing delivers nothing. +`lib.snapshot()` returns a `LibrarySnapshot`, albums newest updated first and +photos newest first. `lib.subscribe({ onChange })` delivers a `LibraryChange` +whenever a refresh changes the records, and returns `{ unsubscribe }`. A +`LibraryChange` holds `albumsChanged`, `photosChanged`, `fileIDsRemoved`, +`albumIDsRemoved` and `refreshedAt`. A refresh that changes nothing delivers +nothing. -### Thumbnails, ML search, and backup +### Thumbnails, machine-learning data and backup -- `lib.thumbnails.ensure({ fileIDs, priority, signal?, onProgress? })` - prefetches thumbnails through the thumbnail pool, deduped by fileID, returning - one `EnsureResult` (`{ fileID, path?, error? }`) per file. `priority` is - `"visible" | "ahead" | "background"`; only `"visible"` preempts background - work. -- `lib.mldata` searches the CLIP index built from Ente's per-file ML data: - `forFile({ fileID })` → `Promise<MLData | undefined>` (the whole stored - payload — face boxes, landmarks, embedding — read from disk on demand); - `similar({ fileID, limit? })` and `searchByEmbedding({ embedding, limit? })` → - `SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first, - default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes - a query vector the caller produced elsewhere. -- `await lib.backup(opts?)` → `BackupResult`. It takes the lock in the download - directory, and fails at once with an error whose `code` is `ELOCKED` while - another backup of it runs. A caller can instead take the lock itself, before - it opens its library, as `quak backup` does: `await lockBackupDirectory(dir)` - takes it, failing the same way, and returns a `BackupLock`, whose `release()` - removes the lock unless it was lost and whose `lost` holds why it was lost. - The caller passes it to the backup as `lock`, and releases it once the backup - ends. It waits for a refresh as `fresh()` does, puts every in-scope original - not already at its save path there as `photo.download()` does (and, with - `includeThumbnails`, fetches thumbnails) through the content cache, waits for - an ML data fetch, and rebuilds the on-disk backup tree with a durable failure - ledger. Each file's JSON holds its ML data and keywords. An image's JSON also - holds the EXIF and XMP of its original, and its dimensions if it is a JPEG. A - live photo's JSON holds these for its image. A video's JSON holds no EXIF, XMP - or dimensions. A fetched original is written straight to its save path and not - into the cache, which then counts it as present; one the cache already held is - copied from there. `BackupOptions`: `downloadDirectory` (falls back to the - library's), `includeOriginals` (default `true`), `includeThumbnails` (default - `false`), `onlyAlbumNames`, `verify` (default `false`), `onProgress`, and - `lock`. See Backup layout above for the tree it writes, and for the albums - `BackupResult` lists in `undecryptableAlbums` and `failedAlbums`. Whether the - backup took the lock or its caller did, once the lock is lost the backup - rejects with `lost the lock on <dir>: <reason>` before it starts another file. +`lib.thumbnails.ensure({ fileIDs, priority, signal?, onProgress? })` fetches +thumbnails ahead of need through the thumbnail pool, each file once, and returns +one `EnsureResult` per file: `{ fileID, path?, error? }`. `priority` is +`"visible"`, `"ahead"` or `"background"`. Only `"visible"` goes ahead of +background work; `"ahead"` is treated as `"background"`. + +`lib.mldata` searches the CLIP index built from Ente's per-file machine-learning +data: + +- `forFile({ fileID })` returns `Promise<MLData | undefined>`, the whole stored + payload (faces, landmarks and embedding), read from disk on demand. `MLData` + is a plain object type and is not exported by name. +- `similar({ fileID, limit? })` and `searchByEmbedding({ embedding, limit? })` + return `SimilarResult[]`, each `{ fileID, score }`, most similar first. The + score is the cosine similarity, from -1 to 1 (see "Keywords"). `limit` + defaults to 20. quak bundles no text encoder, so `searchByEmbedding` takes a + query vector the caller produced elsewhere. + +`await lib.backup(opts?)` returns a `BackupResult` and writes the tree described +under "The backup directory". It takes the lock in the download directory and +fails at once, with an error whose `code` is `ELOCKED`, while another backup of +it runs. Then it waits for a refresh as `fresh()` does. It puts the original of +every file in the albums it backs up at its save path, as `photo.download()` +does. It waits for a machine-learning fetch, and rebuilds the JSON files and +symlinks. Once the lock is lost, it rejects with +`lost the lock on <dir>: <reason>` before it starts another file. + +`BackupOptions`: + +- `downloadDirectory`: where the tree lives. Default: the library's. +- `includeOriginals`: fetch and store originals. Default `true`. With `false`, + the backup also fetches no machine-learning data, and writes no JSON beside + the originals and no symlinks. +- `includeThumbnails`: also store thumbnails as `thumbnails/<fileID>.jpg` beside + `collections/`. Default `false`. `quak backup` does not set it. +- `onlyAlbumNames`: back up only the albums with these names. +- `verify`: as `--verify`. Default `false`. +- `onProgress`: called with each progress line and, for a line that reports a + failure, the level `"error"`. +- `lock`: a lock the caller took itself. + +A caller can take the lock before it opens its library, as `quak backup` does: +`await lockBackupDirectory(dir)` takes it, failing the same way, and returns a +`BackupLock`. Its `release()` removes the lock unless it was lost, and its +`lost` holds why it was lost. The caller passes it to the backup as `lock` and +releases it once the backup ends. ### Request pools -`RequestPools` holds three independent bounded pools — metadata (10), content -(5), thumbnails (25) — because Ente meters these traffic classes differently. -Each pool orders on-demand work ahead of background/precache work and dedups -in-flight fetches by key, and an idle pool never lends its slots to a busy one. +`RequestPools` holds three independent pools, each with its own limit on how +many requests run at once, because Ente meters these three kinds of traffic +differently: -`lib.backup()`, and so `quak backup`, works on up to the content pool's limit of -originals at once, 5 unless the library was opened with other `pools`. The limit -covers everything it does for one original, hashing it for `verify` included, so -however large the account is, no more originals than that are open at once. +- metadata, with 10 slots: only the machine-learning data requests; +- content, with 5: originals; +- thumbnails, with 25: thumbnails. -### On-disk cache layout +Only the requests named above go through the pools. A refresh, +`freshCollections()` and `freshFiles()` send theirs outside them. Construct one +with +`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })` +and pass it as `pools`. Each pool serves on-demand work ahead of background +work. A task submitted under a key that is still pending is not run a second +time. An idle pool never lends its slots to a busy one. -Under `cacheDirectory`: +`lib.backup()` works on as many originals at once as the content pool allows, 5 +by default. `quak backup` uses the default. The limit covers everything it does +for one original, hashing for `verify` included. -``` -<cacheDirectory>/ - metadata.json decrypted account state + refresh cursor - originals/<fileID>.<ext> cached full-resolution files - originals/<fileID>.livephoto.json - which of a live photo's two files is which - thumbnails/<fileID>.jpg cached thumbnails - mldata/ - <fileID>.json one decrypted ML payload per file - clip.f32, clip.json the packed CLIP index and its id list - fetched.json per-file fetch bookkeeping -``` +### Retries and timeouts -When `metadata.json` belongs to a different account than the client's, -`Library.open` deletes it and `mldata/` and starts from an empty cache. Cached -originals and thumbnails are kept; they are reached only through the files the -current account's records name. +Every request in the library goes through one policy. A request is repeated only +when repeating it could produce a different answer: -A live photo's original is cached as at its save path: its image and its video, -each `originals/<fileID>.<ext>` with its own extension, and -`originals/<fileID>.livephoto.json` naming them; the two are evicted together. +- An `ApiError` with a 5xx status is retried. So are `408` and `429`, the two + 4xx codes that are statements about timing rather than about the request. +- Every other 4xx is not retried. A 404 in particular is an answer, and + `list-missing-thumbnails` depends on getting it promptly and once. +- A transport failure is retried: a rejected `fetch`, `ECONNRESET`, `ETIMEDOUT`, + a DNS or TLS failure, and one of quak's own deadlines. Node's `fetch` rejects + with a plain `TypeError`, so every `TypeError` is retried. The error code is + looked for along the error's `cause` chain, which is where Node's `fetch` puts + it. +- A truncated download is retried. +- Anything else is not retried. This includes an authentication failure in the + encrypted stream that is not truncation. -A stored file appears only via an atomic temp-then-rename, so its presence means -it is complete. Every downloaded original (by `quak get`, the cache, or -`backup`) whose metadata records a content hash (`FileMetadata.hash`) is hashed -as it is written: unkeyed BLAKE2b with a 64-byte output, standard base64. A live -photo arrives as a ZIP and is unpacked as it is written; its image and its video -are hashed separately and joined as `<imageHash>:<videoHash>`, and neither is -stored unless both are complete and match. A live photo ZIP holding a second -image or video, or whose two files together come to more than 20 times its size -plus 16 MiB, is refused. A mismatch or a refusal stores nothing and fails the -download with an error naming the file ID. An original with no recorded hash, -from a very old client, is stored unchecked. +The binary that `make install` builds runs under bun. Bun's `fetch` rejects with +a plain `Error` that has no `cause`. That error carries a code of bun's own. +quak retries a refused connection, a name that does not resolve, a reset +connection and a deadline under bun as it does under Node. Bun reports a refused +connection, and a failed TLS handshake, as `ConnectionRefused`. For a name that +does not resolve it gives `ConnectionRefused` on some requests and +`FailedToOpenSocket` on others. A connection reset or closed before the response +ended is `ECONNRESET`, as under Node, and a deadline is the same `TimeoutError`. +A certificate bun rejects, with a code such as `DEPTH_ZERO_SELF_SIGNED_CERT`, is +not retried. + +Backoff is exponential with full jitter. The delay before retry _n_ is +`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term +is the ceiling, and the wait is drawn at random below it. So a client that lost +many parallel downloads to one blip does not send them all again at the same +instant. The numbers are set through `ApiClientOptions.retry`: + +- `attempts`: total calls, not retries. Default `4`; `quak backup` uses `10`. +- `baseDelayMs`: the ceiling for the first retry's delay. Default `500`; + `quak backup` uses `1000`. +- `maxDelayMs`: the upper bound on that ceiling. Default `10000`; `quak backup` + uses `60000`. + +With the defaults, a request that keeps failing gives up after at most three and +a half seconds of waiting. `quak backup` usually runs from cron with nobody +watching, so every request it makes uses its own numbers, exported as +`UNATTENDED_RETRY_OPTIONS`. Under them, a request that keeps failing gives up +after at most 243 seconds of waiting. It usually gives up after about half that, +since each wait is drawn at random below its ceiling. Every other command uses +the defaults. A library user gets the backup's budget by passing +`UNATTENDED_RETRY_OPTIONS` as `ApiClientOptions.retry`. `sleep` and `random` can +be injected through the same option, which is how the test suite exercises the +policy without waiting. `onRetry`, given there too, is called before each wait +with the error, the attempt's number and the wait. + +Two deadlines apply, each renewed for every attempt: + +- `requestTimeoutMs`, default `30000`, bounds a whole JSON request or upload. +- `downloadTimeoutMs`, default `60000`, is an idle deadline on a file or + thumbnail download: the download is aborted when no bytes have arrived for + that long. + +They differ because a download's length depends on the file and the link. A +whole-transfer deadline short enough to catch a hung connection would cancel a +large video on a slow link that is still making progress. The download deadline +restarts each time bytes arrive, so a slow download runs as long as it keeps +moving, and one that stalls is aborted after 60 seconds of silence. It covers +the wait for the headers as well as the body. There is no limit on the total +length of a download. + +A request that changes state is not blindly replayed. Every `POST` and `PUT` in +the endpoint list under "HTTP API" goes through this rule. Verifying a second +factor, for one, uses up one of a small number of attempts. Such a request is +retried only when every error code in its `cause` chain proves that no TCP +connection to the server ever existed. Three codes prove that: + +- `ENOTFOUND`: name resolution found no address; +- `EAI_AGAIN`: name resolution failed for the moment; +- `ECONNREFUSED`: the peer refused the connection. + +A 5xx, a reset mid-flight and a deadline are all left to the caller, because +each of them can happen after the server has already acted. These requests do +not follow redirects either: a redirect means the server already received the +request, so it is reported as an error and not retried. + +The routing errors `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are left out of +that rule although they look like connect-time failures. On Linux they are also +delivered on an established socket, after the request may have been written. +They stay retryable for requests that change nothing. Bun's `ConnectionRefused` +and `FailedToOpenSocket` are treated the same way: a request that changes +nothing is retried on them, and a `POST` or `PUT` is not replayed on them. When +a connection bun kept open from an earlier request closes before any answer, bun +sends the request again on a new connection. If that one fails, bun gives one of +these codes although the server may already have the request. The upload of a +thumbnail to its presigned URL is exempt from the rule. It stores one whole +object at one key in one request, so replaying it has no partial state to +damage. + +A download is retried as a whole: the request, the reading of the body and the +decryption. A socket reset after the response headers have arrived surfaces in +the download layer rather than in the HTTP client. That is the common failure +for a large file over a CDN. The encrypted stream cannot be resumed and the +endpoints have no Range support, so a retry starts the file over. Each attempt +writes its own temporary files and removes them if it fails. Only the attempt +that completes renames anything into place. + +One imprecision is deliberate. When a body ends partway through an encrypted +chunk, the authentication check fails and carries no sign of why, so a cut +connection and corrupt bytes look the same. quak reports that as truncation, +which means it is retried. quak can tell the two apart only in a body of more +than one chunk. There, a chunk that fails while the stream carries on past it +stays an authentication failure and is not retried. Most thumbnails and every +small file are one chunk. There a wrong key, corrupt bytes and a cut connection +look alike, and all are retried. The attempt count limits the extra retries. In +return, quak never keeps a truncated file. + +### The `Client` + +`Client.login({ email, password, totp?, emailOTP?, apiOptions? })` logs in and +resolves to a `Client`. `totp` and `emailOTP` are callbacks that return the code +when the account asks for one. Without the callback the account needs, login +throws. A passkey challenge always throws. After an email code, login expects +the key attributes and the token. When the server asks for a second factor +instead, login throws. `apiOptions` configures the HTTP client (see below). + +The `Client` holds the session token, the master key, the secret key and the +public key in memory. There is no on-disk session store in the library; the +caller decides how to keep a session. `client.toJSON()` returns a +`ClientSnapshot`, a plain object with the keys in base64, which the caller can +write wherever it likes. `Client.fromJSON(snapshot, apiOptions?)` restores a +working client from that snapshot without logging in again. It checks every +field and each key's length first, and throws an error naming the bad field. + +`Client.fromJSON` works only once libsodium, the cryptography library quak uses, +is loaded. Only `Client.login` loads it. A program that restores a session in a +new process first runs `await sodium.ready`, with `sodium` imported from +`libsodium-wrappers-sumo`, which quak depends on. The exported `beginLogin`, +`unwrapAuth`, `decryptCollection`, `decryptFile`, `downloadFile` and +`downloadThumbnail` need the same. + +`client.logout()` clears the token and zeroes the key buffers in place. After +it, every other method on that client throws. It does not contact the server, so +the token stays valid there and in any saved snapshot. Call +`await client.logoutOnServer()` before `logout()` to end the session on the +server. + +`client.whoami()` returns `{ email, userID }`. + +`client.collectionsSince({ sinceTime })` and +`client.filesSince({ collectionID, collectionKey, sinceTime })` enumerate the +changes since a microsecond timestamp, with `0` for everything. Each returns: + +- `collections` or `files`: the decrypted records that are not deleted; +- `deleted`: the IDs of deleted records; +- `undecryptable`: each record that did not decrypt, as `{ id, error }`; +- `cursor`: the timestamp to pass next time. + +`client.listCollections()` and `client.listFiles(collectionID, collectionKey)` +return only the decrypted records that are not deleted, from `sinceTime` `0`. +`client.fetchMLData({ fileIDs, fileKeys })` fetches and decrypts the +machine-learning data of up to 200 files. `client.downloadFile(file, outPath?)` +and `client.downloadThumbnail(file, outPath?)` download and decrypt one file to +`outPath`. For a live photo, `downloadFile` writes the image and the video, each +named `outPath` with its own extension. `quak get --out` instead puts the image +at the path itself. Without `outPath`, they write to the working directory under +the names `quak get` and `quak get-thumb` would use. + +`ApiClientOptions`: + +- `apiOrigin`: the API server. Default `https://api.ente.io`. A self-hosted + server is set here; file and thumbnail downloads then go through it too. +- `filesOrigin` and `thumbsOrigin`: the download hosts used with the default API + server. +- `authToken`: a session token to send from the start. +- `fetch`: a replacement for the global `fetch`. +- `retry`, `requestTimeoutMs` and `downloadTimeoutMs`: see "Retries and + timeouts". +- `onRequest`: called once for each request sent, and once for each attempt of a + retried one. It gets the method, the URL and the status of the answer or the + error. The URL of a thumbnail upload is given without its query, which holds + the upload's signature. + +The package also exports the pieces the `Client` and the `Library` are built +from, and a few constants: + +- the HTTP client: `ApiClient`, `ApiError` and `TruncatedStreamError`, with + `DEFAULT_REQUEST_TIMEOUT_MS` and `DEFAULT_DOWNLOAD_TIMEOUT_MS`; +- the retry policy: `withRetry`, `isRetryable`, `isSafeToReplay`, + `resolveRetryOptions`, `DEFAULT_RETRY_OPTIONS` and `UNATTENDED_RETRY_OPTIONS`; +- the login steps: `beginLogin`, `submitTOTP`, `requestEmailOTP`, + `submitEmailOTP` and `unwrapAuth`; +- the record decryption: `decryptCollection` and `decryptFile`; +- the downloads: `downloadFile` and `downloadThumbnail`; +- the backup: `runBackup` and `lockBackupDirectory`; +- the pools: `RequestPools` and `BoundedPool`, with + `DEFAULT_METADATA_CONCURRENCY`, `DEFAULT_CONTENT_CONCURRENCY` and + `DEFAULT_THUMBNAIL_CONCURRENCY`; +- the library: `Library`, `Album`, `Photo` and + `DEFAULT_REFRESH_INTERVAL_SECONDS`; +- `VERSION`: the version string `quak --version` prints. ### Key types by source file @@ -1408,8 +1524,8 @@ from a very old client, is stored unchecked. - `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`, `LibraryChange` - `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult` -- `src/exif.ts`: `ExifTags`, `PhotoExif` - `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool` +- `src/exif.ts`: `ExifTags`, `PhotoExif` - `src/backup.ts`: `BackupLock`, `BackupOptions`, `BackupResult`, `BackupError` - `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot` - `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `StreamOptions` @@ -1417,62 +1533,417 @@ from a very old client, is stored unchecked. - `src/retry.ts`: `withRetry`, `isRetryable`, `isSafeToReplay`, `RetryOptions` - `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileType`, `CollectionType`, `RawCollection`, `RawEnteFile` -- `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult` + +`src/index.ts` lists what the package exports. `MLDataAPI` and `SimilarResult` +are the types of `lib.mldata` and of its results, and `MLData` is the type of a +stored payload. None of the three is exported by name. + +## Examples + +`examples/download-albums.ts` downloads every album's photos and their metadata +into a directory, `photos` in the working directory unless you name another. It +uses only the library's public API. The build compiles it; run it after +`yarn install`: + +```bash +yarn build +QUAK_EMAIL=… QUAK_PASSWORD=… node dist/examples/download-albums.js [dir] +``` + +It opens the library with `precacheThumbnails` and `precacheOriginals` off, as +`quak backup` does, so the only file content it fetches is the originals it +saves. It asks on the terminal for a second-factor code or an email code when +the account requires one. It writes: + +- each photo's original at its save path under `dir`, as `photo.download()` + writes it, and for a live photo its image, its video and the `.livephoto.json` + file naming them; +- beside each original, a JSON file named after it with `.json` added, such as + `2026-03-01.12345.jpg.json`. It holds the photo's record without its cache + paths, and under `exif` every EXIF tag of the photo; +- `albums/<collectionID>.json` for each album, holding its `collectionID`, its + `name` and, under `savePaths`, the save paths of its photos relative to `dir`, + newest first. + +A photo in several albums is downloaded once. A second run downloads nothing and +rewrites only the JSON files whose content changed. A failed download stops the +run; running it again carries on, since every photo already saved is skipped. + +## Rationale + +Ente encrypts photos end to end, so a backup of an Ente account cannot be made +from the server's side. Only a client that holds the account's keys can read the +photos. Ente's own clients are apps and a command-line tool; none is a library a +program can use. quak is the client for scripts and for cron. + +quak is a library first, so that a program of your own can read an account the +way quak's command line does. The backup it writes is plain files in a tree a +photo viewer can open, with no quak needed to read it. One failed file never +ends a run; it is recorded and tried again next time. + +## Design + +quak is a TypeScript library with a thin command-line wrapper. The library does +the work; the command line is for people and cron. Login uses SRP, a protocol +that proves the password to the server without sending it. + +### Layout + +``` +quak/ + src/ + crypto/ libsodium primitives: boxes, streams, key derivation, + hashing + api/ the HTTP client (ApiClient) + auth/ the login flow (SRP, email code, TOTP) and key unwrap + model/ the decrypted Collection, EnteFile and FileMetadata + types, and the functions that decrypt them + download/ streaming download and decryption of files and + thumbnails + library/ the Library over the cache: metadata store, reads, + records, content cache, background downloads, + machine-learning data and search, request pools + backup.ts the full-account backup + metadata-backup.ts + backup-metadata: the metadata quak keeps, as JSON + exif.ts EXIF read from an image's bytes with exifreader + mldata-fetch.ts fetch and decrypt per-file machine-learning data + clip-keywords.ts + a file's keywords from its CLIP embedding + clip-keyword-vectors.ts + Ente's Discover categories with their text + embeddings, written by bin/clip-keyword-vectors + filename.ts safe file names from server metadata + errors.ts error types shared across layers, and how an error + is described + retry.ts the retry classifier and the backoff loop + thumbnails.ts find and regenerate missing thumbnails + client.ts the Client class assembled from the above + cli-commands.ts the command line's commands as functions returning + exit codes + cli-log.ts the command line's log level and the one function + its commands write stderr lines through + cli-output.ts how the command line prints a file's title and time + cli-read.ts how the read commands get current data + cli-run.ts run a command, print its error, exit with its code + cli-session.ts read the saved session file back into a Client + index.ts the package's exports + bin/ + quak.ts the command-line entrypoint (commander.js) + clip-keyword-vectors, clip-keyword-vectors.py + write src/clip-keyword-vectors.ts and a test + fixture (see "Source attribution") + examples/ + download-albums.ts + download every album's photos and metadata + test/ unit and integration tests (vitest) + script/ the entrypoints (see "Entrypoints") + Makefile + Dockerfile lint phase, test phase, compile + package.json + tsconfig.json +``` + +`make build` compiles `src/`, `bin/` and `examples/` into `dist/`, keeping their +shape: the library lands in `dist/src/`, the examples in `dist/examples/` and +the command line in `dist/bin/quak.js`. `package.json` points `main` and `types` +at `dist/src/index.js` and `dist/src/index.d.ts`, and `bin` at +`dist/bin/quak.js`. The compiler's `rootDir` is the repository root rather than +`src/`, because `bin/` and `examples/` are compiled too. + +### Cryptography + +All cryptography is done by `libsodium-wrappers-sumo`, with two exceptions. The +SRP handshake uses `fast-srp-hap`, and the MD5 checksum sent with a thumbnail +upload uses Node's `node:crypto`. The "sumo" build is required for +`crypto_pwhash`, the Argon2id function. No cryptography is written by hand. + +Login derives the keys in this order: + +1. The user enters the password. +2. Argon2id (`crypto_pwhash`) derives a 32-byte key encryption key, the KEK, + from the password. It uses the server-issued `kekSalt`, `memLimit` and + `opsLimit`. +3. For SRP login, `crypto_kdf_derive_from_key` (BLAKE2b) derives a 32-byte + subkey from the KEK with subkey ID 1 and context `loginctx`. Its first 16 + bytes are the SRP password. +4. When SRP completes, the server returns the key attributes and an encrypted + token, or first asks for a second factor. quak answers a TOTP request with + the code (`POST /users/two-factor/verify`); the server then returns the key + attributes and the token. quak cannot answer a passkey request. When the + account has email verification on (`isEmailMFAEnabled`), an email code + replaces SRP. quak then expects the key attributes and the token in the + answer, and fails the login when the server asks for a second factor instead. +5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK + yields the 32-byte master key. +6. `crypto_secretbox_open_easy` over the encrypted secret key with the master + key yields the user's X25519 private key. The matching public key arrives in + cleartext. +7. `crypto_box_seal_open` over the encrypted token with the user's key pair + yields the token's bytes. Encoded as URL-safe base64 with padding, they are + the `X-Auth-Token` value for every later call. + +A collection the account owns carries its key as a secretbox under the master +key. A collection shared with the account carries it as a sealed box to the +account's public key. Each file's key is a secretbox under its collection's key. +File metadata is a secretstream blob, a single chunk with the final tag, under +the file key. File content is a chunked `crypto_secretstream_xchacha20poly1305` +stream under the file key, with 4 MiB plaintext chunks and 17 bytes of +authentication overhead per chunk. A thumbnail is such a stream too. For a +thumbnail upload, `encryptBlob` does the push side: one chunk with the final +tag, returning the header and the ciphertext. + +### HTTP API + +The production endpoints are: + +- API: `https://api.ente.io` +- File downloads: `https://files.ente.io/?fileID=<id>` +- Thumbnail downloads: `https://thumbnails.ente.io/?fileID=<id>` + +A self-hosted server is set through the `apiOrigin` option. File downloads then +go through `<apiOrigin>/files/download/<id>` and thumbnail downloads through +`<apiOrigin>/files/preview/<id>`. + +Every authenticated call carries two headers: `X-Auth-Token`, the token from +login, and `X-Client-Package`, which names the client as `berlin.sneak.quak`. + +Endpoints used: + +- `GET /users/srp/attributes?email=<email>`: fetch the SRP and key-derivation + parameters. +- `POST /users/srp/create-session`: begin the SRP handshake. +- `POST /users/srp/verify-session`: complete SRP; receive the second-factor + challenge or the encrypted token with the key attributes. +- `POST /users/ott` and `POST /users/verify-email`: the email code, used instead + of SRP when the SRP attributes have `isEmailMFAEnabled` set. +- `POST /users/two-factor/verify`: the TOTP second factor. +- `POST /users/logout`: end the calling token's session (`quak logout`). +- `GET /collections/v2?sinceTime=<usec>`: list the collections changed since a + microsecond timestamp; `0` lists them all. +- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list the files + changed in a collection; quak pages while `hasMore` is true. +- `GET https://files.ente.io/?fileID=<id>`: download a file's encrypted bytes. +- `GET https://thumbnails.ente.io/?fileID=<id>`: download a thumbnail's + encrypted bytes. +- `POST /files/data/fetch`: fetch the encrypted machine-learning data of a batch + of files. +- `POST /files/upload-url`: get a presigned upload URL, for thumbnail repair. +- `PUT /files/thumbnail`: register an uploaded thumbnail's object key. ## Source attribution -The cryptographic protocol and wire format implemented here are Ente's, taken -from the Ente open source clients at <https://github.com/ente-io/ente>. No code -is imported or vendored from those projects; any reference code that is copied -is rewritten in TypeScript in this repository. Protocol fidelity is verified -against the upstream implementations in `web/packages/base/`, -`mobile/apps/photos/lib/`, and `cli/`. +The cryptographic protocol and wire format are Ente's, read from the Ente +clients at <https://github.com/ente-io/ente>. No code from those projects is +included; quak's implementation is written in TypeScript from reading them. The keywords come from Ente too. `src/clip-keyword-vectors.ts` holds the Discover categories of Ente's photos app, from -`mobile/apps/photos/assets/discover.json`, each with its query's text embedding -from Ente's CLIP text model. `bin/clip-keyword-vectors` writes that file and the -CLIP embedding of a public-domain photo that the tests use. It runs Ente's -models in Docker, with every download pinned by sha256. It is run by hand only, -followed by `make fmt`. +`mobile/apps/photos/assets/discover.json`, each with the text embedding of its +query from Ente's CLIP text model. `bin/clip-keyword-vectors` writes that file, +and the CLIP embedding of a public-domain photo that the tests use. It runs +Ente's models in Docker, with every download pinned by SHA-256 hash. It is run +by hand only, followed by `make fmt`. -## For LLMs +## Entrypoints -If you are an LLM agent working on this repository, read and follow these -documents: +This repository adheres to the +[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) +standard. Normalized scripts in `script/` are the entrypoints for the +development workflow, and most Makefile targets are thin shims that call them. +The scripts are POSIX `sh`, not bash, so they run in minimal containers such as +Alpine. Of these, only `bootstrap`, `setup`, `test` and `cibuild` come from the +standard. The others are additions to it. -- **`REPO_POLICIES.md`** in the repo root. It is copied from +- `script/bootstrap`: install all dependencies. It installs `make`, `git`, Node + and `yarn` when they are missing, then runs `yarn install --frozen-lockfile`. +- `script/setup`: set up the repo for development after a fresh clone. It runs + `script/bootstrap`, then `script/install-precommit`. +- `script/projectname`: print the project name. `script/docker` uses it for the + image tag. +- `script/build`: compile the TypeScript sources into `dist/` and stamp the + version into `dist/package.json`. Then check that the entrypoints + `package.json` declares (`main`, `types`, `bin`) were written, and make the + command line executable. +- `script/version`: print the version `script/build` stamps (see "Version"). +- `script/test`: run the test suite, by building the `test` phase of the + `Dockerfile`. Requires Docker. +- `script/lint`: run eslint and a prettier check, by building the `lint` phase + of the `Dockerfile`. Requires Docker. +- `script/fmt`: format all files with prettier. It rewrites files in place. +- `script/fmt-check`: check formatting on the host. It is read-only. +- `script/check`: run all checks: `test`, `lint`, `fmt-check`. +- `script/docker`: build the image, tagged with `script/projectname`. +- `script/cibuild`: what CI runs: `script/bootstrap`, then `script/check`, then + the image build. +- `script/precommit`: run by the git pre-commit hook. It runs `script/lint` and + `script/fmt-check`, and not the tests, so that a commit of failing tests can + land (see "Development workflow"). +- `script/install-precommit`: install the git pre-commit hook. `make hooks` + calls it. + +The Makefile targets are: + +- `make bootstrap`: runs `script/bootstrap`. +- `make setup`: runs `script/setup`. +- `make test`: runs `script/test`. +- `make lint`: runs `script/lint`. +- `make fmt`: runs `script/fmt`. +- `make fmt-check`: runs `script/fmt-check`. +- `make check`: runs `script/check`. +- `make build`: runs `script/build`. +- `make docker`: runs `script/docker`. +- `make hooks`: runs `script/install-precommit`. +- `make build-bin`: bundles the built `dist/` into one binary, `bin/quak`. It + runs bun through `nix-shell -p bun`. +- `make install`: runs `make build-bin` and copies the binary to `~/bin/quak`. +- `make dev`: runs the TypeScript compiler in watch mode. +- `make clean`: removes `dist/`, `coverage/`, `.vitest-cache/` and the + compiler's `*.tsbuildinfo` files. + +### Linting and testing + +Linting and testing are phases of the `Dockerfile`. The `lint` phase copies the +repo into a node image pinned by digest and runs eslint and +`prettier --check .`. The `test` phase runs the test suite the same way, as the +image's non-root `node` user. A 90-second `timeout` stops a hung test. A suite +that fails is run again with verbose output. `script/lint` and `script/test` +each build one phase with `docker build --no-cache --target <phase>`. Neither +script runs the tools on the host, so Docker is required. The scripts also work +where the Docker daemon is remote, because they copy the repo into the image +instead of sharing a directory with the container. + +The last stage of the `Dockerfile` compiles the package. It copies a file from +each phase, so it cannot be built unless lint and the tests pass. The image +build in `script/cibuild` therefore runs lint and the tests a second time, after +`script/check` has run them. + +Every `docker build` in `script/` passes `--no-cache`. On an unchanged tree +Docker would otherwise serve the lint and test steps from its cache, nothing +would run, and the build would still exit 0. + +`script/fmt-check` runs prettier on the host. Its verdict matches the `lint` +phase's, because prettier is pinned to one version and installed from +`yarn.lock` in both places. Prettier reads `.gitignore` as its default ignore +file, which is why `.dockerignore` keeps `.gitignore` in the build context. + +### Version + +`quak --version` reports the `version` in `dist/package.json`, which +`script/build` writes after compiling. The repo's own `package.json` keeps +`0.0.0`. The tests run from source, so they report `0.0.0`. `script/version` +decides what is written: + +- the `VERSION` environment variable, or the `Dockerfile`'s `VERSION` build + argument (`--build-arg VERSION=...`), when one is given and not empty; +- otherwise, in a checkout with `.git`, what `git describe --tags --always` + prints. That is the tag on a tagged commit. On a later commit it is the tag, + the number of commits since it and the short commit, as in + `v1.2.3-4-gabc1234`. When no tag is reachable it is the short commit; +- otherwise, as in a source tarball, the version `package.json` declares. + +The build fails if the checkout has `.git` and the version still comes out +empty, `dev` or `unknown`, because such a build could not be traced to its +commit. + +`.dockerignore` therefore does not leave out `.git`, so any `docker build .` of +a clone stamps the commit it was built from. A clone of depth 1 stamps a tag +only when its commit carries one, and otherwise the short commit. +`.dockerignore` does leave out `.git/config`, which holds the clone's remote URL +and any credential in it; `git describe` does not need that file. +`script/docker` and `script/cibuild` work out the version on the host, with +`--dirty`, and pass it as the `VERSION` build argument. That argument takes +precedence. The image's `org.opencontainers.image.version` label carries that +build argument only, so a build given none leaves it empty. `make build-bin` +bundles the built `dist/`, so the single binary reports the stamped version too. + +## Development workflow + +All work on quak is test-driven. + +1. Every change starts on a feature branch off `next`, and its pull request + targets `next`. +2. The first commit on the branch is the tests for what is being added or + changed. Those tests fail at that commit; the branch is red until the + implementation lands. +3. Later commits add the implementation and any refactors needed to make the + tests pass. +4. A pull request is merged into `next` only when `make check` is green. Once it + has passed review, a maintainer squash-merges it into `next`. Only sneak + merges `next` into `main`. `main` and `next` are always green. CI runs + `script/cibuild`, which builds the `Dockerfile`: its `lint` and `test` + phases, then the compile. So neither a red branch nor one that does not + compile can pass CI. +5. Tests are the canonical API documentation for this library. Every test file + is commented well enough that a reader who has never seen quak can learn how + to use it from the tests alone. Comments explain why a behavior matters, not + just what the assertion checks. +6. Test fixtures, such as cryptographic vectors, recorded HTTP responses and + sample files, are committed beside their tests. Where possible they are + generated by deterministic helpers in `test/`, so a reviewer can reproduce + them by running the helper. +7. `git rebase -i` is allowed on a feature branch before merge, to clean the + tests-then-implementation sequence into reviewable commits. The final history + must still show tests landing before, or with, the matching implementation. +8. The pre-commit hook that `make hooks` installs runs `script/precommit`, which + runs `script/lint` and `script/fmt-check` and not the tests. This is + deliberate, so that the commit of failing tests can land. CI runs the tests, + so a red branch still cannot reach `next`. + +### Rules for contributors + +These apply to people and to coding agents alike. + +- Read and follow `REPO_POLICIES.md` in the repo root. It is copied from <https://git.eeqj.de/sneak/prompts> and covers repository structure, tooling, - Makefile targets, Dockerfile conventions, dependency pinning, and commit - hygiene. All external dependencies must be pinned by cryptographic hash in - `yarn.lock`. Never `git add -A`. Never force-push to main. + Makefile targets, Dockerfile conventions, dependency pinning and commit + hygiene. Never `git add -A`. Never force-push `main`. +- Dependencies are pinned: + - npm packages by hash, in `yarn.lock`; + - container images by digest; + - the nvm archive `script/bootstrap` downloads, and the Python packages + `bin/clip-keyword-vectors` installs, by SHA-256 hash; + - Node and yarn, when `script/bootstrap` installs them, by version. +- Not pinned: + - bun, which `make build-bin` takes from `nix-shell -p bun`; + - `make` and `git`, which `script/bootstrap` installs from the system + package manager when they are missing, as every `Dockerfile` stage does. +- Follow the workflow above: branches off `next`, tests first and committed + failing, and `next` merged into `main` by sneak alone. +- Before every commit, `make lint` and `make fmt-check` must pass. `make lint` + builds the `lint` phase of the `Dockerfile`, so it needs Docker. The + pre-commit hook enforces exactly that. `make check`, which also runs the + tests, must pass before a merge into `next`. Never invoke eslint or prettier + directly; linting runs in the container only. +- Format with `make fmt`: prettier with 4-space indents and `proseWrap: always` + for markdown. Use `yarn`, not `npm`. +- Tests use vitest and live in `test/`, mirroring `src/`. Use `mkdtempSync` for + temporary directories, never a path built from a timestamp. `make test` must + finish in under 60 seconds. The 90-second `timeout` in the `test` phase of the + `Dockerfile` only catches a hung test. +- Code style: `const` for everything, `let` only when a variable is reassigned, + never `var`. Avoid comments that restate the code. No cryptography written by + hand. `LLM_PROSE_TELLS.md` in the prompts repo applies to any prose written in + this repository: the README, comments and commit messages. -- **The "Development workflow" section above.** All changes go on feature - branches off `next`, and every pull request targets `next`; only sneak merges - `next` into `main`. Tests are written first and committed in a failing state - before the implementation. Tests are the canonical API documentation and must - be commented thoroughly. `main` and `next` are always green. +## TODO -- **Required checks before every commit:** `make lint` and `make fmt-check` must - pass. `make lint` is eslint plus the prettier check, and it builds the `lint` - phase of the `Dockerfile`, so it needs docker. The pre-commit hook enforces - exactly that. `make check` (which also runs the tests) must pass before - merging into `next`. Never invoke eslint or prettier directly; linting runs in - the container only. +`TODO.md` in the repo root is the work log: the next step, the steps done and +the steps planned. Open now: -- **Formatting:** prettier with 4-space indents and `proseWrap: always` for - markdown. Use `make fmt` to format. Use `yarn` not `npm`. +- [ ] Check quak against a live Ente account. This needs an account only sneak + has. -- **Testing:** vitest. Tests go in `test/` mirroring the `src/` structure. - `make test` must finish in under 60 seconds (the hard cap) and should finish - in under 20. The 90-second `timeout` in the `test` phase of the `Dockerfile` - is a backstop that catches a hung test, not the time limit. Use `mkdtempSync` - for temporary directories, never manual timestamp paths. +Planned for a desktop client, in a separate repository, on top of this library: -- **Code style:** `const` for everything, `let` if reassignment is needed, never - `var`. Avoid unnecessary comments. No hand-rolled crypto. The - `LLM_PROSE_TELLS.md` document in the prompts repo applies to any prose written - in this repository (README, comments, commit messages). +- [ ] An Electron app skeleton that uses this library. +- [ ] A local cache in SQLite keyed on `(collectionID, fileID, updationTime)`. +- [ ] A background sync worker that streams new files into the cache. +- [ ] A gallery UI: thumbnails, full-image view, basic search. +- [ ] Upload, delete and share operations in the library. + +Tagging and releases are decided by sneak alone, and happen only when he +declares one. ## License diff --git a/src/cli-log.ts b/src/cli-log.ts index 88916c2..f4c3020 100644 --- a/src/cli-log.ts +++ b/src/cli-log.ts @@ -4,7 +4,7 @@ // Every line quak prints on stderr is a log line at one of six levels, from // the quietest: fatal, error, warning, info, debug, trace. A run prints the // lines at its own level and at every quieter one. Its level is info unless -// `-v` or `-q` asks for another (see "CLI surface" in the README). Every +// `-v` or `-q` asks for another (see "What quak prints" in the README). Every // command that exits non-zero logs a fatal line saying why, so even `-q`, which // prints only fatal lines, never leaves a non-zero exit unexplained. An error // line reports something that failed, such as a file or an album, and a diff --git a/test/cli/backup-end-to-end.test.ts b/test/cli/backup-end-to-end.test.ts index 8f075f7..6b2e432 100644 --- a/test/cli/backup-end-to-end.test.ts +++ b/test/cli/backup-end-to-end.test.ts @@ -418,7 +418,7 @@ describe("the first backup of the account", () => { expect(downloads.sort()).toEqual([100, 200, 300]); }); - // The tree "Backup layout" in the README describes. Each original is at + // The tree under "The backup directory" in the README. Each original is at // its save path, `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, with // the extension of its title; the live photo is its image and its video, // each with the extension it had inside the ZIP, and a `.livephoto.json`