quak/README.md

# quak

quak is a WTFPL-licensed TypeScript client library and CLI by
[@sneak](https://sneak.berlin) for the [Ente](https://ente.io) end-to-end
encrypted photo hosting service. It logs in, enumerates collections and files,
and downloads individual images while decrypting them on the way to disk.

quak also includes a resilient backup command that downloads every file in the
account into a deduplicated local directory tree, skipping files that already
exist on disk and continuing past individual download failures instead of
crashing. It decrypts and persists all three metadata layers (basic, private
magic, public magic) per file, including camera info, GPS coordinates, captions,
and any face/keyword labels the Ente clients have added. A helper subcommand can
detect and regenerate missing thumbnails, encrypting and uploading them back to
the server.

## Getting Started

```bash
git clone https://git.eeqj.de/sneak/quak.git
cd quak
yarn install
yarn build

# Log in (prompts for email, password, and OTP/TOTP if required).
yarn quak login

# List the user's collections (albums).
yarn quak collections

# List files in a collection.
yarn quak files --collection 12345

# Download and decrypt a single file.
yarn quak get 67890 --out ./photo.jpg

# Back up every file in the account.
yarn quak backup ./my-backup
```

For library use, the primary surface is the cache-backed `Library`:

```ts
import { Client, Library } from "quak";

// Log in once; the client satisfies the library's client interface.
const client = await Client.login({
    email: "you@example.com",
    password: "your-password",
});

// Open a cache-backed library. On an empty cache this awaits one server
// refresh; on an existing cache it returns immediately and refreshes in the
// background every `refreshIntervalSeconds` (default 3).
const lib = await Library.open({ client });

// Default reads answer synchronously from the local cache — no network.
for (const album of lib.albums.list()) {
    console.log(album.collectionID, album.name);
    for (const photo of album.photos.list()) {
        console.log(`  ${photo.title} [${photo.fileType}]`);
    }
}

// Fresh reads await a server round-trip and answer with current state.
const { albums } = await lib.fresh();
console.log(`${albums.list().length} albums as of now`);

// Fetch (and cache) one photo's full-resolution bytes.
const photo = lib.photos.byID({ fileID: 12345 });
if (photo) {
    const { path } = await photo.original();
    console.log(`original at ${path}`);
}

await lib.close();
```

The lower-level `Client` (login, session serialization, and the raw
enumeration/download calls) is exported too and documented under Design below.

## Entrypoints

This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call them.
The scripts are POSIX sh (not bash) so they run in minimal containers such as
alpine. We provide:

- `script/bootstrap` — install all dependencies (node/yarn if missing, then
  `yarn install --frozen-lockfile`)
- `script/setup` — set up the repo for development after a fresh clone: runs
  `script/bootstrap`, then `script/install-precommit`
- `script/projectname` — output the project name (our own extension); used by
  `script/docker` for the image tag
- `script/build` — compile the TypeScript sources into `dist/`, then verify that
  the entrypoints `package.json` declares (`main`, `types`, `bin`) are among the
  files the compiler wrote, and make the CLI executable (our own extension)
- `script/test` — run the test suite (vitest, hard-capped at 30s where `timeout`
  is available, verbose rerun on failure)
- `script/lint` — run eslint and a prettier check, by building
  `Dockerfile.lint`; requires docker (see Linting below)
- `script/fmt` — format all files with prettier (writes)
- `script/fmt-check` — check formatting on the host (read-only); standalone, and
  not called by `script/check` or `script/precommit`, because `script/lint`
  already checks formatting in the container (see Linting below)
- `script/check` — run all checks: `test`, `lint` (our own extension)
- `script/docker` — build the test and build image, tagged via
  `script/projectname`
- `script/cibuild` — cd to the repo root and build both images (what CI runs):
  `script/lint` first, then the `Dockerfile` image, which runs `make test` and
  `make build`
- `script/precommit` — run by the git pre-commit hook (our own extension); runs
  `script/lint`, which checks both lint and formatting, but deliberately not the
  tests, so the TDD red-phase commit can land
- `script/install-precommit` — installs the git pre-commit hook (our own
  extension); `make hooks` shims to it

`make hooks` installs the pre-commit hook that runs `script/precommit`.

### Linting

Linting runs in a container, one way, everywhere. `script/lint` builds
`Dockerfile.lint`, which copies the repo into a digest-pinned node image and
runs eslint and prettier as build steps, so a successful build is a clean lint.
There is no host lint path: docker is required to lint, and that also works
where the docker daemon is remote and bind mounts are impossible.

The formatting check is part of that, not a step beside it. `script/check` and
`script/precommit` therefore call `script/lint` and stop; neither calls
`script/fmt-check` as well, which would run prettier a second time over the same
tree for the same verdict — and the weaker of the two, since the host's prettier
is whatever the working tree has installed. So `make check` and the pre-commit
hook both still fail on a badly formatted tree, and prettier runs exactly once
in each. `test/packaging/lint-once.test.ts` asserts that count by walking the
invocation graph, so a second pass cannot creep back in unnoticed.

`script/fmt-check` remains as a standalone entrypoint for asking the formatting
question on its own, without docker and without the rest of lint. Its verdict
cannot drift from the container's: prettier is pinned to an exact version,
installed from `yarn.lock` under `--frozen-lockfile` in both places, and reads
`.gitignore` as its default ignore file — which is why `.dockerignore`
deliberately keeps `.gitignore` in the build context.

Lint happens in exactly one place, which constrains the rest of the build.
`script/check` calls `script/lint`, so `make check` cannot run inside a
container without asking for docker inside docker. The image built from
`Dockerfile` therefore runs `make test` and `make build` and does not lint;
`script/cibuild` builds `Dockerfile.lint` first and that image second, so CI
gets both verdicts.

### Build epochs

`script/lint` passes `--build-arg LINT_EPOCH="$(date +%s)"`, and `script/docker`
and `script/cibuild` pass `--build-arg CHECK_EPOCH="$(date +%s)"`. Both
Dockerfiles refuse to build without their argument. This is deliberate: on an
unchanged tree Docker would otherwise serve the linter and test layers from
cache, so nothing would run and the build would still exit 0 — a lint build over
an untouched tree returns success in well under a second, having linted nothing.
A changing epoch invalidates every layer below the guard on every invocation
while leaving the dependency layers above them cached, and the missing-argument
guard means a bare `docker build .` fails loudly instead of quietly reporting a
green it did not earn: an unset build argument is the empty string, which is a
perfectly stable cache key.

## Rationale

Ente is one of very few photo services with a credible end-to-end encryption
story. The shipping clients (mobile Flutter, web React, desktop Electron, and Go
CLI) work, but they are slow, buggy, and difficult to script against. The
Flutter app fails to sync reliably. The web app is heavy. The desktop app is the
web app inside a slow Electron wrapper. The Go CLI is the closest thing to a
usable tool, but it is awkward to integrate from anything that is not a shell.
The Go CLI's backup mode crashes entirely when a single file download fails,
which makes it useless as an actual backup tool.

quak fixes these problems. This repo ships a correct, well-tested implementation
of Ente's cryptographic protocol and API surface, plus a CLI that proves the
library is enough to do real work without a UI. The backup command is resilient
by design: per-file errors are logged and the run continues.

The longer-term goal of this project is a simple desktop client for Ente, built
on this library in Electron (or a comparable runtime), with two priorities above
everything else: correctness and stability. Performance and simplicity follow
from those. Features will be added only after the protocol layer is correct, the
local cache is reliable, and the UI is responsive on a five-year-old laptop.

## Development workflow

All work on quak is test-driven. No exceptions.

1. Every change starts on a feature branch off `main`.
2. The first commit on the branch is the test suite for what is being added or
   changed. Those tests must fail at that commit; the branch is red until the
   implementation lands.
3. Subsequent commits add the implementation and any refactors needed to make
   the tests pass.
4. A feature branch can only be merged into `main` when `make check` is green.
   `main` is always green. CI runs `script/cibuild`, which lints via
   `Dockerfile.lint` and then runs `make test` and `make build` in the
   `Dockerfile` image, so neither a red branch nor one that does not compile can
   pass CI.
5. Tests are the canonical API documentation for this library. Every test file
   is commented thoroughly enough that a reader who has never seen quak can
   learn how to use it from the tests alone. Comments explain why a behavior
   matters, not just what the assertion checks.
6. Test fixtures (cryptographic vectors, recorded HTTP responses, sample files)
   are committed alongside their tests. Where possible they are generated by
   deterministic helpers in the `test/` tree so any reviewer can reproduce them
   by running the helper.
7. `git rebase -i` is allowed on a feature branch before merge to clean up the
   test-then-implementation sequence into reviewable commits, but the final
   history must still show tests landing before (or with) the matching
   implementation.
8. The pre-commit hook installed by `make hooks` runs `script/precommit`, which
   runs `script/lint` — eslint and the prettier check, in the container — but
   not the tests, and so not the full `make check`. This is deliberate so the
   TDD red-phase commit (failing tests, no implementation yet) can land. The
   suite runs as part of the image build, which is what CI executes via
   `script/cibuild`, so a red branch still cannot reach `main`.

## Design

quak is a TypeScript library with a thin CLI wrapper. The library does the work;
the CLI is for humans.

### Layout

```
quak/
    src/
        crypto/         libsodium primitives (boxes, secretstreams, KDF, SRP)
        api/            HTTP client (ApiClient class)
        auth/           login flow (SRP + email OTP + TOTP), key unwrap
        model/          decrypted Collection, File, Metadata types + decrypt fns
        download/       streaming file/thumbnail download + decryption
        backup.ts       resilient full-account backup with dedup
        errors.ts       error types shared across layers
        retry.ts        retry classifier + exponential backoff with jitter
        thumbnails.ts   detect + regenerate missing thumbnails
        client.ts       high-level Client class assembled from the above
        index.ts        public library exports
    bin/
        quak.ts         CLI entrypoint (commander.js)
    test/               unit + integration tests (vitest)
    Makefile
    Dockerfile          test suite and compile
    Dockerfile.lint     eslint and prettier, as build steps
    package.json
    tsconfig.json
```

`make build` compiles that tree into `dist/`, preserving its shape: the library
lands in `dist/src/` and the CLI in `dist/bin/quak.js`, which is what
`package.json` points `main`, `types` and `bin` at. The compiler's `rootDir` is
the repository root rather than `src/`, because `bin/` is compiled too and
`rootDir` has to contain everything that is compiled.

### Cryptography

All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is
required for `crypto_pwhash` / Argon2id). No hand-rolled crypto.

The key hierarchy, derived during login, is:

1. The user enters their password.
2. Argon2id (`crypto_pwhash`) over the password and a server-issued `kekSalt`,
   with server-issued `memLimit` and `opsLimit`, produces a 32-byte Key
   Encryption Key (KEK).
3. SRP login: a 16-byte SRP login subkey is derived from the KEK using
   `crypto_kdf_derive_from_key` (BLAKE2b) with subkey id 1 and context
   `loginctx`. That 16-byte value is the SRP password.
4. After SRP completes (or after email-OTP fallback), the server returns a blob
   of "key attributes" plus an encrypted auth token.
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
   yields the 32-byte master key.
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
   key yields the user's X25519 private key. The matching public key is
   delivered in cleartext.
7. `crypto_box_seal_open` over the encrypted token with the user's keypair
   yields the URL-safe base64 auth token used in `X-Auth-Token` for all
   subsequent calls.

Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the
master key (for owned collections). Per-file keys are decrypted with
`crypto_secretbox_open_easy` using the collection key. File metadata is a
secretstream blob (single chunk, TAG_FINAL) under the file key. File content is
a chunked `crypto_secretstream_xchacha20poly1305` stream under the file key,
with a 4 MiB plaintext chunk size and a 17-byte authentication overhead per
chunk. Thumbnails use the same secretstream blob format as metadata.

For upload (thumbnail repair), `encryptBlob` performs the push side: a single
secretstream chunk with TAG_FINAL, returning the header and ciphertext.

### HTTP API

Production endpoints:

- API: `https://api.ente.io`
- File download CDN: `https://files.ente.io/?fileID=<id>`
- Thumbnail CDN: `https://thumbnails.ente.io/?fileID=<id>`

A custom API endpoint is configurable for self-hosted servers via the
constructor option `apiOrigin`. When set, file downloads route through
`<apiOrigin>/files/download/<id>` instead of the dedicated CDN host.

Required request headers on every authenticated call:

- `X-Auth-Token`: the decrypted auth token from login.
- `X-Client-Package`: identifies the client. quak uses `berlin.sneak.quak`.

Endpoints used:

- `GET /users/srp/attributes?email=<email>`: fetch SRP and KDF parameters.
- `POST /users/srp/create-session`: begin SRP handshake.
- `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or the
  encrypted token plus key attributes.
- `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path.
- `POST /users/two-factor/verify`: TOTP second factor.
- `GET /collections/v2?sinceTime=<usec>`: list collections changed since
  microsecond timestamp; pass 0 for a full enumeration.
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
  collection; paginate while `hasMore` is true.
- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes.
- `POST /files/data/fetch`: fetch encrypted ML data for a batch of files.
- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair).
- `PUT /files/thumbnail`: register an uploaded thumbnail's object key.

### Retries and timeouts

Every request in the library goes through one policy, in `src/retry.ts`. A
request is repeated only when repeating it could produce a different answer:

- `ApiError` with a 5xx status: retried. So are `408` and `429`, the two 4xx
  codes that are statements about timing rather than about the request.
- Every other 4xx: not retried. A 404 in particular is an answer, and
  `listMissingThumbnails` depends on getting it promptly and once.
- Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or
  TLS failure — and deadline aborts: retried. The errno is looked for in the
  error's `cause` chain, because that is where Node's `fetch` puts it.
- A truncated download: retried.
- Anything else, including a secretstream authentication failure that is not
  truncation: not retried. The default answer is no. For a backup tool, retrying
  a permanent failure spends round trips and delays every remaining file, while
  declining to retry a transient one costs a single file that the next run picks
  up.

Backoff is exponential with full jitter: the delay before retry _n_ is
`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term
is the ceiling and the wait is drawn below it, so a client that lost many
parallel downloads to one CDN blip does not send them all again at the same
instant. Defaults, configurable through `ApiClientOptions.retry`:

| Option        | Default | Meaning                             |
| ------------- | ------- | ----------------------------------- |
| `attempts`    | `4`     | total calls, not retries            |
| `baseDelayMs` | `500`   | ceiling for the first retry's delay |
| `maxDelayMs`  | `10000` | upper bound on that ceiling         |

With those defaults a file that is going to fail gives up after at most three
and a half seconds of waiting. `sleep` and `random` are injectable through the
same option, which is how the test suite exercises the whole policy without
waiting.

Two deadlines, renewed for each attempt:

| Option              | Default | Applies to                                  | Kind                                  |
| ------------------- | ------- | ------------------------------------------- | ------------------------------------- |
| `requestTimeoutMs`  | `30000` | `getJSON`, `postJSON`, `putJSON`, `putFile` | the whole request                     |
| `downloadTimeoutMs` | `60000` | file and thumbnail downloads                | idle: no bytes received for this long |

They are different kinds because a download's length depends on the file and the
link: a whole-transfer deadline short enough to catch a hung connection would
cancel a large video on a slow link that is still making progress. The download
deadline restarts every time bytes arrive, so a slow download runs as long as it
keeps moving, and one that stalls is aborted after 60 seconds of silence. It
covers the wait for the headers and the body — `getFileStream` returns as soon
as headers arrive, so a deadline that only guarded the initial request would
leave the same hang one layer down. There is no limit on the total length of a
download.

**Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON`
send every `POST` and `PUT` in the endpoint list above; some of them change
server state, and `/users/two-factor/verify` consumes one of a small number of
second-factor attempts. They are retried only when every errno in the error's
`cause` chain is one of the three that establish no TCP connection to the server
ever existed, so no request byte can have been transmitted: `ENOTFOUND` and
`EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer
refused the connection). A 5xx, a mid-flight reset and a deadline are all left
to the caller, because each of them can happen after the server has already
acted. These two do not follow redirects either: a redirect means the server
already received the request, so it is reported as an error and not retried. The
routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the
same reason, despite looking like connect-time failures: on Linux an ICMP
unreachable arriving mid-flight, or a local interface going down after the
request was written, delivers them on an already-established socket. They stay
retryable for the idempotent calls. `putFile` is exempt: a presigned PUT stores
one whole object at one key in one request, so replaying it has no partial state
to damage.

A download is retried as a whole — request, stream consumption, and decryption —
because a socket reset after the response headers have arrived surfaces in the
download layer rather than in `ApiClient`, and that is the common failure for
multi-megabyte photos over a CDN. The secretstream pull state is not resumable
and these endpoints have no Range support, so a retry starts the file over. The
atomic write stays outside the retry, so a download that needed three attempts
still performs exactly one write and one rename. `runBackup` and
`runMetadataBackup` are unchanged: the retry sits below them, and a file that
fails after exhausting it is still logged, counted, and stepped over.

One imprecision is deliberate and worth knowing about. When a body ends part-way
through a secretstream chunk, Poly1305 fails and carries no framing signal, so a
cut connection and genuinely corrupt bytes are indistinguishable. quak reports
that as truncation, which means it is retried. For a body of more than one chunk
the distinction is real — a chunk that failed while the stream carried on past
it stays an authentication failure and is not retried — but for a single-chunk
body, which is most thumbnails and every small file, a wrong key, server-side
corruption and a mid-chunk cutoff all present alike and all get retried. The
cost is bounded by the attempt count, and it buys never silently keeping a
truncated file.

### Session handling

The `Client` class holds the auth token, master key, secret key, and public key
in memory. There is no on-disk session store in the library; the consumer
decides how to persist sessions.

`client.toJSON()` returns a `ClientSnapshot` (a plain serializable object with
base64-encoded keys) that the consumer can write to disk, a database, or
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
working client from that snapshot without re-authenticating; it checks every
field and each key's length first, and throws an error naming the bad field.
`client.logout()` clears the token and zeroes the key buffers in place; every
later call on that client throws.

The CLI stores the snapshot at the platform-appropriate data directory via
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
`$XDG_DATA_HOME/quak/session.json` on Linux. The file is written with mode
`0600`. The key material is stored in cleartext in the JSON; treat this file as
you would treat the password itself. A missing file is reported as "not logged
in"; a file that exists but is corrupt is reported as such, naming the bad
field. Both exit with status 1.

### CLI surface

```
quak [--cache-dir <path>] <command>             global: local metadata/content cache location
quak login                                      interactive or QUAK_EMAIL/QUAK_PASSWORD
quak whoami                                     print logged-in account as JSON
quak logout                                     delete saved session
quak collections [--json]                       list all collections
quak files --collection <id> [--json]           list files in a collection
quak get <fileID> [--out path] [--collection]   download and decrypt a file
quak get-thumb <fileID> [--out] [--collection]  download and decrypt a thumbnail
quak backup <dir> [--json]                      full incremental backup
quak backup-metadata <dir> [--exif]             dump all decrypted metadata as JSON
quak helper list-missing-thumbnails [--json]    find files with missing thumbnails
quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails
```

Every command runs on the same cache-backed library. The read commands —
`collections`, `files`, `get`, and `get-thumb` — force a fresh server round-trip
before they answer, so they report current account state rather than whatever
the cache last held. `--cache-dir` overrides where the cache lives; without it
each account gets its own directory under the per-user cache path.

`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
accepted for backward compatibility but ignored. `backup-metadata --exif` (alias
`--all`) additionally downloads each file to extract full EXIF/IPTC/XMP
metadata. The listing and backup commands support `--json` for machine-readable
output.

`backup-metadata` fetches ML data in requests of up to 200 files. When a request
still fails after its retries, the error is logged, each of its files is written
with the reason in an `mlDataError` field instead of `mlData`, and the dump goes
on. The exit code is non-zero if any ML data request failed.

`helper fix-missing-thumbnails` regenerates thumbnails for baseline JPEG images
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG
image (PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
distinct from a `failed` repair, and does not affect the exit code; a genuine
failure still exits non-zero.

### Backup layout

`quak backup <dir>` produces:

```
<dir>/
    originals/
        <fileID>.<ext>          actual file content (one per unique file)
        <fileID>.json           all decrypted metadata for that file
    collections/
        <name>/
            <title> -> ../../originals/<fileID>.<ext>   (symlink)
        <name>.json             collection metadata + file list
```

Each file is downloaded exactly once regardless of how many collections it
appears in. On subsequent runs, existing originals are skipped. If a download
fails, the error is logged and the backup continues with the next file. The exit
code is non-zero if any files failed.

Each original is copied to a temporary file named
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp` in the same directory, synced
to disk, and renamed into place, so an original is either complete or absent,
even after a power cut. A run that is killed can leave one of these temporary
files behind; the next backup deletes those whose process is no longer running.
Downloads and the content cache use the same scheme with `.quak-<random>.tmp`
names. The rename replaces whatever was at the destination rather than writing
through it: a symlink there is replaced, not followed, and the new file has the
temporary file's permissions, not those of the file it replaced.

## TODO

- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
      errors
- [x] Update the API reference section below to match the current implementation
- [x] `make docker` green
- [ ] Tag `v1.0.0`

Future (desktop client, separate repo):

- [ ] Electron app skeleton consuming this library
- [ ] Local cache (SQLite) keyed on `(collectionID, fileID, updationTime)`
- [ ] Background sync worker that streams new files into the cache
- [ ] Gallery UI: thumbnails, full-image view, basic search
- [ ] Upload, delete, and share operations in the library

## API reference

The library's primary surface is the cache-backed `Library`; the lower-level
`Client` sits underneath it and is covered by the Design sections above. The
test suite is the canonical, executable documentation — `test/library/` and
`test/client/usage.test.ts` walk every operation, and `yarn test` verifies them.

### Opening a library

`Library.open(options)` loads the on-disk cache, starts the background refresh
loop, and resolves to a `Library`. On an empty cache it awaits the first refresh
so it never opens onto empty data; on an existing cache it returns immediately
and refreshes in the background, so an unreachable server does not block
opening.

`LibraryOptions`:

| Option                   | Default                     | Meaning                                                               |
| ------------------------ | --------------------------- | --------------------------------------------------------------------- |
| `client`                 | required                    | the account client (a `Client`, or any `LibraryClient`)               |
| `cacheDirectory`         | `<XDG cache>/quak/<userID>` | where `metadata.json` and the content cache live                      |
| `downloadDirectory`      | none                        | backup destination; an original already stored there counts as cached |
| `refreshIntervalSeconds` | `3`                         | background refresh cadence                                            |
| `precacheThumbnails`     | `true`                      | prefetch every thumbnail, newest first                                |
| `precacheOriginals`      | `true`                      | prefetch the favorites album and the latest-window originals          |
| `precacheOriginalsDays`  | `7`                         | length in days of that latest window                                  |
| `cacheOriginalsMaxBytes` | 100 GiB                     | hard ceiling on the originals cache                                   |
| `freeBelowBytes`         | 50 GiB                      | free space to protect on the volume; the effective limit adapts down  |
| `isOriginalPinned`       | none                        | extra predicate for originals that must never be evicted              |
| `pools`                  | fresh `RequestPools`        | the bounded request pools (sets concurrency)                          |
| `onProgress`             | none                        | refresh/ML/precache progress callback (`RefreshEvent`)                |
| `contentSource`          | the client's own            | override the byte source (mainly for tests)                           |

Concurrency is set through `pools`: construct
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).

`lib.status()` returns a `LibraryStatus` (collection/file counts, last
refresh/ML times and errors, originals usage and effective limit, precache
progress, and `closed`). `lib.close()` stops the background timer; it is
idempotent, and an in-flight refresh is left to finish. The promise it returns
resolves once that refresh (including its cache write), the ML data fetch and
the precache fetches already running have all finished, so the cache directory
can then be removed.

### Default reads vs. fresh reads

Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
synchronously from the last refreshed copy held in RAM and never touch the
network. The background timer refreshes that copy every
`refreshIntervalSeconds`, so a default read is immediate but may be up to one
interval stale.

`await lib.fresh()` forces a refresh, waits for it to complete and persist, and
returns the same `{ albums, photos, timeline }` namespaces — now guaranteed to
reflect a completed server round-trip. Concurrent `fresh()` calls coalesce onto
one refresh, and a refresh that fails rejects the caller (default reads stay
silent and keep serving the last good copy). The CLI's read commands use fresh
reads (issue https://git.eeqj.de/sneak/quak/issues/75).

### Read surface

- `lib.albums.list()` → `Album[]`, newest-updated first.
  `lib.albums.byID({ collectionID })` and `byName({ albumName })` →
  `Album | undefined`.
- `lib.photos.byID({ fileID })` → `Photo | undefined`.
  `lib.photos.records({ fileIDs })` → `PhotoRecord[]` in the requested order,
  each id once, unknown ids dropped.
- `lib.timeline.groups({ groupBy, filter? })` → `TimelineGroup[]`, grouped by
  `"day" | "week" | "month"` (keys `YYYY-MM-DD`, ISO `YYYY-Www`, `YYYY-MM`),
  newest group first. A `PhotoFilter` combines `albumID`, `text`
  (title/caption/album-name substring), `fileTypes`, `hasLocation`, and
  `includeArchived`; hidden photos are always excluded.

An `Album` exposes its record fields and `album.photos.list()` → `Photo[]`
(newest first). A `Photo` exposes its record fields, `photo.record()` →
`PhotoRecord`, and two content methods:

- `await photo.original(opts?)` → `{ path, bytes }` — the full-resolution file.
- `await photo.thumbnail(opts?)` → `{ path, bytes }`.

Both serve from the on-disk content cache when the bytes are present and
otherwise fetch through the pools; `opts.onProgress` reports per-file progress.
They throw when the library was opened without a content source.

Lower-level accessors that return decrypted model objects (which hold key
material) are also available: `listCollections()`, `getCollection(id)`,
`listFiles(collectionID)`, `getFile(collectionID, fileID)`, and
`getFileByID(fileID)`.

### Records and change notifications

The GUI-facing records hold no key material and no binary, so they survive
`structuredClone`/JSON across the Electron IPC boundary:

- `PhotoRecord`: `fileID`, `albumIDs`, `title`, `takenAt` (milliseconds),
  `fileType`, optional `caption` / `width` / `height` / `latitude` /
  `longitude`, `isArchived`, `isHidden`, and `thumbnailPath` / `originalPath`
  once the bytes are cached.
- `AlbumRecord`: `collectionID`, `name`, `type`, `isShared`, `updationTime`, and
  `fileIDs` (newest first).
- `LibrarySnapshot`: `{ albums, photos, takenAt }`.

`lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first,
photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`,
`refreshedAt`) whenever a refresh alters the projection, and returns
`{ unsubscribe }`; a refresh that changes nothing delivers nothing.

### Thumbnails, ML search, and backup

- `lib.thumbnails.ensure({ fileIDs, priority, signal?, onProgress? })`
  prefetches thumbnails through the thumbnail pool, deduped by fileID, returning
  one `EnsureResult` (`{ fileID, path?, error? }`) per file. `priority` is
  `"visible" | "ahead" | "background"`; only `"visible"` preempts background
  work.
- `lib.mldata` searches the CLIP index built from Ente's per-file ML data:
  `forFile({ fileID })` → `Promise<MLData | undefined>` (the whole stored
  payload — face boxes, landmarks, embedding — read from disk on demand);
  `similar({ fileID, limit? })` and `searchByEmbedding({ embedding, limit? })` →
  `SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
  default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
  a query vector the caller produced elsewhere.
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every
  in-scope original (and, with `includeThumbnails`, thumbnails) through the
  content cache, and rebuilds the on-disk backup tree with a durable failure
  ledger. `BackupOptions`: `downloadDirectory` (falls back to the one `open()`
  was given), `includeOriginals` (default `true`), `includeThumbnails` (default
  `false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above for the
  tree it writes.

### Request pools

`RequestPools` holds three independent bounded pools — metadata (10), content
(5), thumbnails (25) — because Ente meters these traffic classes differently.
Each pool orders on-demand work ahead of background/precache work and dedups
in-flight fetches by key, and an idle pool never lends its slots to a busy one.

### On-disk cache layout

Under `cacheDirectory`:

```
<cacheDirectory>/
    metadata.json                 decrypted account state + refresh cursor
    originals/<fileID>.<ext>      cached full-resolution files
    thumbnails/<fileID>.jpg       cached thumbnails
    mldata/
        <fileID>.json             one decrypted ML payload per file
        clip.f32, clip.json       the packed CLIP index and its id list
        fetched.json              per-file fetch bookkeeping
```

A stored file appears only via an atomic temp-then-rename, so its presence means
it is complete. The design also calls for a content-hash comparison against
`FileMetadata.hash` on each fetched original; that check is deferred (issue
https://git.eeqj.de/sneak/quak/issues/68) because the exact hash construction
cannot yet be confirmed against the repo's fixtures.

### Key types by source file

- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
  `LibraryClient`, `RefreshEvent`
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
  `TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
  `EnsureOptions`, `EnsureResult`, `ContentSource`
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,
  `LibraryChange`
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `StreamOptions`
- `src/errors.ts`: `ApiError`, `TruncatedStreamError`
- `src/retry.ts`: `withRetry`, `isRetryable`, `isSafeToReplay`, `RetryOptions`
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileType`,
  `CollectionType`, `RawCollection`, `RawEnteFile`
- `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult`

## Source attribution

The cryptographic protocol and wire format implemented here are Ente's, taken
from the Ente open source clients at <https://github.com/ente-io/ente>. No code
is imported or vendored from those projects; any reference code that is copied
is rewritten in TypeScript in this repository. Protocol fidelity is verified
against the upstream implementations in `web/packages/base/`,
`mobile/apps/photos/lib/`, and `cli/`.

## For LLMs

If you are an LLM agent working on this repository, read and follow these
documents:

- **`REPO_POLICIES.md`** in the repo root. It is copied from
  <https://git.eeqj.de/sneak/prompts> and covers repository structure, tooling,
  Makefile targets, Dockerfile conventions, dependency pinning, and commit
  hygiene. All external dependencies must be pinned by cryptographic hash in
  `yarn.lock`. Never `git add -A`. Never force-push to main.

- **The "Development workflow" section above.** All changes go on feature
  branches. Tests are written first and committed in a failing state before the
  implementation. Tests are the canonical API documentation and must be
  commented thoroughly. `main` is always green.

- **Required checks before every commit:** `make lint` must pass — that is
  eslint plus the prettier check, and it builds `Dockerfile.lint`, so it needs
  docker. The pre-commit hook enforces exactly that. `make check` (which also
  runs the tests) must pass before merging to `main`. `make fmt-check` is
  available for a host-side formatting check on its own, but it is not a
  separate requirement: `make lint` already covers it, and running both would
  check formatting twice. Never invoke eslint or prettier directly; linting runs
  in the container only.

- **Formatting:** prettier with 4-space indents and `proseWrap: always` for
  markdown. Use `make fmt` to format. Use `yarn` not `npm`.

- **Testing:** vitest. Tests go in `test/` mirroring the `src/` structure.
  `make test` must complete in under 20 seconds. Use `mkdtempSync` for temporary
  directories, never manual timestamp paths.

- **Code style:** `const` for everything, `let` if reassignment is needed, never
  `var`. Avoid unnecessary comments. No hand-rolled crypto. The
  `LLM_PROSE_TELLS.md` document in the prompts repo applies to any prose written
  in this repository (README, comments, commit messages).

## License

WTFPL. See [LICENSE](LICENSE).

## Author

[@sneak](https://sneak.berlin)