quak/README.md

# quak

quak is a WTFPL-licensed TypeScript client library and CLI by
[@sneak](https://sneak.berlin) for the [Ente](https://ente.io) end-to-end
encrypted photo hosting service. It logs in, enumerates collections and files,
and downloads individual images while decrypting them on the way to disk.

quak also includes a resilient backup command that downloads every file in the
account into a deduplicated local directory tree, skipping files that already
exist on disk and continuing past individual download failures instead of
crashing. It decrypts and persists all three metadata layers (basic, private
magic, public magic) per file, including camera info, GPS coordinates, captions,
and any face/keyword labels the Ente clients have added. A helper subcommand can
detect and regenerate missing thumbnails, encrypting and uploading them back to
the server.

## Getting Started

```bash
git clone https://git.eeqj.de/sneak/quak.git
cd quak
yarn install
yarn build

# Log in (prompts for email, password, and OTP/TOTP if required).
yarn quak login

# List the user's collections (albums).
yarn quak collections

# List files in a collection.
yarn quak files --collection 12345

# Download and decrypt a single file.
yarn quak get 67890 --out ./photo.jpg

# Back up every file in the account.
yarn quak backup ./my-backup
```

For library use:

```ts
import { Client } from "quak";

const client = await Client.login({
    email: "you@example.com",
    password: "your-password",
});

for (const c of await client.listCollections()) {
    console.log(c.id, c.name);
    const files = await client.listFiles(c.id, c.key);
    for (const f of files) {
        console.log(`  ${f.metadata.title} [${f.metadata.fileType}]`);
    }
}

// Download a file
const files = await client.listFiles(collectionID, collectionKey);
await client.downloadFile(files[0], "./photo.jpg");

// Serialize session for later (consumer handles persistence)
const snapshot = client.toJSON();
// ... later:
const restored = Client.fromJSON(snapshot);
```

## Rationale

Ente is one of very few photo services with a credible end-to-end encryption
story. The shipping clients (mobile Flutter, web React, desktop Electron, and Go
CLI) work, but they are slow, buggy, and difficult to script against. The
Flutter app fails to sync reliably. The web app is heavy. The desktop app is the
web app inside a slow Electron wrapper. The Go CLI is the closest thing to a
usable tool, but it is awkward to integrate from anything that is not a shell.
The Go CLI's backup mode crashes entirely when a single file download fails,
which makes it useless as an actual backup tool.

quak fixes these problems. This repo ships a correct, well-tested implementation
of Ente's cryptographic protocol and API surface, plus a CLI that proves the
library is enough to do real work without a UI. The backup command is resilient
by design: per-file errors are logged and the run continues.

The longer-term goal of this project is a simple desktop client for Ente, built
on this library in Electron (or a comparable runtime), with two priorities above
everything else: correctness and stability. Performance and simplicity follow
from those. Features will be added only after the protocol layer is correct, the
local cache is reliable, and the UI is responsive on a five-year-old laptop.

## Development workflow

All work on quak is test-driven. No exceptions.

1. Every change starts on a feature branch off `main`.
2. The first commit on the branch is the test suite for what is being added or
   changed. Those tests must fail at that commit; the branch is red until the
   implementation lands.
3. Subsequent commits add the implementation and any refactors needed to make
   the tests pass.
4. A feature branch can only be merged into `main` when `make check` is green.
   `main` is always green. The Dockerfile runs `make check`, so a red branch
   cannot pass CI.
5. Tests are the canonical API documentation for this library. Every test file
   is commented thoroughly enough that a reader who has never seen quak can
   learn how to use it from the tests alone. Comments explain why a behavior
   matters, not just what the assertion checks.
6. Test fixtures (cryptographic vectors, recorded HTTP responses, sample files)
   are committed alongside their tests. Where possible they are generated by
   deterministic helpers in the `test/` tree so any reviewer can reproduce them
   by running the helper.
7. `git rebase -i` is allowed on a feature branch before merge to clean up the
   test-then-implementation sequence into reviewable commits, but the final
   history must still show tests landing before (or with) the matching
   implementation.
8. The pre-commit hook installed by `make hooks` runs
   `make lint && make fmt-check`, not the full `make check`. This is deliberate
   so the TDD red-phase commit (failing tests, no implementation yet) can land.
   The full `make check` runs as part of `docker build .`, which is what CI
   executes, so a red branch still cannot reach `main`.

## Design

quak is a TypeScript library with a thin CLI wrapper. The library does the work;
the CLI is for humans.

### Layout

```
quak/
    src/
        crypto/         libsodium primitives (boxes, secretstreams, KDF, SRP)
        api/            HTTP client (ApiClient class)
        auth/           login flow (SRP + email OTP + TOTP), key unwrap
        model/          decrypted Collection, File, Metadata types + decrypt fns
        download/       streaming file/thumbnail download + decryption
        backup.ts       resilient full-account backup with dedup
        thumbnails.ts   detect + regenerate missing thumbnails
        client.ts       high-level Client class assembled from the above
        index.ts        public library exports
    bin/
        quak.ts         CLI entrypoint (commander.js)
    test/               unit + integration tests (vitest)
    Makefile
    Dockerfile
    package.json
    tsconfig.json
```

### Cryptography

All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is
required for `crypto_pwhash` / Argon2id). No hand-rolled crypto.

The key hierarchy, derived during login, is:

1. The user enters their password.
2. Argon2id (`crypto_pwhash`) over the password and a server-issued `kekSalt`,
   with server-issued `memLimit` and `opsLimit`, produces a 32-byte Key
   Encryption Key (KEK).
3. SRP login: a 16-byte SRP login subkey is derived from the KEK using
   `crypto_kdf_derive_from_key` (BLAKE2b) with subkey id 1 and context
   `loginctx`. That 16-byte value is the SRP password.
4. After SRP completes (or after email-OTP fallback), the server returns a blob
   of "key attributes" plus an encrypted auth token.
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
   yields the 32-byte master key.
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
   key yields the user's X25519 private key. The matching public key is
   delivered in cleartext.
7. `crypto_box_seal_open` over the encrypted token with the user's keypair
   yields the URL-safe base64 auth token used in `X-Auth-Token` for all
   subsequent calls.

Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the
master key (for owned collections). Per-file keys are decrypted with
`crypto_secretbox_open_easy` using the collection key. File metadata is a
secretstream blob (single chunk, TAG_FINAL) under the file key. File content is
a chunked `crypto_secretstream_xchacha20poly1305` stream under the file key,
with a 4 MiB plaintext chunk size and a 17-byte authentication overhead per
chunk. Thumbnails use the same secretstream blob format as metadata.

For upload (thumbnail repair), `encryptBlob` performs the push side: a single
secretstream chunk with TAG_FINAL, returning the header and ciphertext.

### HTTP API

Production endpoints:

- API: `https://api.ente.io`
- File download CDN: `https://files.ente.io/?fileID=<id>`
- Thumbnail CDN: `https://thumbnails.ente.io/?fileID=<id>`

A custom API endpoint is configurable for self-hosted servers via the
constructor option `apiOrigin`. When set, file downloads route through
`<apiOrigin>/files/download/<id>` instead of the dedicated CDN host.

Required request headers on every authenticated call:

- `X-Auth-Token`: the decrypted auth token from login.
- `X-Client-Package`: identifies the client. quak uses `berlin.sneak.quak`.

Endpoints used:

- `GET /users/srp/attributes?email=<email>`: fetch SRP and KDF parameters.
- `POST /users/srp/create-session`: begin SRP handshake.
- `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or the
  encrypted token plus key attributes.
- `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path.
- `POST /users/two-factor/verify`: TOTP second factor.
- `GET /collections/v2?sinceTime=<usec>`: list collections changed since
  microsecond timestamp; pass 0 for a full enumeration.
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
  collection; paginate while `hasMore` is true.
- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes.
- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair).
- `PUT /files/thumbnail`: register an uploaded thumbnail's object key.

### Session handling

The `Client` class holds the auth token, master key, secret key, and public key
in memory. There is no on-disk session store in the library; the consumer
decides how to persist sessions.

`client.toJSON()` returns a `ClientSnapshot` (a plain serializable object with
base64-encoded keys) that the consumer can write to disk, a database, or
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
working client from that snapshot without re-authenticating.

The CLI stores the snapshot at the platform-appropriate data directory via
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
`$XDG_DATA_HOME/quak/session.json` on Linux. The file is written with mode
`0600`. The key material is stored in cleartext in the JSON; treat this file as
you would treat the password itself.

### CLI surface

```
quak login                                      interactive or QUAK_EMAIL/QUAK_PASSWORD
quak whoami                                     print logged-in account as JSON
quak logout                                     delete saved session
quak collections [--json]                       list all collections
quak files --collection <id> [--json]           list files in a collection
quak get <fileID> [--out path] [--collection]   download and decrypt a file
quak get-thumb <fileID> [--out] [--collection]  download and decrypt a thumbnail
quak backup <dir> [--json]                      full incremental backup
quak helper list-missing-thumbnails [--json]    find files with missing thumbnails
quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails
```

`get` and `get-thumb` search all collections for the file ID when `--collection`
is not specified. All listing and backup commands support `--json` for
machine-readable output.

### Backup layout

`quak backup <dir>` produces:

```
<dir>/
    originals/
        <fileID>.<ext>          actual file content (one per unique file)
        <fileID>.json           all decrypted metadata for that file
    collections/
        <name>/
            <title> -> ../../originals/<fileID>.<ext>   (symlink)
        <name>.json             collection metadata + file list
```

Each file is downloaded exactly once regardless of how many collections it
appears in. On subsequent runs, existing originals are skipped. If a download
fails, the error is logged and the backup continues with the next file. The exit
code is non-zero if any files failed.

## TODO

- [ ] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
      errors
- [ ] Update the API reference section below to match the current implementation
- [ ] `make docker` green
- [ ] Tag `v1.0.0`

Future (desktop client, separate repo):

- [ ] Electron app skeleton consuming this library
- [ ] Local cache (SQLite) keyed on `(collectionID, fileID, updationTime)`
- [ ] Background sync worker that streams new files into the cache
- [ ] Gallery UI: thumbnails, full-image view, basic search
- [ ] Upload, delete, and share operations in the library

## API reference

The API reference section below is from an earlier draft and does not fully
reflect the current implementation. The authoritative API documentation is in
the test files, particularly `test/client/usage.test.ts` which is a literate
tutorial walking through every operation. Run `yarn test` to verify the examples
are correct.

The key types and their actual signatures can be found in:

- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `ApiError`
- `src/auth/types.ts`: `KeyAttributes`, `SRPAttributes`,
  `AuthorizationResponse`, `LoginChallenge`
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileBlob`,
  `RawCollection`, `RawEnteFile`, `RawMagicMetadata`
- `src/download/index.ts`: `DownloadResult`
- `src/backup.ts`: `BackupResult`, `BackupError`
- `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult`

## Source attribution

The cryptographic protocol and wire format implemented here are Ente's, taken
from the Ente open source clients at <https://github.com/ente-io/ente>. No code
is imported or vendored from those projects; any reference code that is copied
is rewritten in TypeScript in this repository. Protocol fidelity is verified
against the upstream implementations in `web/packages/base/`,
`mobile/apps/photos/lib/`, and `cli/`.

## For LLMs

If you are an LLM agent working on this repository, read and follow these
documents:

- **`REPO_POLICIES.md`** in the repo root. It is copied from
  <https://git.eeqj.de/sneak/prompts> and covers repository structure, tooling,
  Makefile targets, Dockerfile conventions, dependency pinning, and commit
  hygiene. All external dependencies must be pinned by cryptographic hash in
  `yarn.lock`. Never `git add -A`. Never force-push to main.

- **The "Development workflow" section above.** All changes go on feature
  branches. Tests are written first and committed in a failing state before the
  implementation. Tests are the canonical API documentation and must be
  commented thoroughly. `main` is always green.

- **Required checks before every commit:** `make lint` (eslint + prettier check)
  and `make fmt-check` must pass. The pre-commit hook enforces this.
  `make check` (which also runs tests) must pass before merging to `main`.

- **Formatting:** prettier with 4-space indents and `proseWrap: always` for
  markdown. Use `make fmt` to format. Use `yarn` not `npm`.

- **Testing:** vitest. Tests go in `test/` mirroring the `src/` structure.
  `make test` must complete in under 20 seconds. Use `mkdtempSync` for temporary
  directories, never manual timestamp paths.

- **Code style:** `const` for everything, `let` if reassignment is needed, never
  `var`. Avoid unnecessary comments. No hand-rolled crypto. The
  `LLM_PROSE_TELLS.md` document in the prompts repo applies to any prose written
  in this repository (README, comments, commit messages).

## License

WTFPL. See [LICENSE](LICENSE).

## Author

[@sneak](https://sneak.berlin)