# quak quak is a WTFPL-licensed TypeScript client library and CLI by [@sneak](https://sneak.berlin) for the [Ente](https://ente.io) end-to-end encrypted photo hosting service. It logs in, enumerates collections and files, and downloads individual images while decrypting them on the way to disk. quak also includes a resilient backup command that downloads every file in the account into a deduplicated local directory tree, skipping files that already exist on disk and continuing past individual download failures instead of crashing. It decrypts and persists all three metadata layers (basic, private magic, public magic) per file, including camera info, GPS coordinates, captions, and any face/keyword labels the Ente clients have added. A helper subcommand can detect and regenerate missing thumbnails, encrypting and uploading them back to the server. ## Getting Started ```bash git clone https://git.eeqj.de/sneak/quak.git cd quak yarn install yarn build # Log in (prompts for email, password, and OTP/TOTP if required). yarn quak login # List the user's collections (albums). yarn quak collections # List files in a collection. yarn quak files --collection 12345 # Download and decrypt a single file. yarn quak get 67890 --out ./photo.jpg # Back up every file in the account. yarn quak backup ./my-backup ``` For library use: ```ts import { Client } from "quak"; const client = await Client.login({ email: "you@example.com", password: "your-password", }); for (const c of await client.listCollections()) { console.log(c.id, c.name); const files = await client.listFiles(c.id, c.key); for (const f of files) { console.log(` ${f.metadata.title} [${f.metadata.fileType}]`); } } // Download a file const files = await client.listFiles(collectionID, collectionKey); await client.downloadFile(files[0], "./photo.jpg"); // Serialize session for later (consumer handles persistence) const snapshot = client.toJSON(); // ... later: const restored = Client.fromJSON(snapshot); ``` ## Entrypoints This repository adheres to the [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) standard: normalized scripts in `script/` are the entrypoints for the development workflow, and the Makefile targets are thin shims that call them. The scripts are POSIX sh (not bash) so they run in minimal containers such as alpine. We provide: - `script/bootstrap` — install all dependencies (node/yarn if missing, then `yarn install --frozen-lockfile`) - `script/setup` — set up the repo for development after a fresh clone: runs `script/bootstrap`, then `script/install-precommit` - `script/projectname` — output the project name (our own extension); used by `script/docker` for the image tag - `script/build` — compile the TypeScript sources into `dist/`, then verify that the entrypoints `package.json` declares (`main`, `types`, `bin`) are among the files the compiler wrote, and make the CLI executable (our own extension) - `script/test` — run the test suite (vitest, hard-capped at 30s where `timeout` is available, verbose rerun on failure) - `script/lint` — run eslint and a prettier check - `script/fmt` — format all files with prettier (writes) - `script/fmt-check` — check formatting (read-only) - `script/check` — run all checks: `test`, `lint`, `fmt-check` (our own extension) - `script/docker` — build the Docker image, tagged via `script/projectname` - `script/cibuild` — cd to the repo root and run the image build (what CI runs; the build runs `make fmt-check` and `make lint` in a first stage, then `make check` and `make build` in a second) - `script/precommit` — run by the git pre-commit hook (our own extension); runs `script/lint` and `script/fmt-check` but deliberately not the tests, so the TDD red-phase commit can land - `script/install-precommit` — installs the git pre-commit hook (our own extension); `make hooks` shims to it `make hooks` installs the pre-commit hook that runs `script/precommit`. Both `script/docker` and `script/cibuild` pass `--build-arg CHECK_EPOCH="$(date +%s)"`. The Dockerfile refuses to build without it. This is deliberate: on an unchanged tree Docker would otherwise serve the `make check` layer from cache, so the suite would never run and the build would still exit 0. A changing epoch invalidates the check and build layers on every invocation while leaving the dependency layers below them cached, and the missing-argument guard means a bare `docker build .` fails loudly instead of quietly reporting a green it did not earn. ## Rationale Ente is one of very few photo services with a credible end-to-end encryption story. The shipping clients (mobile Flutter, web React, desktop Electron, and Go CLI) work, but they are slow, buggy, and difficult to script against. The Flutter app fails to sync reliably. The web app is heavy. The desktop app is the web app inside a slow Electron wrapper. The Go CLI is the closest thing to a usable tool, but it is awkward to integrate from anything that is not a shell. The Go CLI's backup mode crashes entirely when a single file download fails, which makes it useless as an actual backup tool. quak fixes these problems. This repo ships a correct, well-tested implementation of Ente's cryptographic protocol and API surface, plus a CLI that proves the library is enough to do real work without a UI. The backup command is resilient by design: per-file errors are logged and the run continues. The longer-term goal of this project is a simple desktop client for Ente, built on this library in Electron (or a comparable runtime), with two priorities above everything else: correctness and stability. Performance and simplicity follow from those. Features will be added only after the protocol layer is correct, the local cache is reliable, and the UI is responsive on a five-year-old laptop. ## Development workflow All work on quak is test-driven. No exceptions. 1. Every change starts on a feature branch off `main`. 2. The first commit on the branch is the test suite for what is being added or changed. Those tests must fail at that commit; the branch is red until the implementation lands. 3. Subsequent commits add the implementation and any refactors needed to make the tests pass. 4. A feature branch can only be merged into `main` when `make check` is green. `main` is always green. The Dockerfile runs `make check` and `make build`, so neither a red branch nor one that does not compile can pass CI. 5. Tests are the canonical API documentation for this library. Every test file is commented thoroughly enough that a reader who has never seen quak can learn how to use it from the tests alone. Comments explain why a behavior matters, not just what the assertion checks. 6. Test fixtures (cryptographic vectors, recorded HTTP responses, sample files) are committed alongside their tests. Where possible they are generated by deterministic helpers in the `test/` tree so any reviewer can reproduce them by running the helper. 7. `git rebase -i` is allowed on a feature branch before merge to clean up the test-then-implementation sequence into reviewable commits, but the final history must still show tests landing before (or with) the matching implementation. 8. The pre-commit hook installed by `make hooks` runs `script/precommit`, which runs the lint and format checks but not the full `make check`. This is deliberate so the TDD red-phase commit (failing tests, no implementation yet) can land. The full `make check` runs as part of the image build, which is what CI executes via `script/cibuild`, so a red branch still cannot reach `main`. ## Design quak is a TypeScript library with a thin CLI wrapper. The library does the work; the CLI is for humans. ### Layout ``` quak/ src/ crypto/ libsodium primitives (boxes, secretstreams, KDF, SRP) api/ HTTP client (ApiClient class) auth/ login flow (SRP + email OTP + TOTP), key unwrap model/ decrypted Collection, File, Metadata types + decrypt fns download/ streaming file/thumbnail download + decryption backup.ts resilient full-account backup with dedup errors.ts error types shared across layers retry.ts retry classifier + exponential backoff with jitter thumbnails.ts detect + regenerate missing thumbnails client.ts high-level Client class assembled from the above index.ts public library exports bin/ quak.ts CLI entrypoint (commander.js) test/ unit + integration tests (vitest) Makefile Dockerfile package.json tsconfig.json ``` `make build` compiles that tree into `dist/`, preserving its shape: the library lands in `dist/src/` and the CLI in `dist/bin/quak.js`, which is what `package.json` points `main`, `types` and `bin` at. The compiler's `rootDir` is the repository root rather than `src/`, because `bin/` is compiled too and `rootDir` has to contain everything that is compiled. ### Cryptography All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is required for `crypto_pwhash` / Argon2id). No hand-rolled crypto. The key hierarchy, derived during login, is: 1. The user enters their password. 2. Argon2id (`crypto_pwhash`) over the password and a server-issued `kekSalt`, with server-issued `memLimit` and `opsLimit`, produces a 32-byte Key Encryption Key (KEK). 3. SRP login: a 16-byte SRP login subkey is derived from the KEK using `crypto_kdf_derive_from_key` (BLAKE2b) with subkey id 1 and context `loginctx`. That 16-byte value is the SRP password. 4. After SRP completes (or after email-OTP fallback), the server returns a blob of "key attributes" plus an encrypted auth token. 5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK yields the 32-byte master key. 6. `crypto_secretbox_open_easy` over the encrypted secret key with the master key yields the user's X25519 private key. The matching public key is delivered in cleartext. 7. `crypto_box_seal_open` over the encrypted token with the user's keypair yields the URL-safe base64 auth token used in `X-Auth-Token` for all subsequent calls. Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the master key (for owned collections). Per-file keys are decrypted with `crypto_secretbox_open_easy` using the collection key. File metadata is a secretstream blob (single chunk, TAG_FINAL) under the file key. File content is a chunked `crypto_secretstream_xchacha20poly1305` stream under the file key, with a 4 MiB plaintext chunk size and a 17-byte authentication overhead per chunk. Thumbnails use the same secretstream blob format as metadata. For upload (thumbnail repair), `encryptBlob` performs the push side: a single secretstream chunk with TAG_FINAL, returning the header and ciphertext. ### HTTP API Production endpoints: - API: `https://api.ente.io` - File download CDN: `https://files.ente.io/?fileID=` - Thumbnail CDN: `https://thumbnails.ente.io/?fileID=` A custom API endpoint is configurable for self-hosted servers via the constructor option `apiOrigin`. When set, file downloads route through `/files/download/` instead of the dedicated CDN host. Required request headers on every authenticated call: - `X-Auth-Token`: the decrypted auth token from login. - `X-Client-Package`: identifies the client. quak uses `berlin.sneak.quak`. Endpoints used: - `GET /users/srp/attributes?email=`: fetch SRP and KDF parameters. - `POST /users/srp/create-session`: begin SRP handshake. - `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or the encrypted token plus key attributes. - `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path. - `POST /users/two-factor/verify`: TOTP second factor. - `GET /collections/v2?sinceTime=`: list collections changed since microsecond timestamp; pass 0 for a full enumeration. - `GET /collections/v2/diff?collectionID=&sinceTime=`: list files in a collection; paginate while `hasMore` is true. - `GET https://files.ente.io/?fileID=`: download encrypted file bytes. - `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair). - `PUT /files/thumbnail`: register an uploaded thumbnail's object key. ### Retries and timeouts Every request in the library goes through one policy, in `src/retry.ts`. A request is repeated only when repeating it could produce a different answer: - `ApiError` with a 5xx status: retried. So are `408` and `429`, the two 4xx codes that are statements about timing rather than about the request. - Every other 4xx: not retried. A 404 in particular is an answer, and `listMissingThumbnails` depends on getting it promptly and once. - Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or TLS failure — and deadline aborts: retried. The errno is looked for in the error's `cause` chain, because that is where Node's `fetch` puts it. - A truncated download: retried. - Anything else, including a secretstream authentication failure that is not truncation: not retried. The default answer is no. For a backup tool, retrying a permanent failure spends round trips and delays every remaining file, while declining to retry a transient one costs a single file that the next run picks up. Backoff is exponential with full jitter: the delay before retry _n_ is `random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term is the ceiling and the wait is drawn below it, so a client that lost many parallel downloads to one CDN blip does not send them all again at the same instant. Defaults, configurable through `ApiClientOptions.retry`: | Option | Default | Meaning | | ------------- | ------- | ----------------------------------- | | `attempts` | `4` | total calls, not retries | | `baseDelayMs` | `500` | ceiling for the first retry's delay | | `maxDelayMs` | `10000` | upper bound on that ceiling | With those defaults a file that is going to fail gives up after at most three and a half seconds of waiting. `sleep` and `random` are injectable through the same option, which is how the test suite exercises the whole policy without waiting. Two deadlines, applied with `AbortSignal.timeout()` and renewed for each attempt: | Option | Default | Applies to | | ------------------- | -------- | ------------------------------------------- | | `requestTimeoutMs` | `30000` | `getJSON`, `postJSON`, `putJSON`, `putFile` | | `downloadTimeoutMs` | `600000` | file and thumbnail body transfers | They are separate because one number cannot serve both: a value short enough to keep a hung API call from stalling a backup would cancel a legitimate multi-gigabyte download. The download deadline covers the body, not just the headers — `getFileStream` returns as soon as headers arrive, so a deadline that only guarded the initial request would leave the same hang one layer down. **Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON` reach `/users/srp/create-session`, `/users/two-factor/verify` — which consumes one of a small number of second-factor attempts — and `/files/thumbnail`. They are retried only on the three failures that establish no TCP connection to the server ever existed, so no request byte can have been transmitted: `ENOTFOUND` and `EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer refused the connection). A 5xx, a mid-flight reset and a deadline are all left to the caller, because each of them can happen after the server has already acted. The routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the same reason, despite looking like connect-time failures: on Linux an ICMP unreachable arriving mid-flight, or a local interface going down after the request was written, delivers them on an already-established socket. They stay retryable for the idempotent calls. `putFile` is exempt: a presigned PUT stores one whole object at one key in one request, so replaying it has no partial state to damage. A download is retried as a whole — request, stream consumption, and decryption — because a socket reset after the response headers have arrived surfaces in the download layer rather than in `ApiClient`, and that is the common failure for multi-megabyte photos over a CDN. The secretstream pull state is not resumable and these endpoints have no Range support, so a retry starts the file over. The atomic write stays outside the retry, so a download that needed three attempts still performs exactly one write and one rename. `runBackup` and `runMetadataBackup` are unchanged: the retry sits below them, and a file that fails after exhausting it is still logged, counted, and stepped over. One imprecision is deliberate and worth knowing about. When a body ends part-way through a secretstream chunk, Poly1305 fails and carries no framing signal, so a cut connection and genuinely corrupt bytes are indistinguishable. quak reports that as truncation, which means it is retried. For a body of more than one chunk the distinction is real — a chunk that failed while the stream carried on past it stays an authentication failure and is not retried — but for a single-chunk body, which is most thumbnails and every small file, a wrong key, server-side corruption and a mid-chunk cutoff all present alike and all get retried. The cost is bounded by the attempt count, and it buys never silently keeping a truncated file. ### Session handling The `Client` class holds the auth token, master key, secret key, and public key in memory. There is no on-disk session store in the library; the consumer decides how to persist sessions. `client.toJSON()` returns a `ClientSnapshot` (a plain serializable object with base64-encoded keys) that the consumer can write to disk, a database, or whatever else fits their use case. `Client.fromJSON(snapshot)` restores a working client from that snapshot without re-authenticating. The CLI stores the snapshot at the platform-appropriate data directory via `env-paths`: `~/Library/Application Support/quak/session.json` on macOS, `$XDG_DATA_HOME/quak/session.json` on Linux. The file is written with mode `0600`. The key material is stored in cleartext in the JSON; treat this file as you would treat the password itself. ### CLI surface ``` quak login interactive or QUAK_EMAIL/QUAK_PASSWORD quak whoami print logged-in account as JSON quak logout delete saved session quak collections [--json] list all collections quak files --collection [--json] list files in a collection quak get [--out path] [--collection] download and decrypt a file quak get-thumb [--out] [--collection] download and decrypt a thumbnail quak backup [--json] full incremental backup quak helper list-missing-thumbnails [--json] find files with missing thumbnails quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails ``` `get` and `get-thumb` search all collections for the file ID when `--collection` is not specified. All listing and backup commands support `--json` for machine-readable output. ### Backup layout `quak backup ` produces: ``` / originals/ . actual file content (one per unique file) .json all decrypted metadata for that file collections/ / -> ../../originals/<fileID>.<ext> (symlink) <name>.json collection metadata + file list ``` Each file is downloaded exactly once regardless of how many collections it appears in. On subsequent runs, existing originals are skipped. If a download fails, the error is logged and the backup continues with the next file. The exit code is non-zero if any files failed. ## TODO - [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network errors - [ ] Update the API reference section below to match the current implementation - [x] `make docker` green - [ ] Tag `v1.0.0` Future (desktop client, separate repo): - [ ] Electron app skeleton consuming this library - [ ] Local cache (SQLite) keyed on `(collectionID, fileID, updationTime)` - [ ] Background sync worker that streams new files into the cache - [ ] Gallery UI: thumbnails, full-image view, basic search - [ ] Upload, delete, and share operations in the library ## API reference The API reference section below is from an earlier draft and does not fully reflect the current implementation. The authoritative API documentation is in the test files, particularly `test/client/usage.test.ts` which is a literate tutorial walking through every operation. Run `yarn test` to verify the examples are correct. The key types and their actual signatures can be found in: - `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot` - `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `ApiError`, `StreamOptions` - `src/errors.ts`: `ApiError`, `TruncatedStreamError` - `src/retry.ts`: `withRetry`, `isRetryable`, `isSafeToReplay`, `RetryOptions` - `src/auth/types.ts`: `KeyAttributes`, `SRPAttributes`, `AuthorizationResponse`, `LoginChallenge` - `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileBlob`, `RawCollection`, `RawEnteFile`, `RawMagicMetadata` - `src/download/index.ts`: `DownloadResult` - `src/backup.ts`: `BackupResult`, `BackupError` - `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult` ## Source attribution The cryptographic protocol and wire format implemented here are Ente's, taken from the Ente open source clients at <https://github.com/ente-io/ente>. No code is imported or vendored from those projects; any reference code that is copied is rewritten in TypeScript in this repository. Protocol fidelity is verified against the upstream implementations in `web/packages/base/`, `mobile/apps/photos/lib/`, and `cli/`. ## For LLMs If you are an LLM agent working on this repository, read and follow these documents: - **`REPO_POLICIES.md`** in the repo root. It is copied from <https://git.eeqj.de/sneak/prompts> and covers repository structure, tooling, Makefile targets, Dockerfile conventions, dependency pinning, and commit hygiene. All external dependencies must be pinned by cryptographic hash in `yarn.lock`. Never `git add -A`. Never force-push to main. - **The "Development workflow" section above.** All changes go on feature branches. Tests are written first and committed in a failing state before the implementation. Tests are the canonical API documentation and must be commented thoroughly. `main` is always green. - **Required checks before every commit:** `make lint` (eslint + prettier check) and `make fmt-check` must pass. The pre-commit hook enforces this. `make check` (which also runs tests) must pass before merging to `main`. - **Formatting:** prettier with 4-space indents and `proseWrap: always` for markdown. Use `make fmt` to format. Use `yarn` not `npm`. - **Testing:** vitest. Tests go in `test/` mirroring the `src/` structure. `make test` must complete in under 20 seconds. Use `mkdtempSync` for temporary directories, never manual timestamp paths. - **Code style:** `const` for everything, `let` if reassignment is needed, never `var`. Avoid unnecessary comments. No hand-rolled crypto. The `LLM_PROSE_TELLS.md` document in the prompts repo applies to any prose written in this repository (README, comments, commit messages). ## License WTFPL. See [LICENSE](LICENSE). ## Author [@sneak](https://sneak.berlin)