diff --git a/README.md b/README.md index b78c8f9..a589bbf 100644 --- a/README.md +++ b/README.md @@ -43,16 +43,29 @@ await client.downloadFile(file, "./out/"); ## Rationale Ente is one of very few photo services with a credible end-to-end encryption -story. The official clients (mobile Flutter app, web React app, and Go CLI) are -high quality but each is bound up with a UI or with sync semantics that aren't -useful for scripted access. Pulling individual photos out of an Ente account -from a script (for backup, migration, archival, or programmatic processing) is -awkward without a small TypeScript library that does just the cryptography and -nothing else. +story. The shipping clients (mobile Flutter, web React, desktop Electron, and +Go CLI) work, but they are slow, buggy, and difficult to script against. The +Flutter app fails to sync reliably. The web app is heavy. The desktop app is +the web app inside a slow Electron wrapper. The Go CLI is the closest thing +to a usable tool, but it is awkward to integrate from anything that is not a +shell. -This project exists to provide that library. It is deliberately scoped to read -operations: log in, walk the account, decrypt and save files. Upload, sharing, -deletion, and sync state management are out of scope for the first release. +quack is the first step in fixing that. This repo ships a small, correct, +well-tested implementation of Ente's cryptographic protocol and its read-only +API surface, plus a CLI that proves the library is enough to do real work +without a UI. + +The longer-term goal of this project is a simple desktop client for Ente, +built on this library in Electron (or a comparable runtime), with two +priorities above everything else: correctness and stability. Performance and +simplicity follow from those. Features will be added only after the protocol +layer is correct, the local cache is reliable, and the UI is responsive on a +five-year-old laptop. + +This first release is deliberately scoped to read operations: log in, walk +the account, decrypt and save files. Upload, sharing, deletion, and +bidirectional sync are out of scope. Adding them later is straightforward; +doing them right requires the protocol layer to be correct first. ## Design @@ -85,7 +98,7 @@ quack/ All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is required for `crypto_pwhash` / Argon2id). No hand-rolled crypto. -The key hierarchy, derived during login, looks like this: +The key hierarchy, derived during login, is: 1. The user enters their password. 2. Argon2id (`crypto_pwhash`) over the password and a server-issued @@ -132,17 +145,17 @@ Required request headers on every authenticated call: Endpoints used: -- `GET /users/srp/attributes?email=` — fetch SRP + KDF parameters. -- `POST /users/srp/create-session` — begin SRP handshake. -- `POST /users/srp/verify-session` — complete SRP, receive 2FA challenge or - the encrypted token + key attributes. -- `POST /users/ott` and `POST /users/verify-email` — email OTP fallback path. -- `POST /users/two-factor/verify` — TOTP second factor. -- `GET /collections/v2?sinceTime=` — list collections changed since +- `GET /users/srp/attributes?email=`: fetch SRP and KDF parameters. +- `POST /users/srp/create-session`: begin SRP handshake. +- `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or + the encrypted token plus key attributes. +- `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path. +- `POST /users/two-factor/verify`: TOTP second factor. +- `GET /collections/v2?sinceTime=`: list collections changed since microsecond timestamp; pass 0 for a full enumeration. -- `GET /collections/v2/diff?collectionID=&sinceTime=` — list +- `GET /collections/v2/diff?collectionID=&sinceTime=`: list files in a collection; paginate while `hasMore` is true. -- `GET https://files.ente.io/?fileID=` — download encrypted file bytes. +- `GET https://files.ente.io/?fileID=`: download encrypted file bytes. ### Session persistence @@ -157,18 +170,382 @@ disk in cleartext. ### CLI surface -- `quack login` — interactive login, writes session. -- `quack logout` — deletes the session. -- `quack whoami` — prints the logged-in email. -- `quack collections` — list collections (id, name, type, file count). -- `quack files --collection ` — list files in a collection (id, name, +- `quack login`: interactive login, writes session. +- `quack logout`: deletes the session. +- `quack whoami`: prints the logged-in email. +- `quack collections`: list collections (id, name, type, file count). +- `quack files --collection `: list files in a collection (id, name, type, creation time, size). -- `quack get --out ` — download and decrypt a file. -- `quack get-thumb --out ` — download and decrypt a +- `quack get --out `: download and decrypt a file. +- `quack get-thumb --out `: download and decrypt a thumbnail. All commands accept `--json` for machine-readable output. +## API reference + +The complete public surface of the library, expressed as TypeScript +declarations. Every exported name is listed here. Anything not listed is +internal. + +### Type aliases + +```ts +// src/model/types.ts +export type Bytes = Uint8Array; +export type Base64 = string; // standard base64 unless noted +export type Base64URL = string; // URL-safe base64 +export type Microseconds = number; // unix epoch microseconds (int64-ish) +``` + +### Crypto module + +```ts +// src/crypto/index.ts + +// Lazily initializes libsodium. Safe to call repeatedly; the first call +// performs the init, subsequent calls are no-ops. +export function init(): Promise; + +// Argon2id over a UTF-8 password and a 16-byte salt, producing a 32-byte +// key. memLimit is in bytes, opsLimit is the iteration count, both as +// returned by the server in SRP / key attributes. +export function deriveKEK( + password: string, + salt: Bytes, + opsLimit: number, + memLimit: number, +): Promise; + +// crypto_kdf_derive_from_key with subkey id 1 and context "loginctx", +// returning the first 16 bytes. Used as the SRP password. +export function deriveLoginSubkey(kek: Bytes): Bytes; + +// crypto_secretbox_open_easy. Returns plaintext or throws on auth failure. +export function decryptBox(ciphertext: Bytes, nonce: Bytes, key: Bytes): Bytes; + +// crypto_box_seal_open. Used to recover the auth token after login. +export function decryptSealed( + ciphertext: Bytes, + publicKey: Bytes, + secretKey: Bytes, +): Bytes; + +// crypto_secretstream_xchacha20poly1305_init_pull. Returned state is opaque +// and threaded through pullStreamChunk. +export function initStreamPull(header: Bytes, key: Bytes): StreamPullState; + +// crypto_secretstream_xchacha20poly1305_pull. Tag values follow libsodium's +// constants: 0=MESSAGE, 1=PUSH, 2=REKEY, 3=FINAL. +export function pullStreamChunk( + state: StreamPullState, + ciphertext: Bytes, +): { plaintext: Bytes; tag: number }; + +// Convenience helpers. fromBase64 accepts both standard and URL-safe. +export function fromBase64(s: Base64 | Base64URL): Bytes; +export function toBase64(b: Bytes): Base64; +export function toBase64URL(b: Bytes): Base64URL; + +// Plaintext chunk size used by Ente for file content streams. +export const STREAM_CHUNK_SIZE: number; // 4 * 1024 * 1024 + +// Encrypted-chunk overhead: secretstream auth tag (16) + tag byte (1). +export const STREAM_CHUNK_OVERHEAD: number; // 17 + +export interface StreamPullState { + /* opaque */ +} +``` + +### Auth module + +```ts +// src/auth/types.ts + +export interface KDFParams { + kekSalt: Base64; + memLimit: number; + opsLimit: number; +} + +export interface KeyAttributes { + kekSalt: Base64; + encryptedKey: Base64; + keyDecryptionNonce: Base64; + publicKey: Base64; + encryptedSecretKey: Base64; + secretKeyDecryptionNonce: Base64; + memLimit: number; + opsLimit: number; + masterKeyEncryptedWithRecoveryKey?: Base64; + masterKeyDecryptionNonce?: Base64; + recoveryKeyEncryptedWithMasterKey?: Base64; + recoveryKeyDecryptionNonce?: Base64; +} + +export interface SRPAttributes { + srpUserID: string; + srpSalt: Base64; + memLimit: number; + opsLimit: number; + kekSalt: Base64; + isEmailMFAEnabled: boolean; +} + +export interface AuthorizationResponse { + id: number; // user ID + keyAttributes?: KeyAttributes; + encryptedToken?: Base64URL; // sealed-box-encrypted to user's pubkey + twoFactorSessionID?: string; + passkeySessionID?: string; +} + +export type LoginChallenge = + | { kind: "complete"; response: AuthorizationResponse } + | { kind: "totp"; sessionID: string } + | { kind: "passkey"; sessionID: string } + | { kind: "emailOTP" }; +``` + +```ts +// src/auth/index.ts + +// Begin login. Returns a challenge that tells the caller what to do next: +// supply a TOTP code, supply an email OTP, follow a passkey URL, or stop +// because login is already complete. +export function beginLogin( + api: ApiClient, + email: string, + password: string, +): Promise; + +// Submit a TOTP code from an authenticator app. Returns the final +// AuthorizationResponse on success. +export function submitTOTP( + api: ApiClient, + sessionID: string, + code: string, +): Promise; + +// Request and submit an email-delivered one-time code. Two calls, because +// the first triggers email delivery and the second verifies it. +export function requestEmailOTP(api: ApiClient, email: string): Promise; +export function submitEmailOTP( + api: ApiClient, + email: string, + code: string, +): Promise; + +// Given an AuthorizationResponse and the user's password, decrypt the master +// key, secret key, and auth token. Throws on bad password or tampered data. +export function unwrapAuth( + response: AuthorizationResponse, + password: string, +): Promise<{ + masterKey: Bytes; + secretKey: Bytes; + publicKey: Bytes; + token: string; // base64 URL-safe; goes into X-Auth-Token +}>; +``` + +### Model module + +```ts +// src/model/index.ts + +export type CollectionType = + | "album" + | "folder" + | "favorites" + | "uncategorized" + | "unknown"; + +export interface Collection { + id: number; + ownerID: number; + key: Bytes; // decrypted + name: string; // decrypted + type: CollectionType; + updationTime: Microseconds; + isShared: boolean; // true if owner != current user +} + +export type FileType = "image" | "video" | "livePhoto" | "unknown"; + +export interface FileMetadata { + title: string; + fileType: FileType; + creationTime: Microseconds; + modificationTime: Microseconds; + latitude?: number; + longitude?: number; + hash?: string; // base64 of file SHA256 +} + +export interface FileBlob { + decryptionHeader: Base64; + size?: number; // size of the encrypted body, if known from server +} + +export interface EnteFile { + id: number; + collectionID: number; + ownerID: number; + key: Bytes; // decrypted file key + metadata: FileMetadata; + file: FileBlob; + thumbnail: FileBlob; + updationTime: Microseconds; +} +``` + +### HTTP client + +```ts +// src/api/client.ts + +export interface ApiClientOptions { + apiOrigin?: string; // default https://api.ente.io + filesOrigin?: string; // default https://files.ente.io + thumbsOrigin?: string; // default https://thumbnails.ente.io + authToken?: string; + fetch?: typeof fetch; // injectable for tests + userAgent?: string; // default "quack/" +} + +export class ApiError extends Error { + readonly status: number; + readonly code?: string; + readonly requestID?: string; + readonly body?: unknown; +} + +export class ApiClient { + constructor(opts?: ApiClientOptions); + setAuthToken(token: string): void; + clearAuthToken(): void; + + getJSON( + path: string, + query?: Record, + ): Promise; + postJSON(path: string, body: unknown): Promise; + + // Streaming download from the file CDN. Caller is responsible for + // consuming the stream. + getFileStream(fileID: number): Promise>; + getThumbnailStream(fileID: number): Promise>; +} +``` + +### Session + +```ts +// src/session/index.ts + +export interface Session { + email: string; + userID: number; + token: string; // base64 URL-safe + masterKey: Bytes; // 32 bytes, never serialized in cleartext + secretKey: Bytes; // 32 bytes, never serialized in cleartext + publicKey: Bytes; // 32 bytes +} + +export interface SessionStoreOptions { + path?: string; // default $XDG_CONFIG_HOME/quack/session.json + keychainService?: string; // default "berlin.sneak.quack" +} + +export class SessionStore { + constructor(opts?: SessionStoreOptions); + load(): Promise; + save(s: Session): Promise; + clear(): Promise; +} +``` + +### Client + +```ts +// src/client.ts + +export interface LoginPrompt { + password: () => Promise; + emailOTP?: () => Promise; // called when account uses email OTP + totp?: () => Promise; // called when TOTP is required +} + +export interface ClientOptions extends ApiClientOptions { + sessionStore?: SessionStore; +} + +export interface DownloadResult { + path: string; + bytesWritten: number; +} + +export class Client { + // Static constructors. Both end with a Client that has a populated + // session and a ready ApiClient. + static login( + email: string, + prompt: LoginPrompt, + opts?: ClientOptions, + ): Promise; + static fromSavedSession(opts?: ClientOptions): Promise; + + readonly api: ApiClient; + readonly session: Readonly; + + // Account. + whoami(): { email: string; userID: number }; + saveSession(): Promise; + logout(): Promise; // clears session on disk and in memory + + // Collections. + listCollections(opts?: { sinceTime?: Microseconds }): Promise; + getCollection(id: number): Promise; + + // Files. + listFiles( + collectionID: number, + opts?: { sinceTime?: Microseconds }, + ): Promise; + getFile(fileID: number): Promise; + + // Downloads. If outPath is omitted, a path is constructed from the + // decrypted metadata title in the current working directory. If outPath + // is a directory, the filename is taken from the metadata title. If + // outPath is a file path, that file is written. + downloadFile( + file: EnteFile | number, + outPath?: string, + ): Promise; + downloadThumbnail( + file: EnteFile | number, + outPath?: string, + ): Promise; +} +``` + +### Public exports (`src/index.ts`) + +```ts +export { Client } from "./client"; +export { ApiClient, ApiError } from "./api/client"; +export { SessionStore } from "./session"; +export * from "./model"; +export type { + Session, + LoginPrompt, + ClientOptions, + DownloadResult, +} from "./client"; +``` + ## TODO Phase 1: scaffolding (this commit and the next) @@ -194,7 +571,7 @@ Phase 2: crypto primitives 16 bytes) - [ ] `decryptBox(ciphertext, nonce, key)` for secretbox - [ ] `decryptSealed(ciphertext, publicKey, secretKey)` for sealed box -- [ ] `decryptStream(reader, header, key, writer)` for chunked secretstream +- [ ] `initStreamPull` and `pullStreamChunk` for chunked secretstream (4 MiB plaintext chunks, 17-byte overhead) - [ ] Round-trip tests against vectors generated by libsodium directly @@ -202,21 +579,20 @@ Phase 3: SRP + auth - [ ] SRP-6a client using `secure-remote-password` with the same group as the server -- [ ] `loginViaSRP(email, password)` returning either a 2FA challenge or - `KeyAttributes + encryptedToken` -- [ ] `loginViaEmailOTP(email)` for accounts without SRP enabled +- [ ] `beginLogin(email, password)` returning a `LoginChallenge` +- [ ] `requestEmailOTP` and `submitEmailOTP` for accounts without SRP - [ ] `submitTOTP(sessionID, code)` -- [ ] `unwrapMasterKey(keyAttributes, password)` returning master key, - secret key, public key, and decrypted token +- [ ] `unwrapAuth(response, password)` returning master key, secret key, + public key, and decrypted token - [ ] Tests against recorded HTTP fixtures Phase 4: HTTP client + endpoints -- [ ] Tiny fetch wrapper that attaches `X-Auth-Token` and `X-Client-Package` +- [ ] `ApiClient` that attaches `X-Auth-Token` and `X-Client-Package` - [ ] Typed wrappers for the endpoints listed above - [ ] Retry policy: no retry on 4xx, exponential backoff on 5xx and network errors -- [ ] Error type that surfaces the server's error code and request id +- [ ] `ApiError` that surfaces the server's error code and request id Phase 5: collections and files @@ -239,8 +615,8 @@ Phase 6: download Phase 7: session persistence -- [ ] Write session blob encrypted with a key from the OS keychain - (`keytar`) or a `0600` keyfile fallback +- [ ] `SessionStore` writing an encrypted session blob with a key from the + OS keychain (`keytar`) or a `0600` keyfile fallback - [ ] `Client.fromSavedSession()` and `Client.saveSession()` - [ ] `quack logout` deletes the session and the keychain entry @@ -257,13 +633,22 @@ Phase 9: docs and 1.0 - [ ] All TODO items above checked - [ ] Tag `v1.0.0` +Phase 10 and beyond: desktop client (separate repo) + +- [ ] Spike Electron app skeleton consuming this library +- [ ] Local cache (SQLite) keyed on `(collectionID, fileID, updationTime)` +- [ ] Background sync worker that streams new files into the cache +- [ ] Read-only gallery UI: thumbnails, full-image view, basic search +- [ ] Add upload, delete, and share back into the library before the + desktop UI exposes them + ## Source attribution -The cryptographic protocol and wire format implemented here are Ente's, taken -from the Ente open source clients at -. No code is imported or vendored from those -projects; any reference code that is copied is rewritten in TypeScript in -this repository. Protocol fidelity is verified against the upstream +The cryptographic protocol and wire format implemented here are Ente's, +taken from the Ente open source clients at +. No code is imported or vendored from +those projects; any reference code that is copied is rewritten in TypeScript +in this repository. Protocol fidelity is verified against the upstream implementations in `web/packages/base/`, `mobile/apps/photos/lib/`, and `cli/`.