Merge: README rewrite to match current implementation
This commit is contained in:
663
README.md
663
README.md
@@ -5,6 +5,15 @@ quak is a WTFPL-licensed TypeScript client library and CLI by
|
||||
encrypted photo hosting service. It logs in, enumerates collections and files,
|
||||
and downloads individual images while decrypting them on the way to disk.
|
||||
|
||||
quak also includes a resilient backup command that downloads every file in the
|
||||
account into a deduplicated local directory tree, skipping files that already
|
||||
exist on disk and continuing past individual download failures instead of
|
||||
crashing. It decrypts and persists all three metadata layers (basic, private
|
||||
magic, public magic) per file, including camera info, GPS coordinates, captions,
|
||||
and any face/keyword labels the Ente clients have added. A helper subcommand can
|
||||
detect and regenerate missing thumbnails, encrypting and uploading them back to
|
||||
the server.
|
||||
|
||||
## Getting Started
|
||||
|
||||
```bash
|
||||
@@ -14,7 +23,6 @@ yarn install
|
||||
yarn build
|
||||
|
||||
# Log in (prompts for email, password, and OTP/TOTP if required).
|
||||
# Stores an encrypted session under $XDG_CONFIG_HOME/quak/.
|
||||
yarn quak login
|
||||
|
||||
# List the user's collections (albums).
|
||||
@@ -23,8 +31,11 @@ yarn quak collections
|
||||
# List files in a collection.
|
||||
yarn quak files --collection 12345
|
||||
|
||||
# Download and decrypt a single file to ./out/.
|
||||
yarn quak get 67890 --out ./out/
|
||||
# Download and decrypt a single file.
|
||||
yarn quak get 67890 --out ./photo.jpg
|
||||
|
||||
# Back up every file in the account.
|
||||
yarn quak backup ./my-backup
|
||||
```
|
||||
|
||||
For library use:
|
||||
@@ -32,12 +43,27 @@ For library use:
|
||||
```ts
|
||||
import { Client } from "quak";
|
||||
|
||||
const client = await Client.fromSavedSession();
|
||||
const client = await Client.login({
|
||||
email: "you@example.com",
|
||||
password: "your-password",
|
||||
});
|
||||
|
||||
for (const c of await client.listCollections()) {
|
||||
console.log(c.id, c.name);
|
||||
const files = await client.listFiles(c.id, c.key);
|
||||
for (const f of files) {
|
||||
console.log(` ${f.metadata.title} [${f.metadata.fileType}]`);
|
||||
}
|
||||
const file = await client.getFile(67890);
|
||||
await client.downloadFile(file, "./out/");
|
||||
}
|
||||
|
||||
// Download a file
|
||||
const files = await client.listFiles(collectionID, collectionKey);
|
||||
await client.downloadFile(files[0], "./photo.jpg");
|
||||
|
||||
// Serialize session for later (consumer handles persistence)
|
||||
const snapshot = client.toJSON();
|
||||
// ... later:
|
||||
const restored = Client.fromJSON(snapshot);
|
||||
```
|
||||
|
||||
## Rationale
|
||||
@@ -48,11 +74,13 @@ CLI) work, but they are slow, buggy, and difficult to script against. The
|
||||
Flutter app fails to sync reliably. The web app is heavy. The desktop app is the
|
||||
web app inside a slow Electron wrapper. The Go CLI is the closest thing to a
|
||||
usable tool, but it is awkward to integrate from anything that is not a shell.
|
||||
The Go CLI's backup mode crashes entirely when a single file download fails,
|
||||
which makes it useless as an actual backup tool.
|
||||
|
||||
quak is the first step in fixing that. This repo ships a small, correct,
|
||||
well-tested implementation of Ente's cryptographic protocol and its read-only
|
||||
API surface, plus a CLI that proves the library is enough to do real work
|
||||
without a UI.
|
||||
quak fixes these problems. This repo ships a correct, well-tested implementation
|
||||
of Ente's cryptographic protocol and API surface, plus a CLI that proves the
|
||||
library is enough to do real work without a UI. The backup command is resilient
|
||||
by design: per-file errors are logged and the run continues.
|
||||
|
||||
The longer-term goal of this project is a simple desktop client for Ente, built
|
||||
on this library in Electron (or a comparable runtime), with two priorities above
|
||||
@@ -60,11 +88,6 @@ everything else: correctness and stability. Performance and simplicity follow
|
||||
from those. Features will be added only after the protocol layer is correct, the
|
||||
local cache is reliable, and the UI is responsive on a five-year-old laptop.
|
||||
|
||||
This first release is deliberately scoped to read operations: log in, walk the
|
||||
account, decrypt and save files. Upload, sharing, deletion, and bidirectional
|
||||
sync are out of scope. Adding them later is straightforward; doing them right
|
||||
requires the protocol layer to be correct first.
|
||||
|
||||
## Development workflow
|
||||
|
||||
All work on quak is test-driven. No exceptions.
|
||||
@@ -107,10 +130,12 @@ the CLI is for humans.
|
||||
quak/
|
||||
src/
|
||||
crypto/ libsodium primitives (boxes, secretstreams, KDF, SRP)
|
||||
api/ HTTP client + typed endpoint wrappers
|
||||
api/ HTTP client (ApiClient class)
|
||||
auth/ login flow (SRP + email OTP + TOTP), key unwrap
|
||||
model/ decrypted Collection, File, Metadata types
|
||||
session/ on-disk session persistence (token + master key)
|
||||
model/ decrypted Collection, File, Metadata types + decrypt fns
|
||||
download/ streaming file/thumbnail download + decryption
|
||||
backup.ts resilient full-account backup with dedup
|
||||
thumbnails.ts detect + regenerate missing thumbnails
|
||||
client.ts high-level Client class assembled from the above
|
||||
index.ts public library exports
|
||||
bin/
|
||||
@@ -150,9 +175,13 @@ The key hierarchy, derived during login, is:
|
||||
Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the
|
||||
master key (for owned collections). Per-file keys are decrypted with
|
||||
`crypto_secretbox_open_easy` using the collection key. File metadata is a
|
||||
secretbox under the file key. File content is a chunked
|
||||
`crypto_secretstream_xchacha20poly1305` stream under the file key, with a 4 MiB
|
||||
plaintext chunk size and a 17-byte authentication overhead per chunk.
|
||||
secretstream blob (single chunk, TAG_FINAL) under the file key. File content is
|
||||
a chunked `crypto_secretstream_xchacha20poly1305` stream under the file key,
|
||||
with a 4 MiB plaintext chunk size and a 17-byte authentication overhead per
|
||||
chunk. Thumbnails use the same secretstream blob format as metadata.
|
||||
|
||||
For upload (thumbnail repair), `encryptBlob` performs the push side: a single
|
||||
secretstream chunk with TAG_FINAL, returning the header and ciphertext.
|
||||
|
||||
### HTTP API
|
||||
|
||||
@@ -163,8 +192,8 @@ Production endpoints:
|
||||
- Thumbnail CDN: `https://thumbnails.ente.io/?fileID=<id>`
|
||||
|
||||
A custom API endpoint is configurable for self-hosted servers via the
|
||||
`ENTE_API_ENDPOINT` environment variable. When set, file downloads route through
|
||||
`<endpoint>/files/download/<id>` instead of the dedicated CDN host.
|
||||
constructor option `apiOrigin`. When set, file downloads route through
|
||||
`<apiOrigin>/files/download/<id>` instead of the dedicated CDN host.
|
||||
|
||||
Required request headers on every authenticated call:
|
||||
|
||||
@@ -184,514 +213,100 @@ Endpoints used:
|
||||
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
|
||||
collection; paginate while `hasMore` is true.
|
||||
- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes.
|
||||
- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair).
|
||||
- `PUT /files/thumbnail`: register an uploaded thumbnail's object key.
|
||||
|
||||
### Session persistence
|
||||
### Session handling
|
||||
|
||||
After login, quak writes an encrypted session blob to
|
||||
`$XDG_CONFIG_HOME/quak/session.json` (default `~/.config/quak/session.json`)
|
||||
containing the auth token, the user's master key, the user's secret key, and the
|
||||
user's email. The session file is itself encrypted with a key derived from a
|
||||
per-machine random value stored in the OS keychain when available, falling back
|
||||
to a key file at mode `0600` in the same config directory. The master key and
|
||||
secret key are never written to disk in cleartext.
|
||||
The `Client` class holds the auth token, master key, secret key, and public key
|
||||
in memory. There is no on-disk session store in the library; the consumer
|
||||
decides how to persist sessions.
|
||||
|
||||
`client.toJSON()` returns a `ClientSnapshot` (a plain serializable object with
|
||||
base64-encoded keys) that the consumer can write to disk, a database, or
|
||||
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
|
||||
working client from that snapshot without re-authenticating.
|
||||
|
||||
The CLI stores the snapshot at the platform-appropriate data directory via
|
||||
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
|
||||
`$XDG_DATA_HOME/quak/session.json` on Linux. The file is written with mode
|
||||
`0600`. The key material is stored in cleartext in the JSON; treat this file as
|
||||
you would treat the password itself.
|
||||
|
||||
### CLI surface
|
||||
|
||||
- `quak login`: interactive login, writes session.
|
||||
- `quak logout`: deletes the session.
|
||||
- `quak whoami`: prints the logged-in email.
|
||||
- `quak collections`: list collections (id, name, type, file count).
|
||||
- `quak files --collection <id>`: list files in a collection (id, name, type,
|
||||
creation time, size).
|
||||
- `quak get <fileID> --out <dir>`: download and decrypt a file.
|
||||
- `quak get-thumb <fileID> --out <dir>`: download and decrypt a thumbnail.
|
||||
|
||||
All commands accept `--json` for machine-readable output.
|
||||
|
||||
## API reference
|
||||
|
||||
The complete public surface of the library, expressed as TypeScript
|
||||
declarations. Every exported name is listed here. Anything not listed is
|
||||
internal.
|
||||
|
||||
### Type aliases
|
||||
|
||||
```ts
|
||||
// src/model/types.ts
|
||||
export type Bytes = Uint8Array;
|
||||
export type Base64 = string; // standard base64 unless noted
|
||||
export type Base64URL = string; // URL-safe base64
|
||||
export type Microseconds = number; // unix epoch microseconds (int64-ish)
|
||||
```
|
||||
quak login interactive or QUAK_EMAIL/QUAK_PASSWORD
|
||||
quak whoami print logged-in account as JSON
|
||||
quak logout delete saved session
|
||||
quak collections [--json] list all collections
|
||||
quak files --collection <id> [--json] list files in a collection
|
||||
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
||||
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
||||
quak backup <dir> [--json] full incremental backup
|
||||
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
|
||||
quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails
|
||||
```
|
||||
|
||||
### Crypto module
|
||||
`get` and `get-thumb` search all collections for the file ID when `--collection`
|
||||
is not specified. All listing and backup commands support `--json` for
|
||||
machine-readable output.
|
||||
|
||||
```ts
|
||||
// src/crypto/index.ts
|
||||
### Backup layout
|
||||
|
||||
// Lazily initializes libsodium. Safe to call repeatedly; the first call
|
||||
// performs the init, subsequent calls are no-ops.
|
||||
export function init(): Promise<void>;
|
||||
`quak backup <dir>` produces:
|
||||
|
||||
// Argon2id over a UTF-8 password and a 16-byte salt, producing a 32-byte
|
||||
// key. memLimit is in bytes, opsLimit is the iteration count, both as
|
||||
// returned by the server in SRP / key attributes.
|
||||
export function deriveKEK(
|
||||
password: string,
|
||||
salt: Bytes,
|
||||
opsLimit: number,
|
||||
memLimit: number,
|
||||
): Promise<Bytes>;
|
||||
|
||||
// crypto_kdf_derive_from_key with subkey id 1 and context "loginctx",
|
||||
// returning the first 16 bytes. Used as the SRP password.
|
||||
export function deriveLoginSubkey(kek: Bytes): Bytes;
|
||||
|
||||
// crypto_secretbox_open_easy. Returns plaintext or throws on auth failure.
|
||||
export function decryptBox(ciphertext: Bytes, nonce: Bytes, key: Bytes): Bytes;
|
||||
|
||||
// crypto_box_seal_open. Used to recover the auth token after login.
|
||||
export function decryptSealed(
|
||||
ciphertext: Bytes,
|
||||
publicKey: Bytes,
|
||||
secretKey: Bytes,
|
||||
): Bytes;
|
||||
|
||||
// crypto_secretstream_xchacha20poly1305_init_pull. Returned state is opaque
|
||||
// and threaded through pullStreamChunk.
|
||||
export function initStreamPull(header: Bytes, key: Bytes): StreamPullState;
|
||||
|
||||
// crypto_secretstream_xchacha20poly1305_pull. Tag values follow libsodium's
|
||||
// constants: 0=MESSAGE, 1=PUSH, 2=REKEY, 3=FINAL.
|
||||
export function pullStreamChunk(
|
||||
state: StreamPullState,
|
||||
ciphertext: Bytes,
|
||||
): { plaintext: Bytes; tag: number };
|
||||
|
||||
// Convenience helpers. fromBase64 accepts both standard and URL-safe.
|
||||
export function fromBase64(s: Base64 | Base64URL): Bytes;
|
||||
export function toBase64(b: Bytes): Base64;
|
||||
export function toBase64URL(b: Bytes): Base64URL;
|
||||
|
||||
// Plaintext chunk size used by Ente for file content streams.
|
||||
export const STREAM_CHUNK_SIZE: number; // 4 * 1024 * 1024
|
||||
|
||||
// Encrypted-chunk overhead: secretstream auth tag (16) + tag byte (1).
|
||||
export const STREAM_CHUNK_OVERHEAD: number; // 17
|
||||
|
||||
export interface StreamPullState {
|
||||
/* opaque */
|
||||
}
|
||||
```
|
||||
<dir>/
|
||||
originals/
|
||||
<fileID>.<ext> actual file content (one per unique file)
|
||||
<fileID>.json all decrypted metadata for that file
|
||||
collections/
|
||||
<name>/
|
||||
<title> -> ../../originals/<fileID>.<ext> (symlink)
|
||||
<name>.json collection metadata + file list
|
||||
```
|
||||
|
||||
### Auth module
|
||||
|
||||
```ts
|
||||
// src/auth/types.ts
|
||||
|
||||
export interface KDFParams {
|
||||
kekSalt: Base64;
|
||||
memLimit: number;
|
||||
opsLimit: number;
|
||||
}
|
||||
|
||||
export interface KeyAttributes {
|
||||
kekSalt: Base64;
|
||||
encryptedKey: Base64;
|
||||
keyDecryptionNonce: Base64;
|
||||
publicKey: Base64;
|
||||
encryptedSecretKey: Base64;
|
||||
secretKeyDecryptionNonce: Base64;
|
||||
memLimit: number;
|
||||
opsLimit: number;
|
||||
masterKeyEncryptedWithRecoveryKey?: Base64;
|
||||
masterKeyDecryptionNonce?: Base64;
|
||||
recoveryKeyEncryptedWithMasterKey?: Base64;
|
||||
recoveryKeyDecryptionNonce?: Base64;
|
||||
}
|
||||
|
||||
export interface SRPAttributes {
|
||||
srpUserID: string;
|
||||
srpSalt: Base64;
|
||||
memLimit: number;
|
||||
opsLimit: number;
|
||||
kekSalt: Base64;
|
||||
isEmailMFAEnabled: boolean;
|
||||
}
|
||||
|
||||
export interface AuthorizationResponse {
|
||||
id: number; // user ID
|
||||
keyAttributes?: KeyAttributes;
|
||||
encryptedToken?: Base64URL; // sealed-box-encrypted to user's pubkey
|
||||
twoFactorSessionID?: string;
|
||||
passkeySessionID?: string;
|
||||
}
|
||||
|
||||
export type LoginChallenge =
|
||||
| { kind: "complete"; response: AuthorizationResponse }
|
||||
| { kind: "totp"; sessionID: string }
|
||||
| { kind: "passkey"; sessionID: string }
|
||||
| { kind: "emailOTP" };
|
||||
```
|
||||
|
||||
```ts
|
||||
// src/auth/index.ts
|
||||
|
||||
// Begin login. Returns a challenge that tells the caller what to do next:
|
||||
// supply a TOTP code, supply an email OTP, follow a passkey URL, or stop
|
||||
// because login is already complete.
|
||||
export function beginLogin(
|
||||
api: ApiClient,
|
||||
email: string,
|
||||
password: string,
|
||||
): Promise<LoginChallenge>;
|
||||
|
||||
// Submit a TOTP code from an authenticator app. Returns the final
|
||||
// AuthorizationResponse on success.
|
||||
export function submitTOTP(
|
||||
api: ApiClient,
|
||||
sessionID: string,
|
||||
code: string,
|
||||
): Promise<AuthorizationResponse>;
|
||||
|
||||
// Request and submit an email-delivered one-time code. Two calls, because
|
||||
// the first triggers email delivery and the second verifies it.
|
||||
export function requestEmailOTP(api: ApiClient, email: string): Promise<void>;
|
||||
export function submitEmailOTP(
|
||||
api: ApiClient,
|
||||
email: string,
|
||||
code: string,
|
||||
): Promise<AuthorizationResponse>;
|
||||
|
||||
// Given an AuthorizationResponse and the user's password, decrypt the master
|
||||
// key, secret key, and auth token. Throws on bad password or tampered data.
|
||||
export function unwrapAuth(
|
||||
response: AuthorizationResponse,
|
||||
password: string,
|
||||
): Promise<{
|
||||
masterKey: Bytes;
|
||||
secretKey: Bytes;
|
||||
publicKey: Bytes;
|
||||
token: string; // base64 URL-safe; goes into X-Auth-Token
|
||||
}>;
|
||||
```
|
||||
|
||||
### Model module
|
||||
|
||||
```ts
|
||||
// src/model/index.ts
|
||||
|
||||
export type CollectionType =
|
||||
| "album"
|
||||
| "folder"
|
||||
| "favorites"
|
||||
| "uncategorized"
|
||||
| "unknown";
|
||||
|
||||
export interface Collection {
|
||||
id: number;
|
||||
ownerID: number;
|
||||
key: Bytes; // decrypted
|
||||
name: string; // decrypted
|
||||
type: CollectionType;
|
||||
updationTime: Microseconds;
|
||||
isShared: boolean; // true if owner != current user
|
||||
}
|
||||
|
||||
export type FileType = "image" | "video" | "livePhoto" | "unknown";
|
||||
|
||||
export interface FileMetadata {
|
||||
title: string;
|
||||
fileType: FileType;
|
||||
creationTime: Microseconds;
|
||||
modificationTime: Microseconds;
|
||||
latitude?: number;
|
||||
longitude?: number;
|
||||
hash?: string; // base64 of file SHA256
|
||||
}
|
||||
|
||||
export interface FileBlob {
|
||||
decryptionHeader: Base64;
|
||||
size?: number; // size of the encrypted body, if known from server
|
||||
}
|
||||
|
||||
export interface EnteFile {
|
||||
id: number;
|
||||
collectionID: number;
|
||||
ownerID: number;
|
||||
key: Bytes; // decrypted file key
|
||||
metadata: FileMetadata;
|
||||
file: FileBlob;
|
||||
thumbnail: FileBlob;
|
||||
updationTime: Microseconds;
|
||||
}
|
||||
```
|
||||
|
||||
### HTTP client
|
||||
|
||||
```ts
|
||||
// src/api/client.ts
|
||||
|
||||
export interface ApiClientOptions {
|
||||
apiOrigin?: string; // default https://api.ente.io
|
||||
filesOrigin?: string; // default https://files.ente.io
|
||||
thumbsOrigin?: string; // default https://thumbnails.ente.io
|
||||
authToken?: string;
|
||||
fetch?: typeof fetch; // injectable for tests
|
||||
userAgent?: string; // default "quak/<version>"
|
||||
}
|
||||
|
||||
export class ApiError extends Error {
|
||||
readonly status: number;
|
||||
readonly code?: string;
|
||||
readonly requestID?: string;
|
||||
readonly body?: unknown;
|
||||
}
|
||||
|
||||
export class ApiClient {
|
||||
constructor(opts?: ApiClientOptions);
|
||||
setAuthToken(token: string): void;
|
||||
clearAuthToken(): void;
|
||||
|
||||
getJSON<T>(
|
||||
path: string,
|
||||
query?: Record<string, string | number | undefined>,
|
||||
): Promise<T>;
|
||||
postJSON<T>(path: string, body: unknown): Promise<T>;
|
||||
|
||||
// Streaming download from the file CDN. Caller is responsible for
|
||||
// consuming the stream.
|
||||
getFileStream(fileID: number): Promise<ReadableStream<Uint8Array>>;
|
||||
getThumbnailStream(fileID: number): Promise<ReadableStream<Uint8Array>>;
|
||||
}
|
||||
```
|
||||
|
||||
### Session
|
||||
|
||||
```ts
|
||||
// src/session/index.ts
|
||||
|
||||
export interface Session {
|
||||
email: string;
|
||||
userID: number;
|
||||
token: string; // base64 URL-safe
|
||||
masterKey: Bytes; // 32 bytes, never serialized in cleartext
|
||||
secretKey: Bytes; // 32 bytes, never serialized in cleartext
|
||||
publicKey: Bytes; // 32 bytes
|
||||
}
|
||||
|
||||
export interface SessionStoreOptions {
|
||||
path?: string; // default $XDG_CONFIG_HOME/quak/session.json
|
||||
keychainService?: string; // default "berlin.sneak.quak"
|
||||
}
|
||||
|
||||
export class SessionStore {
|
||||
constructor(opts?: SessionStoreOptions);
|
||||
load(): Promise<Session | null>;
|
||||
save(s: Session): Promise<void>;
|
||||
clear(): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
### Client
|
||||
|
||||
```ts
|
||||
// src/client.ts
|
||||
|
||||
export interface LoginPrompt {
|
||||
password: () => Promise<string>;
|
||||
emailOTP?: () => Promise<string>; // called when account uses email OTP
|
||||
totp?: () => Promise<string>; // called when TOTP is required
|
||||
}
|
||||
|
||||
export interface ClientOptions extends ApiClientOptions {
|
||||
sessionStore?: SessionStore;
|
||||
}
|
||||
|
||||
export interface DownloadResult {
|
||||
path: string;
|
||||
bytesWritten: number;
|
||||
}
|
||||
|
||||
export class Client {
|
||||
// Static constructors. Both end with a Client that has a populated
|
||||
// session and a ready ApiClient.
|
||||
static login(
|
||||
email: string,
|
||||
prompt: LoginPrompt,
|
||||
opts?: ClientOptions,
|
||||
): Promise<Client>;
|
||||
static fromSavedSession(opts?: ClientOptions): Promise<Client>;
|
||||
|
||||
readonly api: ApiClient;
|
||||
readonly session: Readonly<Session>;
|
||||
|
||||
// Account.
|
||||
whoami(): { email: string; userID: number };
|
||||
saveSession(): Promise<void>;
|
||||
logout(): Promise<void>; // clears session on disk and in memory
|
||||
|
||||
// Collections.
|
||||
listCollections(opts?: { sinceTime?: Microseconds }): Promise<Collection[]>;
|
||||
getCollection(id: number): Promise<Collection>;
|
||||
|
||||
// Files.
|
||||
listFiles(
|
||||
collectionID: number,
|
||||
opts?: { sinceTime?: Microseconds },
|
||||
): Promise<EnteFile[]>;
|
||||
getFile(fileID: number): Promise<EnteFile>;
|
||||
|
||||
// Downloads. If outPath is omitted, a path is constructed from the
|
||||
// decrypted metadata title in the current working directory. If outPath
|
||||
// is a directory, the filename is taken from the metadata title. If
|
||||
// outPath is a file path, that file is written.
|
||||
downloadFile(
|
||||
file: EnteFile | number,
|
||||
outPath?: string,
|
||||
): Promise<DownloadResult>;
|
||||
downloadThumbnail(
|
||||
file: EnteFile | number,
|
||||
outPath?: string,
|
||||
): Promise<DownloadResult>;
|
||||
}
|
||||
```
|
||||
|
||||
### Public exports (`src/index.ts`)
|
||||
|
||||
```ts
|
||||
export { Client } from "./client";
|
||||
export { ApiClient, ApiError } from "./api/client";
|
||||
export { SessionStore } from "./session";
|
||||
export * from "./model";
|
||||
export type {
|
||||
Session,
|
||||
LoginPrompt,
|
||||
ClientOptions,
|
||||
DownloadResult,
|
||||
} from "./client";
|
||||
```
|
||||
Each file is downloaded exactly once regardless of how many collections it
|
||||
appears in. On subsequent runs, existing originals are skipped. If a download
|
||||
fails, the error is logged and the backup continues with the next file. The exit
|
||||
code is non-zero if any files failed.
|
||||
|
||||
## TODO
|
||||
|
||||
Phase 1: scaffolding
|
||||
|
||||
- [x] `git init`, write README
|
||||
- [x] Create `initial-scaffolding` feature branch
|
||||
- [x] Add `LICENSE` (WTFPL), `REPO_POLICIES.md`, `.gitignore`, `.editorconfig`,
|
||||
`.prettierrc`, `.prettierignore`, `.dockerignore`
|
||||
- [x] Add `Makefile` with `test`, `lint`, `fmt`, `fmt-check`, `check`, `docker`,
|
||||
`hooks`, plus `build`, `dev`, `clean`
|
||||
- [x] Add `Dockerfile` running `make check` against pinned node image
|
||||
- [x] Add `.gitea/workflows/check.yml` running `docker build .`
|
||||
- [x] Add `package.json`, `tsconfig.json`, pinned dev versions of `typescript`,
|
||||
`prettier`, `eslint`, `typescript-eslint`, `vitest`, `@types/node` (the
|
||||
runtime deps `libsodium-wrappers-sumo`, `secure-remote-password`,
|
||||
`commander`, etc. land with their respective implementation phases)
|
||||
- [x] Smoke test: `make check` and `make docker` both pass
|
||||
|
||||
Phase 2: crypto primitives
|
||||
|
||||
- [x] Wrap libsodium init as an awaitable singleton
|
||||
- [x] `deriveKEK(password, kekSalt, memLimit, opsLimit)` (Argon2id)
|
||||
- [x] `deriveLoginSubkey(kek)` (KDF with subkey id 1, context `loginctx`, 16
|
||||
bytes)
|
||||
- [x] `decryptBox(ciphertext, nonce, key)` for secretbox
|
||||
- [x] `decryptSealed(ciphertext, publicKey, secretKey)` for sealed box
|
||||
- [x] `initStreamPull` and `pullStreamChunk` for chunked secretstream (4 MiB
|
||||
plaintext chunks, 17-byte overhead)
|
||||
- [x] Round-trip tests against vectors generated by libsodium directly
|
||||
- [x] Base64 helpers (`fromBase64`, `toBase64`, `toBase64URL`) accepting all
|
||||
four sodium variants on input
|
||||
|
||||
Phase 3: SRP + auth
|
||||
|
||||
- [x] SRP-6a client using `fast-srp-hap` with the 4096-bit group (matching the
|
||||
upstream Ente web client)
|
||||
- [x] `beginLogin(api, email, password)` returning a `LoginChallenge`
|
||||
- [x] `requestEmailOTP` and `submitEmailOTP` for accounts without SRP
|
||||
- [x] `submitTOTP(api, sessionID, code)`
|
||||
- [x] `unwrapAuth(response, password)` returning master key, secret key, public
|
||||
key, and decrypted token (URL-safe-no-padding base64)
|
||||
- [x] `src/auth/types.ts` with `KeyAttributes`, `SRPAttributes`,
|
||||
`AuthorizationResponse`, and `LoginChallenge`
|
||||
- [x] Tests with mock SRP server performing real 4096-bit math end-to-end
|
||||
|
||||
Phase 4: HTTP client + endpoints
|
||||
|
||||
- [x] `ApiClient` that attaches `X-Auth-Token` and `X-Client-Package`
|
||||
- [x] `ApiError` that surfaces the server's error code and request id
|
||||
- [x] `getJSON` / `postJSON` with query-param and JSON-body handling
|
||||
- [x] `getFileStream` / `getThumbnailStream` with self-hosted routing
|
||||
- [ ] Typed wrappers for the endpoints listed above
|
||||
- [ ] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
|
||||
errors
|
||||
|
||||
Phase 5: collections and files
|
||||
|
||||
- [x] `decryptCollection(raw, masterKey)` with key + name decryption, type
|
||||
mapping, isShared flag
|
||||
- [x] `decryptFile(raw, collectionKey)` with key + metadata decryption
|
||||
(secretstream blob, not secretbox), fileType mapping, header passthrough
|
||||
- [x] `decryptBlob(ciphertext, header, key)` convenience for single-chunk
|
||||
secretstream decryption (used by file metadata and magic metadata)
|
||||
- [x] Model types: `Collection`, `EnteFile`, `FileMetadata`, `RawCollection`,
|
||||
`RawEnteFile`
|
||||
- [x] Live-tested against real Ente API (collection names + file metadata)
|
||||
- [ ] Higher-level `listCollections()` / `listFiles()` with pagination
|
||||
|
||||
Phase 6: download
|
||||
|
||||
- [x] `downloadFile(api, file, outPath?)` streams encrypted body, buffers to 4
|
||||
MiB chunk boundary, decrypts via secretstream pull, writes to disk. Falls
|
||||
back to `metadata.title` when outPath is omitted.
|
||||
- [x] `downloadThumbnail(api, file, outPath?)` same for thumbnails
|
||||
- [x] Live integration test: logs in, decrypts collections and files, downloads
|
||||
a real JPEG from the dev account and verifies it on disk
|
||||
|
||||
Phase 7: Client class
|
||||
|
||||
- [x] `Client.login({ email, password, totp?, emailOTP? })` performs the full
|
||||
SRP handshake, key unwrap, returns a ready Client
|
||||
- [x] `Client.fromJSON(snapshot)` restores from a serialized snapshot
|
||||
- [x] `client.toJSON()` produces a `ClientSnapshot` the consumer can persist
|
||||
- [x] `client.whoami()`, `client.logout()`
|
||||
- [x] `client.listCollections()` with decryption
|
||||
- [x] `client.listFiles(collectionID, collectionKey)` with pagination
|
||||
- [x] `client.downloadFile(file, outPath?)` and `client.downloadThumbnail()`
|
||||
- [x] Literate test/client/usage.test.ts tutorial covering the entire API
|
||||
|
||||
Phase 8: CLI
|
||||
|
||||
- [x] `quak login` (interactive TTY prompts or QUAK_EMAIL/QUAK_PASSWORD env vars
|
||||
for non-interactive use)
|
||||
- [x] `quak whoami`, `quak logout`
|
||||
- [x] `quak collections` and `quak files --collection <id>`
|
||||
- [x] `quak get <fileID>` and `quak get-thumb <fileID>` with --out and
|
||||
--collection options; searches all collections when --collection omitted
|
||||
- [x] `quak backup <dir>` with originals/ dedup, collections/ symlinks,
|
||||
per-collection and per-file JSON metadata, incremental skip, per-file
|
||||
error resilience, --json flag
|
||||
- [x] `--json` output on every listing/backup command
|
||||
- [x] Progress output to stderr for backup
|
||||
- [x] Session persistence via env-paths (~/Library/Application Support/quak/ on
|
||||
macOS, XDG_DATA_HOME/quak/ on Linux)
|
||||
|
||||
Phase 9: docs and 1.0
|
||||
|
||||
- [ ] Update README Getting Started and Design sections to match current state
|
||||
- [ ] All TODO items above checked
|
||||
- [ ] Update the API reference section below to match the current implementation
|
||||
- [ ] `make docker` green
|
||||
- [ ] Tag `v1.0.0`
|
||||
|
||||
Phase 10 and beyond: desktop client (separate repo)
|
||||
Future (desktop client, separate repo):
|
||||
|
||||
- [ ] Spike Electron app skeleton consuming this library
|
||||
- [ ] Electron app skeleton consuming this library
|
||||
- [ ] Local cache (SQLite) keyed on `(collectionID, fileID, updationTime)`
|
||||
- [ ] Background sync worker that streams new files into the cache
|
||||
- [ ] Read-only gallery UI: thumbnails, full-image view, basic search
|
||||
- [ ] Add upload, delete, and share back into the library before the desktop UI
|
||||
exposes them
|
||||
- [ ] Gallery UI: thumbnails, full-image view, basic search
|
||||
- [ ] Upload, delete, and share operations in the library
|
||||
|
||||
## API reference
|
||||
|
||||
The API reference section below is from an earlier draft and does not fully
|
||||
reflect the current implementation. The authoritative API documentation is in
|
||||
the test files, particularly `test/client/usage.test.ts` which is a literate
|
||||
tutorial walking through every operation. Run `yarn test` to verify the examples
|
||||
are correct.
|
||||
|
||||
The key types and their actual signatures can be found in:
|
||||
|
||||
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
|
||||
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `ApiError`
|
||||
- `src/auth/types.ts`: `KeyAttributes`, `SRPAttributes`,
|
||||
`AuthorizationResponse`, `LoginChallenge`
|
||||
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileBlob`,
|
||||
`RawCollection`, `RawEnteFile`, `RawMagicMetadata`
|
||||
- `src/download/index.ts`: `DownloadResult`
|
||||
- `src/backup.ts`: `BackupResult`, `BackupError`
|
||||
- `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult`
|
||||
|
||||
## Source attribution
|
||||
|
||||
@@ -702,6 +317,38 @@ is rewritten in TypeScript in this repository. Protocol fidelity is verified
|
||||
against the upstream implementations in `web/packages/base/`,
|
||||
`mobile/apps/photos/lib/`, and `cli/`.
|
||||
|
||||
## For LLMs
|
||||
|
||||
If you are an LLM agent working on this repository, read and follow these
|
||||
documents:
|
||||
|
||||
- **`REPO_POLICIES.md`** in the repo root. It is copied from
|
||||
<https://git.eeqj.de/sneak/prompts> and covers repository structure, tooling,
|
||||
Makefile targets, Dockerfile conventions, dependency pinning, and commit
|
||||
hygiene. All external dependencies must be pinned by cryptographic hash in
|
||||
`yarn.lock`. Never `git add -A`. Never force-push to main.
|
||||
|
||||
- **The "Development workflow" section above.** All changes go on feature
|
||||
branches. Tests are written first and committed in a failing state before the
|
||||
implementation. Tests are the canonical API documentation and must be
|
||||
commented thoroughly. `main` is always green.
|
||||
|
||||
- **Required checks before every commit:** `make lint` (eslint + prettier check)
|
||||
and `make fmt-check` must pass. The pre-commit hook enforces this.
|
||||
`make check` (which also runs tests) must pass before merging to `main`.
|
||||
|
||||
- **Formatting:** prettier with 4-space indents and `proseWrap: always` for
|
||||
markdown. Use `make fmt` to format. Use `yarn` not `npm`.
|
||||
|
||||
- **Testing:** vitest. Tests go in `test/` mirroring the `src/` structure.
|
||||
`make test` must complete in under 20 seconds. Use `mkdtempSync` for temporary
|
||||
directories, never manual timestamp paths.
|
||||
|
||||
- **Code style:** `const` for everything, `let` if reassignment is needed, never
|
||||
`var`. Avoid unnecessary comments. No hand-rolled crypto. The
|
||||
`LLM_PROSE_TELLS.md` document in the prompts repo applies to any prose written
|
||||
in this repository (README, comments, commit messages).
|
||||
|
||||
## License
|
||||
|
||||
WTFPL. See [LICENSE](LICENSE).
|
||||
|
||||
Reference in New Issue
Block a user