Initial commit: README with project spec and implementation plan
This commit is contained in:
276
README.md
Normal file
276
README.md
Normal file
@@ -0,0 +1,276 @@
|
||||
# quack
|
||||
|
||||
quack is a WTFPL-licensed TypeScript client library and CLI by
|
||||
[@sneak](https://sneak.berlin) for the [Ente](https://ente.io) end-to-end
|
||||
encrypted photo hosting service. It logs in, enumerates collections and files,
|
||||
and downloads individual images while decrypting them on the way to disk.
|
||||
|
||||
## Getting Started
|
||||
|
||||
```bash
|
||||
git clone https://git.eeqj.de/sneak/quack.git
|
||||
cd quack
|
||||
yarn install
|
||||
yarn build
|
||||
|
||||
# Log in (prompts for email, password, and OTP/TOTP if required).
|
||||
# Stores an encrypted session under $XDG_CONFIG_HOME/quack/.
|
||||
yarn quack login
|
||||
|
||||
# List the user's collections (albums).
|
||||
yarn quack collections
|
||||
|
||||
# List files in a collection.
|
||||
yarn quack files --collection 12345
|
||||
|
||||
# Download and decrypt a single file to ./out/.
|
||||
yarn quack get 67890 --out ./out/
|
||||
```
|
||||
|
||||
For library use:
|
||||
|
||||
```ts
|
||||
import { Client } from "quack";
|
||||
|
||||
const client = await Client.fromSavedSession();
|
||||
for (const c of await client.listCollections()) {
|
||||
console.log(c.id, c.name);
|
||||
}
|
||||
const file = await client.getFile(67890);
|
||||
await client.downloadFile(file, "./out/");
|
||||
```
|
||||
|
||||
## Rationale
|
||||
|
||||
Ente is one of very few photo services with a credible end-to-end encryption
|
||||
story. The official clients (mobile Flutter app, web React app, and Go CLI) are
|
||||
high quality but each is bound up with a UI or with sync semantics that aren't
|
||||
useful for scripted access. Pulling individual photos out of an Ente account
|
||||
from a script (for backup, migration, archival, or programmatic processing) is
|
||||
awkward without a small TypeScript library that does just the cryptography and
|
||||
nothing else.
|
||||
|
||||
This project exists to provide that library. It is deliberately scoped to read
|
||||
operations: log in, walk the account, decrypt and save files. Upload, sharing,
|
||||
deletion, and sync state management are out of scope for the first release.
|
||||
|
||||
## Design
|
||||
|
||||
quack is a TypeScript library with a thin CLI wrapper. The library does the
|
||||
work; the CLI is for humans.
|
||||
|
||||
### Layout
|
||||
|
||||
```
|
||||
quack/
|
||||
src/
|
||||
crypto/ libsodium primitives (boxes, secretstreams, KDF, SRP)
|
||||
api/ HTTP client + typed endpoint wrappers
|
||||
auth/ login flow (SRP + email OTP + TOTP), key unwrap
|
||||
model/ decrypted Collection, File, Metadata types
|
||||
session/ on-disk session persistence (token + master key)
|
||||
client.ts high-level Client class assembled from the above
|
||||
index.ts public library exports
|
||||
bin/
|
||||
quack.ts CLI entrypoint (commander.js)
|
||||
test/ unit + integration tests (vitest)
|
||||
Makefile
|
||||
Dockerfile
|
||||
package.json
|
||||
tsconfig.json
|
||||
```
|
||||
|
||||
### Cryptography
|
||||
|
||||
All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is
|
||||
required for `crypto_pwhash` / Argon2id). No hand-rolled crypto.
|
||||
|
||||
The key hierarchy, derived during login, looks like this:
|
||||
|
||||
1. The user enters their password.
|
||||
2. Argon2id (`crypto_pwhash`) over the password and a server-issued
|
||||
`kekSalt`, with server-issued `memLimit` and `opsLimit`, produces a
|
||||
32-byte Key Encryption Key (KEK).
|
||||
3. SRP login: a 16-byte SRP login subkey is derived from the KEK using
|
||||
`crypto_kdf_derive_from_key` (BLAKE2b) with subkey id 1 and context
|
||||
`loginctx`. That 16-byte value is the SRP password.
|
||||
4. After SRP completes (or after email-OTP fallback), the server returns a
|
||||
blob of "key attributes" plus an encrypted auth token.
|
||||
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
|
||||
yields the 32-byte master key.
|
||||
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
|
||||
key yields the user's X25519 private key. The matching public key is
|
||||
delivered in cleartext.
|
||||
7. `crypto_box_seal_open` over the encrypted token with the user's keypair
|
||||
yields the URL-safe base64 auth token used in `X-Auth-Token` for all
|
||||
subsequent calls.
|
||||
|
||||
Per-collection keys are decrypted with `crypto_secretbox_open_easy` using
|
||||
the master key (for owned collections). Per-file keys are decrypted with
|
||||
`crypto_secretbox_open_easy` using the collection key. File metadata is a
|
||||
secretbox under the file key. File content is a chunked
|
||||
`crypto_secretstream_xchacha20poly1305` stream under the file key, with a
|
||||
4 MiB plaintext chunk size and a 17-byte authentication overhead per chunk.
|
||||
|
||||
### HTTP API
|
||||
|
||||
Production endpoints:
|
||||
|
||||
- API: `https://api.ente.io`
|
||||
- File download CDN: `https://files.ente.io/?fileID=<id>`
|
||||
- Thumbnail CDN: `https://thumbnails.ente.io/?fileID=<id>`
|
||||
|
||||
A custom API endpoint is configurable for self-hosted servers via the
|
||||
`ENTE_API_ENDPOINT` environment variable. When set, file downloads route
|
||||
through `<endpoint>/files/download/<id>` instead of the dedicated CDN host.
|
||||
|
||||
Required request headers on every authenticated call:
|
||||
|
||||
- `X-Auth-Token`: the decrypted auth token from login.
|
||||
- `X-Client-Package`: identifies the client. quack uses
|
||||
`berlin.sneak.quack`.
|
||||
|
||||
Endpoints used:
|
||||
|
||||
- `GET /users/srp/attributes?email=<email>` — fetch SRP + KDF parameters.
|
||||
- `POST /users/srp/create-session` — begin SRP handshake.
|
||||
- `POST /users/srp/verify-session` — complete SRP, receive 2FA challenge or
|
||||
the encrypted token + key attributes.
|
||||
- `POST /users/ott` and `POST /users/verify-email` — email OTP fallback path.
|
||||
- `POST /users/two-factor/verify` — TOTP second factor.
|
||||
- `GET /collections/v2?sinceTime=<usec>` — list collections changed since
|
||||
microsecond timestamp; pass 0 for a full enumeration.
|
||||
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>` — list
|
||||
files in a collection; paginate while `hasMore` is true.
|
||||
- `GET https://files.ente.io/?fileID=<id>` — download encrypted file bytes.
|
||||
|
||||
### Session persistence
|
||||
|
||||
After login, quack writes an encrypted session blob to
|
||||
`$XDG_CONFIG_HOME/quack/session.json` (default
|
||||
`~/.config/quack/session.json`) containing the auth token, the user's master
|
||||
key, the user's secret key, and the user's email. The session file is itself
|
||||
encrypted with a key derived from a per-machine random value stored in the OS
|
||||
keychain when available, falling back to a key file at mode `0600` in the
|
||||
same config directory. The master key and secret key are never written to
|
||||
disk in cleartext.
|
||||
|
||||
### CLI surface
|
||||
|
||||
- `quack login` — interactive login, writes session.
|
||||
- `quack logout` — deletes the session.
|
||||
- `quack whoami` — prints the logged-in email.
|
||||
- `quack collections` — list collections (id, name, type, file count).
|
||||
- `quack files --collection <id>` — list files in a collection (id, name,
|
||||
type, creation time, size).
|
||||
- `quack get <fileID> --out <dir>` — download and decrypt a file.
|
||||
- `quack get-thumb <fileID> --out <dir>` — download and decrypt a
|
||||
thumbnail.
|
||||
|
||||
All commands accept `--json` for machine-readable output.
|
||||
|
||||
## TODO
|
||||
|
||||
Phase 1: scaffolding (this commit and the next)
|
||||
|
||||
- [x] `git init`, write README
|
||||
- [ ] Create `initial-scaffolding` feature branch
|
||||
- [ ] Add `LICENSE` (WTFPL), `REPO_POLICIES.md`, `.gitignore`,
|
||||
`.editorconfig`, `.prettierrc`, `.prettierignore`, `.dockerignore`
|
||||
- [ ] Add `Makefile` with `test`, `lint`, `fmt`, `fmt-check`, `check`,
|
||||
`docker`, `hooks`, plus `build`, `dev`, `clean`
|
||||
- [ ] Add `Dockerfile` running `make check` against pinned node image
|
||||
- [ ] Add `.gitea/workflows/check.yml` running `docker build .`
|
||||
- [ ] Add `package.json`, `tsconfig.json`, install pinned versions of
|
||||
`typescript`, `libsodium-wrappers-sumo`, `secure-remote-password`,
|
||||
`commander`, `vitest`, `prettier`, `eslint`, `@types/node`
|
||||
- [ ] Smoke test: `make check` and `make docker` both pass
|
||||
|
||||
Phase 2: crypto primitives
|
||||
|
||||
- [ ] Wrap libsodium init as an awaitable singleton
|
||||
- [ ] `deriveKEK(password, kekSalt, memLimit, opsLimit)` (Argon2id)
|
||||
- [ ] `deriveLoginSubkey(kek)` (KDF with subkey id 1, context `loginctx`,
|
||||
16 bytes)
|
||||
- [ ] `decryptBox(ciphertext, nonce, key)` for secretbox
|
||||
- [ ] `decryptSealed(ciphertext, publicKey, secretKey)` for sealed box
|
||||
- [ ] `decryptStream(reader, header, key, writer)` for chunked secretstream
|
||||
(4 MiB plaintext chunks, 17-byte overhead)
|
||||
- [ ] Round-trip tests against vectors generated by libsodium directly
|
||||
|
||||
Phase 3: SRP + auth
|
||||
|
||||
- [ ] SRP-6a client using `secure-remote-password` with the same group as
|
||||
the server
|
||||
- [ ] `loginViaSRP(email, password)` returning either a 2FA challenge or
|
||||
`KeyAttributes + encryptedToken`
|
||||
- [ ] `loginViaEmailOTP(email)` for accounts without SRP enabled
|
||||
- [ ] `submitTOTP(sessionID, code)`
|
||||
- [ ] `unwrapMasterKey(keyAttributes, password)` returning master key,
|
||||
secret key, public key, and decrypted token
|
||||
- [ ] Tests against recorded HTTP fixtures
|
||||
|
||||
Phase 4: HTTP client + endpoints
|
||||
|
||||
- [ ] Tiny fetch wrapper that attaches `X-Auth-Token` and `X-Client-Package`
|
||||
- [ ] Typed wrappers for the endpoints listed above
|
||||
- [ ] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
|
||||
errors
|
||||
- [ ] Error type that surfaces the server's error code and request id
|
||||
|
||||
Phase 5: collections and files
|
||||
|
||||
- [ ] `listCollections()` paginating on `sinceTime` until empty
|
||||
- [ ] Decrypt per-collection key with master key
|
||||
- [ ] Decrypt collection name with collection key
|
||||
- [ ] `listFiles(collectionID)` paginating on `sinceTime` while `hasMore`
|
||||
- [ ] Decrypt per-file key with collection key
|
||||
- [ ] Decrypt file metadata blob with file key, expose typed `FileMetadata`
|
||||
|
||||
Phase 6: download
|
||||
|
||||
- [ ] `downloadFile(fileID, outPath)` streams the encrypted body, decrypts
|
||||
it chunk by chunk, writes plaintext to `outPath`. Resolves the
|
||||
filename from the decrypted metadata title when no `outPath` is
|
||||
supplied.
|
||||
- [ ] `downloadThumbnail(fileID, outPath)` for the thumbnail CDN
|
||||
- [ ] Live integration test against a throwaway Ente account if one is
|
||||
available
|
||||
|
||||
Phase 7: session persistence
|
||||
|
||||
- [ ] Write session blob encrypted with a key from the OS keychain
|
||||
(`keytar`) or a `0600` keyfile fallback
|
||||
- [ ] `Client.fromSavedSession()` and `Client.saveSession()`
|
||||
- [ ] `quack logout` deletes the session and the keychain entry
|
||||
|
||||
Phase 8: CLI
|
||||
|
||||
- [ ] `commander`-based CLI that matches the surface in the Design section
|
||||
- [ ] `--json` output for every command
|
||||
- [ ] Reasonable progress output for long downloads (only when stdout is a
|
||||
TTY)
|
||||
|
||||
Phase 9: docs and 1.0
|
||||
|
||||
- [ ] README usage examples for both library and CLI verified by hand
|
||||
- [ ] All TODO items above checked
|
||||
- [ ] Tag `v1.0.0`
|
||||
|
||||
## Source attribution
|
||||
|
||||
The cryptographic protocol and wire format implemented here are Ente's, taken
|
||||
from the Ente open source clients at
|
||||
<https://github.com/ente-io/ente>. No code is imported or vendored from those
|
||||
projects; any reference code that is copied is rewritten in TypeScript in
|
||||
this repository. Protocol fidelity is verified against the upstream
|
||||
implementations in `web/packages/base/`, `mobile/apps/photos/lib/`, and
|
||||
`cli/`.
|
||||
|
||||
## License
|
||||
|
||||
WTFPL. See [LICENSE](LICENSE).
|
||||
|
||||
## Author
|
||||
|
||||
[@sneak](https://sneak.berlin)
|
||||
Reference in New Issue
Block a user