Reflow README to prettier 80-column prose wrap
This commit is contained in:
141
README.md
141
README.md
@@ -43,29 +43,27 @@ await client.downloadFile(file, "./out/");
|
|||||||
## Rationale
|
## Rationale
|
||||||
|
|
||||||
Ente is one of very few photo services with a credible end-to-end encryption
|
Ente is one of very few photo services with a credible end-to-end encryption
|
||||||
story. The shipping clients (mobile Flutter, web React, desktop Electron, and
|
story. The shipping clients (mobile Flutter, web React, desktop Electron, and Go
|
||||||
Go CLI) work, but they are slow, buggy, and difficult to script against. The
|
CLI) work, but they are slow, buggy, and difficult to script against. The
|
||||||
Flutter app fails to sync reliably. The web app is heavy. The desktop app is
|
Flutter app fails to sync reliably. The web app is heavy. The desktop app is the
|
||||||
the web app inside a slow Electron wrapper. The Go CLI is the closest thing
|
web app inside a slow Electron wrapper. The Go CLI is the closest thing to a
|
||||||
to a usable tool, but it is awkward to integrate from anything that is not a
|
usable tool, but it is awkward to integrate from anything that is not a shell.
|
||||||
shell.
|
|
||||||
|
|
||||||
quack is the first step in fixing that. This repo ships a small, correct,
|
quack is the first step in fixing that. This repo ships a small, correct,
|
||||||
well-tested implementation of Ente's cryptographic protocol and its read-only
|
well-tested implementation of Ente's cryptographic protocol and its read-only
|
||||||
API surface, plus a CLI that proves the library is enough to do real work
|
API surface, plus a CLI that proves the library is enough to do real work
|
||||||
without a UI.
|
without a UI.
|
||||||
|
|
||||||
The longer-term goal of this project is a simple desktop client for Ente,
|
The longer-term goal of this project is a simple desktop client for Ente, built
|
||||||
built on this library in Electron (or a comparable runtime), with two
|
on this library in Electron (or a comparable runtime), with two priorities above
|
||||||
priorities above everything else: correctness and stability. Performance and
|
everything else: correctness and stability. Performance and simplicity follow
|
||||||
simplicity follow from those. Features will be added only after the protocol
|
from those. Features will be added only after the protocol layer is correct, the
|
||||||
layer is correct, the local cache is reliable, and the UI is responsive on a
|
local cache is reliable, and the UI is responsive on a five-year-old laptop.
|
||||||
five-year-old laptop.
|
|
||||||
|
|
||||||
This first release is deliberately scoped to read operations: log in, walk
|
This first release is deliberately scoped to read operations: log in, walk the
|
||||||
the account, decrypt and save files. Upload, sharing, deletion, and
|
account, decrypt and save files. Upload, sharing, deletion, and bidirectional
|
||||||
bidirectional sync are out of scope. Adding them later is straightforward;
|
sync are out of scope. Adding them later is straightforward; doing them right
|
||||||
doing them right requires the protocol layer to be correct first.
|
requires the protocol layer to be correct first.
|
||||||
|
|
||||||
## Design
|
## Design
|
||||||
|
|
||||||
@@ -101,14 +99,14 @@ required for `crypto_pwhash` / Argon2id). No hand-rolled crypto.
|
|||||||
The key hierarchy, derived during login, is:
|
The key hierarchy, derived during login, is:
|
||||||
|
|
||||||
1. The user enters their password.
|
1. The user enters their password.
|
||||||
2. Argon2id (`crypto_pwhash`) over the password and a server-issued
|
2. Argon2id (`crypto_pwhash`) over the password and a server-issued `kekSalt`,
|
||||||
`kekSalt`, with server-issued `memLimit` and `opsLimit`, produces a
|
with server-issued `memLimit` and `opsLimit`, produces a 32-byte Key
|
||||||
32-byte Key Encryption Key (KEK).
|
Encryption Key (KEK).
|
||||||
3. SRP login: a 16-byte SRP login subkey is derived from the KEK using
|
3. SRP login: a 16-byte SRP login subkey is derived from the KEK using
|
||||||
`crypto_kdf_derive_from_key` (BLAKE2b) with subkey id 1 and context
|
`crypto_kdf_derive_from_key` (BLAKE2b) with subkey id 1 and context
|
||||||
`loginctx`. That 16-byte value is the SRP password.
|
`loginctx`. That 16-byte value is the SRP password.
|
||||||
4. After SRP completes (or after email-OTP fallback), the server returns a
|
4. After SRP completes (or after email-OTP fallback), the server returns a blob
|
||||||
blob of "key attributes" plus an encrypted auth token.
|
of "key attributes" plus an encrypted auth token.
|
||||||
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
|
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
|
||||||
yields the 32-byte master key.
|
yields the 32-byte master key.
|
||||||
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
|
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
|
||||||
@@ -118,12 +116,12 @@ The key hierarchy, derived during login, is:
|
|||||||
yields the URL-safe base64 auth token used in `X-Auth-Token` for all
|
yields the URL-safe base64 auth token used in `X-Auth-Token` for all
|
||||||
subsequent calls.
|
subsequent calls.
|
||||||
|
|
||||||
Per-collection keys are decrypted with `crypto_secretbox_open_easy` using
|
Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the
|
||||||
the master key (for owned collections). Per-file keys are decrypted with
|
master key (for owned collections). Per-file keys are decrypted with
|
||||||
`crypto_secretbox_open_easy` using the collection key. File metadata is a
|
`crypto_secretbox_open_easy` using the collection key. File metadata is a
|
||||||
secretbox under the file key. File content is a chunked
|
secretbox under the file key. File content is a chunked
|
||||||
`crypto_secretstream_xchacha20poly1305` stream under the file key, with a
|
`crypto_secretstream_xchacha20poly1305` stream under the file key, with a 4 MiB
|
||||||
4 MiB plaintext chunk size and a 17-byte authentication overhead per chunk.
|
plaintext chunk size and a 17-byte authentication overhead per chunk.
|
||||||
|
|
||||||
### HTTP API
|
### HTTP API
|
||||||
|
|
||||||
@@ -134,39 +132,37 @@ Production endpoints:
|
|||||||
- Thumbnail CDN: `https://thumbnails.ente.io/?fileID=<id>`
|
- Thumbnail CDN: `https://thumbnails.ente.io/?fileID=<id>`
|
||||||
|
|
||||||
A custom API endpoint is configurable for self-hosted servers via the
|
A custom API endpoint is configurable for self-hosted servers via the
|
||||||
`ENTE_API_ENDPOINT` environment variable. When set, file downloads route
|
`ENTE_API_ENDPOINT` environment variable. When set, file downloads route through
|
||||||
through `<endpoint>/files/download/<id>` instead of the dedicated CDN host.
|
`<endpoint>/files/download/<id>` instead of the dedicated CDN host.
|
||||||
|
|
||||||
Required request headers on every authenticated call:
|
Required request headers on every authenticated call:
|
||||||
|
|
||||||
- `X-Auth-Token`: the decrypted auth token from login.
|
- `X-Auth-Token`: the decrypted auth token from login.
|
||||||
- `X-Client-Package`: identifies the client. quack uses
|
- `X-Client-Package`: identifies the client. quack uses `berlin.sneak.quack`.
|
||||||
`berlin.sneak.quack`.
|
|
||||||
|
|
||||||
Endpoints used:
|
Endpoints used:
|
||||||
|
|
||||||
- `GET /users/srp/attributes?email=<email>`: fetch SRP and KDF parameters.
|
- `GET /users/srp/attributes?email=<email>`: fetch SRP and KDF parameters.
|
||||||
- `POST /users/srp/create-session`: begin SRP handshake.
|
- `POST /users/srp/create-session`: begin SRP handshake.
|
||||||
- `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or
|
- `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or the
|
||||||
the encrypted token plus key attributes.
|
encrypted token plus key attributes.
|
||||||
- `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path.
|
- `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path.
|
||||||
- `POST /users/two-factor/verify`: TOTP second factor.
|
- `POST /users/two-factor/verify`: TOTP second factor.
|
||||||
- `GET /collections/v2?sinceTime=<usec>`: list collections changed since
|
- `GET /collections/v2?sinceTime=<usec>`: list collections changed since
|
||||||
microsecond timestamp; pass 0 for a full enumeration.
|
microsecond timestamp; pass 0 for a full enumeration.
|
||||||
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list
|
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
|
||||||
files in a collection; paginate while `hasMore` is true.
|
collection; paginate while `hasMore` is true.
|
||||||
- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes.
|
- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes.
|
||||||
|
|
||||||
### Session persistence
|
### Session persistence
|
||||||
|
|
||||||
After login, quack writes an encrypted session blob to
|
After login, quack writes an encrypted session blob to
|
||||||
`$XDG_CONFIG_HOME/quack/session.json` (default
|
`$XDG_CONFIG_HOME/quack/session.json` (default `~/.config/quack/session.json`)
|
||||||
`~/.config/quack/session.json`) containing the auth token, the user's master
|
containing the auth token, the user's master key, the user's secret key, and the
|
||||||
key, the user's secret key, and the user's email. The session file is itself
|
user's email. The session file is itself encrypted with a key derived from a
|
||||||
encrypted with a key derived from a per-machine random value stored in the OS
|
per-machine random value stored in the OS keychain when available, falling back
|
||||||
keychain when available, falling back to a key file at mode `0600` in the
|
to a key file at mode `0600` in the same config directory. The master key and
|
||||||
same config directory. The master key and secret key are never written to
|
secret key are never written to disk in cleartext.
|
||||||
disk in cleartext.
|
|
||||||
|
|
||||||
### CLI surface
|
### CLI surface
|
||||||
|
|
||||||
@@ -174,11 +170,10 @@ disk in cleartext.
|
|||||||
- `quack logout`: deletes the session.
|
- `quack logout`: deletes the session.
|
||||||
- `quack whoami`: prints the logged-in email.
|
- `quack whoami`: prints the logged-in email.
|
||||||
- `quack collections`: list collections (id, name, type, file count).
|
- `quack collections`: list collections (id, name, type, file count).
|
||||||
- `quack files --collection <id>`: list files in a collection (id, name,
|
- `quack files --collection <id>`: list files in a collection (id, name, type,
|
||||||
type, creation time, size).
|
creation time, size).
|
||||||
- `quack get <fileID> --out <dir>`: download and decrypt a file.
|
- `quack get <fileID> --out <dir>`: download and decrypt a file.
|
||||||
- `quack get-thumb <fileID> --out <dir>`: download and decrypt a
|
- `quack get-thumb <fileID> --out <dir>`: download and decrypt a thumbnail.
|
||||||
thumbnail.
|
|
||||||
|
|
||||||
All commands accept `--json` for machine-readable output.
|
All commands accept `--json` for machine-readable output.
|
||||||
|
|
||||||
@@ -552,10 +547,10 @@ Phase 1: scaffolding (this commit and the next)
|
|||||||
|
|
||||||
- [x] `git init`, write README
|
- [x] `git init`, write README
|
||||||
- [ ] Create `initial-scaffolding` feature branch
|
- [ ] Create `initial-scaffolding` feature branch
|
||||||
- [ ] Add `LICENSE` (WTFPL), `REPO_POLICIES.md`, `.gitignore`,
|
- [ ] Add `LICENSE` (WTFPL), `REPO_POLICIES.md`, `.gitignore`, `.editorconfig`,
|
||||||
`.editorconfig`, `.prettierrc`, `.prettierignore`, `.dockerignore`
|
`.prettierrc`, `.prettierignore`, `.dockerignore`
|
||||||
- [ ] Add `Makefile` with `test`, `lint`, `fmt`, `fmt-check`, `check`,
|
- [ ] Add `Makefile` with `test`, `lint`, `fmt`, `fmt-check`, `check`, `docker`,
|
||||||
`docker`, `hooks`, plus `build`, `dev`, `clean`
|
`hooks`, plus `build`, `dev`, `clean`
|
||||||
- [ ] Add `Dockerfile` running `make check` against pinned node image
|
- [ ] Add `Dockerfile` running `make check` against pinned node image
|
||||||
- [ ] Add `.gitea/workflows/check.yml` running `docker build .`
|
- [ ] Add `.gitea/workflows/check.yml` running `docker build .`
|
||||||
- [ ] Add `package.json`, `tsconfig.json`, install pinned versions of
|
- [ ] Add `package.json`, `tsconfig.json`, install pinned versions of
|
||||||
@@ -567,23 +562,23 @@ Phase 2: crypto primitives
|
|||||||
|
|
||||||
- [ ] Wrap libsodium init as an awaitable singleton
|
- [ ] Wrap libsodium init as an awaitable singleton
|
||||||
- [ ] `deriveKEK(password, kekSalt, memLimit, opsLimit)` (Argon2id)
|
- [ ] `deriveKEK(password, kekSalt, memLimit, opsLimit)` (Argon2id)
|
||||||
- [ ] `deriveLoginSubkey(kek)` (KDF with subkey id 1, context `loginctx`,
|
- [ ] `deriveLoginSubkey(kek)` (KDF with subkey id 1, context `loginctx`, 16
|
||||||
16 bytes)
|
bytes)
|
||||||
- [ ] `decryptBox(ciphertext, nonce, key)` for secretbox
|
- [ ] `decryptBox(ciphertext, nonce, key)` for secretbox
|
||||||
- [ ] `decryptSealed(ciphertext, publicKey, secretKey)` for sealed box
|
- [ ] `decryptSealed(ciphertext, publicKey, secretKey)` for sealed box
|
||||||
- [ ] `initStreamPull` and `pullStreamChunk` for chunked secretstream
|
- [ ] `initStreamPull` and `pullStreamChunk` for chunked secretstream (4 MiB
|
||||||
(4 MiB plaintext chunks, 17-byte overhead)
|
plaintext chunks, 17-byte overhead)
|
||||||
- [ ] Round-trip tests against vectors generated by libsodium directly
|
- [ ] Round-trip tests against vectors generated by libsodium directly
|
||||||
|
|
||||||
Phase 3: SRP + auth
|
Phase 3: SRP + auth
|
||||||
|
|
||||||
- [ ] SRP-6a client using `secure-remote-password` with the same group as
|
- [ ] SRP-6a client using `secure-remote-password` with the same group as the
|
||||||
the server
|
server
|
||||||
- [ ] `beginLogin(email, password)` returning a `LoginChallenge`
|
- [ ] `beginLogin(email, password)` returning a `LoginChallenge`
|
||||||
- [ ] `requestEmailOTP` and `submitEmailOTP` for accounts without SRP
|
- [ ] `requestEmailOTP` and `submitEmailOTP` for accounts without SRP
|
||||||
- [ ] `submitTOTP(sessionID, code)`
|
- [ ] `submitTOTP(sessionID, code)`
|
||||||
- [ ] `unwrapAuth(response, password)` returning master key, secret key,
|
- [ ] `unwrapAuth(response, password)` returning master key, secret key, public
|
||||||
public key, and decrypted token
|
key, and decrypted token
|
||||||
- [ ] Tests against recorded HTTP fixtures
|
- [ ] Tests against recorded HTTP fixtures
|
||||||
|
|
||||||
Phase 4: HTTP client + endpoints
|
Phase 4: HTTP client + endpoints
|
||||||
@@ -605,18 +600,16 @@ Phase 5: collections and files
|
|||||||
|
|
||||||
Phase 6: download
|
Phase 6: download
|
||||||
|
|
||||||
- [ ] `downloadFile(fileID, outPath)` streams the encrypted body, decrypts
|
- [ ] `downloadFile(fileID, outPath)` streams the encrypted body, decrypts it
|
||||||
it chunk by chunk, writes plaintext to `outPath`. Resolves the
|
chunk by chunk, writes plaintext to `outPath`. Resolves the filename from
|
||||||
filename from the decrypted metadata title when no `outPath` is
|
the decrypted metadata title when no `outPath` is supplied.
|
||||||
supplied.
|
|
||||||
- [ ] `downloadThumbnail(fileID, outPath)` for the thumbnail CDN
|
- [ ] `downloadThumbnail(fileID, outPath)` for the thumbnail CDN
|
||||||
- [ ] Live integration test against a throwaway Ente account if one is
|
- [ ] Live integration test against a throwaway Ente account if one is available
|
||||||
available
|
|
||||||
|
|
||||||
Phase 7: session persistence
|
Phase 7: session persistence
|
||||||
|
|
||||||
- [ ] `SessionStore` writing an encrypted session blob with a key from the
|
- [ ] `SessionStore` writing an encrypted session blob with a key from the OS
|
||||||
OS keychain (`keytar`) or a `0600` keyfile fallback
|
keychain (`keytar`) or a `0600` keyfile fallback
|
||||||
- [ ] `Client.fromSavedSession()` and `Client.saveSession()`
|
- [ ] `Client.fromSavedSession()` and `Client.saveSession()`
|
||||||
- [ ] `quack logout` deletes the session and the keychain entry
|
- [ ] `quack logout` deletes the session and the keychain entry
|
||||||
|
|
||||||
@@ -624,8 +617,7 @@ Phase 8: CLI
|
|||||||
|
|
||||||
- [ ] `commander`-based CLI that matches the surface in the Design section
|
- [ ] `commander`-based CLI that matches the surface in the Design section
|
||||||
- [ ] `--json` output for every command
|
- [ ] `--json` output for every command
|
||||||
- [ ] Reasonable progress output for long downloads (only when stdout is a
|
- [ ] Reasonable progress output for long downloads (only when stdout is a TTY)
|
||||||
TTY)
|
|
||||||
|
|
||||||
Phase 9: docs and 1.0
|
Phase 9: docs and 1.0
|
||||||
|
|
||||||
@@ -639,18 +631,17 @@ Phase 10 and beyond: desktop client (separate repo)
|
|||||||
- [ ] Local cache (SQLite) keyed on `(collectionID, fileID, updationTime)`
|
- [ ] Local cache (SQLite) keyed on `(collectionID, fileID, updationTime)`
|
||||||
- [ ] Background sync worker that streams new files into the cache
|
- [ ] Background sync worker that streams new files into the cache
|
||||||
- [ ] Read-only gallery UI: thumbnails, full-image view, basic search
|
- [ ] Read-only gallery UI: thumbnails, full-image view, basic search
|
||||||
- [ ] Add upload, delete, and share back into the library before the
|
- [ ] Add upload, delete, and share back into the library before the desktop UI
|
||||||
desktop UI exposes them
|
exposes them
|
||||||
|
|
||||||
## Source attribution
|
## Source attribution
|
||||||
|
|
||||||
The cryptographic protocol and wire format implemented here are Ente's,
|
The cryptographic protocol and wire format implemented here are Ente's, taken
|
||||||
taken from the Ente open source clients at
|
from the Ente open source clients at <https://github.com/ente-io/ente>. No code
|
||||||
<https://github.com/ente-io/ente>. No code is imported or vendored from
|
is imported or vendored from those projects; any reference code that is copied
|
||||||
those projects; any reference code that is copied is rewritten in TypeScript
|
is rewritten in TypeScript in this repository. Protocol fidelity is verified
|
||||||
in this repository. Protocol fidelity is verified against the upstream
|
against the upstream implementations in `web/packages/base/`,
|
||||||
implementations in `web/packages/base/`, `mobile/apps/photos/lib/`, and
|
`mobile/apps/photos/lib/`, and `cli/`.
|
||||||
`cli/`.
|
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user