Removes the only native dependency (sharp). All image processing
now uses jpeg-js (pure JS JPEG codec) + a bilinear RGBA resize for
thumbnail generation, and raw JPEG APP1/XMP byte parsing + exif-reader
for EXIF extraction. Every dependency is now pure JS or Emscripten.
make build-bin compiles the entire CLI into a 59MB self-contained
binary via bun build --compile (bun from nix-shell). No runtime
dependencies needed to run the binary.
sharp was the only native dependency preventing a single-file binary.
Replaced with:
- jpeg-js (pure JS) for JPEG decode/resize/encode in thumbnail gen
- exif-reader (pure JS) for EXIF tag parsing
- Raw JPEG APP1 marker extraction for EXIF segment discovery
- Raw XMP packet extraction from file bytes
make build-bin produces a ~59MB self-contained Mach-O binary via
bun build --compile (bun installed via nix-shell). Zero runtime
dependencies. Tested: login, whoami, collections, files all work
from the compiled binary.
bin/quak.ts: init() called once at program start before commander
parses, so libsodium is ready for all commands including those that
restore sessions from disk.
118 tests pass.
ML metadata (face detections, CLIP embeddings) is not a separate
category from the rest of the metadata. It is always fetched and
included. The only opt-in is --exif (or --all) which requires
downloading every file for EXIF extraction.
ML data (face detections, CLIP embeddings) is now fetched by default
in backup-metadata. Use --no-ml to skip it. EXIF extraction (which
requires downloading every file) remains opt-in via --exif. --all is
an alias for --exif.
--ml: fetches face detections (bounding boxes, landmarks, embeddings)
and CLIP search embeddings from the /files/data/fetch endpoint. These
are encrypted with the file key and gzipped; quak decrypts and
decompresses them into the per-file JSON output.
--exif: downloads each original file, extracts full image metadata
via sharp (format, dimensions, color space, orientation) and parses
raw EXIF tags via exif-reader (lens, ISO, shutter, aperture, GPS
altitude, software, etc.). Also captures IPTC, XMP, and ICC data.
3 new tests. 119 total, all green.
--ml fetches face detections and CLIP embeddings from the /files/data/fetch
endpoint (type 'mldata'). Each blob is encrypted with the file's key and
gzipped; we decrypt with decryptBlob, gunzip, and include the parsed JSON
as 'mlData' in the per-file output. Fetched in batches of 200 file IDs.
--exif downloads each file, runs sharp().metadata() to extract image
properties (format, dimensions, color space, orientation), then parses
the raw EXIF buffer with exif-reader for structured tags (lens, ISO,
shutter, aperture, GPS altitude, etc.). Also captures raw IPTC, XMP,
and ICC profile data. Included as 'imageMetadata' in the per-file output.
Without either flag, behavior is unchanged (fast metadata-only dump).
Adds exif-reader 2.0.3 as a runtime dependency.
3 new tests (ML data decrypted, ML data absent when flag not set, EXIF
extraction). 119 total tests, all green.
Dumps all decrypted account metadata to a directory tree of plain JSON
files. No file content downloads. Includes collection-level magic
metadata decryption (visibility, sort order, cover photo) which was
previously missing. 6 new tests, 116 total.
New command: quak backup-metadata <dir>
Dumps every piece of decrypted account metadata into a directory tree
of plain JSON files without downloading any file content. Layout:
<dir>/
account.json { email, userID }
collections/
<id>-<name>/
_collection.json { id, name, type, pubMagicMetadata?, ... }
<fileID>.json { id, metadata, magicMetadata?, pubMagicMetadata? }
Also adds collection-level magic metadata decryption (magicMetadata,
pubMagicMetadata, sharedMagicMetadata) to decryptCollection, which was
previously only done for files. The server sends these for visibility
settings, sort order, cover photo selection, etc.
6 new tests covering: account.json, per-collection dirs with
_collection.json, collection pubMagicMetadata decryption, per-file
JSON with all three metadata layers, graceful handling of files with
no magic metadata, and incremental re-run safety. 116 total.
Fixes accumulated drift from the original spec-first README:
- Intro: added paragraph describing backup, metadata decryption, and
thumbnail repair capabilities
- Rationale: removed the 'deliberately scoped to read operations'
claim (no longer true since thumbnail upload exists); called out
the Go CLI's crash-on-failure bug as explicit motivation
- Getting Started: fixed library example to use actual Client.login()
API, removed nonexistent fromSavedSession/getFile methods, added
backup CLI example
- Layout: fixed to match actual directory structure (download/,
backup.ts, thumbnails.ts; removed nonexistent session/)
- Session handling: replaced the fictional encrypted-keychain
SessionStore description with the actual implementation (plain JSON
via env-paths, consumer-managed persistence via toJSON/fromJSON)
- CLI surface: added backup, helper list-missing-thumbnails, and
helper fix-missing-thumbnails commands
- Backup layout: new section documenting the originals/ + collections/
symlink structure
- API reference: replaced the stale type declarations with pointers
to the actual source files and a note that test/client/usage.test.ts
is the authoritative API tutorial
- TODO: collapsed completed phases, kept only open items
- For LLMs: new section summarizing repo policies, TDD workflow,
required checks, formatting rules, and pointers to REPO_POLICIES.md
and LLM_PROSE_TELLS.md
quak helper list-missing-thumbnails: scans all files, fetches each
thumbnail, reports missing/empty ones with deduplication.
quak helper fix-missing-thumbnails: downloads originals, generates
720px JPEG via sharp, encrypts with secretstream push (encryptBlob),
uploads via presigned URL, registers via PUT /files/thumbnail.
New crypto: encryptBlob (secretstream push, single chunk TAG_FINAL).
New ApiClient: putJSON, putFile (no auth headers for S3), getUploadURL,
updateThumbnail. New Client: getApiClient() accessor.
20 new tests covering: encryptBlob round-trips and edge cases (8),
upload API methods including putFile auth-header leakage check (4),
listMissingThumbnails detection and dedup (2), fixMissingThumbnails
full pipeline with JPEG magic byte verification on decrypted upload (3),
getApiClient logged-in/logged-out behavior (2), error resilience (1).
110 total tests, all green.
test/crypto/encrypt-blob.test.ts (8 tests):
Round-trip with decryptBlob, zero-length payload, ciphertext
overhead check, header size check, different keys produce different
output, same key produces different output each call (random nonce),
wrong-key rejection, tamper detection.
test/api/upload.test.ts (4 tests):
putJSON sends PUT with auth headers and JSON body. putFile sends
PUT to the exact presigned URL with Content-Type octet-stream and
does NOT send X-Auth-Token or X-Client-Package (S3 would reject
them). getUploadURL POSTs with contentLength and contentMD5.
updateThumbnail PUTs to /files/thumbnail with correct body shape.
test/thumbnails/thumbnails.test.ts (8 tests):
listMissingThumbnails identifies empty (0 byte) and 404 thumbnails
while ignoring working ones; deduplicates across collections.
fixMissingThumbnails verifies the full pipeline: download original,
generate JPEG via sharp, encrypt with encryptBlob, upload via
presigned URL, register via PUT /files/thumbnail. The test
decrypts the uploaded ciphertext and verifies it starts with JPEG
magic bytes (FF D8 FF). Also tests: nonexistent file ID reports
failure without crashing; mixed success/failure across multiple
files; Client.getApiClient() works when logged in, throws after
logout.
list-missing-thumbnails: iterates all files across all collections,
fetches each thumbnail from the CDN, reports any that are missing or
empty. Deduplicates by file ID across collections.
fix-missing-thumbnails: for each missing thumbnail, downloads the
original file, generates a 720px JPEG thumbnail via sharp, encrypts
it with secretstream push (encryptBlob), uploads to a presigned URL,
and registers the new thumbnail via PUT /files/thumbnail.
New crypto: encryptBlob (secretstream push, single chunk TAG_FINAL).
New ApiClient methods: getUploadURL, putFile, putJSON, updateThumbnail.
New Client method: getApiClient() for modules that need raw API access.
Deps: sharp 0.34.5 (image processing), @types/sharp 0.32.0.
Complete CLI surface:
quak login interactive or QUAK_EMAIL/QUAK_PASSWORD
quak whoami print logged-in account
quak logout delete session
quak collections list all albums (--json)
quak files list files in a collection (--json)
quak get <id> download+decrypt a file (--out, --collection)
quak get-thumb <id> download+decrypt a thumbnail
quak backup <dir> full incremental backup
get/get-thumb search all collections for the file ID when --collection
is not specified. All listing commands support --json.
Live-tested: collections list, file list, single file download (472 KB
JPEG from the dev account, verified as valid JPEG with EXIF intact).
All three metadata layers (basic, magicMetadata, pubMagicMetadata) are
now decrypted from secretstream blobs and exposed on EnteFile. Backup
writes originals/<fileID>.json with the full decrypted metadata
including camera make/model, dimensions, datetime, and any face/keyword
data the Ente clients have added.
Extends RawEnteFile and EnteFile with optional magicMetadata and
pubMagicMetadata fields. Both are secretstream blobs under the file
key, decrypted to arbitrary JSON (Record<string, unknown>).
pubMagicMetadata carries ML-derived data from the Ente clients:
camera make/model, image dimensions, datetime with timezone offset,
and (when present) captions, editedName, face labels, keywords.
magicMetadata carries private mutable fields like visibility.
Backup now writes per-file JSON at originals/<fileID>.json containing
all three metadata layers (basic + magic + pubMagic).
Live-tested: all 11 files in the dev account have pubMagicMetadata
with SONY DSC-RX1RM3 camera info and 3000x2000 dimensions.
bin/quak.ts: commander-based CLI with login (interactive + QUAK_EMAIL/
QUAK_PASSWORD env vars), whoami, logout, backup commands. Session
stored at env-paths('quak').data/session.json (~/Library/Application
Support/quak/ on macOS, XDG on Linux).
src/backup.ts: runBackup downloads all files into originals/<id>.<ext>,
symlinks into collections/<name>/<title>, writes per-collection JSON
metadata at collections/<name>.json. Deduplicates across collections
(each file downloaded once). Skips existing originals on incremental
runs. Never crashes on single-file failure.
4 backup tests + live-tested against real Ente account.
4 tests for runBackup: full download into collection-named dirs,
incremental skip of existing files, resilient continuation after
single-file HTTP 500, and metadata.json output.
Adds commander 14.0.3 and env-paths 4.0.0 as runtime deps.
German for 'quack', matching the Ente (German for 'duck') naming. All
references updated: package name, CLI binary, X-Client-Package header,
test descriptions, temp dir prefixes, README, Makefile docker tag.
Client.login() performs full SRP + key unwrap and returns a ready
object. toJSON/fromJSON for consumer-managed persistence.
listCollections, listFiles (with pagination), downloadFile,
downloadThumbnail, whoami, logout. 8 new tests in a literate
tutorial-as-test format. 86 total tests, all green.
Exports Client, all lower-level modules, and all types from
src/index.ts. Replaces Phase 7 (on-disk session persistence) with
the Client class phase: session lives in the object, consumer
handles persistence via toJSON/fromJSON.
Client.login(email, password) performs the full SRP handshake, key
unwrap, and returns a ready Client. Session lives in the object.
Client.fromJSON(snapshot) restores from a serialized snapshot.
client.toJSON() produces a plain object with base64-encoded keys
that the consumer can persist however they like.
Methods: whoami, listCollections, listFiles (with pagination),
downloadFile, downloadThumbnail, logout.
All 86 tests pass including the 8-part literate usage tutorial.
test/client/usage.test.ts is a tutorial-as-test-suite that walks
through the entire quack API in order: login, whoami, listCollections,
listFiles, downloadFile, downloadThumbnail, toJSON/fromJSON, logout.
Each it() block is a self-contained example with prose commentary
explaining what the code does and why, with code samples showing the
API as a consumer would use it. The mock server performs real SRP and
crypto so the test data is structurally identical to production.
8 tests, all failing against the stub.
downloadFile/downloadThumbnail: stream from CDN, buffer to 4 MiB chunk
boundary, secretstream pull decrypt, write to disk. 4 unit tests + live
integration test downloading a real JPEG from the dev Ente account.
downloadFile streams the encrypted body from the CDN, buffers it to
the 4 MiB + 17 encrypted chunk boundary, decrypts each chunk via
secretstream pull, and writes the concatenated plaintext to disk.
downloadThumbnail does the same for the thumbnail CDN.
4 unit tests (single-chunk, large single-chunk, filename fallback,
thumbnail) + live integration test that downloads a real 472 KB JPEG
from the dev account and verifies it lands on disk.
Uses mkdtempSync for temp directories (not manual timestamp paths).
4 tests: single-chunk download, metadata-title fallback filename,
multi-chunk stream (50-byte chunks to exercise the buffering/split
logic without allocating 4 MiB), and thumbnail download.
Tests encrypt payloads with sodium's secretstream push, serve them
from a mock fetch, and verify the decrypted file on disk.
decryptCollection: secretbox key+name from raw server JSON, type mapping,
isShared. decryptFile: secretbox key + secretstream-blob metadata, fileType
mapping, header passthrough. decryptBlob: convenience for single-chunk
secretstream (file metadata uses this form, not secretbox).
10 unit tests + live integration test against real Ente API. All 74 pass.
File metadata is encrypted as a single-chunk secretstream blob (the
'decryptionHeader' is the secretstream init header, not a secretbox
nonce). Collection keys and names correctly use secretbox.
Adds decryptBlob(ciphertext, header, key) to the crypto module as a
convenience wrapper for single-chunk secretstream decryption (init +
pull + verify TAG_FINAL).
Live-tested: collection names and file metadata (titles, types, dates)
decrypt correctly from the real Ente API.
10 tests covering decryptCollection (key + name decryption from raw
server JSON, type mapping, isShared, missing name, wrong key) and
decryptFile (key + metadata decryption, fileType number-to-string
mapping, file/thumbnail header passthrough, wrong key).
Adds src/model/types.ts with both raw (server) and decrypted (library)
type definitions. src/model/decrypt.ts has throwing stubs.
The Ente server validates the auth token as URL-safe base64 with
padding (matching Go's base64.URLEncoding). Our toBase64URL strips
padding, producing a 43-char token where the server expects 44. This
caused HTTP 401 'invalid token' on every authenticated call.
Adds toBase64URLPadded to the crypto module and uses it in unwrapAuth
for the token specifically. toBase64URL (no-padding) is kept for
general use (JWT-style contexts).
Adds test/integration/live-login.ts which logs into the dev account
(entedev2026jp@acidhou.se), unwraps keys, and fetches collections
from the real Ente API. Verified: 4 collections returned successfully.
Complete login flow: beginLogin (SRP-6a with 4096-bit group via
fast-srp-hap, with email-OTP and TOTP 2FA fallback paths),
submitTOTP, requestEmailOTP, submitEmailOTP. Built test-first with
a mock server performing real SRP math; 7 new tests, 64 total.
beginLogin(api, email, password):
1. Fetches SRP attributes for the email
2. If isEmailMFAEnabled, returns { kind: 'emailOTP' } immediately
3. Otherwise: derives KEK + login subkey, creates SRP client with
4096-bit group, runs the two-round handshake (create-session,
verify-session), verifies server M2
4. Returns { kind: 'complete' } with AuthorizationResponse, or
{ kind: 'totp' } / { kind: 'passkey' } if 2FA is required
submitTOTP(api, sessionID, code): POST /users/two-factor/verify
requestEmailOTP(api, email): POST /users/ott
submitEmailOTP(api, email, code): POST /users/verify-email
All 64 tests pass including real SRP-6a handshakes against a mock
server built with fast-srp-hap's SrpServer.
Adds fast-srp-hap (the same SRP library Ente's web client uses, pinned
to 2.0.4) as a runtime dependency.
Tests build a full mock Ente server using fast-srp-hap's SrpServer to
exercise real SRP-6a math end-to-end. The mock handles:
GET /users/srp/attributes
POST /users/srp/create-session
POST /users/srp/verify-session
POST /users/two-factor/verify
POST /users/ott
POST /users/verify-email
7 tests covering:
* SRP login completing successfully
* SRP login requiring TOTP (returns { kind: 'totp' })
* Wrong password (SRP M1 fails server-side checkM1)
* Email MFA fallback (returns { kind: 'emailOTP' })
* submitTOTP
* requestEmailOTP + submitEmailOTP
Production origins: api.ente.io, files.ente.io, thumbnails.ente.io.
Custom apiOrigin routes file/thumbnail downloads through the same host
at /files/download/<id> and /files/preview/<id>.
Every request carries X-Client-Package (berlin.sneak.quack).
Authenticated requests carry X-Auth-Token, toggled via setAuthToken /
clearAuthToken or the constructor authToken option.
getJSON appends query params (skipping undefined values), parses JSON.
postJSON sends JSON with Content-Type header, parses JSON response.
getFileStream / getThumbnailStream return the response body stream.
Non-2xx responses throw ApiError with status, code (from JSON body),
requestID (from x-request-id header), and raw body. Retry logic is
deferred to a follow-on branch.
All 57 tests pass.
19 tests covering ApiClient's full public surface: default and custom
origins, X-Client-Package and X-Auth-Token headers, getJSON with query
params, postJSON with JSON body, ApiError on 4xx/5xx, streaming file
and thumbnail downloads, and self-hosted origin routing.
Tests inject a recording fetch via the constructor, so nothing hits the
network. The test file is documented to serve as canonical usage
reference per the development workflow.
unwrapAuth: password to master/secret/public key and auth token. The
password-only side of Ente login. Built test-first; 6 tests describe
the protocol and verify byte-for-byte recovery of the inputs from a
synthetic AuthorizationResponse.
The implementation is exactly the decryption chain documented in the
test file: deriveKEK -> decryptBox(masterKey) -> decryptBox(secretKey)
-> decryptSealed(token) -> toBase64URL. Errors from the underlying
crypto primitives propagate; the only added validation is the up-front
check that the response actually contains both keyAttributes and
encryptedToken (caller bug if not).
Also re-exports the auth/unwrap and auth/types public surface from
src/index.ts.
All 38 tests pass; make check and make docker are green.
Tests for the password-only decryption chain that follows a successful
login (SRP or email OTP, with or without 2FA). The unwrap covers:
password -> KEK (Argon2id) -> masterKey (secretbox) ->
secretKey (secretbox) -> tokenBytes (sealed box) -> base64url token
Each test builds a synthetic AuthorizationResponse using libsodium
directly and asserts unwrapAuth recovers the inputs byte for byte. The
test file also functions as the canonical description of the protocol.
Adds src/auth/types.ts with KeyAttributes, SRPAttributes,
AuthorizationResponse, and LoginChallenge declarations matching the
README's API reference. src/auth/unwrap.ts is the throwing stub; the
real implementation lands next.
Implements all the libsodium primitives quack needs to unwrap key
material and decrypt files (KDF, secretbox, sealed box, secretstream
pull). Built test-first per the development workflow; 32 tests pass.
