/collections/v2 is a sync API: deleted collections remain in the
response forever with isDeleted: true, and their /collections/v2/diff
endpoint returns HTTP 404. A long-lived account accumulates hundreds
of tombstones, so any caller that iterates listCollections() output
(backup, backup-metadata) dies on the first one.
decryptCollection now takes the full key material {masterKey,
publicKey, secretKey} and dispatches on keyDecryptionNonce: present
means an owned collection (secretbox under the master key), absent
means a shared collection (sealed box to our public key). Client
already held the keypair for unsealing the auth token, so it just
passes it through.
Collections shared with the account are not encrypted with the master
key: the sharer only knows the recipient's public key, so the server
delivers encryptedKey as crypto_box_seal to that key and omits
keyDecryptionNonce entirely. decryptCollection assumed the owned-only
wire format and crashed on fromBase64(undefined) for any account with
an incoming shared album, taking down listCollections and every
command built on it (backup, backup-metadata, ...).
The previous "shared" fixture was unfaithful (secretbox + nonce with a
foreign ownerID, a shape the server never sends), which is why the
suite stayed green. These tests model the real wire format and change
decryptCollection to take the full key material {masterKey, publicKey,
secretKey} so it can unseal shared collection keys.
Use || instead of ?? when picking the TOTP session ID. The server
sends "" for unset 2FA fields (no omitempty), and "" ?? v2
short-circuits to "", which made dual-2FA accounts fall through to
the unsupported passkey branch.
The museum server's EmailAuthorizationResponse declares
passkeySessionID, accountsUrl, twoFactorSessionID, and
twoFactorSessionIDV2 without `omitempty`, so Go always sends them,
as "" when unset. The previous mock omitted the unset fields
entirely, which let the ?? -based dispatch pass in tests while the
real server's "" defeated it and dual-2FA logins fell through to
the unsupported passkey branch.
The server signals dual-2FA accounts with passkeySessionID +
twoFactorSessionIDV2. quak now takes the TOTP path in that case
instead of failing with 'Passkey authentication is not supported'.
beginLogin now checks twoFactorSessionID ?? twoFactorSessionIDV2 before
the passkey branch. Accounts with both passkeys and TOTP can now log in
from the CLI using their authenticator app.
When an account has both passkeys and TOTP enrolled, the Ente server
returns passkeySessionID + twoFactorSessionIDV2 (deliberately not the
V1 twoFactorSessionID field, so old clients keep using passkeys).
quak only checked the V1 field, saw the passkey session, and threw
'Passkey authentication is not supported'. A CLI cannot do WebAuthn,
so it must take the TOTP path via the V2 session ID.
Replaces the hand-rolled readline/raw-mode password prompt with
@inquirer/prompts (input + password). The manual approach broke in
bun-compiled binaries because bun doesn't properly re-open stdin
after closing a readline instance. inquirer handles TTY, raw mode,
and password masking correctly across both node and bun runtimes.
Removes the only native dependency (sharp). All image processing
now uses jpeg-js (pure JS JPEG codec) + a bilinear RGBA resize for
thumbnail generation, and raw JPEG APP1/XMP byte parsing + exif-reader
for EXIF extraction. Every dependency is now pure JS or Emscripten.
make build-bin compiles the entire CLI into a 59MB self-contained
binary via bun build --compile (bun from nix-shell). No runtime
dependencies needed to run the binary.
sharp was the only native dependency preventing a single-file binary.
Replaced with:
- jpeg-js (pure JS) for JPEG decode/resize/encode in thumbnail gen
- exif-reader (pure JS) for EXIF tag parsing
- Raw JPEG APP1 marker extraction for EXIF segment discovery
- Raw XMP packet extraction from file bytes
make build-bin produces a ~59MB self-contained Mach-O binary via
bun build --compile (bun installed via nix-shell). Zero runtime
dependencies. Tested: login, whoami, collections, files all work
from the compiled binary.
bin/quak.ts: init() called once at program start before commander
parses, so libsodium is ready for all commands including those that
restore sessions from disk.
118 tests pass.
ML metadata (face detections, CLIP embeddings) is not a separate
category from the rest of the metadata. It is always fetched and
included. The only opt-in is --exif (or --all) which requires
downloading every file for EXIF extraction.
ML data (face detections, CLIP embeddings) is now fetched by default
in backup-metadata. Use --no-ml to skip it. EXIF extraction (which
requires downloading every file) remains opt-in via --exif. --all is
an alias for --exif.
--ml: fetches face detections (bounding boxes, landmarks, embeddings)
and CLIP search embeddings from the /files/data/fetch endpoint. These
are encrypted with the file key and gzipped; quak decrypts and
decompresses them into the per-file JSON output.
--exif: downloads each original file, extracts full image metadata
via sharp (format, dimensions, color space, orientation) and parses
raw EXIF tags via exif-reader (lens, ISO, shutter, aperture, GPS
altitude, software, etc.). Also captures IPTC, XMP, and ICC data.
3 new tests. 119 total, all green.
--ml fetches face detections and CLIP embeddings from the /files/data/fetch
endpoint (type 'mldata'). Each blob is encrypted with the file's key and
gzipped; we decrypt with decryptBlob, gunzip, and include the parsed JSON
as 'mlData' in the per-file output. Fetched in batches of 200 file IDs.
--exif downloads each file, runs sharp().metadata() to extract image
properties (format, dimensions, color space, orientation), then parses
the raw EXIF buffer with exif-reader for structured tags (lens, ISO,
shutter, aperture, GPS altitude, etc.). Also captures raw IPTC, XMP,
and ICC profile data. Included as 'imageMetadata' in the per-file output.
Without either flag, behavior is unchanged (fast metadata-only dump).
Adds exif-reader 2.0.3 as a runtime dependency.
3 new tests (ML data decrypted, ML data absent when flag not set, EXIF
extraction). 119 total tests, all green.
Dumps all decrypted account metadata to a directory tree of plain JSON
files. No file content downloads. Includes collection-level magic
metadata decryption (visibility, sort order, cover photo) which was
previously missing. 6 new tests, 116 total.
New command: quak backup-metadata <dir>
Dumps every piece of decrypted account metadata into a directory tree
of plain JSON files without downloading any file content. Layout:
<dir>/
account.json { email, userID }
collections/
<id>-<name>/
_collection.json { id, name, type, pubMagicMetadata?, ... }
<fileID>.json { id, metadata, magicMetadata?, pubMagicMetadata? }
Also adds collection-level magic metadata decryption (magicMetadata,
pubMagicMetadata, sharedMagicMetadata) to decryptCollection, which was
previously only done for files. The server sends these for visibility
settings, sort order, cover photo selection, etc.
6 new tests covering: account.json, per-collection dirs with
_collection.json, collection pubMagicMetadata decryption, per-file
JSON with all three metadata layers, graceful handling of files with
no magic metadata, and incremental re-run safety. 116 total.
Fixes accumulated drift from the original spec-first README:
- Intro: added paragraph describing backup, metadata decryption, and
thumbnail repair capabilities
- Rationale: removed the 'deliberately scoped to read operations'
claim (no longer true since thumbnail upload exists); called out
the Go CLI's crash-on-failure bug as explicit motivation
- Getting Started: fixed library example to use actual Client.login()
API, removed nonexistent fromSavedSession/getFile methods, added
backup CLI example
- Layout: fixed to match actual directory structure (download/,
backup.ts, thumbnails.ts; removed nonexistent session/)
- Session handling: replaced the fictional encrypted-keychain
SessionStore description with the actual implementation (plain JSON
via env-paths, consumer-managed persistence via toJSON/fromJSON)
- CLI surface: added backup, helper list-missing-thumbnails, and
helper fix-missing-thumbnails commands
- Backup layout: new section documenting the originals/ + collections/
symlink structure
- API reference: replaced the stale type declarations with pointers
to the actual source files and a note that test/client/usage.test.ts
is the authoritative API tutorial
- TODO: collapsed completed phases, kept only open items
- For LLMs: new section summarizing repo policies, TDD workflow,
required checks, formatting rules, and pointers to REPO_POLICIES.md
and LLM_PROSE_TELLS.md
quak helper list-missing-thumbnails: scans all files, fetches each
thumbnail, reports missing/empty ones with deduplication.
quak helper fix-missing-thumbnails: downloads originals, generates
720px JPEG via sharp, encrypts with secretstream push (encryptBlob),
uploads via presigned URL, registers via PUT /files/thumbnail.
New crypto: encryptBlob (secretstream push, single chunk TAG_FINAL).
New ApiClient: putJSON, putFile (no auth headers for S3), getUploadURL,
updateThumbnail. New Client: getApiClient() accessor.
20 new tests covering: encryptBlob round-trips and edge cases (8),
upload API methods including putFile auth-header leakage check (4),
listMissingThumbnails detection and dedup (2), fixMissingThumbnails
full pipeline with JPEG magic byte verification on decrypted upload (3),
getApiClient logged-in/logged-out behavior (2), error resilience (1).
110 total tests, all green.
test/crypto/encrypt-blob.test.ts (8 tests):
Round-trip with decryptBlob, zero-length payload, ciphertext
overhead check, header size check, different keys produce different
output, same key produces different output each call (random nonce),
wrong-key rejection, tamper detection.
test/api/upload.test.ts (4 tests):
putJSON sends PUT with auth headers and JSON body. putFile sends
PUT to the exact presigned URL with Content-Type octet-stream and
does NOT send X-Auth-Token or X-Client-Package (S3 would reject
them). getUploadURL POSTs with contentLength and contentMD5.
updateThumbnail PUTs to /files/thumbnail with correct body shape.
test/thumbnails/thumbnails.test.ts (8 tests):
listMissingThumbnails identifies empty (0 byte) and 404 thumbnails
while ignoring working ones; deduplicates across collections.
fixMissingThumbnails verifies the full pipeline: download original,
generate JPEG via sharp, encrypt with encryptBlob, upload via
presigned URL, register via PUT /files/thumbnail. The test
decrypts the uploaded ciphertext and verifies it starts with JPEG
magic bytes (FF D8 FF). Also tests: nonexistent file ID reports
failure without crashing; mixed success/failure across multiple
files; Client.getApiClient() works when logged in, throws after
logout.
list-missing-thumbnails: iterates all files across all collections,
fetches each thumbnail from the CDN, reports any that are missing or
empty. Deduplicates by file ID across collections.
fix-missing-thumbnails: for each missing thumbnail, downloads the
original file, generates a 720px JPEG thumbnail via sharp, encrypts
it with secretstream push (encryptBlob), uploads to a presigned URL,
and registers the new thumbnail via PUT /files/thumbnail.
New crypto: encryptBlob (secretstream push, single chunk TAG_FINAL).
New ApiClient methods: getUploadURL, putFile, putJSON, updateThumbnail.
New Client method: getApiClient() for modules that need raw API access.
Deps: sharp 0.34.5 (image processing), @types/sharp 0.32.0.
Complete CLI surface:
quak login interactive or QUAK_EMAIL/QUAK_PASSWORD
quak whoami print logged-in account
quak logout delete session
quak collections list all albums (--json)
quak files list files in a collection (--json)
quak get <id> download+decrypt a file (--out, --collection)
quak get-thumb <id> download+decrypt a thumbnail
quak backup <dir> full incremental backup
get/get-thumb search all collections for the file ID when --collection
is not specified. All listing commands support --json.
Live-tested: collections list, file list, single file download (472 KB
JPEG from the dev account, verified as valid JPEG with EXIF intact).
All three metadata layers (basic, magicMetadata, pubMagicMetadata) are
now decrypted from secretstream blobs and exposed on EnteFile. Backup
writes originals/<fileID>.json with the full decrypted metadata
including camera make/model, dimensions, datetime, and any face/keyword
data the Ente clients have added.
Extends RawEnteFile and EnteFile with optional magicMetadata and
pubMagicMetadata fields. Both are secretstream blobs under the file
key, decrypted to arbitrary JSON (Record<string, unknown>).
pubMagicMetadata carries ML-derived data from the Ente clients:
camera make/model, image dimensions, datetime with timezone offset,
and (when present) captions, editedName, face labels, keywords.
magicMetadata carries private mutable fields like visibility.
Backup now writes per-file JSON at originals/<fileID>.json containing
all three metadata layers (basic + magic + pubMagic).
Live-tested: all 11 files in the dev account have pubMagicMetadata
with SONY DSC-RX1RM3 camera info and 3000x2000 dimensions.
bin/quak.ts: commander-based CLI with login (interactive + QUAK_EMAIL/
QUAK_PASSWORD env vars), whoami, logout, backup commands. Session
stored at env-paths('quak').data/session.json (~/Library/Application
Support/quak/ on macOS, XDG on Linux).
src/backup.ts: runBackup downloads all files into originals/<id>.<ext>,
symlinks into collections/<name>/<title>, writes per-collection JSON
metadata at collections/<name>.json. Deduplicates across collections
(each file downloaded once). Skips existing originals on incremental
runs. Never crashes on single-file failure.
4 backup tests + live-tested against real Ente account.
4 tests for runBackup: full download into collection-named dirs,
incremental skip of existing files, resilient continuation after
single-file HTTP 500, and metadata.json output.
Adds commander 14.0.3 and env-paths 4.0.0 as runtime deps.
German for 'quack', matching the Ente (German for 'duck') naming. All
references updated: package name, CLI binary, X-Client-Package header,
test descriptions, temp dir prefixes, README, Makefile docker tag.
Client.login() performs full SRP + key unwrap and returns a ready
object. toJSON/fromJSON for consumer-managed persistence.
listCollections, listFiles (with pagination), downloadFile,
downloadThumbnail, whoami, logout. 8 new tests in a literate
tutorial-as-test format. 86 total tests, all green.
Exports Client, all lower-level modules, and all types from
src/index.ts. Replaces Phase 7 (on-disk session persistence) with
the Client class phase: session lives in the object, consumer
handles persistence via toJSON/fromJSON.
Client.login(email, password) performs the full SRP handshake, key
unwrap, and returns a ready Client. Session lives in the object.
Client.fromJSON(snapshot) restores from a serialized snapshot.
client.toJSON() produces a plain object with base64-encoded keys
that the consumer can persist however they like.
Methods: whoami, listCollections, listFiles (with pagination),
downloadFile, downloadThumbnail, logout.
All 86 tests pass including the 8-part literate usage tutorial.
test/client/usage.test.ts is a tutorial-as-test-suite that walks
through the entire quack API in order: login, whoami, listCollections,
listFiles, downloadFile, downloadThumbnail, toJSON/fromJSON, logout.
Each it() block is a self-contained example with prose commentary
explaining what the code does and why, with code samples showing the
API as a consumer would use it. The mock server performs real SRP and
crypto so the test data is structurally identical to production.
8 tests, all failing against the stub.
downloadFile/downloadThumbnail: stream from CDN, buffer to 4 MiB chunk
boundary, secretstream pull decrypt, write to disk. 4 unit tests + live
integration test downloading a real JPEG from the dev Ente account.
downloadFile streams the encrypted body from the CDN, buffers it to
the 4 MiB + 17 encrypted chunk boundary, decrypts each chunk via
secretstream pull, and writes the concatenated plaintext to disk.
downloadThumbnail does the same for the thumbnail CDN.
4 unit tests (single-chunk, large single-chunk, filename fallback,
thumbnail) + live integration test that downloads a real 472 KB JPEG
from the dev account and verifies it lands on disk.
Uses mkdtempSync for temp directories (not manual timestamp paths).
