Owner amendment to design #36: alongside the default reads, which answer
from RAM and refresh in the background, `Library.fresh()` forces a refresh,
awaits it, and only then hands back the albums/photos/timeline namespaces, so
a caller (the CLI, in #52) gets server-current data.
The background loop and the default reads are unchanged. Both paths now share
one in-flight-cycle slot: the loop skips when a cycle runs and swallows its
errors as before; a fresh read coalesces onto that cycle or starts one, and
propagates a failure so a refresh that could not complete rejects the caller
instead of silently serving stale data. Concurrent fresh reads therefore
collapse to a single server round-trip.
Model: opus-4-8
Precaches aggressively inside open(): every thumbnail (newest first, through the shared thumbnail pool, never evicted; visible/ahead requests preempt the background fill) and the pinned originals — the favorites album plus every file within precacheOriginalsDays (default 7) of the newest takenAt — through the content pool. The pinned set integrates with the #47 eviction hook; when the window moves or a favorite is removed, files become ordinary evictable originals. open() options precacheThumbnails/precacheOriginals/precacheOriginalsDays; progress via onProgress/status().
Model: opus-4-8
Adds lib.mldata search over the CLIP index (#49): forFile returns a file's stored payload; similar ranks nearest files by cosine on the CLIP embedding; searchByEmbedding ranks the index against a caller-supplied query vector. All RAM-only, reusing the packed Float32Array index and id list. No text encoder is bundled — the caller provides the query embedding. (Redo of the reverted first attempt, now tsc-clean.)
Model: opus-4-8
Rewrites backup on the library API. backup() refreshes, fetches pending originals (and optionally thumbnails) through the pools reusing the content cache, then materialises the unchanged collections/ symlink views + per-collection JSON + sidecars from the model. A symlink failure no longer aborts the run (closes#8). failures.json reconciles against each run's attempted set — since-deleted/out-of-scope/resolved entries clear, still-failing retained, one attempt per file per run — and the exit-code contract is preserved.
Model: opus-4-8
Bounds the originals cache and evicts least-recently-used. cacheOriginalsMaxBytes default 100 GiB; the effective limit adapts down via fs.statfs to keep freeBelowBytes (default 50 GiB) free. Over-limit writes evict unpinned originals oldest-mtime-first (mtime touched on read); pinned files (favorites + latest week) are skipped and an over-budget on-demand fetch proceeds over-limit. Every in-flight write is excluded from eviction, so concurrent fetches never delete each other's just-stored file. Only cacheDirectory/originals is evicted; downloadDirectory and thumbnails never.
Model: opus-4-8
Adds the on-disk content and thumbnail cache keyed by fileID: originals/ and thumbnails/ under cacheDirectory, present-means-complete (streaming atomic rename), orphan temp reaping on open. Photo.original/thumbnail return a cached path with no network when present, else fetch through the shared request pool; thumbnails.ensure drives the thumbnail pool with priority, dedup and AbortSignal. One shared RequestPools set serves both the ML fetch and the content cache. Content-hash integrity is deferred (#68); authenticated streaming decrypt guarantees integrity now.
Model: opus-4-8
Reverts the #50 merge (224bd101): mlsearch.ts failed tsc in the CI build (make check does not run make build, so it slipped past review). next restored to green; #50 to be redone with make build in its gate.
Model: opus-4-8
Adds lib.mldata search over the CLIP index (#49): forFile returns a file's stored payload; similar ranks nearest files by cosine on the CLIP embedding; searchByEmbedding ranks the index against a caller-supplied query vector. All RAM-only, reusing the packed Float32Array index and id list. No text encoder is bundled — the caller provides the query embedding.
Model: opus-4-8
Adds the machine-learning (magic) data layer: fetches per-file ML payloads (face detections + CLIP embeddings) via the existing metadata-backup fetch through the metadata pool after each refresh, decrypts and gunzips them, and stores one mldata/<fileID>.json per file by rename (present-means-complete). A derived index (mldata/clip.f32 + clip.json) loads in one read and is rebuilt whenever it disagrees with the payloads on disk in either direction, so an interrupted backfill self-heals. Never in metadata.json; incremental on later refreshes; progress via onProgress/status.
Model: opus-4-8
Adds the in-process read surface, served from RAM with args-object signatures: albums (list/byName/byID), photos (byID/records -> plain PhotoRecord[]), thin Album/Photo wrappers, and timeline.groups (day/week/month) with a PhotoFilter (albumID/text/fileTypes/hasLocation/includeArchived; hidden excluded). Week keys use ISO YYYY-Www; each file appears once per group, newest first. Built on the #43 snapshot projection; no network.
Model: opus-4-8
Adds three independent bounded request pools — metadata (10 in flight), content (5), thumbnails (25), each overridable — with on-demand-before-background priority and in-flight dedup (a shared key runs once); an idle pool never lends slots, and retries run inside a slot. Self-contained module; the content cache wires it into the library later.
Model: opus-4-8
Adds a synchronous, key-free snapshot() returning LibrarySnapshot (one PhotoRecord per fileID, deduped, newest first) with edited-name/time precedence (pubMagicMetadata over basic metadata) in milliseconds, plus AlbumRecord (favorites identified by type). subscribe({onChange}) delivers LibraryChange (changed/removed albums and files, refreshedAt) only when a refresh changes something; unsubscribe stops delivery. Built on the existing refresh loop; served from RAM, safe to send over IPC.
Model: opus-4-8
streamDecrypt no longer recopies the whole accumulation buffer on every network read. Reads are queued with a running byte count and a contiguous buffer is materialised only at each ENC_CHUNK_SIZE boundary, with a straddling read split via a subarray view, so each byte is copied once instead of O(n^2). TAG_FINAL truncation detection, final-chunk handling, retry, and the per-chunk progress hook are unchanged. A new test feeds a multi-chunk body through a ReadableStream that yields many small pieces, exercising the fragmented-read path.
Model: opus-4-8
Library.open loads metadata.json and serves reads from RAM: an empty cache awaits the first refresh, an existing cache returns at once and refreshes in the background so an unreachable server never stalls open(). A background timer refreshes every refreshIntervalSeconds (default 3), diffing only changed albums via the resumable cursor+tombstone enumerators and rewriting metadata.json only when something changed. A refresh failure is invisible to reads and surfaced via status()/onProgress; a failed save keeps status().lastError set and retries until one lands, so a stale disk is never masked. No sync()/refresh()/serverReachable surface; status() and close() included.
Model: opus-4-8
Originals no longer buffer the whole decrypted file in RAM: streamDecrypt hands each secretstream chunk to a sink and the download path writes it to the staged temp file, so peak memory is one chunk regardless of file size. The atomic write moved inside the retry loop; only a TAG_FINAL-authenticated attempt renames; truncation leaves no destination file. Does not close#21 (the streamDecrypt accumulation-buffer recopy stays open).
Model: opus-4-8
Adds the metadata.json store: loads whole into RAM with id-lookup Maps, rewrites whole through the exported fsync atomic writer (temp, fsync, rename, dir fsync); a missing, unparseable, or wrong-schema file loads as empty (it is a cache); directory 0700, file 0600. No lock file, no public sync().
Model: opus-4-8
Adds collectionsSince/filesSince taking a starting cursor, returning the resumable max-updationTime cursor and a separate list of tombstoned ids; filesSince throws instead of looping when the server reports hasMore without advancing (closes#7). listCollections/listFiles stay as thin, unaffected wrappers.
Model: opus-4-8
The atomic writer fsyncs the staged temp file before rename and the directory after, and is exported for reuse. downloadFile/downloadThumbnail gain an optional per-chunk onProgress hook (non-decreasing, final equals bytesWritten; no-op when absent). Retry and TAG_FINAL checks unchanged.
Model: opus-4-8
decryptFile now sets file.size/thumbnail.size from raw.info (undefined when absent) and carries an optional isDeleted on EnteFile. Plain EnteFile return, no caller changes; listFiles keeps filtering tombstones. Tests cover the three fields and the size-absent case.
Model: opus-4-8
rootDir was ./src while include also matched bin/**/*, which is TS6059: tsc
refuses to emit at all when a compiled file sits outside rootDir. rootDir is
now the repository root, which is the smallest change that makes the two
agree and leaves the source layout the README documents alone. Output keeps
the shape of the source tree, so main and types move to dist/src/index.js and
dist/src/index.d.ts while bin.quak stays at dist/bin/quak.js. The alternative,
moving the CLI body into src/ behind a shim in bin/, would hold main at
dist/index.js at the cost of churning the CLI and contradicting the layout
diagram in the README.
Clearing TS6059 exposed two type errors that had never been reached, because
the config error aborts before checking: StateAddress was read as a namespace
member off the default import, and the secretstream pull was called without
the additional-data argument, which libsodium does not make optional. The type
is now taken from the module's named export and the pull passes null for ad,
matching the null already passed on the push side in encryptBlob. Neither
changes what runs.
noEmitOnError stops a failed build from leaving output behind. It emitted
despite the error before, which is how a stale bin/quak.js came to sit next to
bin/quak.ts in a working tree, where eslint then read it and failed make check
on a generated file.
script/build compiles and then checks that the files package.json advertises
are among the ones the compiler wrote, since tsc knows nothing about the
manifest and a green build could still ship a package whose main resolves to
nothing. It also sets the executable bit on the bin entries, which tsc does
not carry over from the source even though it does copy the shebang. The
Makefile target is now a shim over it, as the other targets are, and
package.json's build script points at it so yarn build gets the same checks.
The Dockerfile runs make build after make check, so a branch that does not
compile cannot reach main. What make check itself runs is unchanged.
package.json gains a quak script, so the yarn quak commands the README's
Getting Started block has always listed resolve to the built CLI.
`CONNECT_CODES` drives `isSafeToReplay`, which is the only thing standing
between a transport failure and a replayed `POST /users/two-factor/verify`.
It included `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` on the stated
grounds that those errnos can only be reported before any request byte was
written. That is not true on Linux: an ICMP destination-unreachable
delivered on an already-established connection sets the socket error and
the next read or write returns `EHOSTUNREACH` or `ENETUNREACH`, and a local
interface going down after the request was fully written surfaces as
`ENETDOWN` the same way. In each case the server may already have received
and acted on the request -- exactly the ambiguity the rule exists to
exclude, on the paths that consume a second-factor attempt or register a
thumbnail.
The three are dropped from `CONNECT_CODES` and stay in `TRANSPORT_CODES`,
so they remain retryable for the idempotent calls; only replay eligibility
narrows. What is left -- `ENOTFOUND`, `EAI_AGAIN`, `ECONNREFUSED` -- means
no TCP connection to the server ever existed, so no request byte can have
been transmitted.
The justification is corrected everywhere it was stated: the comment on
`CONNECT_CODES`, the one on `isSafeToReplay`, the `postJSON` call site, the
README's idempotency section and the `client.test.ts` docblock. All of them
now describe what the narrowed set actually establishes rather than
claiming a proof it did not support.
The narrowing is enforced by the suite rather than asserted in a comment:
the three errnos join `ECONNRESET`/`EPIPE`/`ETIMEDOUT` in the
`isSafeToReplay`-returns-false test, with companion `isRetryable` assertions
so a future edit cannot make them non-retryable by accident. Putting the
three back into `CONNECT_CODES` turns that test red (1 failure, verified).
No retry on 4xx, backoff on 5xx and transport failures, and a deadline on
every request. Before this, one transient 503 or TCP reset failed a file
for good, and a CDN connection that went quiet after accepting the request
blocked `quak backup` forever, because there was no timeout anywhere.
src/retry.ts holds the policy: a classifier that decides whether another
attempt could produce a different answer, and a loop that acts on it with
exponential backoff and full jitter. Retried: 5xx, 408, 429, transport
failures (the errno is read out of the cause chain, which is where Node's
fetch puts it), deadline aborts, and truncated transfers. Not retried:
every other 4xx, and anything unrecognised — a wrongly retried permanent
failure delays every remaining file, while a wrongly abandoned transient
one costs a single file the next run picks up. Attempt count, delays,
sleep and jitter source are all configurable through ApiClientOptions;
sleep being injectable is what lets the suite exercise the policy without
waiting.
Truncation needed a type before it could be classified. streamDecrypt
threw plain Errors whose messages began "download: stream truncated", and
classifying on message text would mean the next reword silently turned
every truncated download into a permanent failure. It now throws
TruncatedStreamError, which lives in src/errors.ts alongside ApiError so
the classifier can recognise both without importing the modules that
import it; api/client.ts re-exports ApiError, so it stays one class and
every existing import path still resolves.
Downloads retry the request, the stream consumption and the decryption
together. Only the first of those happens inside ApiClient: a socket
reset after the headers arrived throws in streamDecrypt, and retrying the
request alone would never see it. The client's own retry is switched off
for those two calls so the budgets do not multiply into sixteen requests
per file, and the atomic write stays outside the loop so a download that
took three attempts still performs one write and one rename.
Non-idempotent requests are not blindly replayed. postJSON and putJSON
reach create-session, two-factor/verify — which burns one of a few
second-factor attempts — and files/thumbnail, so they retry only when the
connection was never established and the server provably never saw the
request. putFile is exempt and retries fully: a presigned PUT stores one
whole object at one key, with no partial state to damage. It now throws
ApiError with the status, as do the two null-body paths, which previously
threw bare Errors that nothing could classify.
Timeouts come from AbortSignal.timeout(), renewed per attempt: 30s for
JSON and upload calls, 10 minutes for file bodies, since a value short
enough to keep a hung API call from stalling a backup would cancel a
legitimate multi-gigabyte download. The download deadline is enforced
over the body rather than only the headers, by racing each read against
the signal, so the guarantee does not depend on the fetch implementation
tearing down a stream it already handed over.
listMissingThumbnails now separates a genuine 404 from an exhausted
retry. Its bare catch reported both as missing, which after this change
would have let a few minutes of 500s talk fix-missing-thumbnails into
regenerating and re-uploading thumbnails that were fine. runBackup and
runMetadataBackup are untouched: the retry sits below them and their
per-file resilience is unchanged.
Rework against review finding B-A on the branch.
The claim was false as written. Commit 8a200be's message, the PR body and
the comments on both sides said the repurposed pinning test in
test/crypto/stream.test.ts fails if streamTagFinal() is ever made eager.
It does not: substituting
const EAGER: number = sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL;
export const streamTagFinal = (): number => EAGER;
leaves the whole suite green. Under vitest, sodium is already initialised
in the worker process by the time a source module is evaluated, so an
eager read picks up a real value there and value equality cannot see the
difference. The danger it claimed to cover is real: a direct probe of the
vendored libsodium-wrappers-sumo gives undefined before await sodium.ready
and 3 after, so an eager read would ship a library that rejects every
valid download as truncated for anyone importing it in plain Node ESM,
while make check stayed green.
Rather than drop the claim, this makes it true. A new test reproduces the
plain-Node ESM ordering that vitest hides: vi.resetModules() plus a
doMock'd stand-in sodium whose TAG_FINAL property is absent while
src/crypto/stream.ts is evaluated and appears only afterwards, exactly as
libsodium attaches its constants inside ready.then(...). A call-time read
sees the value that appeared after evaluation; a module-level read binds
undefined and the test fails. The stand-in hands back a sentinel rather
than 3, so a read that somehow reached the real library, or a return to a
hardcoded literal, fails too.
Demonstrated rather than argued: with the two-line eager variant above in
place, make test reports 1 failed | 140 passed, "expected undefined to be
42", exit 2. Restored with git checkout -- src/crypto/stream.ts, make test
reports 141 passed. The eager variant was applied with the editor and
reverted with git, not by scripted substitution.
The pinning test keeps its original job — value equality against the tag
observed on a real final chunk — and its doc comment now says only that.
The comment on the accessor in src/crypto/stream.ts names the guard test,
and records why the ordinary tests cannot see the bug on their own.
Rework against review findings on the branch.
Test runtime (B1): the suite's cost was never the 4 MiB fixture, it was
filling that fixture from the CSPRNG. sodium.randombytes_buf goes through
the wasm wrapper a byte at a time and takes ~20s for 4 MiB, against ~108ms
to encrypt the same buffer. Fixture content is not load-bearing anywhere in
the file — only length and tag are — so payloads now come from a seeded LCG
instead, which also makes them deterministic and reproducible as the README
asks. A seeded generator rather than a constant fill, so a downloader that
reordered or repeated chunks would still be caught. make test goes from
over the 30s cap in script/test (it was failing outright, then rerunning
verbose) back to 8.96s, against 10.02s on main.
Atomic write coverage (B2): every failure the suite injected originated in
streamDecrypt, which runs before anything is written, so no test observed a
temp file existing or being cleaned up and the catch block in writeAtomic
was dead code. rename is now intercepted in the test file, which adds two
cases per entry point: one asserting the staged file exists at rename time
and is a sibling of the destination, and one failing the rename itself so
the cleanup path runs with a temp file genuinely on disk. Deleting
writeAtomic in favour of a plain writeFile now turns the suite red.
TAG_FINAL (M1): STREAM_TAG_FINAL was a hardcoded 3 plus a test to detect
drift. The premise was right — libsodium attaches its constants inside
ready.then(...), so an eager module-level read binds undefined — but a lazy
read works, and decryptBlob was doing exactly that before. Replaced with
streamTagFinal(), which reads the library's own value at call time. The
drift test is repurposed to pin the accessor against the tag observed on a
real final chunk, which fails if it is ever made eager again.
Partial trailing chunk (M2): a transfer that stopped mid-chunk surfaced as
"authentication failed", which reads as corruption and sends the user after
the wrong problem. A final chunk that arrived in full always authenticates,
so trailing bytes that do not are reported as the truncation they almost
always are, with the authentication failure kept as the error's cause.
Poly1305 cannot separate a partial chunk from a corrupt one, so the message
names both possibilities; a corrupt whole chunk mid-stream is still
reported as an authentication failure, and both are now tested.
The pre-existing test that wrote its output to the process working
directory, i.e. the repo root, now writes into the test's temp directory.
streamDecrypt discarded the secretstream tag, so a download cut short
by a dropped connection decrypted cleanly up to the last whole chunk
and was returned as a success. downloadFile and downloadThumbnail then
wrote straight to the destination, and runBackup skips any existing
non-empty file, so a truncated original was treated as complete on
every subsequent run and never repaired.
streamDecrypt now tracks the tag of each chunk it pulls and throws if
the stream ended on anything other than TAG_FINAL, or if the body
carried no chunks at all — Ente always emits at least one chunk, as
encryptBlob shows by producing a TAG_FINAL chunk even for zero-length
plaintext, so an empty body is a failed transfer rather than an empty
file. Both error messages say the stream was truncated.
Plaintext is now staged in a temporary sibling file (same directory,
so the rename cannot cross a filesystem boundary; random UUID suffix,
so concurrent downloads cannot collide) and renamed into place only
after the whole stream has decrypted and verified. On any error the
temporary file is removed and the original error is rethrown
unchanged, so a cleanup failure never masks the real diagnosis. A
failed download therefore leaves the destination exactly as it was.
Public signatures and the DownloadResult shape are unchanged.
The download layer keeps its no-direct-sodium-import shape: TAG_FINAL
is re-exported from src/crypto as STREAM_TAG_FINAL, which decryptBlob
now uses too. Also moves the pullStreamChunk doc comment off
decryptBlob, where it had been sitting.
Retry and backoff remain out of scope; they stay the Next Step in
TODO.md and are tracked separately.
decryptCollection now takes the full key material {masterKey,
publicKey, secretKey} and dispatches on keyDecryptionNonce: present
means an owned collection (secretbox under the master key), absent
means a shared collection (sealed box to our public key). Client
already held the keypair for unsealing the auth token, so it just
passes it through.
Use || instead of ?? when picking the TOTP session ID. The server
sends "" for unset 2FA fields (no omitempty), and "" ?? v2
short-circuits to "", which made dual-2FA accounts fall through to
the unsupported passkey branch.
beginLogin now checks twoFactorSessionID ?? twoFactorSessionIDV2 before
the passkey branch. Accounts with both passkeys and TOTP can now log in
from the CLI using their authenticator app.
sharp was the only native dependency preventing a single-file binary.
Replaced with:
- jpeg-js (pure JS) for JPEG decode/resize/encode in thumbnail gen
- exif-reader (pure JS) for EXIF tag parsing
- Raw JPEG APP1 marker extraction for EXIF segment discovery
- Raw XMP packet extraction from file bytes
make build-bin produces a ~59MB self-contained Mach-O binary via
bun build --compile (bun installed via nix-shell). Zero runtime
dependencies. Tested: login, whoami, collections, files all work
from the compiled binary.
bin/quak.ts: init() called once at program start before commander
parses, so libsodium is ready for all commands including those that
restore sessions from disk.
118 tests pass.
ML metadata (face detections, CLIP embeddings) is not a separate
category from the rest of the metadata. It is always fetched and
included. The only opt-in is --exif (or --all) which requires
downloading every file for EXIF extraction.
ML data (face detections, CLIP embeddings) is now fetched by default
in backup-metadata. Use --no-ml to skip it. EXIF extraction (which
requires downloading every file) remains opt-in via --exif. --all is
an alias for --exif.
--ml fetches face detections and CLIP embeddings from the /files/data/fetch
endpoint (type 'mldata'). Each blob is encrypted with the file's key and
gzipped; we decrypt with decryptBlob, gunzip, and include the parsed JSON
as 'mlData' in the per-file output. Fetched in batches of 200 file IDs.
--exif downloads each file, runs sharp().metadata() to extract image
properties (format, dimensions, color space, orientation), then parses
the raw EXIF buffer with exif-reader for structured tags (lens, ISO,
shutter, aperture, GPS altitude, etc.). Also captures raw IPTC, XMP,
and ICC profile data. Included as 'imageMetadata' in the per-file output.
Without either flag, behavior is unchanged (fast metadata-only dump).
Adds exif-reader 2.0.3 as a runtime dependency.
3 new tests (ML data decrypted, ML data absent when flag not set, EXIF
extraction). 119 total tests, all green.
New command: quak backup-metadata <dir>
Dumps every piece of decrypted account metadata into a directory tree
of plain JSON files without downloading any file content. Layout:
<dir>/
account.json { email, userID }
collections/
<id>-<name>/
_collection.json { id, name, type, pubMagicMetadata?, ... }
<fileID>.json { id, metadata, magicMetadata?, pubMagicMetadata? }
Also adds collection-level magic metadata decryption (magicMetadata,
pubMagicMetadata, sharedMagicMetadata) to decryptCollection, which was
previously only done for files. The server sends these for visibility
settings, sort order, cover photo selection, etc.
6 new tests covering: account.json, per-collection dirs with
_collection.json, collection pubMagicMetadata decryption, per-file
JSON with all three metadata layers, graceful handling of files with
no magic metadata, and incremental re-run safety. 116 total.
list-missing-thumbnails: iterates all files across all collections,
fetches each thumbnail from the CDN, reports any that are missing or
empty. Deduplicates by file ID across collections.
fix-missing-thumbnails: for each missing thumbnail, downloads the
original file, generates a 720px JPEG thumbnail via sharp, encrypts
it with secretstream push (encryptBlob), uploads to a presigned URL,
and registers the new thumbnail via PUT /files/thumbnail.
New crypto: encryptBlob (secretstream push, single chunk TAG_FINAL).
New ApiClient methods: getUploadURL, putFile, putJSON, updateThumbnail.
New Client method: getApiClient() for modules that need raw API access.
Deps: sharp 0.34.5 (image processing), @types/sharp 0.32.0.
Extends RawEnteFile and EnteFile with optional magicMetadata and
pubMagicMetadata fields. Both are secretstream blobs under the file
key, decrypted to arbitrary JSON (Record<string, unknown>).
pubMagicMetadata carries ML-derived data from the Ente clients:
camera make/model, image dimensions, datetime with timezone offset,
and (when present) captions, editedName, face labels, keywords.
magicMetadata carries private mutable fields like visibility.
Backup now writes per-file JSON at originals/<fileID>.json containing
all three metadata layers (basic + magic + pubMagic).
Live-tested: all 11 files in the dev account have pubMagicMetadata
with SONY DSC-RX1RM3 camera info and 3000x2000 dimensions.
bin/quak.ts: commander-based CLI with login (interactive + QUAK_EMAIL/
QUAK_PASSWORD env vars), whoami, logout, backup commands. Session
stored at env-paths('quak').data/session.json (~/Library/Application
Support/quak/ on macOS, XDG on Linux).
src/backup.ts: runBackup downloads all files into originals/<id>.<ext>,
symlinks into collections/<name>/<title>, writes per-collection JSON
metadata at collections/<name>.json. Deduplicates across collections
(each file downloaded once). Skips existing originals on incremental
runs. Never crashes on single-file failure.
4 backup tests + live-tested against real Ente account.
4 tests for runBackup: full download into collection-named dirs,
incremental skip of existing files, resilient continuation after
single-file HTTP 500, and metadata.json output.
Adds commander 14.0.3 and env-paths 4.0.0 as runtime deps.
German for 'quack', matching the Ente (German for 'duck') naming. All
references updated: package name, CLI binary, X-Client-Package header,
test descriptions, temp dir prefixes, README, Makefile docker tag.
Exports Client, all lower-level modules, and all types from
src/index.ts. Replaces Phase 7 (on-disk session persistence) with
the Client class phase: session lives in the object, consumer
handles persistence via toJSON/fromJSON.
Client.login(email, password) performs the full SRP handshake, key
unwrap, and returns a ready Client. Session lives in the object.
Client.fromJSON(snapshot) restores from a serialized snapshot.
client.toJSON() produces a plain object with base64-encoded keys
that the consumer can persist however they like.
Methods: whoami, listCollections, listFiles (with pagination),
downloadFile, downloadThumbnail, logout.
All 86 tests pass including the 8-part literate usage tutorial.
test/client/usage.test.ts is a tutorial-as-test-suite that walks
through the entire quack API in order: login, whoami, listCollections,
listFiles, downloadFile, downloadThumbnail, toJSON/fromJSON, logout.
Each it() block is a self-contained example with prose commentary
explaining what the code does and why, with code samples showing the
API as a consumer would use it. The mock server performs real SRP and
crypto so the test data is structurally identical to production.
8 tests, all failing against the stub.
downloadFile streams the encrypted body from the CDN, buffers it to
the 4 MiB + 17 encrypted chunk boundary, decrypts each chunk via
secretstream pull, and writes the concatenated plaintext to disk.
downloadThumbnail does the same for the thumbnail CDN.
4 unit tests (single-chunk, large single-chunk, filename fallback,
thumbnail) + live integration test that downloads a real 472 KB JPEG
from the dev account and verifies it lands on disk.
Uses mkdtempSync for temp directories (not manual timestamp paths).
4 tests: single-chunk download, metadata-title fallback filename,
multi-chunk stream (50-byte chunks to exercise the buffering/split
logic without allocating 4 MiB), and thumbnail download.
Tests encrypt payloads with sodium's secretstream push, serve them
from a mock fetch, and verify the decrypted file on disk.
File metadata is encrypted as a single-chunk secretstream blob (the
'decryptionHeader' is the secretstream init header, not a secretbox
nonce). Collection keys and names correctly use secretbox.
Adds decryptBlob(ciphertext, header, key) to the crypto module as a
convenience wrapper for single-chunk secretstream decryption (init +
pull + verify TAG_FINAL).
Live-tested: collection names and file metadata (titles, types, dates)
decrypt correctly from the real Ente API.
10 tests covering decryptCollection (key + name decryption from raw
server JSON, type mapping, isShared, missing name, wrong key) and
decryptFile (key + metadata decryption, fileType number-to-string
mapping, file/thumbnail header passthrough, wrong key).
Adds src/model/types.ts with both raw (server) and decrypted (library)
type definitions. src/model/decrypt.ts has throwing stubs.
The Ente server validates the auth token as URL-safe base64 with
padding (matching Go's base64.URLEncoding). Our toBase64URL strips
padding, producing a 43-char token where the server expects 44. This
caused HTTP 401 'invalid token' on every authenticated call.
Adds toBase64URLPadded to the crypto module and uses it in unwrapAuth
for the token specifically. toBase64URL (no-padding) is kept for
general use (JWT-style contexts).
Adds test/integration/live-login.ts which logs into the dev account
(entedev2026jp@acidhou.se), unwraps keys, and fetches collections
from the real Ente API. Verified: 4 collections returned successfully.
beginLogin(api, email, password):
1. Fetches SRP attributes for the email
2. If isEmailMFAEnabled, returns { kind: 'emailOTP' } immediately
3. Otherwise: derives KEK + login subkey, creates SRP client with
4096-bit group, runs the two-round handshake (create-session,
verify-session), verifies server M2
4. Returns { kind: 'complete' } with AuthorizationResponse, or
{ kind: 'totp' } / { kind: 'passkey' } if 2FA is required
submitTOTP(api, sessionID, code): POST /users/two-factor/verify
requestEmailOTP(api, email): POST /users/ott
submitEmailOTP(api, email, code): POST /users/verify-email
All 64 tests pass including real SRP-6a handshakes against a mock
server built with fast-srp-hap's SrpServer.
Adds fast-srp-hap (the same SRP library Ente's web client uses, pinned
to 2.0.4) as a runtime dependency.
Tests build a full mock Ente server using fast-srp-hap's SrpServer to
exercise real SRP-6a math end-to-end. The mock handles:
GET /users/srp/attributes
POST /users/srp/create-session
POST /users/srp/verify-session
POST /users/two-factor/verify
POST /users/ott
POST /users/verify-email
7 tests covering:
* SRP login completing successfully
* SRP login requiring TOTP (returns { kind: 'totp' })
* Wrong password (SRP M1 fails server-side checkM1)
* Email MFA fallback (returns { kind: 'emailOTP' })
* submitTOTP
* requestEmailOTP + submitEmailOTP
Production origins: api.ente.io, files.ente.io, thumbnails.ente.io.
Custom apiOrigin routes file/thumbnail downloads through the same host
at /files/download/<id> and /files/preview/<id>.
Every request carries X-Client-Package (berlin.sneak.quack).
Authenticated requests carry X-Auth-Token, toggled via setAuthToken /
clearAuthToken or the constructor authToken option.
getJSON appends query params (skipping undefined values), parses JSON.
postJSON sends JSON with Content-Type header, parses JSON response.
getFileStream / getThumbnailStream return the response body stream.
Non-2xx responses throw ApiError with status, code (from JSON body),
requestID (from x-request-id header), and raw body. Retry logic is
deferred to a follow-on branch.
All 57 tests pass.
sneak
clawbot