Rework against review finding B-A on the branch.
The claim was false as written. Commit 8a200be's message, the PR body and
the comments on both sides said the repurposed pinning test in
test/crypto/stream.test.ts fails if streamTagFinal() is ever made eager.
It does not: substituting
const EAGER: number = sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL;
export const streamTagFinal = (): number => EAGER;
leaves the whole suite green. Under vitest, sodium is already initialised
in the worker process by the time a source module is evaluated, so an
eager read picks up a real value there and value equality cannot see the
difference. The danger it claimed to cover is real: a direct probe of the
vendored libsodium-wrappers-sumo gives undefined before await sodium.ready
and 3 after, so an eager read would ship a library that rejects every
valid download as truncated for anyone importing it in plain Node ESM,
while make check stayed green.
Rather than drop the claim, this makes it true. A new test reproduces the
plain-Node ESM ordering that vitest hides: vi.resetModules() plus a
doMock'd stand-in sodium whose TAG_FINAL property is absent while
src/crypto/stream.ts is evaluated and appears only afterwards, exactly as
libsodium attaches its constants inside ready.then(...). A call-time read
sees the value that appeared after evaluation; a module-level read binds
undefined and the test fails. The stand-in hands back a sentinel rather
than 3, so a read that somehow reached the real library, or a return to a
hardcoded literal, fails too.
Demonstrated rather than argued: with the two-line eager variant above in
place, make test reports 1 failed | 140 passed, "expected undefined to be
42", exit 2. Restored with git checkout -- src/crypto/stream.ts, make test
reports 141 passed. The eager variant was applied with the editor and
reverted with git, not by scripted substitution.
The pinning test keeps its original job — value equality against the tag
observed on a real final chunk — and its doc comment now says only that.
The comment on the accessor in src/crypto/stream.ts names the guard test,
and records why the ordinary tests cannot see the bug on their own.
Rework against review findings on the branch.
Test runtime (B1): the suite's cost was never the 4 MiB fixture, it was
filling that fixture from the CSPRNG. sodium.randombytes_buf goes through
the wasm wrapper a byte at a time and takes ~20s for 4 MiB, against ~108ms
to encrypt the same buffer. Fixture content is not load-bearing anywhere in
the file — only length and tag are — so payloads now come from a seeded LCG
instead, which also makes them deterministic and reproducible as the README
asks. A seeded generator rather than a constant fill, so a downloader that
reordered or repeated chunks would still be caught. make test goes from
over the 30s cap in script/test (it was failing outright, then rerunning
verbose) back to 8.96s, against 10.02s on main.
Atomic write coverage (B2): every failure the suite injected originated in
streamDecrypt, which runs before anything is written, so no test observed a
temp file existing or being cleaned up and the catch block in writeAtomic
was dead code. rename is now intercepted in the test file, which adds two
cases per entry point: one asserting the staged file exists at rename time
and is a sibling of the destination, and one failing the rename itself so
the cleanup path runs with a temp file genuinely on disk. Deleting
writeAtomic in favour of a plain writeFile now turns the suite red.
TAG_FINAL (M1): STREAM_TAG_FINAL was a hardcoded 3 plus a test to detect
drift. The premise was right — libsodium attaches its constants inside
ready.then(...), so an eager module-level read binds undefined — but a lazy
read works, and decryptBlob was doing exactly that before. Replaced with
streamTagFinal(), which reads the library's own value at call time. The
drift test is repurposed to pin the accessor against the tag observed on a
real final chunk, which fails if it is ever made eager again.
Partial trailing chunk (M2): a transfer that stopped mid-chunk surfaced as
"authentication failed", which reads as corruption and sends the user after
the wrong problem. A final chunk that arrived in full always authenticates,
so trailing bytes that do not are reported as the truncation they almost
always are, with the authentication failure kept as the error's cause.
Poly1305 cannot separate a partial chunk from a corrupt one, so the message
names both possibilities; a corrupt whole chunk mid-stream is still
reported as an authentication failure, and both are now tested.
The pre-existing test that wrote its output to the process working
directory, i.e. the repo root, now writes into the test's temp directory.
streamDecrypt discarded the secretstream tag, so a download cut short
by a dropped connection decrypted cleanly up to the last whole chunk
and was returned as a success. downloadFile and downloadThumbnail then
wrote straight to the destination, and runBackup skips any existing
non-empty file, so a truncated original was treated as complete on
every subsequent run and never repaired.
streamDecrypt now tracks the tag of each chunk it pulls and throws if
the stream ended on anything other than TAG_FINAL, or if the body
carried no chunks at all — Ente always emits at least one chunk, as
encryptBlob shows by producing a TAG_FINAL chunk even for zero-length
plaintext, so an empty body is a failed transfer rather than an empty
file. Both error messages say the stream was truncated.
Plaintext is now staged in a temporary sibling file (same directory,
so the rename cannot cross a filesystem boundary; random UUID suffix,
so concurrent downloads cannot collide) and renamed into place only
after the whole stream has decrypted and verified. On any error the
temporary file is removed and the original error is rethrown
unchanged, so a cleanup failure never masks the real diagnosis. A
failed download therefore leaves the destination exactly as it was.
Public signatures and the DownloadResult shape are unchanged.
The download layer keeps its no-direct-sodium-import shape: TAG_FINAL
is re-exported from src/crypto as STREAM_TAG_FINAL, which decryptBlob
now uses too. Also moves the pullStreamChunk doc comment off
decryptBlob, where it had been sitting.
Retry and backoff remain out of scope; they stay the Next Step in
TODO.md and are tracked separately.
Covers, for both downloadFile and downloadThumbnail:
- a multi-chunk body whose TAG_FINAL chunk never arrived is rejected
with a truncation error;
- an empty body is rejected as truncation rather than written as a
zero-byte file;
- after a truncation or chunk-authentication failure the destination
path does not exist and no temporary scratch file is left behind;
- an existing file at the destination survives a failed download
byte for byte, and is replaced atomically by a successful one;
- the existing success cases still produce identical bytes and an
identical DownloadResult.
Adds an encryptMultiChunkBody helper that frames leading chunks at
exactly STREAM_CHUNK_SIZE so the downloader's fixed-size re-splitting
lines up, plus a multi-chunk success case as the positive control.
Also pins the new STREAM_TAG_FINAL crypto export against libsodium's
own constant, since it must be declared as a literal: libsodium
attaches its constants only after sodium.ready resolves, well after
this library's modules are evaluated.
These fail until the implementation lands, per the repo's TDD workflow.
/collections/v2 is a sync API: deleted collections remain in the
response forever with isDeleted: true, and their /collections/v2/diff
endpoint returns HTTP 404. A long-lived account accumulates hundreds
of tombstones, so any caller that iterates listCollections() output
(backup, backup-metadata) dies on the first one.
decryptCollection now takes the full key material {masterKey,
publicKey, secretKey} and dispatches on keyDecryptionNonce: present
means an owned collection (secretbox under the master key), absent
means a shared collection (sealed box to our public key). Client
already held the keypair for unsealing the auth token, so it just
passes it through.
Collections shared with the account are not encrypted with the master
key: the sharer only knows the recipient's public key, so the server
delivers encryptedKey as crypto_box_seal to that key and omits
keyDecryptionNonce entirely. decryptCollection assumed the owned-only
wire format and crashed on fromBase64(undefined) for any account with
an incoming shared album, taking down listCollections and every
command built on it (backup, backup-metadata, ...).
The previous "shared" fixture was unfaithful (secretbox + nonce with a
foreign ownerID, a shape the server never sends), which is why the
suite stayed green. These tests model the real wire format and change
decryptCollection to take the full key material {masterKey, publicKey,
secretKey} so it can unseal shared collection keys.
The museum server's EmailAuthorizationResponse declares
passkeySessionID, accountsUrl, twoFactorSessionID, and
twoFactorSessionIDV2 without `omitempty`, so Go always sends them,
as "" when unset. The previous mock omitted the unset fields
entirely, which let the ?? -based dispatch pass in tests while the
real server's "" defeated it and dual-2FA logins fell through to
the unsupported passkey branch.
When an account has both passkeys and TOTP enrolled, the Ente server
returns passkeySessionID + twoFactorSessionIDV2 (deliberately not the
V1 twoFactorSessionID field, so old clients keep using passkeys).
quak only checked the V1 field, saw the passkey session, and threw
'Passkey authentication is not supported'. A CLI cannot do WebAuthn,
so it must take the TOTP path via the V2 session ID.
sharp was the only native dependency preventing a single-file binary.
Replaced with:
- jpeg-js (pure JS) for JPEG decode/resize/encode in thumbnail gen
- exif-reader (pure JS) for EXIF tag parsing
- Raw JPEG APP1 marker extraction for EXIF segment discovery
- Raw XMP packet extraction from file bytes
make build-bin produces a ~59MB self-contained Mach-O binary via
bun build --compile (bun installed via nix-shell). Zero runtime
dependencies. Tested: login, whoami, collections, files all work
from the compiled binary.
bin/quak.ts: init() called once at program start before commander
parses, so libsodium is ready for all commands including those that
restore sessions from disk.
118 tests pass.
ML metadata (face detections, CLIP embeddings) is not a separate
category from the rest of the metadata. It is always fetched and
included. The only opt-in is --exif (or --all) which requires
downloading every file for EXIF extraction.
ML data (face detections, CLIP embeddings) is now fetched by default
in backup-metadata. Use --no-ml to skip it. EXIF extraction (which
requires downloading every file) remains opt-in via --exif. --all is
an alias for --exif.
--ml fetches face detections and CLIP embeddings from the /files/data/fetch
endpoint (type 'mldata'). Each blob is encrypted with the file's key and
gzipped; we decrypt with decryptBlob, gunzip, and include the parsed JSON
as 'mlData' in the per-file output. Fetched in batches of 200 file IDs.
--exif downloads each file, runs sharp().metadata() to extract image
properties (format, dimensions, color space, orientation), then parses
the raw EXIF buffer with exif-reader for structured tags (lens, ISO,
shutter, aperture, GPS altitude, etc.). Also captures raw IPTC, XMP,
and ICC profile data. Included as 'imageMetadata' in the per-file output.
Without either flag, behavior is unchanged (fast metadata-only dump).
Adds exif-reader 2.0.3 as a runtime dependency.
3 new tests (ML data decrypted, ML data absent when flag not set, EXIF
extraction). 119 total tests, all green.
New command: quak backup-metadata <dir>
Dumps every piece of decrypted account metadata into a directory tree
of plain JSON files without downloading any file content. Layout:
<dir>/
account.json { email, userID }
collections/
<id>-<name>/
_collection.json { id, name, type, pubMagicMetadata?, ... }
<fileID>.json { id, metadata, magicMetadata?, pubMagicMetadata? }
Also adds collection-level magic metadata decryption (magicMetadata,
pubMagicMetadata, sharedMagicMetadata) to decryptCollection, which was
previously only done for files. The server sends these for visibility
settings, sort order, cover photo selection, etc.
6 new tests covering: account.json, per-collection dirs with
_collection.json, collection pubMagicMetadata decryption, per-file
JSON with all three metadata layers, graceful handling of files with
no magic metadata, and incremental re-run safety. 116 total.
test/crypto/encrypt-blob.test.ts (8 tests):
Round-trip with decryptBlob, zero-length payload, ciphertext
overhead check, header size check, different keys produce different
output, same key produces different output each call (random nonce),
wrong-key rejection, tamper detection.
test/api/upload.test.ts (4 tests):
putJSON sends PUT with auth headers and JSON body. putFile sends
PUT to the exact presigned URL with Content-Type octet-stream and
does NOT send X-Auth-Token or X-Client-Package (S3 would reject
them). getUploadURL POSTs with contentLength and contentMD5.
updateThumbnail PUTs to /files/thumbnail with correct body shape.
test/thumbnails/thumbnails.test.ts (8 tests):
listMissingThumbnails identifies empty (0 byte) and 404 thumbnails
while ignoring working ones; deduplicates across collections.
fixMissingThumbnails verifies the full pipeline: download original,
generate JPEG via sharp, encrypt with encryptBlob, upload via
presigned URL, register via PUT /files/thumbnail. The test
decrypts the uploaded ciphertext and verifies it starts with JPEG
magic bytes (FF D8 FF). Also tests: nonexistent file ID reports
failure without crashing; mixed success/failure across multiple
files; Client.getApiClient() works when logged in, throws after
logout.
bin/quak.ts: commander-based CLI with login (interactive + QUAK_EMAIL/
QUAK_PASSWORD env vars), whoami, logout, backup commands. Session
stored at env-paths('quak').data/session.json (~/Library/Application
Support/quak/ on macOS, XDG on Linux).
src/backup.ts: runBackup downloads all files into originals/<id>.<ext>,
symlinks into collections/<name>/<title>, writes per-collection JSON
metadata at collections/<name>.json. Deduplicates across collections
(each file downloaded once). Skips existing originals on incremental
runs. Never crashes on single-file failure.
4 backup tests + live-tested against real Ente account.
4 tests for runBackup: full download into collection-named dirs,
incremental skip of existing files, resilient continuation after
single-file HTTP 500, and metadata.json output.
Adds commander 14.0.3 and env-paths 4.0.0 as runtime deps.
German for 'quack', matching the Ente (German for 'duck') naming. All
references updated: package name, CLI binary, X-Client-Package header,
test descriptions, temp dir prefixes, README, Makefile docker tag.
test/client/usage.test.ts is a tutorial-as-test-suite that walks
through the entire quack API in order: login, whoami, listCollections,
listFiles, downloadFile, downloadThumbnail, toJSON/fromJSON, logout.
Each it() block is a self-contained example with prose commentary
explaining what the code does and why, with code samples showing the
API as a consumer would use it. The mock server performs real SRP and
crypto so the test data is structurally identical to production.
8 tests, all failing against the stub.
downloadFile streams the encrypted body from the CDN, buffers it to
the 4 MiB + 17 encrypted chunk boundary, decrypts each chunk via
secretstream pull, and writes the concatenated plaintext to disk.
downloadThumbnail does the same for the thumbnail CDN.
4 unit tests (single-chunk, large single-chunk, filename fallback,
thumbnail) + live integration test that downloads a real 472 KB JPEG
from the dev account and verifies it lands on disk.
Uses mkdtempSync for temp directories (not manual timestamp paths).
4 tests: single-chunk download, metadata-title fallback filename,
multi-chunk stream (50-byte chunks to exercise the buffering/split
logic without allocating 4 MiB), and thumbnail download.
Tests encrypt payloads with sodium's secretstream push, serve them
from a mock fetch, and verify the decrypted file on disk.
File metadata is encrypted as a single-chunk secretstream blob (the
'decryptionHeader' is the secretstream init header, not a secretbox
nonce). Collection keys and names correctly use secretbox.
Adds decryptBlob(ciphertext, header, key) to the crypto module as a
convenience wrapper for single-chunk secretstream decryption (init +
pull + verify TAG_FINAL).
Live-tested: collection names and file metadata (titles, types, dates)
decrypt correctly from the real Ente API.
10 tests covering decryptCollection (key + name decryption from raw
server JSON, type mapping, isShared, missing name, wrong key) and
decryptFile (key + metadata decryption, fileType number-to-string
mapping, file/thumbnail header passthrough, wrong key).
Adds src/model/types.ts with both raw (server) and decrypted (library)
type definitions. src/model/decrypt.ts has throwing stubs.
The Ente server validates the auth token as URL-safe base64 with
padding (matching Go's base64.URLEncoding). Our toBase64URL strips
padding, producing a 43-char token where the server expects 44. This
caused HTTP 401 'invalid token' on every authenticated call.
Adds toBase64URLPadded to the crypto module and uses it in unwrapAuth
for the token specifically. toBase64URL (no-padding) is kept for
general use (JWT-style contexts).
Adds test/integration/live-login.ts which logs into the dev account
(entedev2026jp@acidhou.se), unwraps keys, and fetches collections
from the real Ente API. Verified: 4 collections returned successfully.
Adds fast-srp-hap (the same SRP library Ente's web client uses, pinned
to 2.0.4) as a runtime dependency.
Tests build a full mock Ente server using fast-srp-hap's SrpServer to
exercise real SRP-6a math end-to-end. The mock handles:
GET /users/srp/attributes
POST /users/srp/create-session
POST /users/srp/verify-session
POST /users/two-factor/verify
POST /users/ott
POST /users/verify-email
7 tests covering:
* SRP login completing successfully
* SRP login requiring TOTP (returns { kind: 'totp' })
* Wrong password (SRP M1 fails server-side checkM1)
* Email MFA fallback (returns { kind: 'emailOTP' })
* submitTOTP
* requestEmailOTP + submitEmailOTP
19 tests covering ApiClient's full public surface: default and custom
origins, X-Client-Package and X-Auth-Token headers, getJSON with query
params, postJSON with JSON body, ApiError on 4xx/5xx, streaming file
and thumbnail downloads, and self-hosted origin routing.
Tests inject a recording fetch via the constructor, so nothing hits the
network. The test file is documented to serve as canonical usage
reference per the development workflow.
Tests for the password-only decryption chain that follows a successful
login (SRP or email OTP, with or without 2FA). The unwrap covers:
password -> KEK (Argon2id) -> masterKey (secretbox) ->
secretKey (secretbox) -> tokenBytes (sealed box) -> base64url token
Each test builds a synthetic AuthorizationResponse using libsodium
directly and asserts unwrapAuth recovers the inputs byte for byte. The
test file also functions as the canonical description of the protocol.
Adds src/auth/types.ts with KeyAttributes, SRPAttributes,
AuthorizationResponse, and LoginChallenge declarations matching the
README's API reference. src/auth/unwrap.ts is the throwing stub; the
real implementation lands next.
Tests for the entire crypto/ public surface, written against the API
shape declared in the README. The accompanying src/crypto/ modules are
stubs that throw 'not implemented' so the test files compile and tests
fail with clear errors rather than module-not-found.
Tests cover:
* init() resolves and is idempotent
* fromBase64 / toBase64 / toBase64URL round-trips, including URL-safe
input with stripped padding (the form Ente uses for auth tokens)
* deriveKEK matches sodium.crypto_pwhash with Argon2id parameters
* deriveLoginSubkey matches sodium.crypto_kdf_derive_from_key with
subkey id 1 and ctx 'loginctx', truncated to 16 bytes
* decryptBox round-trips, rejects tampering, wrong key, wrong nonce
* decryptSealed round-trips, rejects wrong keypair and tampering
* Secretstream pull decrypts multi-chunk streams in order, exposes
per-chunk tags, rejects tampering, wrong key, and out-of-order chunks
* Constants STREAM_CHUNK_SIZE (4 MiB) and STREAM_CHUNK_OVERHEAD (17)
Tests are commented to serve as the canonical API documentation per the
README development workflow policy. Verified: 29 tests fail (red), 3
trivial constant tests pass; lint and fmt-check are green.
eslint.config.mjs is updated to honour the leading-underscore convention
for intentionally unused parameters (the stubs).
package.json declares the project as ESM with NodeNext module resolution,
exposing dist/index.js as the library entry and dist/bin/quack.js as the
CLI binary. Dev dependencies are pinned to exact versions (yarn.lock holds
the integrity hashes per repo policy on hash-pinned external references).
Adds a placeholder src/index.ts and a single smoke test so make check is
not a no-op.
