37 Commits

Author SHA1 Message Date
f3cf4af833 Retry transient network failures with exponential backoff (closes #2)
All checks were successful
check / check (push) Successful in 20s
No retry on 4xx, backoff on 5xx and transport failures, and a deadline on
every request. Before this, one transient 503 or TCP reset failed a file
for good, and a CDN connection that went quiet after accepting the request
blocked `quak backup` forever, because there was no timeout anywhere.

src/retry.ts holds the policy: a classifier that decides whether another
attempt could produce a different answer, and a loop that acts on it with
exponential backoff and full jitter. Retried: 5xx, 408, 429, transport
failures (the errno is read out of the cause chain, which is where Node's
fetch puts it), deadline aborts, and truncated transfers. Not retried:
every other 4xx, and anything unrecognised — a wrongly retried permanent
failure delays every remaining file, while a wrongly abandoned transient
one costs a single file the next run picks up. Attempt count, delays,
sleep and jitter source are all configurable through ApiClientOptions;
sleep being injectable is what lets the suite exercise the policy without
waiting.

Truncation needed a type before it could be classified. streamDecrypt
threw plain Errors whose messages began "download: stream truncated", and
classifying on message text would mean the next reword silently turned
every truncated download into a permanent failure. It now throws
TruncatedStreamError, which lives in src/errors.ts alongside ApiError so
the classifier can recognise both without importing the modules that
import it; api/client.ts re-exports ApiError, so it stays one class and
every existing import path still resolves.

Downloads retry the request, the stream consumption and the decryption
together. Only the first of those happens inside ApiClient: a socket
reset after the headers arrived throws in streamDecrypt, and retrying the
request alone would never see it. The client's own retry is switched off
for those two calls so the budgets do not multiply into sixteen requests
per file, and the atomic write stays outside the loop so a download that
took three attempts still performs one write and one rename.

Non-idempotent requests are not blindly replayed. postJSON and putJSON
reach create-session, two-factor/verify — which burns one of a few
second-factor attempts — and files/thumbnail, so they retry only when the
connection was never established and the server provably never saw the
request. putFile is exempt and retries fully: a presigned PUT stores one
whole object at one key, with no partial state to damage. It now throws
ApiError with the status, as do the two null-body paths, which previously
threw bare Errors that nothing could classify.

Timeouts come from AbortSignal.timeout(), renewed per attempt: 30s for
JSON and upload calls, 10 minutes for file bodies, since a value short
enough to keep a hung API call from stalling a backup would cancel a
legitimate multi-gigabyte download. The download deadline is enforced
over the body rather than only the headers, by racing each read against
the signal, so the guarantee does not depend on the fetch implementation
tearing down a stream it already handed over.

listMissingThumbnails now separates a genuine 404 from an exhausted
retry. Its bare catch reported both as missing, which after this change
would have let a few minutes of 500s talk fix-missing-thumbnails into
regenerating and re-uploading thumbnails that were fine. runBackup and
runMetadataBackup are untouched: the retry sits below them and their
per-file resilience is unchanged.
2026-08-09 05:21:31 +00:00
0cbe338b58 Add failing tests for the download retry policy
Tests only; the retry module they import does not exist yet, so the
branch is red at this commit.

New test/retry/retry.test.ts documents the classifier and the backoff:
which errors are worth another attempt, which are not, and how the delay
before each retry is derived. It asserts on the arguments handed to an
injected sleep function rather than on elapsed time, so the suite never
waits and the numbers are exact.

test/api/client.test.ts gains the request-count contract for each of the
six call sites, the deadline behaviour, the ApiError typing that the
presigned PUT and the null-body paths need in order to be classified at
all, and the replay rule for the two non-idempotent methods.

test/download/download.test.ts gains the case that motivates the whole
design: a socket reset after the response headers arrived, which happens
below ApiClient and can only be caught by retrying the request, the
stream consumption and the decryption together. It also pins that a
retried download stages exactly one temp file, and that the two retry
layers do not compose into a multiplied request budget. The existing
truncation tests now assert on the error type rather than its wording,
since that type is what the classifier reads.

test/thumbnails/thumbnails.test.ts separates a genuine 404 from an
exhausted retry, so a failing server can no longer make
fix-missing-thumbnails re-upload thumbnails that already exist.
2026-08-09 05:11:07 +00:00
937bcb7aee Guard streamTagFinal() against being made eager
All checks were successful
check / check (push) Successful in 4s
Rework against review finding B-A on the branch.

The claim was false as written. Commit 8a200be's message, the PR body and
the comments on both sides said the repurposed pinning test in
test/crypto/stream.test.ts fails if streamTagFinal() is ever made eager.
It does not: substituting

    const EAGER: number = sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL;
    export const streamTagFinal = (): number => EAGER;

leaves the whole suite green. Under vitest, sodium is already initialised
in the worker process by the time a source module is evaluated, so an
eager read picks up a real value there and value equality cannot see the
difference. The danger it claimed to cover is real: a direct probe of the
vendored libsodium-wrappers-sumo gives undefined before await sodium.ready
and 3 after, so an eager read would ship a library that rejects every
valid download as truncated for anyone importing it in plain Node ESM,
while make check stayed green.

Rather than drop the claim, this makes it true. A new test reproduces the
plain-Node ESM ordering that vitest hides: vi.resetModules() plus a
doMock'd stand-in sodium whose TAG_FINAL property is absent while
src/crypto/stream.ts is evaluated and appears only afterwards, exactly as
libsodium attaches its constants inside ready.then(...). A call-time read
sees the value that appeared after evaluation; a module-level read binds
undefined and the test fails. The stand-in hands back a sentinel rather
than 3, so a read that somehow reached the real library, or a return to a
hardcoded literal, fails too.

Demonstrated rather than argued: with the two-line eager variant above in
place, make test reports 1 failed | 140 passed, "expected undefined to be
42", exit 2. Restored with git checkout -- src/crypto/stream.ts, make test
reports 141 passed. The eager variant was applied with the editor and
reverted with git, not by scripted substitution.

The pinning test keeps its original job — value equality against the tag
observed on a real final chunk — and its doc comment now says only that.
The comment on the accessor in src/crypto/stream.ts names the guard test,
and records why the ordinary tests cannot see the bug on their own.
2026-08-09 02:45:35 +00:00
8a200be8a7 Read TAG_FINAL from libsodium, detect partial chunks, cover the atomic write
All checks were successful
check / check (push) Successful in 22s
Rework against review findings on the branch.

Test runtime (B1): the suite's cost was never the 4 MiB fixture, it was
filling that fixture from the CSPRNG. sodium.randombytes_buf goes through
the wasm wrapper a byte at a time and takes ~20s for 4 MiB, against ~108ms
to encrypt the same buffer. Fixture content is not load-bearing anywhere in
the file — only length and tag are — so payloads now come from a seeded LCG
instead, which also makes them deterministic and reproducible as the README
asks. A seeded generator rather than a constant fill, so a downloader that
reordered or repeated chunks would still be caught. make test goes from
over the 30s cap in script/test (it was failing outright, then rerunning
verbose) back to 8.96s, against 10.02s on main.

Atomic write coverage (B2): every failure the suite injected originated in
streamDecrypt, which runs before anything is written, so no test observed a
temp file existing or being cleaned up and the catch block in writeAtomic
was dead code. rename is now intercepted in the test file, which adds two
cases per entry point: one asserting the staged file exists at rename time
and is a sibling of the destination, and one failing the rename itself so
the cleanup path runs with a temp file genuinely on disk. Deleting
writeAtomic in favour of a plain writeFile now turns the suite red.

TAG_FINAL (M1): STREAM_TAG_FINAL was a hardcoded 3 plus a test to detect
drift. The premise was right — libsodium attaches its constants inside
ready.then(...), so an eager module-level read binds undefined — but a lazy
read works, and decryptBlob was doing exactly that before. Replaced with
streamTagFinal(), which reads the library's own value at call time. The
drift test is repurposed to pin the accessor against the tag observed on a
real final chunk, which fails if it is ever made eager again.

Partial trailing chunk (M2): a transfer that stopped mid-chunk surfaced as
"authentication failed", which reads as corruption and sends the user after
the wrong problem. A final chunk that arrived in full always authenticates,
so trailing bytes that do not are reported as the truncation they almost
always are, with the authentication failure kept as the error's cause.
Poly1305 cannot separate a partial chunk from a corrupt one, so the message
names both possibilities; a corrupt whole chunk mid-stream is still
reported as an authentication failure, and both are now tested.

The pre-existing test that wrote its output to the process working
directory, i.e. the repo root, now writes into the test's temp directory.
2026-08-09 02:23:39 +00:00
99905277a3 Verify secretstream TAG_FINAL and write downloads atomically (closes #1)
Some checks failed
check / check (push) Failing after 1m19s
streamDecrypt discarded the secretstream tag, so a download cut short
by a dropped connection decrypted cleanly up to the last whole chunk
and was returned as a success. downloadFile and downloadThumbnail then
wrote straight to the destination, and runBackup skips any existing
non-empty file, so a truncated original was treated as complete on
every subsequent run and never repaired.

streamDecrypt now tracks the tag of each chunk it pulls and throws if
the stream ended on anything other than TAG_FINAL, or if the body
carried no chunks at all — Ente always emits at least one chunk, as
encryptBlob shows by producing a TAG_FINAL chunk even for zero-length
plaintext, so an empty body is a failed transfer rather than an empty
file. Both error messages say the stream was truncated.

Plaintext is now staged in a temporary sibling file (same directory,
so the rename cannot cross a filesystem boundary; random UUID suffix,
so concurrent downloads cannot collide) and renamed into place only
after the whole stream has decrypted and verified. On any error the
temporary file is removed and the original error is rethrown
unchanged, so a cleanup failure never masks the real diagnosis. A
failed download therefore leaves the destination exactly as it was.

Public signatures and the DownloadResult shape are unchanged.

The download layer keeps its no-direct-sodium-import shape: TAG_FINAL
is re-exported from src/crypto as STREAM_TAG_FINAL, which decryptBlob
now uses too. Also moves the pullStreamChunk doc comment off
decryptBlob, where it had been sitting.

Retry and backoff remain out of scope; they stay the Next Step in
TODO.md and are tracked separately.
2026-08-09 01:59:28 +00:00
1f894bad0e Add failing tests for download truncation detection and atomic writes
Covers, for both downloadFile and downloadThumbnail:

- a multi-chunk body whose TAG_FINAL chunk never arrived is rejected
  with a truncation error;
- an empty body is rejected as truncation rather than written as a
  zero-byte file;
- after a truncation or chunk-authentication failure the destination
  path does not exist and no temporary scratch file is left behind;
- an existing file at the destination survives a failed download
  byte for byte, and is replaced atomically by a successful one;
- the existing success cases still produce identical bytes and an
  identical DownloadResult.

Adds an encryptMultiChunkBody helper that frames leading chunks at
exactly STREAM_CHUNK_SIZE so the downloader's fixed-size re-splitting
lines up, plus a multi-chunk success case as the positive control.

Also pins the new STREAM_TAG_FINAL crypto export against libsodium's
own constant, since it must be declared as a literal: libsodium
attaches its constants only after sodium.ready resolves, well after
this library's modules are evaluated.

These fail until the implementation lands, per the repo's TDD workflow.
2026-08-09 01:53:43 +00:00
2039608c07 Refresh vendored REPO_POLICIES.md from prompts repo
All checks were successful
check / check (push) Successful in 4s
2026-07-07 00:21:00 +02:00
4f506b0155 Adopt scripts-to-rule-them-all: script/ entrypoints, Makefile shims 2026-07-07 00:20:19 +02:00
dc0dd11f19 Format TODO.md with prettier (make fmt)
Some checks failed
check / check (push) Failing after 5s
2026-07-06 21:28:18 +02:00
84554a85ad Add standard Workflow section to TODO.md
Some checks failed
check / check (push) Failing after 6s
2026-07-06 21:06:40 +02:00
88510a3ff5 Add TODO.md
Some checks failed
check / check (push) Failing after 5s
2026-07-06 20:35:46 +02:00
6c26e3ccb7 Merge: skip deleted-collection tombstones in listCollections 2026-06-10 11:49:48 -07:00
d0b4ee979e Green: filter isDeleted tombstones out of listCollections 2026-06-10 11:49:45 -07:00
cb9ac29cb4 Red: listCollections must drop deleted-collection tombstones
/collections/v2 is a sync API: deleted collections remain in the
response forever with isDeleted: true, and their /collections/v2/diff
endpoint returns HTTP 404. A long-lived account accumulates hundreds
of tombstones, so any caller that iterates listCollections() output
(backup, backup-metadata) dies on the first one.
2026-06-10 11:49:11 -07:00
6a9e41a2ee Merge: decrypt collections shared by other users (sealed-box keys) 2026-06-10 11:46:54 -07:00
15d2effc2d Green: unseal shared collection keys with the account keypair
decryptCollection now takes the full key material {masterKey,
publicKey, secretKey} and dispatches on keyDecryptionNonce: present
means an owned collection (secretbox under the master key), absent
means a shared collection (sealed box to our public key). Client
already held the keypair for unsealing the auth token, so it just
passes it through.
2026-06-10 11:46:51 -07:00
59e0aa7d47 Red: shared collections arrive as sealed boxes with no keyDecryptionNonce
Collections shared with the account are not encrypted with the master
key: the sharer only knows the recipient's public key, so the server
delivers encryptedKey as crypto_box_seal to that key and omits
keyDecryptionNonce entirely. decryptCollection assumed the owned-only
wire format and crashed on fromBase64(undefined) for any account with
an incoming shared album, taking down listCollections and every
command built on it (backup, backup-metadata, ...).

The previous "shared" fixture was unfaithful (secretbox + nonce with a
foreign ownerID, a shape the server never sends), which is why the
suite stayed green. These tests model the real wire format and change
decryptCollection to take the full key material {masterKey, publicKey,
secretKey} so it can unseal shared collection keys.
2026-06-10 11:41:36 -07:00
b86ac2cd20 Merge: fix dual-2FA login against real server (empty-string fields) 2026-06-10 11:26:18 -07:00
68d8cfb7fe Green: treat empty-string 2FA session fields as absent
Use || instead of ?? when picking the TOTP session ID. The server
sends "" for unset 2FA fields (no omitempty), and "" ?? v2
short-circuits to "", which made dual-2FA accounts fall through to
the unsupported passkey branch.
2026-06-10 11:26:15 -07:00
2c51074294 Red: mock server must serialize empty 2FA fields like Go does
The museum server's EmailAuthorizationResponse declares
passkeySessionID, accountsUrl, twoFactorSessionID, and
twoFactorSessionIDV2 without `omitempty`, so Go always sends them,
as "" when unset. The previous mock omitted the unset fields
entirely, which let the ?? -based dispatch pass in tests while the
real server's "" defeated it and dual-2FA logins fell through to
the unsupported passkey branch.
2026-06-10 11:25:25 -07:00
0d0dcf5987 Merge: TOTP login for accounts that also have passkeys
The server signals dual-2FA accounts with passkeySessionID +
twoFactorSessionIDV2. quak now takes the TOTP path in that case
instead of failing with 'Passkey authentication is not supported'.
2026-06-10 11:05:39 -07:00
9fd7b6a857 Green: prefer TOTP via twoFactorSessionIDV2 when passkey also enrolled
beginLogin now checks twoFactorSessionID ?? twoFactorSessionIDV2 before
the passkey branch. Accounts with both passkeys and TOTP can now log in
from the CLI using their authenticator app.
2026-06-10 11:05:36 -07:00
0024631ef3 Red: test TOTP preference when account has both passkey and TOTP
When an account has both passkeys and TOTP enrolled, the Ente server
returns passkeySessionID + twoFactorSessionIDV2 (deliberately not the
V1 twoFactorSessionID field, so old clients keep using passkeys).
quak only checked the V1 field, saw the passkey session, and threw
'Passkey authentication is not supported'. A CLI cannot do WebAuthn,
so it must take the TOTP path via the V2 session ID.
2026-06-10 11:04:49 -07:00
3d76bd092f Use @inquirer/prompts for interactive login input
Replaces the hand-rolled readline/raw-mode password prompt with
@inquirer/prompts (input + password). The manual approach broke in
bun-compiled binaries because bun doesn't properly re-open stdin
after closing a readline instance. inquirer handles TTY, raw mode,
and password masking correctly across both node and bun runtimes.
2026-06-10 10:54:17 -07:00
3d6742945b Add make install: copies compiled binary to ~/bin 2026-06-10 10:45:59 -07:00
6171d275e9 Merge: replace sharp with pure JS, add single-binary build
Some checks failed
check / check (push) Failing after 7s
Removes the only native dependency (sharp). All image processing
now uses jpeg-js (pure JS JPEG codec) + a bilinear RGBA resize for
thumbnail generation, and raw JPEG APP1/XMP byte parsing + exif-reader
for EXIF extraction. Every dependency is now pure JS or Emscripten.

make build-bin compiles the entire CLI into a 59MB self-contained
binary via bun build --compile (bun from nix-shell). No runtime
dependencies needed to run the binary.
2026-06-10 10:44:37 -07:00
25d3c612cf Replace sharp with jpeg-js + exif-reader; add bun compile binary
sharp was the only native dependency preventing a single-file binary.
Replaced with:
  - jpeg-js (pure JS) for JPEG decode/resize/encode in thumbnail gen
  - exif-reader (pure JS) for EXIF tag parsing
  - Raw JPEG APP1 marker extraction for EXIF segment discovery
  - Raw XMP packet extraction from file bytes

make build-bin produces a ~59MB self-contained Mach-O binary via
bun build --compile (bun installed via nix-shell). Zero runtime
dependencies. Tested: login, whoami, collections, files all work
from the compiled binary.

bin/quak.ts: init() called once at program start before commander
parses, so libsodium is ready for all commands including those that
restore sessions from disk.

118 tests pass.
2026-06-10 10:44:26 -07:00
5e6069f574 ML data always included in backup-metadata; remove --no-ml
ML metadata (face detections, CLIP embeddings) is not a separate
category from the rest of the metadata. It is always fetched and
included. The only opt-in is --exif (or --all) which requires
downloading every file for EXIF extraction.
2026-06-09 17:42:30 -04:00
21a1a78f07 ML data included by default, --exif is the opt-in, --all aliases --exif
ML data (face detections, CLIP embeddings) is now fetched by default
in backup-metadata. Use --no-ml to skip it. EXIF extraction (which
requires downloading every file) remains opt-in via --exif. --all is
an alias for --exif.
2026-06-09 17:38:15 -04:00
8cd57f4d12 Merge: backup-metadata --ml and --exif flags
--ml: fetches face detections (bounding boxes, landmarks, embeddings)
and CLIP search embeddings from the /files/data/fetch endpoint. These
are encrypted with the file key and gzipped; quak decrypts and
decompresses them into the per-file JSON output.

--exif: downloads each original file, extracts full image metadata
via sharp (format, dimensions, color space, orientation) and parses
raw EXIF tags via exif-reader (lens, ISO, shutter, aperture, GPS
altitude, software, etc.). Also captures IPTC, XMP, and ICC data.

3 new tests. 119 total, all green.
2026-06-09 17:35:44 -04:00
c8e7971445 Add --ml and --exif flags to backup-metadata
--ml fetches face detections and CLIP embeddings from the /files/data/fetch
endpoint (type 'mldata'). Each blob is encrypted with the file's key and
gzipped; we decrypt with decryptBlob, gunzip, and include the parsed JSON
as 'mlData' in the per-file output. Fetched in batches of 200 file IDs.

--exif downloads each file, runs sharp().metadata() to extract image
properties (format, dimensions, color space, orientation), then parses
the raw EXIF buffer with exif-reader for structured tags (lens, ISO,
shutter, aperture, GPS altitude, etc.). Also captures raw IPTC, XMP,
and ICC profile data. Included as 'imageMetadata' in the per-file output.

Without either flag, behavior is unchanged (fast metadata-only dump).

Adds exif-reader 2.0.3 as a runtime dependency.
3 new tests (ML data decrypted, ML data absent when flag not set, EXIF
extraction). 119 total tests, all green.
2026-06-09 17:35:35 -04:00
73bfec5a9e Merge: quak backup-metadata command
Dumps all decrypted account metadata to a directory tree of plain JSON
files. No file content downloads. Includes collection-level magic
metadata decryption (visibility, sort order, cover photo) which was
previously missing. 6 new tests, 116 total.
2026-06-09 12:41:43 -04:00
f3958e911d Add quak backup-metadata: dump all decrypted metadata to plain JSON
New command: quak backup-metadata <dir>

Dumps every piece of decrypted account metadata into a directory tree
of plain JSON files without downloading any file content. Layout:

    <dir>/
        account.json                    { email, userID }
        collections/
            <id>-<name>/
                _collection.json        { id, name, type, pubMagicMetadata?, ... }
                <fileID>.json           { id, metadata, magicMetadata?, pubMagicMetadata? }

Also adds collection-level magic metadata decryption (magicMetadata,
pubMagicMetadata, sharedMagicMetadata) to decryptCollection, which was
previously only done for files. The server sends these for visibility
settings, sort order, cover photo selection, etc.

6 new tests covering: account.json, per-collection dirs with
_collection.json, collection pubMagicMetadata decryption, per-file
JSON with all three metadata layers, graceful handling of files with
no magic metadata, and incremental re-run safety. 116 total.
2026-06-09 12:41:34 -04:00
6729e8bdc3 Merge: README rewrite to match current implementation 2026-06-09 12:35:47 -04:00
16ea7b1f03 Rewrite README to match current implementation
Fixes accumulated drift from the original spec-first README:

- Intro: added paragraph describing backup, metadata decryption, and
  thumbnail repair capabilities
- Rationale: removed the 'deliberately scoped to read operations'
  claim (no longer true since thumbnail upload exists); called out
  the Go CLI's crash-on-failure bug as explicit motivation
- Getting Started: fixed library example to use actual Client.login()
  API, removed nonexistent fromSavedSession/getFile methods, added
  backup CLI example
- Layout: fixed to match actual directory structure (download/,
  backup.ts, thumbnails.ts; removed nonexistent session/)
- Session handling: replaced the fictional encrypted-keychain
  SessionStore description with the actual implementation (plain JSON
  via env-paths, consumer-managed persistence via toJSON/fromJSON)
- CLI surface: added backup, helper list-missing-thumbnails, and
  helper fix-missing-thumbnails commands
- Backup layout: new section documenting the originals/ + collections/
  symlink structure
- API reference: replaced the stale type declarations with pointers
  to the actual source files and a note that test/client/usage.test.ts
  is the authoritative API tutorial
- TODO: collapsed completed phases, kept only open items
- For LLMs: new section summarizing repo policies, TDD workflow,
  required checks, formatting rules, and pointers to REPO_POLICIES.md
  and LLM_PROSE_TELLS.md
2026-06-09 12:35:40 -04:00
ebd247696b Merge: thumbnail helpers with tests
Some checks failed
check / check (push) Failing after 6s
quak helper list-missing-thumbnails: scans all files, fetches each
thumbnail, reports missing/empty ones with deduplication.

quak helper fix-missing-thumbnails: downloads originals, generates
720px JPEG via sharp, encrypts with secretstream push (encryptBlob),
uploads via presigned URL, registers via PUT /files/thumbnail.

New crypto: encryptBlob (secretstream push, single chunk TAG_FINAL).
New ApiClient: putJSON, putFile (no auth headers for S3), getUploadURL,
updateThumbnail. New Client: getApiClient() accessor.

20 new tests covering: encryptBlob round-trips and edge cases (8),
upload API methods including putFile auth-header leakage check (4),
listMissingThumbnails detection and dedup (2), fixMissingThumbnails
full pipeline with JPEG magic byte verification on decrypted upload (3),
getApiClient logged-in/logged-out behavior (2), error resilience (1).

110 total tests, all green.
2026-06-09 12:29:41 -04:00
6cb679d62f Add tests for thumbnail-helpers branch (20 new tests)
test/crypto/encrypt-blob.test.ts (8 tests):
  Round-trip with decryptBlob, zero-length payload, ciphertext
  overhead check, header size check, different keys produce different
  output, same key produces different output each call (random nonce),
  wrong-key rejection, tamper detection.

test/api/upload.test.ts (4 tests):
  putJSON sends PUT with auth headers and JSON body. putFile sends
  PUT to the exact presigned URL with Content-Type octet-stream and
  does NOT send X-Auth-Token or X-Client-Package (S3 would reject
  them). getUploadURL POSTs with contentLength and contentMD5.
  updateThumbnail PUTs to /files/thumbnail with correct body shape.

test/thumbnails/thumbnails.test.ts (8 tests):
  listMissingThumbnails identifies empty (0 byte) and 404 thumbnails
  while ignoring working ones; deduplicates across collections.
  fixMissingThumbnails verifies the full pipeline: download original,
  generate JPEG via sharp, encrypt with encryptBlob, upload via
  presigned URL, register via PUT /files/thumbnail. The test
  decrypts the uploaded ciphertext and verifies it starts with JPEG
  magic bytes (FF D8 FF). Also tests: nonexistent file ID reports
  failure without crashing; mixed success/failure across multiple
  files; Client.getApiClient() works when logged in, throws after
  logout.
2026-06-09 12:29:24 -04:00
50 changed files with 5872 additions and 987 deletions

View File

@@ -6,4 +6,4 @@ jobs:
steps: steps:
# actions/checkout v4.2.2, 2026-02-22 # actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- run: docker build . - run: script/cibuild

3
.gitignore vendored
View File

@@ -30,6 +30,9 @@ coverage/
*.pem *.pem
*.key *.key
# Compiled binary (built by make build-bin)
bin/quak
# quak runtime data (in case anyone runs the CLI from inside the repo) # quak runtime data (in case anyone runs the CLI from inside the repo)
.quak/ .quak/

View File

@@ -1,11 +1,10 @@
# node 22-alpine, 2026-02-22 # node 22-alpine, 2026-02-22
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34
RUN apk add --no-cache make
WORKDIR /app WORKDIR /app
COPY script/ script/
COPY package.json yarn.lock ./ COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile RUN script/bootstrap
COPY . . COPY . .
RUN make check RUN make check

View File

@@ -1,31 +1,39 @@
.PHONY: test lint fmt fmt-check check build dev clean docker hooks .PHONY: bootstrap setup test lint fmt fmt-check check build build-bin install dev clean docker hooks
# Use `timeout` (GNU coreutils) when available so `make test` is hard-capped.
# On macOS without coreutils this is empty and the cap is skipped.
TIMEOUT := $(shell command -v timeout 2>/dev/null || command -v gtimeout 2>/dev/null)
YARN := yarn run YARN := yarn run
bootstrap:
@script/bootstrap
setup:
@script/setup
test: test:
@$(TIMEOUT) $(if $(TIMEOUT),30s,) $(YARN) vitest run --reporter=dot || \ @script/test
{ echo "--- Rerunning with verbose for details ---"; \
$(YARN) vitest run --reporter=verbose; exit 1; }
lint: lint:
@$(YARN) eslint . @script/lint
@$(YARN) prettier --check .
fmt: fmt:
@$(YARN) prettier --write . @script/fmt
fmt-check: fmt-check:
@$(YARN) prettier --check . @script/fmt-check
check: test lint fmt-check check:
@script/check
build: build:
@$(YARN) tsc @$(YARN) tsc
build-bin:
nix-shell -p bun --run "bun build bin/quak.ts --compile --outfile bin/quak"
install: build-bin
mkdir -p ~/bin
cp bin/quak ~/bin/quak
chmod +x ~/bin/quak
dev: dev:
@$(YARN) tsc --watch @$(YARN) tsc --watch
@@ -33,13 +41,7 @@ clean:
@rm -rf dist coverage .vitest-cache *.tsbuildinfo @rm -rf dist coverage .vitest-cache *.tsbuildinfo
docker: docker:
docker build -t quak . @script/docker
hooks: hooks:
@printf '#!/bin/sh\nset -e\nmake lint\nmake fmt-check\n' > .git/hooks/pre-commit @script/install-precommit
@chmod +x .git/hooks/pre-commit
@echo "Installed pre-commit hook (runs make lint && make fmt-check)."
@echo "Note: tests are deliberately not in the pre-commit hook so the"
@echo "TDD red-phase commit (failing tests, no implementation yet)"
@echo "can land. CI runs make check via docker build, which catches"
@echo "any branch that ships red."

797
README.md
View File

@@ -5,6 +5,15 @@ quak is a WTFPL-licensed TypeScript client library and CLI by
encrypted photo hosting service. It logs in, enumerates collections and files, encrypted photo hosting service. It logs in, enumerates collections and files,
and downloads individual images while decrypting them on the way to disk. and downloads individual images while decrypting them on the way to disk.
quak also includes a resilient backup command that downloads every file in the
account into a deduplicated local directory tree, skipping files that already
exist on disk and continuing past individual download failures instead of
crashing. It decrypts and persists all three metadata layers (basic, private
magic, public magic) per file, including camera info, GPS coordinates, captions,
and any face/keyword labels the Ente clients have added. A helper subcommand can
detect and regenerate missing thumbnails, encrypting and uploading them back to
the server.
## Getting Started ## Getting Started
```bash ```bash
@@ -14,7 +23,6 @@ yarn install
yarn build yarn build
# Log in (prompts for email, password, and OTP/TOTP if required). # Log in (prompts for email, password, and OTP/TOTP if required).
# Stores an encrypted session under $XDG_CONFIG_HOME/quak/.
yarn quak login yarn quak login
# List the user's collections (albums). # List the user's collections (albums).
@@ -23,8 +31,11 @@ yarn quak collections
# List files in a collection. # List files in a collection.
yarn quak files --collection 12345 yarn quak files --collection 12345
# Download and decrypt a single file to ./out/. # Download and decrypt a single file.
yarn quak get 67890 --out ./out/ yarn quak get 67890 --out ./photo.jpg
# Back up every file in the account.
yarn quak backup ./my-backup
``` ```
For library use: For library use:
@@ -32,14 +43,63 @@ For library use:
```ts ```ts
import { Client } from "quak"; import { Client } from "quak";
const client = await Client.fromSavedSession(); const client = await Client.login({
email: "you@example.com",
password: "your-password",
});
for (const c of await client.listCollections()) { for (const c of await client.listCollections()) {
console.log(c.id, c.name); console.log(c.id, c.name);
const files = await client.listFiles(c.id, c.key);
for (const f of files) {
console.log(` ${f.metadata.title} [${f.metadata.fileType}]`);
}
} }
const file = await client.getFile(67890);
await client.downloadFile(file, "./out/"); // Download a file
const files = await client.listFiles(collectionID, collectionKey);
await client.downloadFile(files[0], "./photo.jpg");
// Serialize session for later (consumer handles persistence)
const snapshot = client.toJSON();
// ... later:
const restored = Client.fromJSON(snapshot);
``` ```
## Entrypoints
This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call them.
The scripts are POSIX sh (not bash) so they run in minimal containers such as
alpine. We provide:
- `script/bootstrap` — install all dependencies (node/yarn if missing, then
`yarn install --frozen-lockfile`)
- `script/setup` — set up the repo for development after a fresh clone: runs
`script/bootstrap`, then `script/install-precommit`
- `script/projectname` — output the project name (our own extension); used by
`script/docker` for the image tag
- `script/test` — run the test suite (vitest, hard-capped at 30s where `timeout`
is available, verbose rerun on failure)
- `script/lint` — run eslint and a prettier check
- `script/fmt` — format all files with prettier (writes)
- `script/fmt-check` — check formatting (read-only)
- `script/check` — run all checks: `test`, `lint`, `fmt-check` (our own
extension)
- `script/docker` — build the Docker image, tagged via `script/projectname`
(byte-identical across repos)
- `script/cibuild` — cd to the repo root and `docker build .` (what CI runs; the
image build runs `make check`)
- `script/precommit` — run by the git pre-commit hook (our own extension); runs
`script/lint` and `script/fmt-check` but deliberately not the tests, so the
TDD red-phase commit can land
- `script/install-precommit` — installs the git pre-commit hook (our own
extension); `make hooks` shims to it
`make hooks` installs the pre-commit hook that runs `script/precommit`.
## Rationale ## Rationale
Ente is one of very few photo services with a credible end-to-end encryption Ente is one of very few photo services with a credible end-to-end encryption
@@ -48,11 +108,13 @@ CLI) work, but they are slow, buggy, and difficult to script against. The
Flutter app fails to sync reliably. The web app is heavy. The desktop app is the Flutter app fails to sync reliably. The web app is heavy. The desktop app is the
web app inside a slow Electron wrapper. The Go CLI is the closest thing to a web app inside a slow Electron wrapper. The Go CLI is the closest thing to a
usable tool, but it is awkward to integrate from anything that is not a shell. usable tool, but it is awkward to integrate from anything that is not a shell.
The Go CLI's backup mode crashes entirely when a single file download fails,
which makes it useless as an actual backup tool.
quak is the first step in fixing that. This repo ships a small, correct, quak fixes these problems. This repo ships a correct, well-tested implementation
well-tested implementation of Ente's cryptographic protocol and its read-only of Ente's cryptographic protocol and API surface, plus a CLI that proves the
API surface, plus a CLI that proves the library is enough to do real work library is enough to do real work without a UI. The backup command is resilient
without a UI. by design: per-file errors are logged and the run continues.
The longer-term goal of this project is a simple desktop client for Ente, built The longer-term goal of this project is a simple desktop client for Ente, built
on this library in Electron (or a comparable runtime), with two priorities above on this library in Electron (or a comparable runtime), with two priorities above
@@ -60,11 +122,6 @@ everything else: correctness and stability. Performance and simplicity follow
from those. Features will be added only after the protocol layer is correct, the from those. Features will be added only after the protocol layer is correct, the
local cache is reliable, and the UI is responsive on a five-year-old laptop. local cache is reliable, and the UI is responsive on a five-year-old laptop.
This first release is deliberately scoped to read operations: log in, walk the
account, decrypt and save files. Upload, sharing, deletion, and bidirectional
sync are out of scope. Adding them later is straightforward; doing them right
requires the protocol layer to be correct first.
## Development workflow ## Development workflow
All work on quak is test-driven. No exceptions. All work on quak is test-driven. No exceptions.
@@ -90,11 +147,11 @@ All work on quak is test-driven. No exceptions.
test-then-implementation sequence into reviewable commits, but the final test-then-implementation sequence into reviewable commits, but the final
history must still show tests landing before (or with) the matching history must still show tests landing before (or with) the matching
implementation. implementation.
8. The pre-commit hook installed by `make hooks` runs 8. The pre-commit hook installed by `make hooks` runs `script/precommit`, which
`make lint && make fmt-check`, not the full `make check`. This is deliberate runs the lint and format checks but not the full `make check`. This is
so the TDD red-phase commit (failing tests, no implementation yet) can land. deliberate so the TDD red-phase commit (failing tests, no implementation yet)
The full `make check` runs as part of `docker build .`, which is what CI can land. The full `make check` runs as part of `docker build .`, which is
executes, so a red branch still cannot reach `main`. what CI executes, so a red branch still cannot reach `main`.
## Design ## Design
@@ -107,14 +164,18 @@ the CLI is for humans.
quak/ quak/
src/ src/
crypto/ libsodium primitives (boxes, secretstreams, KDF, SRP) crypto/ libsodium primitives (boxes, secretstreams, KDF, SRP)
api/ HTTP client + typed endpoint wrappers api/ HTTP client (ApiClient class)
auth/ login flow (SRP + email OTP + TOTP), key unwrap auth/ login flow (SRP + email OTP + TOTP), key unwrap
model/ decrypted Collection, File, Metadata types model/ decrypted Collection, File, Metadata types + decrypt fns
session/ on-disk session persistence (token + master key) download/ streaming file/thumbnail download + decryption
backup.ts resilient full-account backup with dedup
errors.ts error types shared across layers
retry.ts retry classifier + exponential backoff with jitter
thumbnails.ts detect + regenerate missing thumbnails
client.ts high-level Client class assembled from the above client.ts high-level Client class assembled from the above
index.ts public library exports index.ts public library exports
bin/ bin/
quak.ts CLI entrypoint (commander.js) quak.ts CLI entrypoint (commander.js)
test/ unit + integration tests (vitest) test/ unit + integration tests (vitest)
Makefile Makefile
Dockerfile Dockerfile
@@ -150,9 +211,13 @@ The key hierarchy, derived during login, is:
Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the
master key (for owned collections). Per-file keys are decrypted with master key (for owned collections). Per-file keys are decrypted with
`crypto_secretbox_open_easy` using the collection key. File metadata is a `crypto_secretbox_open_easy` using the collection key. File metadata is a
secretbox under the file key. File content is a chunked secretstream blob (single chunk, TAG_FINAL) under the file key. File content is
`crypto_secretstream_xchacha20poly1305` stream under the file key, with a 4 MiB a chunked `crypto_secretstream_xchacha20poly1305` stream under the file key,
plaintext chunk size and a 17-byte authentication overhead per chunk. with a 4 MiB plaintext chunk size and a 17-byte authentication overhead per
chunk. Thumbnails use the same secretstream blob format as metadata.
For upload (thumbnail repair), `encryptBlob` performs the push side: a single
secretstream chunk with TAG_FINAL, returning the header and ciphertext.
### HTTP API ### HTTP API
@@ -163,8 +228,8 @@ Production endpoints:
- Thumbnail CDN: `https://thumbnails.ente.io/?fileID=<id>` - Thumbnail CDN: `https://thumbnails.ente.io/?fileID=<id>`
A custom API endpoint is configurable for self-hosted servers via the A custom API endpoint is configurable for self-hosted servers via the
`ENTE_API_ENDPOINT` environment variable. When set, file downloads route through constructor option `apiOrigin`. When set, file downloads route through
`<endpoint>/files/download/<id>` instead of the dedicated CDN host. `<apiOrigin>/files/download/<id>` instead of the dedicated CDN host.
Required request headers on every authenticated call: Required request headers on every authenticated call:
@@ -184,514 +249,184 @@ Endpoints used:
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a - `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
collection; paginate while `hasMore` is true. collection; paginate while `hasMore` is true.
- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes. - `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes.
- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair).
- `PUT /files/thumbnail`: register an uploaded thumbnail's object key.
### Session persistence ### Retries and timeouts
After login, quak writes an encrypted session blob to Every request in the library goes through one policy, in `src/retry.ts`. A
`$XDG_CONFIG_HOME/quak/session.json` (default `~/.config/quak/session.json`) request is repeated only when repeating it could produce a different answer:
containing the auth token, the user's master key, the user's secret key, and the
user's email. The session file is itself encrypted with a key derived from a - `ApiError` with a 5xx status: retried. So are `408` and `429`, the two 4xx
per-machine random value stored in the OS keychain when available, falling back codes that are statements about timing rather than about the request.
to a key file at mode `0600` in the same config directory. The master key and - Every other 4xx: not retried. A 404 in particular is an answer, and
secret key are never written to disk in cleartext. `listMissingThumbnails` depends on getting it promptly and once.
- Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or
TLS failure — and deadline aborts: retried. The errno is looked for in the
error's `cause` chain, because that is where Node's `fetch` puts it.
- A truncated download: retried.
- Anything else, including a secretstream authentication failure that is not
truncation: not retried. The default answer is no. For a backup tool, retrying
a permanent failure spends round trips and delays every remaining file, while
declining to retry a transient one costs a single file that the next run picks
up.
Backoff is exponential with full jitter: the delay before retry _n_ is
`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term
is the ceiling and the wait is drawn below it, so a client that lost many
parallel downloads to one CDN blip does not send them all again at the same
instant. Defaults, configurable through `ApiClientOptions.retry`:
| Option | Default | Meaning |
| ------------- | ------- | ----------------------------------- |
| `attempts` | `4` | total calls, not retries |
| `baseDelayMs` | `500` | ceiling for the first retry's delay |
| `maxDelayMs` | `10000` | upper bound on that ceiling |
With those defaults a file that is going to fail gives up after at most three
and a half seconds of waiting. `sleep` and `random` are injectable through the
same option, which is how the test suite exercises the whole policy without
waiting.
Two deadlines, applied with `AbortSignal.timeout()` and renewed for each
attempt:
| Option | Default | Applies to |
| ------------------- | -------- | ------------------------------------------- |
| `requestTimeoutMs` | `30000` | `getJSON`, `postJSON`, `putJSON`, `putFile` |
| `downloadTimeoutMs` | `600000` | file and thumbnail body transfers |
They are separate because one number cannot serve both: a value short enough to
keep a hung API call from stalling a backup would cancel a legitimate
multi-gigabyte download. The download deadline covers the body, not just the
headers — `getFileStream` returns as soon as headers arrive, so a deadline that
only guarded the initial request would leave the same hang one layer down.
**Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON`
reach `/users/srp/create-session`, `/users/two-factor/verify` — which consumes
one of a small number of second-factor attempts — and `/files/thumbnail`. They
are retried only on failures that prove no request byte reached the server,
which means the connection was never established (`ECONNREFUSED`, `ENOTFOUND`,
and the like). A 5xx, a mid-flight reset and a deadline are all left to the
caller, because each of them can happen after the server has already acted.
`putFile` is exempt: a presigned PUT stores one whole object at one key in one
request, so replaying it has no partial state to damage.
A download is retried as a whole — request, stream consumption, and decryption —
because a socket reset after the response headers have arrived surfaces in the
download layer rather than in `ApiClient`, and that is the common failure for
multi-megabyte photos over a CDN. The secretstream pull state is not resumable
and these endpoints have no Range support, so a retry starts the file over. The
atomic write stays outside the retry, so a download that needed three attempts
still performs exactly one write and one rename. `runBackup` and
`runMetadataBackup` are unchanged: the retry sits below them, and a file that
fails after exhausting it is still logged, counted, and stepped over.
One imprecision is deliberate and worth knowing about. When a body ends part-way
through a secretstream chunk, Poly1305 fails and carries no framing signal, so a
cut connection and genuinely corrupt bytes are indistinguishable. quak reports
that as truncation, which means it is retried. For a body of more than one chunk
the distinction is real — a chunk that failed while the stream carried on past
it stays an authentication failure and is not retried — but for a single-chunk
body, which is most thumbnails and every small file, a wrong key, server-side
corruption and a mid-chunk cutoff all present alike and all get retried. The
cost is bounded by the attempt count, and it buys never silently keeping a
truncated file.
### Session handling
The `Client` class holds the auth token, master key, secret key, and public key
in memory. There is no on-disk session store in the library; the consumer
decides how to persist sessions.
`client.toJSON()` returns a `ClientSnapshot` (a plain serializable object with
base64-encoded keys) that the consumer can write to disk, a database, or
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
working client from that snapshot without re-authenticating.
The CLI stores the snapshot at the platform-appropriate data directory via
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
`$XDG_DATA_HOME/quak/session.json` on Linux. The file is written with mode
`0600`. The key material is stored in cleartext in the JSON; treat this file as
you would treat the password itself.
### CLI surface ### CLI surface
- `quak login`: interactive login, writes session. ```
- `quak logout`: deletes the session. quak login interactive or QUAK_EMAIL/QUAK_PASSWORD
- `quak whoami`: prints the logged-in email. quak whoami print logged-in account as JSON
- `quak collections`: list collections (id, name, type, file count). quak logout delete saved session
- `quak files --collection <id>`: list files in a collection (id, name, type, quak collections [--json] list all collections
creation time, size). quak files --collection <id> [--json] list files in a collection
- `quak get <fileID> --out <dir>`: download and decrypt a file. quak get <fileID> [--out path] [--collection] download and decrypt a file
- `quak get-thumb <fileID> --out <dir>`: download and decrypt a thumbnail. quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
quak backup <dir> [--json] full incremental backup
All commands accept `--json` for machine-readable output. quak helper list-missing-thumbnails [--json] find files with missing thumbnails
quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails
## API reference
The complete public surface of the library, expressed as TypeScript
declarations. Every exported name is listed here. Anything not listed is
internal.
### Type aliases
```ts
// src/model/types.ts
export type Bytes = Uint8Array;
export type Base64 = string; // standard base64 unless noted
export type Base64URL = string; // URL-safe base64
export type Microseconds = number; // unix epoch microseconds (int64-ish)
``` ```
### Crypto module `get` and `get-thumb` search all collections for the file ID when `--collection`
is not specified. All listing and backup commands support `--json` for
machine-readable output.
```ts ### Backup layout
// src/crypto/index.ts
// Lazily initializes libsodium. Safe to call repeatedly; the first call `quak backup <dir>` produces:
// performs the init, subsequent calls are no-ops.
export function init(): Promise<void>;
// Argon2id over a UTF-8 password and a 16-byte salt, producing a 32-byte ```
// key. memLimit is in bytes, opsLimit is the iteration count, both as <dir>/
// returned by the server in SRP / key attributes. originals/
export function deriveKEK( <fileID>.<ext> actual file content (one per unique file)
password: string, <fileID>.json all decrypted metadata for that file
salt: Bytes, collections/
opsLimit: number, <name>/
memLimit: number, <title> -> ../../originals/<fileID>.<ext> (symlink)
): Promise<Bytes>; <name>.json collection metadata + file list
// crypto_kdf_derive_from_key with subkey id 1 and context "loginctx",
// returning the first 16 bytes. Used as the SRP password.
export function deriveLoginSubkey(kek: Bytes): Bytes;
// crypto_secretbox_open_easy. Returns plaintext or throws on auth failure.
export function decryptBox(ciphertext: Bytes, nonce: Bytes, key: Bytes): Bytes;
// crypto_box_seal_open. Used to recover the auth token after login.
export function decryptSealed(
ciphertext: Bytes,
publicKey: Bytes,
secretKey: Bytes,
): Bytes;
// crypto_secretstream_xchacha20poly1305_init_pull. Returned state is opaque
// and threaded through pullStreamChunk.
export function initStreamPull(header: Bytes, key: Bytes): StreamPullState;
// crypto_secretstream_xchacha20poly1305_pull. Tag values follow libsodium's
// constants: 0=MESSAGE, 1=PUSH, 2=REKEY, 3=FINAL.
export function pullStreamChunk(
state: StreamPullState,
ciphertext: Bytes,
): { plaintext: Bytes; tag: number };
// Convenience helpers. fromBase64 accepts both standard and URL-safe.
export function fromBase64(s: Base64 | Base64URL): Bytes;
export function toBase64(b: Bytes): Base64;
export function toBase64URL(b: Bytes): Base64URL;
// Plaintext chunk size used by Ente for file content streams.
export const STREAM_CHUNK_SIZE: number; // 4 * 1024 * 1024
// Encrypted-chunk overhead: secretstream auth tag (16) + tag byte (1).
export const STREAM_CHUNK_OVERHEAD: number; // 17
export interface StreamPullState {
/* opaque */
}
``` ```
### Auth module Each file is downloaded exactly once regardless of how many collections it
appears in. On subsequent runs, existing originals are skipped. If a download
```ts fails, the error is logged and the backup continues with the next file. The exit
// src/auth/types.ts code is non-zero if any files failed.
export interface KDFParams {
kekSalt: Base64;
memLimit: number;
opsLimit: number;
}
export interface KeyAttributes {
kekSalt: Base64;
encryptedKey: Base64;
keyDecryptionNonce: Base64;
publicKey: Base64;
encryptedSecretKey: Base64;
secretKeyDecryptionNonce: Base64;
memLimit: number;
opsLimit: number;
masterKeyEncryptedWithRecoveryKey?: Base64;
masterKeyDecryptionNonce?: Base64;
recoveryKeyEncryptedWithMasterKey?: Base64;
recoveryKeyDecryptionNonce?: Base64;
}
export interface SRPAttributes {
srpUserID: string;
srpSalt: Base64;
memLimit: number;
opsLimit: number;
kekSalt: Base64;
isEmailMFAEnabled: boolean;
}
export interface AuthorizationResponse {
id: number; // user ID
keyAttributes?: KeyAttributes;
encryptedToken?: Base64URL; // sealed-box-encrypted to user's pubkey
twoFactorSessionID?: string;
passkeySessionID?: string;
}
export type LoginChallenge =
| { kind: "complete"; response: AuthorizationResponse }
| { kind: "totp"; sessionID: string }
| { kind: "passkey"; sessionID: string }
| { kind: "emailOTP" };
```
```ts
// src/auth/index.ts
// Begin login. Returns a challenge that tells the caller what to do next:
// supply a TOTP code, supply an email OTP, follow a passkey URL, or stop
// because login is already complete.
export function beginLogin(
api: ApiClient,
email: string,
password: string,
): Promise<LoginChallenge>;
// Submit a TOTP code from an authenticator app. Returns the final
// AuthorizationResponse on success.
export function submitTOTP(
api: ApiClient,
sessionID: string,
code: string,
): Promise<AuthorizationResponse>;
// Request and submit an email-delivered one-time code. Two calls, because
// the first triggers email delivery and the second verifies it.
export function requestEmailOTP(api: ApiClient, email: string): Promise<void>;
export function submitEmailOTP(
api: ApiClient,
email: string,
code: string,
): Promise<AuthorizationResponse>;
// Given an AuthorizationResponse and the user's password, decrypt the master
// key, secret key, and auth token. Throws on bad password or tampered data.
export function unwrapAuth(
response: AuthorizationResponse,
password: string,
): Promise<{
masterKey: Bytes;
secretKey: Bytes;
publicKey: Bytes;
token: string; // base64 URL-safe; goes into X-Auth-Token
}>;
```
### Model module
```ts
// src/model/index.ts
export type CollectionType =
| "album"
| "folder"
| "favorites"
| "uncategorized"
| "unknown";
export interface Collection {
id: number;
ownerID: number;
key: Bytes; // decrypted
name: string; // decrypted
type: CollectionType;
updationTime: Microseconds;
isShared: boolean; // true if owner != current user
}
export type FileType = "image" | "video" | "livePhoto" | "unknown";
export interface FileMetadata {
title: string;
fileType: FileType;
creationTime: Microseconds;
modificationTime: Microseconds;
latitude?: number;
longitude?: number;
hash?: string; // base64 of file SHA256
}
export interface FileBlob {
decryptionHeader: Base64;
size?: number; // size of the encrypted body, if known from server
}
export interface EnteFile {
id: number;
collectionID: number;
ownerID: number;
key: Bytes; // decrypted file key
metadata: FileMetadata;
file: FileBlob;
thumbnail: FileBlob;
updationTime: Microseconds;
}
```
### HTTP client
```ts
// src/api/client.ts
export interface ApiClientOptions {
apiOrigin?: string; // default https://api.ente.io
filesOrigin?: string; // default https://files.ente.io
thumbsOrigin?: string; // default https://thumbnails.ente.io
authToken?: string;
fetch?: typeof fetch; // injectable for tests
userAgent?: string; // default "quak/<version>"
}
export class ApiError extends Error {
readonly status: number;
readonly code?: string;
readonly requestID?: string;
readonly body?: unknown;
}
export class ApiClient {
constructor(opts?: ApiClientOptions);
setAuthToken(token: string): void;
clearAuthToken(): void;
getJSON<T>(
path: string,
query?: Record<string, string | number | undefined>,
): Promise<T>;
postJSON<T>(path: string, body: unknown): Promise<T>;
// Streaming download from the file CDN. Caller is responsible for
// consuming the stream.
getFileStream(fileID: number): Promise<ReadableStream<Uint8Array>>;
getThumbnailStream(fileID: number): Promise<ReadableStream<Uint8Array>>;
}
```
### Session
```ts
// src/session/index.ts
export interface Session {
email: string;
userID: number;
token: string; // base64 URL-safe
masterKey: Bytes; // 32 bytes, never serialized in cleartext
secretKey: Bytes; // 32 bytes, never serialized in cleartext
publicKey: Bytes; // 32 bytes
}
export interface SessionStoreOptions {
path?: string; // default $XDG_CONFIG_HOME/quak/session.json
keychainService?: string; // default "berlin.sneak.quak"
}
export class SessionStore {
constructor(opts?: SessionStoreOptions);
load(): Promise<Session | null>;
save(s: Session): Promise<void>;
clear(): Promise<void>;
}
```
### Client
```ts
// src/client.ts
export interface LoginPrompt {
password: () => Promise<string>;
emailOTP?: () => Promise<string>; // called when account uses email OTP
totp?: () => Promise<string>; // called when TOTP is required
}
export interface ClientOptions extends ApiClientOptions {
sessionStore?: SessionStore;
}
export interface DownloadResult {
path: string;
bytesWritten: number;
}
export class Client {
// Static constructors. Both end with a Client that has a populated
// session and a ready ApiClient.
static login(
email: string,
prompt: LoginPrompt,
opts?: ClientOptions,
): Promise<Client>;
static fromSavedSession(opts?: ClientOptions): Promise<Client>;
readonly api: ApiClient;
readonly session: Readonly<Session>;
// Account.
whoami(): { email: string; userID: number };
saveSession(): Promise<void>;
logout(): Promise<void>; // clears session on disk and in memory
// Collections.
listCollections(opts?: { sinceTime?: Microseconds }): Promise<Collection[]>;
getCollection(id: number): Promise<Collection>;
// Files.
listFiles(
collectionID: number,
opts?: { sinceTime?: Microseconds },
): Promise<EnteFile[]>;
getFile(fileID: number): Promise<EnteFile>;
// Downloads. If outPath is omitted, a path is constructed from the
// decrypted metadata title in the current working directory. If outPath
// is a directory, the filename is taken from the metadata title. If
// outPath is a file path, that file is written.
downloadFile(
file: EnteFile | number,
outPath?: string,
): Promise<DownloadResult>;
downloadThumbnail(
file: EnteFile | number,
outPath?: string,
): Promise<DownloadResult>;
}
```
### Public exports (`src/index.ts`)
```ts
export { Client } from "./client";
export { ApiClient, ApiError } from "./api/client";
export { SessionStore } from "./session";
export * from "./model";
export type {
Session,
LoginPrompt,
ClientOptions,
DownloadResult,
} from "./client";
```
## TODO ## TODO
Phase 1: scaffolding - [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
- [x] `git init`, write README
- [x] Create `initial-scaffolding` feature branch
- [x] Add `LICENSE` (WTFPL), `REPO_POLICIES.md`, `.gitignore`, `.editorconfig`,
`.prettierrc`, `.prettierignore`, `.dockerignore`
- [x] Add `Makefile` with `test`, `lint`, `fmt`, `fmt-check`, `check`, `docker`,
`hooks`, plus `build`, `dev`, `clean`
- [x] Add `Dockerfile` running `make check` against pinned node image
- [x] Add `.gitea/workflows/check.yml` running `docker build .`
- [x] Add `package.json`, `tsconfig.json`, pinned dev versions of `typescript`,
`prettier`, `eslint`, `typescript-eslint`, `vitest`, `@types/node` (the
runtime deps `libsodium-wrappers-sumo`, `secure-remote-password`,
`commander`, etc. land with their respective implementation phases)
- [x] Smoke test: `make check` and `make docker` both pass
Phase 2: crypto primitives
- [x] Wrap libsodium init as an awaitable singleton
- [x] `deriveKEK(password, kekSalt, memLimit, opsLimit)` (Argon2id)
- [x] `deriveLoginSubkey(kek)` (KDF with subkey id 1, context `loginctx`, 16
bytes)
- [x] `decryptBox(ciphertext, nonce, key)` for secretbox
- [x] `decryptSealed(ciphertext, publicKey, secretKey)` for sealed box
- [x] `initStreamPull` and `pullStreamChunk` for chunked secretstream (4 MiB
plaintext chunks, 17-byte overhead)
- [x] Round-trip tests against vectors generated by libsodium directly
- [x] Base64 helpers (`fromBase64`, `toBase64`, `toBase64URL`) accepting all
four sodium variants on input
Phase 3: SRP + auth
- [x] SRP-6a client using `fast-srp-hap` with the 4096-bit group (matching the
upstream Ente web client)
- [x] `beginLogin(api, email, password)` returning a `LoginChallenge`
- [x] `requestEmailOTP` and `submitEmailOTP` for accounts without SRP
- [x] `submitTOTP(api, sessionID, code)`
- [x] `unwrapAuth(response, password)` returning master key, secret key, public
key, and decrypted token (URL-safe-no-padding base64)
- [x] `src/auth/types.ts` with `KeyAttributes`, `SRPAttributes`,
`AuthorizationResponse`, and `LoginChallenge`
- [x] Tests with mock SRP server performing real 4096-bit math end-to-end
Phase 4: HTTP client + endpoints
- [x] `ApiClient` that attaches `X-Auth-Token` and `X-Client-Package`
- [x] `ApiError` that surfaces the server's error code and request id
- [x] `getJSON` / `postJSON` with query-param and JSON-body handling
- [x] `getFileStream` / `getThumbnailStream` with self-hosted routing
- [ ] Typed wrappers for the endpoints listed above
- [ ] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
errors errors
- [ ] Update the API reference section below to match the current implementation
Phase 5: collections and files
- [x] `decryptCollection(raw, masterKey)` with key + name decryption, type
mapping, isShared flag
- [x] `decryptFile(raw, collectionKey)` with key + metadata decryption
(secretstream blob, not secretbox), fileType mapping, header passthrough
- [x] `decryptBlob(ciphertext, header, key)` convenience for single-chunk
secretstream decryption (used by file metadata and magic metadata)
- [x] Model types: `Collection`, `EnteFile`, `FileMetadata`, `RawCollection`,
`RawEnteFile`
- [x] Live-tested against real Ente API (collection names + file metadata)
- [ ] Higher-level `listCollections()` / `listFiles()` with pagination
Phase 6: download
- [x] `downloadFile(api, file, outPath?)` streams encrypted body, buffers to 4
MiB chunk boundary, decrypts via secretstream pull, writes to disk. Falls
back to `metadata.title` when outPath is omitted.
- [x] `downloadThumbnail(api, file, outPath?)` same for thumbnails
- [x] Live integration test: logs in, decrypts collections and files, downloads
a real JPEG from the dev account and verifies it on disk
Phase 7: Client class
- [x] `Client.login({ email, password, totp?, emailOTP? })` performs the full
SRP handshake, key unwrap, returns a ready Client
- [x] `Client.fromJSON(snapshot)` restores from a serialized snapshot
- [x] `client.toJSON()` produces a `ClientSnapshot` the consumer can persist
- [x] `client.whoami()`, `client.logout()`
- [x] `client.listCollections()` with decryption
- [x] `client.listFiles(collectionID, collectionKey)` with pagination
- [x] `client.downloadFile(file, outPath?)` and `client.downloadThumbnail()`
- [x] Literate test/client/usage.test.ts tutorial covering the entire API
Phase 8: CLI
- [x] `quak login` (interactive TTY prompts or QUAK_EMAIL/QUAK_PASSWORD env vars
for non-interactive use)
- [x] `quak whoami`, `quak logout`
- [x] `quak collections` and `quak files --collection <id>`
- [x] `quak get <fileID>` and `quak get-thumb <fileID>` with --out and
--collection options; searches all collections when --collection omitted
- [x] `quak backup <dir>` with originals/ dedup, collections/ symlinks,
per-collection and per-file JSON metadata, incremental skip, per-file
error resilience, --json flag
- [x] `--json` output on every listing/backup command
- [x] Progress output to stderr for backup
- [x] Session persistence via env-paths (~/Library/Application Support/quak/ on
macOS, XDG_DATA_HOME/quak/ on Linux)
Phase 9: docs and 1.0
- [ ] Update README Getting Started and Design sections to match current state
- [ ] All TODO items above checked
- [ ] `make docker` green - [ ] `make docker` green
- [ ] Tag `v1.0.0` - [ ] Tag `v1.0.0`
Phase 10 and beyond: desktop client (separate repo) Future (desktop client, separate repo):
- [ ] Spike Electron app skeleton consuming this library - [ ] Electron app skeleton consuming this library
- [ ] Local cache (SQLite) keyed on `(collectionID, fileID, updationTime)` - [ ] Local cache (SQLite) keyed on `(collectionID, fileID, updationTime)`
- [ ] Background sync worker that streams new files into the cache - [ ] Background sync worker that streams new files into the cache
- [ ] Read-only gallery UI: thumbnails, full-image view, basic search - [ ] Gallery UI: thumbnails, full-image view, basic search
- [ ] Add upload, delete, and share back into the library before the desktop UI - [ ] Upload, delete, and share operations in the library
exposes them
## API reference
The API reference section below is from an earlier draft and does not fully
reflect the current implementation. The authoritative API documentation is in
the test files, particularly `test/client/usage.test.ts` which is a literate
tutorial walking through every operation. Run `yarn test` to verify the examples
are correct.
The key types and their actual signatures can be found in:
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `ApiError`,
`StreamOptions`
- `src/errors.ts`: `ApiError`, `TruncatedStreamError`
- `src/retry.ts`: `withRetry`, `isRetryable`, `isSafeToReplay`, `RetryOptions`
- `src/auth/types.ts`: `KeyAttributes`, `SRPAttributes`,
`AuthorizationResponse`, `LoginChallenge`
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileBlob`,
`RawCollection`, `RawEnteFile`, `RawMagicMetadata`
- `src/download/index.ts`: `DownloadResult`
- `src/backup.ts`: `BackupResult`, `BackupError`
- `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult`
## Source attribution ## Source attribution
@@ -702,6 +437,38 @@ is rewritten in TypeScript in this repository. Protocol fidelity is verified
against the upstream implementations in `web/packages/base/`, against the upstream implementations in `web/packages/base/`,
`mobile/apps/photos/lib/`, and `cli/`. `mobile/apps/photos/lib/`, and `cli/`.
## For LLMs
If you are an LLM agent working on this repository, read and follow these
documents:
- **`REPO_POLICIES.md`** in the repo root. It is copied from
<https://git.eeqj.de/sneak/prompts> and covers repository structure, tooling,
Makefile targets, Dockerfile conventions, dependency pinning, and commit
hygiene. All external dependencies must be pinned by cryptographic hash in
`yarn.lock`. Never `git add -A`. Never force-push to main.
- **The "Development workflow" section above.** All changes go on feature
branches. Tests are written first and committed in a failing state before the
implementation. Tests are the canonical API documentation and must be
commented thoroughly. `main` is always green.
- **Required checks before every commit:** `make lint` (eslint + prettier check)
and `make fmt-check` must pass. The pre-commit hook enforces this.
`make check` (which also runs tests) must pass before merging to `main`.
- **Formatting:** prettier with 4-space indents and `proseWrap: always` for
markdown. Use `make fmt` to format. Use `yarn` not `npm`.
- **Testing:** vitest. Tests go in `test/` mirroring the `src/` structure.
`make test` must complete in under 20 seconds. Use `mkdtempSync` for temporary
directories, never manual timestamp paths.
- **Code style:** `const` for everything, `let` if reassignment is needed, never
`var`. Avoid unnecessary comments. No hand-rolled crypto. The
`LLM_PROSE_TELLS.md` document in the prompts repo applies to any prose written
in this repository (README, comments, commit messages).
## License ## License
WTFPL. See [LICENSE](LICENSE). WTFPL. See [LICENSE](LICENSE).

View File

@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-03-18 last_modified: 2026-07-06
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -34,10 +34,46 @@ style conventions are in separate documents:
every file before committing. There are zero exceptions to this rule. every file before committing. There are zero exceptions to this rule.
- Every repo with software must have a root `Makefile` with these targets: - Every repo with software must have a root `Makefile` with these targets:
`make test`, `make lint`, `make fmt` (writes), `make fmt-check` (read-only), `make bootstrap`, `make setup`, `make test`, `make lint`, `make fmt` (writes),
`make check` (prereqs: `test`, `lint`, `fmt-check`), `make docker`, and `make fmt-check` (read-only), `make check` (runs `test`, `lint`, `fmt-check`),
`make hooks` (installs pre-commit hook). A model Makefile is at `make docker`, and `make hooks` (installs pre-commit hook). A model Makefile
`https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`. is at `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`.
- Repos follow the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
pattern: the implementation of each Makefile target lives in an executable
script in `script/` (`script/bootstrap`, `script/setup`, `script/test`,
`script/lint`, `script/fmt`, `script/fmt-check`, `script/check`,
`script/docker`), and the Makefile targets are thin shims that call them. The
scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they run in
minimal containers (e.g. alpine images have no bash); locate the repo root
with `$(cd "$(dirname "$0")/.." && pwd -P)` and `cd` there before acting. From
the standard's canonical set we use `bootstrap`, `setup` (make the repo ready
for development after a fresh clone: runs `bootstrap`, then
`install-precommit`, plus any repo-specific initialization), `test`, and
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
assumes nothing is present: base tools come from nix, apt, brew, or apk
(detected in that order; apt runs noninteractive). For node it uses the
installed node if present; otherwise it installs a PINNED node version via
nvm, first installing nvm itself if missing — from a hash-verified GitHub
release archive (never `curl | sh`), with bash installed as an explicit
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root and runs `docker build .`; the Gitea workflow calls it. Four further
scripts are our own extensions to the standard: `script/check` runs
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
what the git pre-commit hook runs, and it calls `script/check`;
`script/install-precommit` installs the git pre-commit hook (the `make hooks`
target shims to it); and `script/projectname` (literally that filename) simply
outputs the project's name. Scripts that need the name call
`script/projectname` — e.g. `script/docker` assembles its image tag from it —
so those scripts stay byte-identical across all repos. Repo-type-specific
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/precommit`, not in the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
- Always use Makefile targets (`make fmt`, `make test`, `make lint`, etc.) - Always use Makefile targets (`make fmt`, `make test`, `make lint`, etc.)
instead of invoking the underlying tools directly. The Makefile is the single instead of invoking the underlying tools directly. The Makefile is the single
@@ -57,7 +93,11 @@ style conventions are in separate documents:
as a build step so the build fails if the branch is not green. For non-server as a build step so the build fails if the branch is not green. For non-server
repos, the Dockerfile should bring up a development environment and run repos, the Dockerfile should bring up a development environment and run
`make check`. For server repos, `make check` should run as an early build `make check`. For server repos, `make check` should run as an early build
stage before the final image is assembled. stage before the final image is assembled. Dockerfiles install development
prerequisites by running `script/bootstrap` rather than duplicating installs
inline; COPY `script/` and the dependency manifests (`package.json` +
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
layer stays cached until dependencies change.
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go - **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
repos use a multistage build where linting runs in an independent stage based repos use a multistage build where linting runs in an independent stage based
@@ -127,8 +167,9 @@ style conventions are in separate documents:
artifacts or heavier dependencies. artifacts or heavier dependencies.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `docker build .` on push. Since the Dockerfile already runs `make check`, runs `script/cibuild` (which runs `docker build .`) on push. Since the
a successful build implies all checks pass. Dockerfile already runs `make check`, a successful build implies all checks
pass.
- Use platform-standard formatters: `black` for Python, `prettier` for - Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -136,9 +177,11 @@ style conventions are in separate documents:
Markdown (hard-wrap at 80 columns). Documentation and writing repos (Markdown, Markdown (hard-wrap at 80 columns). Documentation and writing repos (Markdown,
HTML, CSS) should also have `.prettierrc` and `.prettierignore`. HTML, CSS) should also have `.prettierrc` and `.prettierignore`.
- Pre-commit hook: `make check` if local testing is possible, otherwise - Pre-commit hook: runs `script/precommit`, which calls `script/check`. If local
`make lint && make fmt-check`. The Makefile should provide a `make hooks` testing is not possible in the repo, `script/precommit` may skip `script/test`
target to install the pre-commit hook. and run only `script/lint` and `script/fmt-check`. The hook is installed by
`script/install-precommit`; the Makefile must provide a `make hooks` target
that shims to it.
- All repos with software must have tests that run via the platform-standard - All repos with software must have tests that run via the platform-standard
test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful
@@ -297,6 +340,10 @@ style conventions are in separate documents:
"µPaaS is an MIT-licensed Go web application by @sneak that receives "µPaaS is an MIT-licensed Go web application by @sneak that receives
git-frontend webhooks and deploys applications via Docker in realtime." git-frontend webhooks and deploys applications via Docker in realtime."
- **Getting Started**: Copy-pasteable install/usage code block. - **Getting Started**: Copy-pasteable install/usage code block.
- **Entrypoints**: Opens by stating that the repo adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard (with that link), then documents each provided `script/`
entrypoint and its purpose.
- **Rationale**: Why does this exist? - **Rationale**: Why does this exist?
- **Design**: How is the program structured? - **Design**: How is the program structured?
- **TODO**: Update meticulously, even between commits. When planning, put - **TODO**: Update meticulously, even between commits. When planning, put
@@ -351,6 +398,9 @@ style conventions are in separate documents:
- `README.md`, `.git`, `.gitignore`, `.editorconfig` - `README.md`, `.git`, `.gitignore`, `.editorconfig`
- `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo) - `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo)
- `Makefile` - `Makefile`
- `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`,
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`install-precommit`)
- `Dockerfile`, `.dockerignore` - `Dockerfile`, `.dockerignore`
- `.gitea/workflows/check.yml` - `.gitea/workflows/check.yml`
- Go: `go.mod`, `go.sum`, `.golangci.yml` - Go: `go.mod`, `go.sum`, `.golangci.yml`

59
TODO.md Normal file
View File

@@ -0,0 +1,59 @@
# Workflow
- branch (from `main`)
- do the work in Next Step
- move Next Step to the top of Completed Steps
- move the top item of Future Steps into Next Step
- commit (`TODO.md` changes in the same commit as the work)
- merge to `main` if the branch is not protected, otherwise open a PR
- push
# Status
pre-1.0
# Next Step
Update the README API reference section to match the current implementation.
# Completed Steps
- 2026-08-09: Retry policy: no retry on 4xx (except `408` and `429`),
exponential backoff with full jitter on 5xx, transport failures and truncated
transfers, under per-attempt deadlines that cover the response body as well as
the request. Downloads retry request, stream consumption and decryption as one
unit; `postJSON` and `putJSON` are replayed only when the connection was never
established.
- 2026-08-09: Downloads verify the secretstream terminated on `TAG_FINAL` and
write output atomically: a truncated body is rejected instead of landing on
disk as a short file, and plaintext is staged in a sibling temp file and
renamed into place, so a failed download leaves the destination untouched.
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
shims, README Entrypoints section
- 2026-06-10: Decrypted collections shared by other users (sealed-box keys);
listCollections drops deleted-collection tombstones.
- 2026-06-10: Login hardening: dual-2FA empty-string fields handled, TOTP
preferred when a passkey is also enrolled, interactive input via
@inquirer/prompts.
- 2026-06-10: Replaced sharp with pure JS (jpeg-js + exif-reader); added
single-binary bun build and make install.
- 2026-06-09: Added backup-metadata command (ML data always included, --exif
opt-in); rewrote README to match the implementation; added thumbnail helper
tests.
- 2026-05-13: Full CLI surface: login, backup with dedup symlink layout,
collections, files, get, get-thumb, thumbnail repair helpers.
- 2026-05-13: Client OO API with literate usage tests; file download and
decryption; all three metadata layers decrypted and persisted; renamed quack
to quak.
- 2026-05-11: SRP login flow (email OTP + TOTP) and ApiClient.
# Future Steps
- Make `make docker` green.
- Tag v1.0.0.
- Future desktop client, separate repo:
- Electron app skeleton consuming this library.
- Local SQLite cache keyed on (collectionID, fileID, updationTime).
- Background sync worker streaming new files into the cache.
- Gallery UI: thumbnails, full-image view, basic search.
- Upload, delete, and share operations in the library.

View File

@@ -1,7 +1,7 @@
#!/usr/bin/env node #!/usr/bin/env node
import { createInterface } from "node:readline/promises"; import { input, password as passwordPrompt } from "@inquirer/prompts";
import { stdin, stdout, stderr } from "node:process"; import { stdout, stderr } from "node:process";
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { join } from "node:path"; import { join } from "node:path";
import { Command } from "commander"; import { Command } from "commander";
@@ -9,6 +9,7 @@ import envPaths from "env-paths";
import { Client, type ClientSnapshot } from "../src/client.js"; import { Client, type ClientSnapshot } from "../src/client.js";
import { init } from "../src/crypto/index.js"; import { init } from "../src/crypto/index.js";
import { runBackup } from "../src/backup.js"; import { runBackup } from "../src/backup.js";
import { runMetadataBackup } from "../src/metadata-backup.js";
import { import {
listMissingThumbnails, listMissingThumbnails,
fixMissingThumbnails, fixMissingThumbnails,
@@ -44,46 +45,10 @@ const requireSession = (): Client => {
return Client.fromJSON(snapshot); return Client.fromJSON(snapshot);
}; };
const prompt = async (message: string): Promise<string> => { const prompt = async (message: string): Promise<string> => input({ message });
const rl = createInterface({ input: stdin, output: stderr });
const answer = await rl.question(message);
rl.close();
return answer;
};
const promptPassword = async (message: string): Promise<string> => { const promptSecret = async (message: string): Promise<string> =>
if (!stdin.isTTY) { passwordPrompt({ message, mask: true });
return prompt(message);
}
stderr.write(message);
stdin.setRawMode(true);
stdin.resume();
const chars: string[] = [];
return new Promise((resolve) => {
const onData = (buf: Buffer) => {
for (const byte of buf) {
if (byte === 3) {
stdin.setRawMode(false);
process.exit(130);
}
if (byte === 13 || byte === 10) {
stdin.setRawMode(false);
stdin.removeListener("data", onData);
stdin.pause();
stderr.write("\n");
resolve(chars.join(""));
return;
}
if (byte === 127 || byte === 8) {
chars.pop();
} else {
chars.push(String.fromCharCode(byte));
}
}
};
stdin.on("data", onData);
});
};
const program = new Command(); const program = new Command();
@@ -97,18 +62,9 @@ program
.description("Log in to an Ente account and save the session") .description("Log in to an Ente account and save the session")
.action(async () => { .action(async () => {
await init(); await init();
const email = const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
process.env.QUAK_EMAIL ??
(stdin.isTTY ? await prompt("Email: ") : null);
const password = const password =
process.env.QUAK_PASSWORD ?? process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
(stdin.isTTY ? await promptPassword("Password: ") : null);
if (!email || !password) {
stderr.write(
"Set QUAK_EMAIL and QUAK_PASSWORD env vars for non-interactive use.\n",
);
process.exit(1);
}
stderr.write("Authenticating...\n"); stderr.write("Authenticating...\n");
try { try {
@@ -333,6 +289,26 @@ program
}, },
); );
program
.command("backup-metadata")
.description(
"Dump all decrypted account metadata to a directory of JSON files",
)
.argument("<dir>", "Output directory")
.option(
"--exif",
"Download each file and extract full EXIF/IPTC/XMP metadata (slow)",
)
.option("--all", "Alias for --exif")
.action(async (dir: string, opts: { exif?: boolean; all?: boolean }) => {
await init();
const client = requireSession();
await runMetadataBackup(client, dir, {
exif: opts.exif || opts.all,
onProgress: (msg) => stderr.write(msg + "\n"),
});
});
program program
.command("backup") .command("backup")
.description( .description(
@@ -456,4 +432,5 @@ helper
process.exit(results.some((r) => !r.success) ? 1 : 0); process.exit(results.some((r) => !r.success) ? 1 : 0);
}); });
await init();
program.parse(); program.parse();

View File

@@ -31,7 +31,6 @@
"@eslint/js": "9.38.0", "@eslint/js": "9.38.0",
"@types/libsodium-wrappers-sumo": "0.8.2", "@types/libsodium-wrappers-sumo": "0.8.2",
"@types/node": "22.18.13", "@types/node": "22.18.13",
"@types/sharp": "0.32.0",
"eslint": "9.38.0", "eslint": "9.38.0",
"prettier": "3.8.1", "prettier": "3.8.1",
"typescript": "5.9.3", "typescript": "5.9.3",
@@ -39,10 +38,12 @@
"vitest": "2.1.9" "vitest": "2.1.9"
}, },
"dependencies": { "dependencies": {
"@inquirer/prompts": "8.5.2",
"commander": "14.0.3", "commander": "14.0.3",
"env-paths": "4.0.0", "env-paths": "4.0.0",
"exif-reader": "2.0.3",
"fast-srp-hap": "2.0.4", "fast-srp-hap": "2.0.4",
"libsodium-wrappers-sumo": "0.8.4", "jpeg-js": "0.4.4",
"sharp": "0.34.5" "libsodium-wrappers-sumo": "0.8.4"
} }
} }

136
script/bootstrap Executable file
View File

@@ -0,0 +1,136 @@
#!/bin/sh
# script/bootstrap: install all dependencies needed to build and develop
# this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes nothing is present. Node is
# used directly if installed; otherwise it is installed at a pinned
# version via nvm (installing nvm itself first, from a hash-verified
# release archive, never curl | sh).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-07-06
NODE_VERSION="22.17.0"
NVM_VERSION="0.40.3"
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
YARN_VERSION="1.22.22"
PKGMGR=""
SUDO=""
detect_pkgmgr() {
[ -n "$PKGMGR" ] && return 0
if command -v nix-env >/dev/null 2>&1; then
PKGMGR="nix"
elif command -v apt-get >/dev/null 2>&1; then
PKGMGR="apt"
elif command -v brew >/dev/null 2>&1; then
PKGMGR="brew"
elif command -v apk >/dev/null 2>&1; then
PKGMGR="apk"
else
echo "bootstrap: no supported package manager (nix, apt, brew, apk)" >&2
exit 1
fi
if [ "$PKGMGR" = "apt" ]; then
export DEBIAN_FRONTEND=noninteractive
if [ "$(id -u)" != "0" ]; then
SUDO="sudo"
fi
fi
}
# pkg_install <nix-attr> <apt-pkg> <brew-formula> <apk-pkg>
pkg_install() {
detect_pkgmgr
case "$PKGMGR" in
nix) nix-env -iA "nixpkgs.$1" ;;
apt) $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2" ;;
brew) brew install "$3" ;;
apk) apk add --no-cache "$4" ;;
esac
}
missing() {
! command -v "$1" >/dev/null 2>&1
}
# verify_sha256 <file> <expected-hash>
verify_sha256() {
if command -v sha256sum >/dev/null 2>&1; then
actual="$(sha256sum "$1" | cut -d' ' -f1)"
else
actual="$(shasum -a 256 "$1" | cut -d' ' -f1)"
fi
if [ "$actual" != "$2" ]; then
echo "bootstrap: sha256 mismatch for $1" >&2
echo " expected: $2" >&2
echo " actual: $actual" >&2
exit 1
fi
}
# nvm is a bash script; run a command in a bash with nvm loaded
nvm_sh() {
bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*"
}
ensure_nvm() {
[ -s "$HOME/.nvm/nvm.sh" ] && return 0
# nvm prerequisites; nvm itself requires bash
if missing bash; then pkg_install bash bash bash bash; fi
if missing curl; then pkg_install curl curl curl curl; fi
if missing git; then pkg_install git git git git; fi
tmp="$(mktemp -d)"
curl -fsSL -o "$tmp/nvm.tar.gz" \
"https://github.com/nvm-sh/nvm/archive/refs/tags/v${NVM_VERSION}.tar.gz"
verify_sha256 "$tmp/nvm.tar.gz" "$NVM_SHA256"
mkdir -p "$HOME/.nvm"
tar -xzf "$tmp/nvm.tar.gz" -C "$HOME/.nvm" --strip-components=1
rm -rf "$tmp"
}
ensure_node() {
if ! missing node; then return 0; fi
ensure_nvm
nvm_sh "nvm install $NODE_VERSION"
}
ensure_yarn() {
if ! missing yarn; then return 0; fi
if ! missing corepack; then
corepack enable
corepack prepare "yarn@$YARN_VERSION" --activate
elif [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
corepack prepare yarn@$YARN_VERSION --activate"
else
npm install -g "yarn@$YARN_VERSION"
fi
}
install_js_deps() {
if missing yarn && [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
yarn install --frozen-lockfile"
else
yarn install --frozen-lockfile
fi
}
main() {
cd "$ROOT"
if missing make; then pkg_install gnumake make make make; fi
if missing git; then pkg_install git git git git; fi
ensure_node
ensure_yarn
install_js_deps
echo "bootstrap complete"
}
main "$@"

14
script/check Executable file
View File

@@ -0,0 +1,14 @@
#!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. Must not modify any files.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/test"
"$SCRIPT_DIR/lint"
"$SCRIPT_DIR/fmt-check"
}
main "$@"

13
script/cibuild Executable file
View File

@@ -0,0 +1,13 @@
#!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs script/check, so
# a successful build implies all checks pass.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
docker build .
}
main "$@"

14
script/docker Executable file
View File

@@ -0,0 +1,14 @@
#!/bin/sh
# script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build -t "$("$SCRIPT_DIR/projectname")" .
}
main "$@"

12
script/fmt Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/sh
# script/fmt: format all files (writes).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
yarn run prettier --write .
}
main "$@"

12
script/fmt-check Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/sh
# script/fmt-check: check formatting (read-only).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
yarn run prettier --check .
}
main "$@"

16
script/install-precommit Executable file
View File

@@ -0,0 +1,16 @@
#!/bin/sh
# script/install-precommit: install the git pre-commit hook that runs
# script/precommit. Our own extension to scripts-to-rule-them-all.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
hook=".git/hooks/pre-commit"
printf '#!/bin/sh\nset -e\nscript/precommit\n' > .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo "pre-commit hook installed: runs script/precommit"
}
main "$@"

13
script/lint Executable file
View File

@@ -0,0 +1,13 @@
#!/bin/sh
# script/lint: run the linter (eslint plus a prettier check).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
yarn run eslint .
yarn run prettier --check .
}
main "$@"

18
script/precommit Executable file
View File

@@ -0,0 +1,18 @@
#!/bin/sh
# script/precommit: run by the git pre-commit hook; fails the commit if
# checks fail. Our own extension to scripts-to-rule-them-all.
#
# Runs lint and fmt-check but deliberately NOT the tests, so the TDD
# red-phase commit (failing tests, no implementation yet) can land. CI
# runs make check via docker build, which catches any branch that
# ships red.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/lint"
"$SCRIPT_DIR/fmt-check"
}
main "$@"

12
script/projectname Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/sh
# script/projectname: output the name of this project. Our own
# extension to scripts-to-rule-them-all. Other scripts that need the
# name (e.g. script/docker) call this, so they can stay identical
# across all repos.
set -eu
main() {
echo "quack"
}
main "$@"

13
script/setup Executable file
View File

@@ -0,0 +1,13 @@
#!/bin/sh
# script/setup: set up the repo for development after a fresh clone:
# installs dependencies and the git pre-commit hook.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/install-precommit"
}
main "$@"

25
script/test Executable file
View File

@@ -0,0 +1,25 @@
#!/bin/sh
# script/test: run the test suite. Uses `timeout` (GNU coreutils) when
# available so the run is hard-capped at 30s; on macOS without
# coreutils the cap is skipped.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
rerun_verbose() {
echo "--- Rerunning with verbose for details ---"
yarn run vitest run --reporter=verbose
exit 1
}
main() {
cd "$ROOT"
TIMEOUT="$(command -v timeout 2>/dev/null || command -v gtimeout 2>/dev/null || true)"
if [ -n "$TIMEOUT" ]; then
"$TIMEOUT" 30s yarn run vitest run --reporter=dot || rerun_verbose
else
yarn run vitest run --reporter=dot || rerun_verbose
fi
}
main "$@"

View File

@@ -1,8 +1,33 @@
import { ApiError } from "../errors.js";
import {
isSafeToReplay,
resolveRetryOptions,
withRetry,
type ResolvedRetryOptions,
type RetryOptions,
} from "../retry.js";
// `ApiError` is defined in `src/errors.ts` so that the retry classifier can
// recognise it without importing this module, which imports the classifier.
// It is re-exported here because this is where callers have always imported it
// from, and it must remain one class: a second copy would make `instanceof`
// fail in the classifier and every 5xx would look permanent.
export { ApiError };
const DEFAULT_API_ORIGIN = "https://api.ente.io"; const DEFAULT_API_ORIGIN = "https://api.ente.io";
const DEFAULT_FILES_ORIGIN = "https://files.ente.io"; const DEFAULT_FILES_ORIGIN = "https://files.ente.io";
const DEFAULT_THUMBS_ORIGIN = "https://thumbnails.ente.io"; const DEFAULT_THUMBS_ORIGIN = "https://thumbnails.ente.io";
const CLIENT_PACKAGE = "berlin.sneak.quak"; const CLIENT_PACKAGE = "berlin.sneak.quak";
// Two deadlines rather than one, because a single number cannot serve both
// jobs. Thirty seconds is generous for a JSON call and short enough that a
// hung API connection cannot stall a backup for long. A file body is a
// different shape of problem: the deadline has to cover the whole transfer,
// which for a large video on a slow link is minutes, so a value sane for JSON
// would cancel legitimate downloads.
export const DEFAULT_REQUEST_TIMEOUT_MS = 30_000;
export const DEFAULT_DOWNLOAD_TIMEOUT_MS = 600_000;
export interface ApiClientOptions { export interface ApiClientOptions {
apiOrigin?: string; apiOrigin?: string;
filesOrigin?: string; filesOrigin?: string;
@@ -10,33 +35,78 @@ export interface ApiClientOptions {
authToken?: string; authToken?: string;
fetch?: typeof globalThis.fetch; fetch?: typeof globalThis.fetch;
userAgent?: string; userAgent?: string;
retry?: RetryOptions;
requestTimeoutMs?: number;
downloadTimeoutMs?: number;
} }
export class ApiError extends Error { export interface StreamOptions {
readonly status: number; // Opt out of this client's own retry. Exactly one caller wants that: the
readonly code?: string; // download layer, which retries the request, the stream consumption and
readonly requestID?: string; // the decryption as one unit. Leaving both layers enabled would multiply
readonly body?: unknown; // the budgets — four attempts each becoming sixteen requests per file.
constructor( retry?: boolean;
message: string,
status: number,
opts?: { code?: string; requestID?: string; body?: unknown },
) {
super(message);
this.name = "ApiError";
this.status = status;
this.code = opts?.code;
this.requestID = opts?.requestID;
this.body = opts?.body;
}
} }
// Enforce a deadline over a response body, not merely over its headers.
//
// `getFileStream` returns as soon as headers arrive; the bytes are pulled
// later, in the download layer. Whether the signal passed to `fetch` also
// tears down the body afterwards is up to the fetch implementation, so this
// wrapper makes it a property of quak instead: every read races the signal,
// and an abort errors the stream with the abort reason — which the retry
// classifier recognises.
const deadlineStream = (
body: ReadableStream<Uint8Array>,
signal: AbortSignal,
): ReadableStream<Uint8Array> => {
const reader = body.getReader();
let rejectOnAbort: (reason: unknown) => void = () => undefined;
const aborted = new Promise<never>((_resolve, reject) => {
rejectOnAbort = reject;
});
// The abort may fire when nothing is awaiting `aborted` — after the body
// has been read in full, say. Without this, that rejection would surface
// as an unhandled rejection and take the process down.
void aborted.catch(() => undefined);
const onAbort = (): void => rejectOnAbort(signal.reason);
if (signal.aborted) onAbort();
else signal.addEventListener("abort", onAbort, { once: true });
const release = (): void => signal.removeEventListener("abort", onAbort);
return new ReadableStream<Uint8Array>({
async pull(controller) {
try {
const next = await Promise.race([reader.read(), aborted]);
if (next.done) {
release();
controller.close();
return;
}
controller.enqueue(next.value);
} catch (err) {
release();
await reader.cancel(err).catch(() => undefined);
throw err;
}
},
async cancel(reason) {
release();
await reader.cancel(reason);
},
});
};
export class ApiClient { export class ApiClient {
private readonly apiOrigin: string; private readonly apiOrigin: string;
private readonly isCustomOrigin: boolean; private readonly isCustomOrigin: boolean;
private readonly filesOrigin: string; private readonly filesOrigin: string;
private readonly thumbsOrigin: string; private readonly thumbsOrigin: string;
private readonly _fetch: typeof globalThis.fetch; private readonly _fetch: typeof globalThis.fetch;
private readonly retry: ResolvedRetryOptions;
private readonly requestTimeoutMs: number;
private readonly downloadTimeoutMs: number;
private token: string | undefined; private token: string | undefined;
constructor(opts?: ApiClientOptions) { constructor(opts?: ApiClientOptions) {
@@ -53,6 +123,11 @@ export class ApiClient {
opts?.thumbsOrigin ?? DEFAULT_THUMBS_ORIGIN opts?.thumbsOrigin ?? DEFAULT_THUMBS_ORIGIN
).replace(/\/+$/, ""); ).replace(/\/+$/, "");
this._fetch = opts?.fetch ?? globalThis.fetch; this._fetch = opts?.fetch ?? globalThis.fetch;
this.retry = resolveRetryOptions(opts?.retry);
this.requestTimeoutMs =
opts?.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS;
this.downloadTimeoutMs =
opts?.downloadTimeoutMs ?? DEFAULT_DOWNLOAD_TIMEOUT_MS;
this.token = opts?.authToken; this.token = opts?.authToken;
} }
@@ -64,6 +139,13 @@ export class ApiClient {
this.token = undefined; this.token = undefined;
} }
// The policy this client was configured with, so that a caller wrapping a
// whole operation in its own `withRetry` — the download layer — runs under
// the same settings rather than under the library defaults.
getRetryOptions(): ResolvedRetryOptions {
return this.retry;
}
private headers(extra?: Record<string, string>): Record<string, string> { private headers(extra?: Record<string, string>): Record<string, string> {
const h: Record<string, string> = { const h: Record<string, string> = {
"X-Client-Package": CLIENT_PACKAGE, "X-Client-Package": CLIENT_PACKAGE,
@@ -128,38 +210,53 @@ export class ApiClient {
} }
} }
} }
const resp = await this._fetch(url.href, { // A GET changes nothing, so it is retried under the full policy.
method: "GET", return withRetry(async () => {
headers: this.headers(), const resp = await this._fetch(url.href, {
}); method: "GET",
await this.throwIfError(resp); headers: this.headers(),
return (await resp.json()) as T; signal: AbortSignal.timeout(this.requestTimeoutMs),
});
await this.throwIfError(resp);
return (await resp.json()) as T;
}, this.retry);
} }
async postJSON<T>(path: string, body: unknown): Promise<T> { async postJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`; const url = `${this.apiOrigin}${path}`;
const resp = await this._fetch(url, { // Idempotency: this reaches `/users/srp/create-session`,
method: "POST", // `/users/two-factor/verify` and `/users/ott`, all of which change
headers: this.headers({ "Content-Type": "application/json" }), // server state — verifying a second factor consumes one of a small
body: JSON.stringify(body), // number of attempts. So a POST is replayed only when the failure
}); // proves the request never reached the server, which in practice means
await this.throwIfError(resp); // the connection was never established. A 5xx, a mid-flight reset and
return (await resp.json()) as T; // a timeout are all left to the caller, because each of them can occur
// after the server has already acted.
return withRetry(
async () => {
const resp = await this._fetch(url, {
method: "POST",
headers: this.headers({
"Content-Type": "application/json",
}),
body: JSON.stringify(body),
signal: AbortSignal.timeout(this.requestTimeoutMs),
});
await this.throwIfError(resp);
return (await resp.json()) as T;
},
{ ...this.retry, isRetryable: isSafeToReplay },
);
} }
async getFileStream(fileID: number): Promise<ReadableStream<Uint8Array>> { async getFileStream(
fileID: number,
opts?: StreamOptions,
): Promise<ReadableStream<Uint8Array>> {
const url = this.isCustomOrigin const url = this.isCustomOrigin
? `${this.apiOrigin}/files/download/${fileID}` ? `${this.apiOrigin}/files/download/${fileID}`
: `${this.filesOrigin}/?fileID=${fileID}`; : `${this.filesOrigin}/?fileID=${fileID}`;
const resp = await this._fetch(url, { return this.streamRequest(url, opts);
method: "GET",
headers: this.headers(),
});
await this.throwIfError(resp);
if (!resp.body) {
throw new Error("response body is null");
}
return resp.body;
} }
async getUploadURL( async getUploadURL(
@@ -173,28 +270,52 @@ export class ApiClient {
} }
async putFile(presignedURL: string, data: Uint8Array): Promise<void> { async putFile(presignedURL: string, data: Uint8Array): Promise<void> {
const resp = await this._fetch(presignedURL, { // Idempotent despite being a write: a presigned PUT stores one whole
method: "PUT", // object at one key in one request, so replaying it either overwrites
headers: { // the same bytes or lands them for the first time. There is no partial
"Content-Type": "application/octet-stream", // state to protect, hence the full policy rather than the POST rule.
"Content-Length": String(data.length), await withRetry(async () => {
}, const resp = await this._fetch(presignedURL, {
body: data, method: "PUT",
}); headers: {
if (!resp.ok) { "Content-Type": "application/octet-stream",
throw new Error(`PUT to presigned URL failed: HTTP ${resp.status}`); "Content-Length": String(data.length),
} },
body: data,
signal: AbortSignal.timeout(this.requestTimeoutMs),
});
if (!resp.ok) {
// An ApiError, not a bare Error: without the status on the
// error the upload path cannot be classified at all, and a
// 503 from S3 would be indistinguishable from a bug.
throw new ApiError(
`PUT to presigned URL failed: HTTP ${resp.status}`,
resp.status,
);
}
}, this.retry);
} }
async putJSON<T>(path: string, body: unknown): Promise<T> { async putJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`; const url = `${this.apiOrigin}${path}`;
const resp = await this._fetch(url, { // Same idempotency rule as `postJSON`, for the same reason: this
method: "PUT", // reaches `/files/thumbnail`, which registers an uploaded thumbnail
headers: this.headers({ "Content-Type": "application/json" }), // against a file.
body: JSON.stringify(body), return withRetry(
}); async () => {
await this.throwIfError(resp); const resp = await this._fetch(url, {
return (await resp.json()) as T; method: "PUT",
headers: this.headers({
"Content-Type": "application/json",
}),
body: JSON.stringify(body),
signal: AbortSignal.timeout(this.requestTimeoutMs),
});
await this.throwIfError(resp);
return (await resp.json()) as T;
},
{ ...this.retry, isRetryable: isSafeToReplay },
);
} }
async updateThumbnail( async updateThumbnail(
@@ -210,18 +331,36 @@ export class ApiClient {
async getThumbnailStream( async getThumbnailStream(
fileID: number, fileID: number,
opts?: StreamOptions,
): Promise<ReadableStream<Uint8Array>> { ): Promise<ReadableStream<Uint8Array>> {
const url = this.isCustomOrigin const url = this.isCustomOrigin
? `${this.apiOrigin}/files/preview/${fileID}` ? `${this.apiOrigin}/files/preview/${fileID}`
: `${this.thumbsOrigin}/?fileID=${fileID}`; : `${this.thumbsOrigin}/?fileID=${fileID}`;
const resp = await this._fetch(url, { return this.streamRequest(url, opts);
method: "GET", }
headers: this.headers(),
}); private async streamRequest(
await this.throwIfError(resp); url: string,
if (!resp.body) { opts?: StreamOptions,
throw new Error("response body is null"); ): Promise<ReadableStream<Uint8Array>> {
} const once = async (): Promise<ReadableStream<Uint8Array>> => {
return resp.body; // A fresh deadline per attempt, so a retry gets the whole budget
// rather than the remainder of the one that just expired.
const signal = AbortSignal.timeout(this.downloadTimeoutMs);
const resp = await this._fetch(url, {
method: "GET",
headers: this.headers(),
signal,
});
await this.throwIfError(resp);
if (!resp.body) {
// Carries the status, and is not retryable: a response that
// arrived without a body is malformed, and asking again
// produces the same malformed response.
throw new ApiError("response body is null", resp.status);
}
return deadlineStream(resp.body, signal);
};
return opts?.retry === false ? once() : withRetry(once, this.retry);
} }
} }

View File

@@ -73,8 +73,18 @@ const srpHandshake = async (
srpClient.checkM2(Buffer.from(verifyResponse.srpM2, "base64")); srpClient.checkM2(Buffer.from(verifyResponse.srpM2, "base64"));
if (verifyResponse.twoFactorSessionID) { // twoFactorSessionIDV2 is set (instead of twoFactorSessionID) when the
return { kind: "totp", sessionID: verifyResponse.twoFactorSessionID }; // account has BOTH passkeys and TOTP. Prefer TOTP: a CLI cannot perform
// a WebAuthn ceremony, and the user has a TOTP secret enrolled.
//
// The server marshals these fields without `omitempty`, so unset fields
// arrive as "" rather than being absent. Use || (not ??) so empty
// strings are treated as not-set.
const totpSessionID =
verifyResponse.twoFactorSessionID ||
verifyResponse.twoFactorSessionIDV2;
if (totpSessionID) {
return { kind: "totp", sessionID: totpSessionID };
} }
if (verifyResponse.passkeySessionID) { if (verifyResponse.passkeySessionID) {
return { return {

View File

@@ -34,13 +34,18 @@ export interface SRPAttributes {
// The body of a successful login response. Exactly one of the following // The body of a successful login response. Exactly one of the following
// situations applies, and the caller dispatches on the populated fields: // situations applies, and the caller dispatches on the populated fields:
// - both keyAttributes and encryptedToken present: login is complete // - both keyAttributes and encryptedToken present: login is complete
// - twoFactorSessionID present: caller must submit a TOTP code // - twoFactorSessionID present: caller must submit a TOTP code (TOTP-only
// - passkeySessionID present: caller must complete a passkey ceremony // account)
// - passkeySessionID + twoFactorSessionIDV2 present: account has both
// passkeys and TOTP; the V2 field is set instead of twoFactorSessionID
// so that older clients keep using the passkey flow
// - only passkeySessionID present: caller must complete a passkey ceremony
export interface AuthorizationResponse { export interface AuthorizationResponse {
id: number; id: number;
keyAttributes?: KeyAttributes; keyAttributes?: KeyAttributes;
encryptedToken?: Base64; encryptedToken?: Base64;
twoFactorSessionID?: string; twoFactorSessionID?: string;
twoFactorSessionIDV2?: string;
passkeySessionID?: string; passkeySessionID?: string;
} }

View File

@@ -155,9 +155,21 @@ export class Client {
const { collections } = await this.api.getJSON<{ const { collections } = await this.api.getJSON<{
collections: RawCollection[]; collections: RawCollection[];
}>("/collections/v2", { sinceTime: 0 }); }>("/collections/v2", { sinceTime: 0 });
return collections.map((raw) => // The sync API keeps returning deleted collections as tombstones
decryptCollection(raw, this.masterKey, this.userID), // (isDeleted: true); their diff endpoint 404s, so drop them.
); return collections
.filter((raw) => !raw.isDeleted)
.map((raw) =>
decryptCollection(
raw,
{
masterKey: this.masterKey,
publicKey: this.publicKey,
secretKey: this.secretKey,
},
this.userID,
),
);
} }
async listFiles( async listFiles(

View File

@@ -14,5 +14,6 @@ export {
pullStreamChunk, pullStreamChunk,
STREAM_CHUNK_OVERHEAD, STREAM_CHUNK_OVERHEAD,
STREAM_CHUNK_SIZE, STREAM_CHUNK_SIZE,
streamTagFinal,
type StreamPullState, type StreamPullState,
} from "./stream.js"; } from "./stream.js";

View File

@@ -8,6 +8,27 @@ export const STREAM_CHUNK_SIZE = 4 * 1024 * 1024;
// 16 bytes of Poly1305 tag plus 1 byte of secretstream tag. // 16 bytes of Poly1305 tag plus 1 byte of secretstream tag.
export const STREAM_CHUNK_OVERHEAD = 17; export const STREAM_CHUNK_OVERHEAD = 17;
// libsodium's crypto_secretstream_xchacha20poly1305_TAG_FINAL: the tag that
// marks the last chunk of a stream. Exported so callers (the download layer)
// can detect truncation without importing sodium themselves.
//
// This is a function rather than a constant, and that is load-bearing:
// libsodium attaches its constants to the module object inside
// `ready.then(...)`, which resolves long after this module is evaluated. A
// module-level read would bind `undefined`, every downstream comparison
// against it would then be false, and every valid download would be rejected
// as truncated. Reading at call time returns the library's own value, so
// there is also no second copy of a protocol constant to keep in sync.
//
// Under vitest sodium is already initialised in the worker before this module
// is evaluated, so an eager read would pick up a real value there and the
// ordinary tests could not tell the difference. The regression guard is
// "streamTagFinal() reads the constant at call time, not at import time" in
// test/crypto/stream.test.ts, which reproduces the plain-Node ESM ordering
// against a stand-in sodium module; it goes red if this becomes eager.
export const streamTagFinal = (): number =>
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL;
// Encrypt a small blob as a single secretstream chunk with TAG_FINAL. // Encrypt a small blob as a single secretstream chunk with TAG_FINAL.
// Returns the header and ciphertext. Used for encrypting thumbnails // Returns the header and ciphertext. Used for encrypting thumbnails
// and metadata before upload. // and metadata before upload.
@@ -37,9 +58,6 @@ export const initStreamPull = (
): StreamPullState => ): StreamPullState =>
sodium.crypto_secretstream_xchacha20poly1305_init_pull(header, key); sodium.crypto_secretstream_xchacha20poly1305_init_pull(header, key);
// Decrypt one ciphertext chunk. Returns the plaintext and the secretstream
// tag (0=MESSAGE, 1=PUSH, 2=REKEY, 3=FINAL). The caller should verify the
// stream ended on TAG_FINAL to detect truncation.
// Decrypt a small blob that was encrypted as a single secretstream chunk // Decrypt a small blob that was encrypted as a single secretstream chunk
// with TAG_FINAL. Ente uses this form ("blob") for file metadata and // with TAG_FINAL. Ente uses this form ("blob") for file metadata and
// magic metadata — anything under ~1 MiB that isn't chunked. // magic metadata — anything under ~1 MiB that isn't chunked.
@@ -50,12 +68,18 @@ export const decryptBlob = (
): Uint8Array => { ): Uint8Array => {
const state = initStreamPull(header, key); const state = initStreamPull(header, key);
const { plaintext, tag } = pullStreamChunk(state, ciphertext); const { plaintext, tag } = pullStreamChunk(state, ciphertext);
if (tag !== sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL) { const tagFinal = streamTagFinal();
throw new Error(`decryptBlob: expected TAG_FINAL (3), got tag ${tag}`); if (tag !== tagFinal) {
throw new Error(
`decryptBlob: expected TAG_FINAL (${tagFinal}), got tag ${tag}`,
);
} }
return plaintext; return plaintext;
}; };
// Decrypt one ciphertext chunk. Returns the plaintext and the secretstream
// tag (0=MESSAGE, 1=PUSH, 2=REKEY, 3=FINAL). The caller must verify the
// stream ended on TAG_FINAL to detect truncation.
export const pullStreamChunk = ( export const pullStreamChunk = (
state: StreamPullState, state: StreamPullState,
ciphertext: Uint8Array, ciphertext: Uint8Array,

View File

@@ -1,11 +1,16 @@
import { writeFile } from "node:fs/promises"; import { randomUUID } from "node:crypto";
import { rename, rm, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { import {
fromBase64, fromBase64,
initStreamPull, initStreamPull,
pullStreamChunk, pullStreamChunk,
STREAM_CHUNK_OVERHEAD, STREAM_CHUNK_OVERHEAD,
STREAM_CHUNK_SIZE, STREAM_CHUNK_SIZE,
streamTagFinal,
} from "../crypto/index.js"; } from "../crypto/index.js";
import { TruncatedStreamError } from "../errors.js";
import { withRetry } from "../retry.js";
import type { ApiClient } from "../api/client.js"; import type { ApiClient } from "../api/client.js";
import type { EnteFile } from "../model/types.js"; import type { EnteFile } from "../model/types.js";
@@ -26,6 +31,8 @@ const streamDecrypt = async (
let buffer = new Uint8Array(0); let buffer = new Uint8Array(0);
const plainChunks: Uint8Array[] = []; const plainChunks: Uint8Array[] = [];
let totalPlain = 0; let totalPlain = 0;
let chunksPulled = 0;
let lastTag = -1;
for (;;) { for (;;) {
const { done, value } = await reader.read(); const { done, value } = await reader.read();
@@ -39,21 +46,58 @@ const streamDecrypt = async (
while (buffer.length >= ENC_CHUNK_SIZE) { while (buffer.length >= ENC_CHUNK_SIZE) {
const encChunk = buffer.slice(0, ENC_CHUNK_SIZE); const encChunk = buffer.slice(0, ENC_CHUNK_SIZE);
buffer = buffer.slice(ENC_CHUNK_SIZE); buffer = buffer.slice(ENC_CHUNK_SIZE);
const { plaintext } = pullStreamChunk(state, encChunk); const { plaintext, tag } = pullStreamChunk(state, encChunk);
plainChunks.push(plaintext); plainChunks.push(plaintext);
totalPlain += plaintext.length; totalPlain += plaintext.length;
chunksPulled++;
lastTag = tag;
} }
if (done) { if (done) {
if (buffer.length > 0) { if (buffer.length > 0) {
const { plaintext } = pullStreamChunk(state, buffer); // Whatever is left over once every whole chunk has been
plainChunks.push(plaintext); // consumed must be the stream's final chunk, and a final
totalPlain += plaintext.length; // chunk that actually arrived in full authenticates. If it
// does not, the body stopped part-way through a chunk — the
// ordinary shape of a dropped connection. Poly1305 cannot
// tell a partial chunk from a corrupt one, so this is
// reported as the truncation it almost always is, with the
// authentication failure kept as the error's cause.
let pulled;
try {
pulled = pullStreamChunk(state, buffer);
} catch (err) {
throw new TruncatedStreamError(
`download: stream truncated: response body ended with ${buffer.length} trailing bytes that did not authenticate as a final chunk (transfer stopped mid-chunk, or the data is corrupt)`,
{ cause: err },
);
}
plainChunks.push(pulled.plaintext);
totalPlain += pulled.plaintext.length;
chunksPulled++;
lastTag = pulled.tag;
} }
break; break;
} }
} }
// Only the last chunk of a secretstream carries TAG_FINAL. Everything a
// dropped connection did deliver still decrypts and authenticates, so the
// absence of TAG_FINAL is the only evidence that the body was cut short.
// Returning a short plaintext here would put a corrupt file on disk that
// later backup runs would treat as complete.
if (chunksPulled === 0) {
throw new TruncatedStreamError(
"download: stream truncated: response body contained no secretstream chunks",
);
}
const tagFinal = streamTagFinal();
if (lastTag !== tagFinal) {
throw new TruncatedStreamError(
`download: stream truncated: last chunk tag ${lastTag}, expected TAG_FINAL (${tagFinal})`,
);
}
const result = new Uint8Array(totalPlain); const result = new Uint8Array(totalPlain);
let offset = 0; let offset = 0;
for (const chunk of plainChunks) { for (const chunk of plainChunks) {
@@ -63,16 +107,73 @@ const streamDecrypt = async (
return result; return result;
}; };
// Write `plaintext` to `destination` atomically: stage it in a temporary
// sibling file (same directory, so the rename cannot cross a filesystem
// boundary) and rename it into place. Callers therefore never observe a
// partially written destination, and a pre-existing file at that path is
// replaced only once the new contents are complete on disk.
const writeAtomic = async (
destination: string,
plaintext: Uint8Array,
): Promise<void> => {
// The random suffix keeps concurrent downloads of the same destination
// from stepping on each other's temporary file.
const tmpPath = join(dirname(destination), `.quak-${randomUUID()}.tmp`);
try {
await writeFile(tmpPath, plaintext);
await rename(tmpPath, destination);
} catch (err) {
// Best-effort cleanup. A failure to remove the temporary file must
// never replace the error that actually explains what went wrong.
await rm(tmpPath, { force: true }).catch(() => undefined);
throw err;
}
};
// Fetch a stream and decrypt it, retrying the whole sequence.
//
// The request is only the first third of a download. `getXStream` returns as
// soon as headers arrive, and the bytes are pulled here, so a socket reset
// mid-body — the dominant failure mode for multi-megabyte photos over a CDN —
// throws in `streamDecrypt` and never reaches `ApiClient` at all. Retrying the
// request alone would miss it entirely.
//
// The client's own retry is therefore switched off for these two calls: with
// both layers active the budgets would multiply, and the library default of
// four attempts would mean sixteen requests for one file. The policy comes
// from the client so a caller that configured one gets it here too.
//
// A retry starts the file over from byte zero: the secretstream pull state is
// not resumable and there is no Range support on these endpoints.
const fetchAndDecrypt = async (
api: ApiClient,
openStream: () => Promise<ReadableStream<Uint8Array>>,
header: Uint8Array,
key: Uint8Array,
): Promise<Uint8Array> =>
withRetry(async () => {
const stream = await openStream();
return streamDecrypt(stream, header, key);
}, api.getRetryOptions());
export const downloadFile = async ( export const downloadFile = async (
api: ApiClient, api: ApiClient,
file: EnteFile, file: EnteFile,
outPath?: string, outPath?: string,
): Promise<DownloadResult> => { ): Promise<DownloadResult> => {
const resolvedPath = outPath ?? file.metadata.title; const resolvedPath = outPath ?? file.metadata.title;
const stream = await api.getFileStream(file.id);
const header = fromBase64(file.file.decryptionHeader); const header = fromBase64(file.file.decryptionHeader);
const plaintext = await streamDecrypt(stream, header, file.key); const plaintext = await fetchAndDecrypt(
await writeFile(resolvedPath, plaintext); api,
() => api.getFileStream(file.id, { retry: false }),
header,
file.key,
);
// Outside the retry, deliberately: only the attempt that produced a
// complete, authenticated plaintext gets to stage a temporary file, so a
// download that needed three tries still performs exactly one write and
// one rename.
await writeAtomic(resolvedPath, plaintext);
return { path: resolvedPath, bytesWritten: plaintext.length }; return { path: resolvedPath, bytesWritten: plaintext.length };
}; };
@@ -82,9 +183,13 @@ export const downloadThumbnail = async (
outPath?: string, outPath?: string,
): Promise<DownloadResult> => { ): Promise<DownloadResult> => {
const resolvedPath = outPath ?? `thumb_${file.metadata.title}`; const resolvedPath = outPath ?? `thumb_${file.metadata.title}`;
const stream = await api.getThumbnailStream(file.id);
const header = fromBase64(file.thumbnail.decryptionHeader); const header = fromBase64(file.thumbnail.decryptionHeader);
const plaintext = await streamDecrypt(stream, header, file.key); const plaintext = await fetchAndDecrypt(
await writeFile(resolvedPath, plaintext); api,
() => api.getThumbnailStream(file.id, { retry: false }),
header,
file.key,
);
await writeAtomic(resolvedPath, plaintext);
return { path: resolvedPath, bytesWritten: plaintext.length }; return { path: resolvedPath, bytesWritten: plaintext.length };
}; };

45
src/errors.ts Normal file
View File

@@ -0,0 +1,45 @@
// Error types that more than one layer of quak needs to recognise.
//
// They live here rather than beside the code that throws them so that the
// retry classifier can identify them without importing the HTTP client or the
// download layer — both of which import the classifier. `ApiError` is
// re-exported from `src/api/client.ts`, which is where callers have always
// imported it from and where it still belongs conceptually.
export class ApiError extends Error {
readonly status: number;
readonly code?: string;
readonly requestID?: string;
readonly body?: unknown;
constructor(
message: string,
status: number,
opts?: { code?: string; requestID?: string; body?: unknown },
) {
super(message);
this.name = "ApiError";
this.status = status;
this.code = opts?.code;
this.requestID = opts?.requestID;
this.body = opts?.body;
}
}
// A response body that ended before the secretstream did.
//
// This is a type rather than a message prefix because it is a decision, not a
// diagnostic: the retry policy asks "was this a short transfer?" and acts on
// the answer. Matching on the wording of an error message would make the next
// person to reword a diagnostic silently turn every truncated download into a
// permanent failure, and the failure would look like a corrupt file rather
// than like a bug.
//
// `cause` carries the underlying authentication failure on the one path where
// there is one — a body that stopped part-way through a chunk, which Poly1305
// cannot distinguish from corruption.
export class TruncatedStreamError extends Error {
constructor(message: string, opts?: ErrorOptions) {
super(message, opts);
this.name = "TruncatedStreamError";
}
}

View File

@@ -1,7 +1,25 @@
export const VERSION = "0.0.0"; export const VERSION = "0.0.0";
export { Client, type LoginOptions, type ClientSnapshot } from "./client.js"; export { Client, type LoginOptions, type ClientSnapshot } from "./client.js";
export { ApiClient, ApiError, type ApiClientOptions } from "./api/client.js"; export {
ApiClient,
ApiError,
DEFAULT_DOWNLOAD_TIMEOUT_MS,
DEFAULT_REQUEST_TIMEOUT_MS,
type ApiClientOptions,
type StreamOptions,
} from "./api/client.js";
export { TruncatedStreamError } from "./errors.js";
export {
DEFAULT_RETRY_OPTIONS,
isRetryable,
isSafeToReplay,
resolveRetryOptions,
withRetry,
type ResolvedRetryOptions,
type RetryOptions,
type WithRetryOptions,
} from "./retry.js";
export { unwrapAuth, type UnwrapResult } from "./auth/unwrap.js"; export { unwrapAuth, type UnwrapResult } from "./auth/unwrap.js";
export { export {
beginLogin, beginLogin,
@@ -24,6 +42,7 @@ export type {
FileBlob, FileBlob,
FileMetadata, FileMetadata,
FileType, FileType,
KeyMaterial,
Microseconds, Microseconds,
RawCollection, RawCollection,
RawEnteFile, RawEnteFile,

270
src/metadata-backup.ts Normal file
View File

@@ -0,0 +1,270 @@
import { gunzipSync } from "node:zlib";
import {
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import * as jpeg from "jpeg-js";
import exifReader from "exif-reader";
import type { Client } from "./client.js";
import { decryptBlob, fromBase64 } from "./crypto/index.js";
import type { EnteFile } from "./model/types.js";
export type ProgressCallback = (message: string) => void;
export interface MetadataBackupOptions {
exif?: boolean;
onProgress?: ProgressCallback;
}
const sanitizePath = (name: string): string =>
name.replace(/[/\\:*?"<>|]/g, "_").replace(/^\.+/, "_");
interface RawRemoteFileData {
fileID: number;
encryptedData: string;
decryptionHeader: string;
updatedAt?: number;
}
const fetchMLDataForFiles = async (
client: Client,
fileIDs: number[],
fileKeys: Map<number, Uint8Array>,
): Promise<Map<number, Record<string, unknown>>> => {
const api = client.getApiClient();
const result = new Map<number, Record<string, unknown>>();
const batchSize = 200;
for (let i = 0; i < fileIDs.length; i += batchSize) {
const batch = fileIDs.slice(i, i + batchSize);
const { data } = await api.postJSON<{ data: RawRemoteFileData[] }>(
"/files/data/fetch",
{ type: "mldata", fileIDs: batch },
);
for (const entry of data ?? []) {
const key = fileKeys.get(entry.fileID);
if (!key) continue;
try {
const decrypted = decryptBlob(
fromBase64(entry.encryptedData),
fromBase64(entry.decryptionHeader),
key,
);
const jsonStr = gunzipSync(Buffer.from(decrypted)).toString(
"utf-8",
);
result.set(entry.fileID, JSON.parse(jsonStr));
} catch {
// Corrupted ML data for this file; skip it
}
}
}
return result;
};
// Extract the raw EXIF APP1 segment from JPEG bytes. Returns the EXIF
// data buffer (starting after the APP1 length field, at the "Exif\0\0"
// header) or undefined if no APP1 marker is found.
const extractExifFromJpeg = (buf: Uint8Array): Buffer | undefined => {
if (buf[0] !== 0xff || buf[1] !== 0xd8) return undefined;
let offset = 2;
while (offset < buf.length - 1) {
if (buf[offset] !== 0xff) return undefined;
const marker = buf[offset + 1]!;
if (marker === 0xda) break; // start of scan, no more markers
if (offset + 3 >= buf.length) break;
const len = (buf[offset + 2]! << 8) | buf[offset + 3]!;
if (marker === 0xe1) {
// APP1 — check for "Exif\0\0" header
if (
buf[offset + 4] === 0x45 &&
buf[offset + 5] === 0x78 &&
buf[offset + 6] === 0x69 &&
buf[offset + 7] === 0x66
) {
return Buffer.from(
buf.buffer,
buf.byteOffset + offset + 4,
len - 2,
);
}
}
offset += 2 + len;
}
return undefined;
};
const extractImageMetadata = (
fileBytes: Uint8Array,
): Record<string, unknown> | undefined => {
try {
const result: Record<string, unknown> = {};
// Try to get dimensions from JPEG decode
try {
const decoded = jpeg.decode(fileBytes, {
useTArray: true,
formatAsRGBA: false,
});
result.format = "jpeg";
result.width = decoded.width;
result.height = decoded.height;
} catch {
// Not a JPEG or corrupt; still try EXIF extraction
}
const exifBuf = extractExifFromJpeg(fileBytes);
if (exifBuf) {
try {
result.exif = exifReader(exifBuf);
} catch {
result.exifRaw = exifBuf.toString("base64");
}
}
// Extract XMP (look for "http://ns.adobe.com/xap" in the bytes)
const xmpStart = Buffer.from(fileBytes).indexOf("<?xpacket begin");
if (xmpStart !== -1) {
const xmpEnd = Buffer.from(fileBytes).indexOf(
"<?xpacket end",
xmpStart,
);
if (xmpEnd !== -1) {
const end = Buffer.from(fileBytes).indexOf("?>", xmpEnd);
result.xmp = Buffer.from(fileBytes)
.subarray(xmpStart, end !== -1 ? end + 2 : xmpEnd + 50)
.toString("utf-8");
}
}
return Object.keys(result).length > 0 ? result : undefined;
} catch {
return undefined;
}
};
const extractExif = async (
client: Client,
file: EnteFile,
): Promise<Record<string, unknown> | undefined> => {
const tmpDir = mkdtempSync(join(tmpdir(), "quak-exif-"));
try {
const origPath = join(tmpDir, "original");
await client.downloadFile(file, origPath);
const fileBytes = new Uint8Array(readFileSync(origPath));
return extractImageMetadata(fileBytes);
} catch {
return undefined;
} finally {
rmSync(tmpDir, { recursive: true, force: true });
}
};
export const runMetadataBackup = async (
client: Client,
outDir: string,
opts?: MetadataBackupOptions,
): Promise<void> => {
const log = opts?.onProgress ?? (() => {});
const wantExif = opts?.exif ?? false;
mkdirSync(outDir, { recursive: true });
mkdirSync(join(outDir, "collections"), { recursive: true });
const { email, userID } = client.whoami();
writeFileSync(
join(outDir, "account.json"),
JSON.stringify({ email, userID }, null, 2),
);
log("Fetching collections...");
const collections = await client.listCollections();
const allFiles: { file: EnteFile; colDirName: string }[] = [];
const fileKeys = new Map<number, Uint8Array>();
const seenFileIDs = new Set<number>();
for (const col of collections) {
const dirName = `${col.id}-${sanitizePath(col.name || "unnamed")}`;
const colDir = join(outDir, "collections", dirName);
mkdirSync(colDir, { recursive: true });
const collectionMeta: Record<string, unknown> = {
id: col.id,
name: col.name,
type: col.type,
ownerID: col.ownerID,
isShared: col.isShared,
updationTime: col.updationTime,
};
if (col.magicMetadata) collectionMeta.magicMetadata = col.magicMetadata;
if (col.pubMagicMetadata)
collectionMeta.pubMagicMetadata = col.pubMagicMetadata;
if (col.sharedMagicMetadata)
collectionMeta.sharedMagicMetadata = col.sharedMagicMetadata;
writeFileSync(
join(colDir, "_collection.json"),
JSON.stringify(collectionMeta, null, 2),
);
log(`[${col.name}] Fetching files...`);
const files = await client.listFiles(col.id, col.key);
log(`[${col.name}] ${files.length} file(s)`);
for (const file of files) {
allFiles.push({ file, colDirName: dirName });
if (!seenFileIDs.has(file.id)) {
fileKeys.set(file.id, file.key);
seenFileIDs.add(file.id);
}
}
}
log("Fetching ML data (face detections, CLIP embeddings)...");
const mlDataMap = await fetchMLDataForFiles(
client,
[...fileKeys.keys()],
fileKeys,
);
log(`Got ML data for ${mlDataMap.size} file(s)`);
const writtenFileIDs = new Set<number>();
for (const { file, colDirName } of allFiles) {
const colDir = join(outDir, "collections", colDirName);
const fileMeta: Record<string, unknown> = {
id: file.id,
collectionID: file.collectionID,
ownerID: file.ownerID,
metadata: file.metadata,
updationTime: file.updationTime,
};
if (file.magicMetadata) fileMeta.magicMetadata = file.magicMetadata;
if (file.pubMagicMetadata)
fileMeta.pubMagicMetadata = file.pubMagicMetadata;
const ml = mlDataMap.get(file.id);
if (ml) fileMeta.mlData = ml;
if (wantExif && !writtenFileIDs.has(file.id)) {
log(`[${file.metadata.title}] Extracting EXIF...`);
const exifData = await extractExif(client, file);
if (exifData) fileMeta.imageMetadata = exifData;
}
writtenFileIDs.add(file.id);
writeFileSync(
join(colDir, `${file.id}.json`),
JSON.stringify(fileMeta, null, 2),
);
}
log("Metadata backup complete.");
};

View File

@@ -1,10 +1,16 @@
import { decryptBlob, decryptBox, fromBase64 } from "../crypto/index.js"; import {
decryptBlob,
decryptBox,
decryptSealed,
fromBase64,
} from "../crypto/index.js";
import type { import type {
Collection, Collection,
CollectionType, CollectionType,
EnteFile, EnteFile,
FileMetadata, FileMetadata,
FileType, FileType,
KeyMaterial,
RawCollection, RawCollection,
RawEnteFile, RawEnteFile,
RawMagicMetadata, RawMagicMetadata,
@@ -30,14 +36,25 @@ const parseFileType = (n: number): FileType => FILE_TYPE_MAP[n] ?? "unknown";
export const decryptCollection = ( export const decryptCollection = (
raw: RawCollection, raw: RawCollection,
masterKey: Uint8Array, keys: KeyMaterial,
currentUserID?: number, currentUserID?: number,
): Collection => { ): Collection => {
const key = decryptBox( // Owned collections carry their key as a secretbox under our master
fromBase64(raw.encryptedKey), // key, with the nonce in keyDecryptionNonce. Collections shared with
fromBase64(raw.keyDecryptionNonce), // us carry it as an anonymous sealed box to our public key and have
masterKey, // no keyDecryptionNonce at all (sealed boxes embed an ephemeral
); // public key instead).
const key = raw.keyDecryptionNonce
? decryptBox(
fromBase64(raw.encryptedKey),
fromBase64(raw.keyDecryptionNonce),
keys.masterKey,
)
: decryptSealed(
fromBase64(raw.encryptedKey),
keys.publicKey,
keys.secretKey,
);
let name = ""; let name = "";
if (raw.encryptedName && raw.nameDecryptionNonce) { if (raw.encryptedName && raw.nameDecryptionNonce) {
@@ -57,6 +74,9 @@ export const decryptCollection = (
type: parseCollectionType(raw.type), type: parseCollectionType(raw.type),
updationTime: raw.updationTime, updationTime: raw.updationTime,
isShared: currentUserID !== undefined && raw.owner.id !== currentUserID, isShared: currentUserID !== undefined && raw.owner.id !== currentUserID,
magicMetadata: decryptMagicMetadata(raw.magicMetadata, key),
pubMagicMetadata: decryptMagicMetadata(raw.pubMagicMetadata, key),
sharedMagicMetadata: decryptMagicMetadata(raw.sharedMagicMetadata, key),
}; };
}; };

View File

@@ -6,6 +6,7 @@ export type {
FileBlob, FileBlob,
FileMetadata, FileMetadata,
FileType, FileType,
KeyMaterial,
Microseconds, Microseconds,
RawCollection, RawCollection,
RawEnteFile, RawEnteFile,

View File

@@ -15,6 +15,9 @@ export interface Collection {
type: CollectionType; type: CollectionType;
updationTime: Microseconds; updationTime: Microseconds;
isShared: boolean; isShared: boolean;
magicMetadata?: Record<string, unknown>;
pubMagicMetadata?: Record<string, unknown>;
sharedMagicMetadata?: Record<string, unknown>;
} }
export type FileType = "image" | "video" | "livePhoto" | "unknown"; export type FileType = "image" | "video" | "livePhoto" | "unknown";
@@ -47,18 +50,33 @@ export interface EnteFile {
updationTime: Microseconds; updationTime: Microseconds;
} }
// The key material a logged-in client holds, everything needed to decrypt
// any collection: the master key (secretbox for owned collection keys) and
// the X25519 keypair (sealed box for collection keys shared with us).
export interface KeyMaterial {
masterKey: Uint8Array;
publicKey: Uint8Array;
secretKey: Uint8Array;
}
// Raw shapes as they arrive from the Ente API, before decryption. // Raw shapes as they arrive from the Ente API, before decryption.
export interface RawCollection { export interface RawCollection {
id: number; id: number;
owner: { id: number }; owner: { id: number };
encryptedKey: string; encryptedKey: string;
keyDecryptionNonce: string; // Absent for collections shared with us: their encryptedKey is a
// sealed box to our public key, which embeds an ephemeral public key
// instead of using a nonce.
keyDecryptionNonce?: string;
encryptedName?: string; encryptedName?: string;
nameDecryptionNonce?: string; nameDecryptionNonce?: string;
type: string; type: string;
updationTime: number; updationTime: number;
isDeleted?: boolean; isDeleted?: boolean;
magicMetadata?: RawMagicMetadata;
pubMagicMetadata?: RawMagicMetadata;
sharedMagicMetadata?: RawMagicMetadata;
} }
export interface RawMagicMetadata { export interface RawMagicMetadata {

193
src/retry.ts Normal file
View File

@@ -0,0 +1,193 @@
// The retry policy shared by every network operation in quak.
//
// Two independent pieces: a classifier that decides whether an error is worth
// another attempt, and a loop that acts on that decision with exponential
// backoff. Keeping them apart is what lets the non-idempotent call sites reuse
// the loop under a stricter question (see `isSafeToReplay`).
//
// The classifier's default answer is no. For a backup tool, retrying a
// permanent failure spends round trips and delays every remaining file, while
// declining to retry a transient one costs a single file that the next run
// picks up anyway. When the evidence is ambiguous, fail fast.
import { ApiError, TruncatedStreamError } from "./errors.js";
export interface RetryOptions {
// Total calls, not retries: `attempts: 1` disables retrying.
attempts?: number;
// Ceiling for the first retry's delay; doubles with each retry.
baseDelayMs?: number;
// Upper bound on that ceiling, so a long outage settles into a steady
// poll instead of growing without limit.
maxDelayMs?: number;
// Injected so tests exercise the whole policy without waiting.
sleep?: (ms: number) => Promise<void>;
// Injected so the jitter is reproducible under test.
random?: () => number;
}
export type ResolvedRetryOptions = Required<RetryOptions>;
export const DEFAULT_RETRY_OPTIONS: ResolvedRetryOptions = {
attempts: 4,
baseDelayMs: 500,
maxDelayMs: 10_000,
sleep: (ms: number): Promise<void> =>
new Promise((resolve) => setTimeout(resolve, ms)),
random: Math.random,
};
export const resolveRetryOptions = (
opts?: RetryOptions,
): ResolvedRetryOptions => ({
attempts: opts?.attempts ?? DEFAULT_RETRY_OPTIONS.attempts,
baseDelayMs: opts?.baseDelayMs ?? DEFAULT_RETRY_OPTIONS.baseDelayMs,
maxDelayMs: opts?.maxDelayMs ?? DEFAULT_RETRY_OPTIONS.maxDelayMs,
sleep: opts?.sleep ?? DEFAULT_RETRY_OPTIONS.sleep,
random: opts?.random ?? DEFAULT_RETRY_OPTIONS.random,
});
// Transport failures: the request did not complete, for reasons below HTTP.
const TRANSPORT_CODES = new Set([
"ECONNRESET",
"ECONNABORTED",
"ETIMEDOUT",
"EPIPE",
"ENOTFOUND",
"EAI_AGAIN",
"ECONNREFUSED",
"EHOSTUNREACH",
"ENETUNREACH",
"ENETRESET",
"ENETDOWN",
]);
// The subset of the above that can only happen before any request byte was
// written: name resolution failed, or the connection was refused or never
// routed. See `isSafeToReplay`.
const CONNECT_CODES = new Set([
"ENOTFOUND",
"EAI_AGAIN",
"ECONNREFUSED",
"EHOSTUNREACH",
"ENETUNREACH",
"ENETDOWN",
]);
// `cause` is an arbitrary user-settable property and nothing prevents it from
// forming a cycle, so the walk is bounded. Hanging the process would be a
// worse outcome than any misclassification.
const MAX_CAUSE_DEPTH = 8;
// Collect every `code` in an error's cause chain. undici does not put the
// errno on the error it throws — it hangs the underlying socket error off
// `cause`, sometimes more than one level down — so a classifier that only read
// the top-level error would see a bare `Error` and call every dropped
// connection permanent.
const causeCodes = (err: unknown): string[] => {
const codes: string[] = [];
let current: unknown = err;
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
if (current === null || typeof current !== "object") break;
const { code, cause } = current as { code?: unknown; cause?: unknown };
if (typeof code === "string") codes.push(code);
if (cause === current) break;
current = cause;
}
return codes;
};
const isAbort = (err: unknown): boolean => {
if (err === null || typeof err !== "object") return false;
const { name } = err as { name?: unknown };
return name === "AbortError" || name === "TimeoutError";
};
// Is another attempt capable of producing a different answer?
//
// A note on the truncation case, recorded on issue #2. `TruncatedStreamError`
// is retried, and for a body of more than one chunk that is exactly right: a
// chunk that failed to authenticate while the stream carried on past it is
// corruption, stays an ordinary authentication failure, and is not retried.
//
// For a single-chunk body — most thumbnails, every small file — the split is
// not achievable. A wrong file key, server-side corruption and a connection
// cut mid-chunk are cryptographically identical: Poly1305 fails and carries no
// framing signal. All three are reported as truncation and therefore retried.
// That imprecision is deliberate and bounded by the attempt count: one wasted
// round trip on a genuinely corrupt file is a fair price for never silently
// keeping a truncated one, and the alternative — treating a single-chunk
// authentication failure as permanent — would reintroduce exactly the
// silent-corruption risk the truncation check exists to remove.
export const isRetryable = (err: unknown): boolean => {
if (err instanceof ApiError) {
// 408 and 429 are the two 4xx codes that are statements about timing
// rather than about the request, and backoff is the right answer to
// both. Every other 4xx will answer the same way however often it is
// asked.
if (err.status === 408 || err.status === 429) return true;
return err.status >= 500 && err.status <= 599;
}
if (err instanceof TruncatedStreamError) return true;
// quak only ever aborts a request on its own deadline, so an abort means
// this attempt ran out of time — which a later one may not.
if (isAbort(err)) return true;
// Node's fetch rejects with `TypeError: fetch failed` for everything below
// HTTP: DNS failure, refused connection, TLS error, reset socket. Nothing
// on the object separates it from a TypeError thrown by a bug, so this is
// deliberately literal. Demanding a recognised `cause` instead would
// classify real network failures as permanent and fail backups that should
// have succeeded; the cost of the imprecision is bounded by the attempt
// count.
if (err instanceof TypeError) return true;
return causeCodes(err).some((code) => TRANSPORT_CODES.has(code));
};
// Could the first attempt already have taken effect on the server?
//
// `isRetryable` is the wrong question for a request that changes state.
// quak's non-idempotent calls are `/users/srp/create-session`,
// `/users/two-factor/verify` — which consumes one of a small number of 2FA
// attempts — and `/files/thumbnail`. They are replayed only when the failure
// proves no request byte reached the server, which means the connection was
// never established.
//
// Everything else is ambiguous. A 5xx proves the server did process the
// request. A reset or a broken pipe can arrive after it was fully sent and
// acted on. A deadline says nothing at all about the server's state.
export const isSafeToReplay = (err: unknown): boolean =>
isRetryable(err) && causeCodes(err).some((code) => CONNECT_CODES.has(code));
export interface WithRetryOptions extends RetryOptions {
isRetryable?: (err: unknown) => boolean;
}
// Exponential backoff with full jitter: the exponential term is the ceiling,
// and the actual wait is drawn uniformly below it. Full jitter, rather than a
// fixed delay plus noise, is what stops a client that lost a hundred parallel
// downloads to one CDN blip from re-sending all hundred at the same instant.
const backoffMs = (retryNumber: number, policy: ResolvedRetryOptions): number =>
policy.random() *
Math.min(policy.maxDelayMs, policy.baseDelayMs * 2 ** (retryNumber - 1));
export const withRetry = async <T>(
fn: () => Promise<T>,
opts?: WithRetryOptions,
): Promise<T> => {
const policy = resolveRetryOptions(opts);
const retryable = opts?.isRetryable ?? isRetryable;
for (let attempt = 1; ; attempt++) {
try {
return await fn();
} catch (err) {
if (attempt >= policy.attempts || !retryable(err)) {
// The error escapes unwrapped, and it is the one from the
// final attempt: callers classify what they catch, and
// `listMissingThumbnails` in particular needs the `ApiError`
// and its status intact.
throw err;
}
await policy.sleep(backoffMs(attempt, policy));
}
}
};

View File

@@ -1,6 +1,8 @@
import { createHash } from "node:crypto"; import { createHash } from "node:crypto";
import sharp from "sharp"; import { readFileSync } from "node:fs";
import * as jpeg from "jpeg-js";
import type { Client } from "./client.js"; import type { Client } from "./client.js";
import { ApiError } from "./api/client.js";
import { encryptBlob, toBase64 } from "./crypto/index.js"; import { encryptBlob, toBase64 } from "./crypto/index.js";
import { downloadFile } from "./download/index.js"; import { downloadFile } from "./download/index.js";
import type { EnteFile } from "./model/types.js"; import type { EnteFile } from "./model/types.js";
@@ -61,29 +63,102 @@ export const listMissingThumbnails = async (
reason: "empty thumbnail (0 bytes)", reason: "empty thumbnail (0 bytes)",
}); });
} }
} catch { } catch (err) {
missing.push({ // A 404 is the server stating the thumbnail is not there:
fileID: file.id, // that, and an empty body, are the only two answers that mean
title: file.metadata.title, // "missing". Anything else reaching this point is a failure
collection: col.name, // that already exhausted its retries — a failing server, a
reason: "thumbnail fetch failed", // dropped connection, a deadline — and says nothing about
}); // whether the thumbnail exists.
//
// The distinction is what stops `helper
// fix-missing-thumbnails` from downloading originals,
// regenerating thumbnails and uploading them over thumbnails
// that were fine all along, because the CDN was briefly
// returning 500s while this ran.
if (err instanceof ApiError && err.status === 404) {
missing.push({
fileID: file.id,
title: file.metadata.title,
collection: col.name,
reason: "thumbnail not found (HTTP 404)",
});
} else {
log(
`[${col.name}] Could not check ${file.metadata.title}: ${err instanceof Error ? err.message : String(err)} (not reported as missing)`,
);
}
} }
} }
} }
return missing; return missing;
}; };
const generateThumbnail = async (originalPath: string): Promise<Uint8Array> => { // Bilinear resize of RGBA pixel buffer
const result = await sharp(originalPath) const resizeRGBA = (
.rotate() src: Uint8Array,
.resize(THUMB_MAX_DIMENSION, THUMB_MAX_DIMENSION, { srcW: number,
fit: "inside", srcH: number,
withoutEnlargement: true, dstW: number,
}) dstH: number,
.jpeg({ quality: THUMB_JPEG_QUALITY }) ): Uint8Array => {
.toBuffer(); const dst = new Uint8Array(dstW * dstH * 4);
return new Uint8Array(result); const xRatio = srcW / dstW;
const yRatio = srcH / dstH;
for (let y = 0; y < dstH; y++) {
const srcY = y * yRatio;
const y0 = Math.floor(srcY);
const y1 = Math.min(y0 + 1, srcH - 1);
const fy = srcY - y0;
for (let x = 0; x < dstW; x++) {
const srcX = x * xRatio;
const x0 = Math.floor(srcX);
const x1 = Math.min(x0 + 1, srcW - 1);
const fx = srcX - x0;
const i00 = (y0 * srcW + x0) * 4;
const i10 = (y0 * srcW + x1) * 4;
const i01 = (y1 * srcW + x0) * 4;
const i11 = (y1 * srcW + x1) * 4;
const di = (y * dstW + x) * 4;
for (let c = 0; c < 4; c++) {
dst[di + c] = Math.round(
src[i00 + c]! * (1 - fx) * (1 - fy) +
src[i10 + c]! * fx * (1 - fy) +
src[i01 + c]! * (1 - fx) * fy +
src[i11 + c]! * fx * fy,
);
}
}
}
return dst;
};
const generateThumbnail = (fileBytes: Uint8Array): Uint8Array => {
const decoded = jpeg.decode(fileBytes, {
useTArray: true,
formatAsRGBA: true,
});
const { width: srcW, height: srcH } = decoded;
const scale = Math.min(
THUMB_MAX_DIMENSION / srcW,
THUMB_MAX_DIMENSION / srcH,
1,
);
const dstW = Math.round(srcW * scale);
const dstH = Math.round(srcH * scale);
let pixels: Uint8Array;
if (scale < 1) {
pixels = resizeRGBA(decoded.data, srcW, srcH, dstW, dstH);
} else {
pixels = decoded.data;
}
const encoded = jpeg.encode(
{ data: pixels, width: dstW, height: dstH },
THUMB_JPEG_QUALITY,
);
return new Uint8Array(encoded.data);
}; };
export const fixMissingThumbnails = async ( export const fixMissingThumbnails = async (
@@ -139,7 +214,8 @@ export const fixMissingThumbnails = async (
log( log(
`[${collectionName}] Generating thumbnail for ${file.metadata.title}...`, `[${collectionName}] Generating thumbnail for ${file.metadata.title}...`,
); );
const thumbJpeg = await generateThumbnail(origPath); const fileBytes = readFileSync(origPath);
const thumbJpeg = generateThumbnail(new Uint8Array(fileBytes));
log( log(
`[${collectionName}] Encrypting and uploading thumbnail (${thumbJpeg.length} bytes)...`, `[${collectionName}] Encrypting and uploading thumbnail (${thumbJpeg.length} bytes)...`,

View File

@@ -29,12 +29,23 @@
* return a `ReadableStream<Uint8Array>` from the appropriate CDN * return a `ReadableStream<Uint8Array>` from the appropriate CDN
* (or the self-hosted fallback path). * (or the self-hosted fallback path).
* *
* - Retries and timeouts. Every request is issued under a deadline and,
* where it is safe to do so, retried with exponential backoff. The
* policy is `src/retry.ts`; what the last section of this file
* documents is which requests get it and which deliberately do not.
*
* All tests inject a fake `fetch` via the constructor so nothing touches * All tests inject a fake `fetch` via the constructor so nothing touches
* the network. The fake records every call for assertion. * the network. The fake records every call for assertion.
*/ */
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import { ApiClient, ApiError } from "../../src/api/client.js"; import {
ApiClient,
ApiError,
DEFAULT_DOWNLOAD_TIMEOUT_MS,
DEFAULT_REQUEST_TIMEOUT_MS,
} from "../../src/api/client.js";
import type { RetryOptions } from "../../src/retry.js";
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Test helpers // Test helpers
@@ -102,6 +113,92 @@ const recordingFetch = (
return { fetch: fake as typeof globalThis.fetch, calls }; return { fetch: fake as typeof globalThis.fetch, calls };
}; };
/**
* One scripted outcome for a single `fetch` call:
*
* - a `Response`, returned as-is;
* - an `Error`, thrown — this is how `fetch` reports a network failure;
* - `HANG`, a request that never answers until its own deadline aborts it.
*
* `HANG` is what makes the timeout tests honest. A fake that resolved after a
* delay would be testing the clock; this one resolves *only* when the signal
* quak attached fires. If no signal is attached, or the signal is not wired to
* the body, the promise never settles and the test fails on its own timeout
* rather than passing by accident.
*/
const HANG = Symbol("hang until aborted");
type FetchStep = Response | Error | typeof HANG;
const scriptedFetch = (
...steps: FetchStep[]
): {
fetch: typeof globalThis.fetch;
calls: { url: string; init: RequestInit | undefined }[];
} => {
const calls: { url: string; init: RequestInit | undefined }[] = [];
let i = 0;
const fake = async (
input: RequestInfo | URL,
init?: RequestInit,
): Promise<Response> => {
const url =
typeof input === "string"
? input
: input instanceof URL
? input.href
: input.url;
calls.push({ url, init });
const step = steps[i++];
if (step === undefined) {
throw new Error(`scriptedFetch: no step for call #${i - 1}`);
}
if (step === HANG) {
return new Promise<Response>((_resolve, reject) => {
const signal = init?.signal;
if (!signal) return;
if (signal.aborted) {
reject(signal.reason as Error);
return;
}
signal.addEventListener(
"abort",
() => reject(signal.reason as Error),
{ once: true },
);
});
}
if (step instanceof Error) throw step;
return step;
};
return { fetch: fake as typeof globalThis.fetch, calls };
};
/** An error shaped like a Node transport failure: the errno is on `.code`. */
const errnoError = (code: string, message = code): Error =>
Object.assign(new Error(message), { code });
/**
* A retry policy with the waiting removed. Backoff arithmetic is covered in
* `test/retry/retry.test.ts`; what the tests below are about is *how many
* requests* each call site issues, so they inject a `sleep` that returns
* immediately. Nothing in this file waits.
*/
const noWait: RetryOptions = {
sleep: () => Promise.resolve(),
random: () => 0,
};
/** Drain a stream and return the bytes, so body-level failures surface. */
const readAll = async (stream: ReadableStream<Uint8Array>): Promise<number> => {
const reader = stream.getReader();
let total = 0;
for (;;) {
const { done, value } = await reader.read();
if (value) total += value.length;
if (done) return total;
}
};
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Tests // Tests
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -383,3 +480,445 @@ describe("ApiClient.getFileStream / getThumbnailStream", () => {
expect(headers.get("X-Auth-Token")).toBe("tk"); expect(headers.get("X-Auth-Token")).toBe("tk");
}); });
}); });
// ---------------------------------------------------------------------------
// Retries
//
// The rule the whole section turns on: a request is repeated only when
// repeating it could produce a different answer, and only when repeating it
// cannot do harm. Those are two separate questions and the second one is why
// `postJSON` and `putJSON` behave differently from everything else here.
//
// Every assertion below counts requests. None of them measures how long
// anything took: the retry policy's `sleep` is injected and returns
// immediately, so a machine under load and an idle one produce identical
// results.
// ---------------------------------------------------------------------------
describe("ApiClient retries", () => {
it("issues exactly one request for a 404", async () => {
// A 404 is an answer, not a failure to get one. Repeating it wastes
// a round trip and — for `listMissingThumbnails`, which reads a 404
// as "this thumbnail really is missing" — delays a correct result.
// The script holds five responses; only the first may be consumed.
const { fetch, calls } = scriptedFetch(
textResponse("Not Found", 404),
textResponse("Not Found", 404),
textResponse("Not Found", 404),
textResponse("Not Found", 404),
textResponse("Not Found", 404),
);
const client = new ApiClient({ fetch, retry: noWait });
await expect(client.getJSON("/missing")).rejects.toBeInstanceOf(
ApiError,
);
expect(calls).toHaveLength(1);
});
it("retries a 500 up to the configured attempt count, then throws", async () => {
// `attempts` is a total, not a number of retries: three attempts mean
// three requests. The script offers five responses so that a client
// which ignored the limit would be visible as a count of 4 or 5
// rather than as a crash.
const { fetch, calls } = scriptedFetch(
textResponse("boom", 500),
textResponse("boom", 500),
textResponse("boom", 500),
textResponse("boom", 500),
textResponse("boom", 500),
);
const client = new ApiClient({
fetch,
retry: { ...noWait, attempts: 3 },
});
const err: unknown = await client
.getJSON("/flaky")
.catch((e: unknown) => e);
expect(err).toBeInstanceOf(ApiError);
expect((err as ApiError).status).toBe(500);
expect(calls).toHaveLength(3);
});
it("returns the first successful response after a 503", async () => {
const { fetch, calls } = scriptedFetch(
textResponse("unavailable", 503),
jsonResponse({ ok: true }),
);
const client = new ApiClient({ fetch, retry: noWait });
await expect(
client.getJSON<{ ok: boolean }>("/health"),
).resolves.toEqual({ ok: true });
expect(calls).toHaveLength(2);
});
it("retries 408 and 429", async () => {
// The two 4xx codes that are about timing rather than about the
// request. Backoff is precisely the right response to both.
for (const status of [408, 429]) {
const { fetch, calls } = scriptedFetch(
textResponse("wait", status),
jsonResponse({ ok: true }),
);
const client = new ApiClient({ fetch, retry: noWait });
await expect(
client.getJSON("/rate-limited"),
).resolves.toBeDefined();
expect(calls).toHaveLength(2);
}
});
it("retries a fetch rejection and succeeds on a later attempt", async () => {
// A dropped or refused connection is the failure this policy exists
// for: the request never got an answer, so asking again is free of
// consequence and likely to work.
const { fetch, calls } = scriptedFetch(
new TypeError("fetch failed"),
errnoError("ECONNRESET", "read ECONNRESET"),
jsonResponse({ collections: [] }),
);
const client = new ApiClient({ fetch, retry: noWait });
await expect(client.getJSON("/collections/v2")).resolves.toEqual({
collections: [],
});
expect(calls).toHaveLength(3);
});
it("applies the policy to file and thumbnail streams", async () => {
// Both CDN endpoints go through the same wrapper. A 503 from
// files.ente.io during a large backup is common enough that not
// retrying it would fail files for no reason.
for (const get of ["getFileStream", "getThumbnailStream"] as const) {
const { fetch, calls } = scriptedFetch(
textResponse("unavailable", 503),
streamResponse(new Uint8Array([1, 2, 3])),
);
const client = new ApiClient({ fetch, retry: noWait });
const stream = await client[get](7);
expect(await readAll(stream)).toBe(3);
expect(calls).toHaveLength(2);
}
});
it("does not retry when the caller opts out", async () => {
// `{ retry: false }` exists for one caller: the download layer, which
// wraps request *and* body consumption *and* decryption in a single
// retry of its own. Without the opt-out the two budgets would
// multiply — four attempts each becoming sixteen requests for one
// file.
const { fetch, calls } = scriptedFetch(
textResponse("unavailable", 503),
streamResponse(new Uint8Array([1, 2, 3])),
);
const client = new ApiClient({ fetch, retry: noWait });
await expect(
client.getFileStream(7, { retry: false }),
).rejects.toBeInstanceOf(ApiError);
expect(calls).toHaveLength(1);
});
it("exposes its resolved retry policy to the download layer", async () => {
// The download layer runs its own `withRetry` and must run it under
// the same policy the client was configured with, not under the
// library defaults.
const { fetch } = scriptedFetch(jsonResponse({}));
const client = new ApiClient({
fetch,
retry: { ...noWait, attempts: 9, baseDelayMs: 7, maxDelayMs: 11 },
});
const policy = client.getRetryOptions();
expect(policy.attempts).toBe(9);
expect(policy.baseDelayMs).toBe(7);
expect(policy.maxDelayMs).toBe(11);
});
});
describe("ApiClient timeouts", () => {
it("ships bounded default deadlines", () => {
// Asserted here so the README and the code cannot drift. Two numbers
// rather than one, because a deadline that is sane for a JSON call is
// nowhere near enough for a multi-gigabyte body, and a deadline long
// enough for that body would let a hung API call stall a backup for
// ten minutes.
expect(DEFAULT_REQUEST_TIMEOUT_MS).toBe(30_000);
expect(DEFAULT_DOWNLOAD_TIMEOUT_MS).toBe(600_000);
});
it("attaches an abort signal to every request", async () => {
const { fetch, calls } = recordingFetch(
jsonResponse({}),
jsonResponse({}),
new Response(null, { status: 200 }),
jsonResponse({}),
);
const client = new ApiClient({ fetch, retry: noWait });
await client.getJSON("/a");
await client.postJSON("/b", {});
await client.putFile("https://s3.example/x", new Uint8Array([1]));
await client.putJSON("/c", {});
for (const call of calls) {
expect(call.init?.signal).toBeInstanceOf(AbortSignal);
}
});
it("gives up on a request that never answers, and retries it", async () => {
// Before this policy existed there was no timeout anywhere in quak: a
// CDN connection that accepted the request and then went quiet would
// hang `quak backup` forever. `HANG` reproduces exactly that — the
// fake never answers, so the only thing that can end the call is the
// deadline quak attached.
const { fetch, calls } = scriptedFetch(HANG, HANG, HANG);
const client = new ApiClient({
fetch,
requestTimeoutMs: 20,
retry: { ...noWait, attempts: 3 },
});
await expect(client.getJSON("/black-hole")).rejects.toThrow();
// A timeout is retryable, so all three attempts were spent...
expect(calls).toHaveLength(3);
// ...each under its own fresh deadline, not one shared one that
// expired during the first attempt.
const signals = calls.map((c) => c.init?.signal);
expect(new Set(signals).size).toBe(3);
}, 5000);
it("recovers when a later attempt answers in time", async () => {
const { fetch, calls } = scriptedFetch(HANG, jsonResponse({ ok: 1 }));
const client = new ApiClient({
fetch,
requestTimeoutMs: 20,
retry: noWait,
});
await expect(client.getJSON("/slow-then-fast")).resolves.toEqual({
ok: 1,
});
expect(calls).toHaveLength(2);
}, 5000);
it("aborts a body that stalls after the headers arrived", async () => {
// The failure mode that a naive timeout misses. `getFileStream`
// returns as soon as headers arrive; the bytes are pulled later, in
// the download layer. A deadline that only guarded the initial fetch
// would leave the identical hang one layer down — which is where
// multi-megabyte photo downloads actually stall.
//
// This fake resolves its headers immediately and then serves a body
// that never produces a chunk and never observes the signal, so the
// only thing that can unblock the read is quak's own enforcement of
// the deadline over the stream it hands out.
const stalling = new Response(
new ReadableStream<Uint8Array>({
pull: () => new Promise<void>(() => {}),
}),
{ status: 200 },
);
const { fetch } = scriptedFetch(stalling);
const client = new ApiClient({
fetch,
downloadTimeoutMs: 20,
retry: { ...noWait, attempts: 1 },
});
const stream = await client.getFileStream(42);
const err: unknown = await readAll(stream).catch((e: unknown) => e);
expect(err).toBeInstanceOf(Error);
expect((err as Error).name).toBe("TimeoutError");
}, 5000);
it("lets a body that arrives in time through untouched", async () => {
// The counterpart to the previous test: enforcing the deadline over
// the stream must not corrupt or truncate a body that is simply being
// read normally.
const payload = new Uint8Array([9, 8, 7, 6, 5]);
const { fetch } = scriptedFetch(streamResponse(payload));
const client = new ApiClient({ fetch, retry: noWait });
const stream = await client.getFileStream(42);
const reader = stream.getReader();
const chunks: Uint8Array[] = [];
for (;;) {
const { done, value } = await reader.read();
if (done) break;
chunks.push(value);
}
const joined = new Uint8Array(chunks.reduce((n, c) => n + c.length, 0));
let offset = 0;
for (const c of chunks) {
joined.set(c, offset);
offset += c.length;
}
expect(joined).toEqual(payload);
});
});
describe("ApiClient error typing", () => {
it("throws ApiError with the status when a presigned PUT fails", async () => {
// `putFile` used to throw a bare Error with the status baked into a
// string. Nothing downstream could classify it, so a 500 from S3 was
// indistinguishable from a bug and could never be retried.
const { fetch, calls } = scriptedFetch(
new Response("Forbidden", { status: 403 }),
);
const client = new ApiClient({ fetch, retry: noWait });
const err: unknown = await client
.putFile("https://s3.example/obj", new Uint8Array([1, 2]))
.catch((e: unknown) => e);
expect(err).toBeInstanceOf(ApiError);
expect((err as ApiError).status).toBe(403);
expect(calls).toHaveLength(1);
});
it("retries a presigned PUT on a 5xx", async () => {
// A presigned PUT writes the whole object at one key in one request,
// so repeating it either overwrites the same bytes or lands them for
// the first time. There is no partial state to protect.
const { fetch, calls } = scriptedFetch(
new Response("slow down", { status: 503 }),
new Response(null, { status: 200 }),
);
const client = new ApiClient({ fetch, retry: noWait });
await client.putFile("https://s3.example/obj", new Uint8Array([1, 2]));
expect(calls).toHaveLength(2);
});
it("throws ApiError when a download response has no body", async () => {
// Also previously a bare Error. It carries the response status so a
// caller can see what arrived — and it is *not* retried: a 200 with
// no body is a malformed response, and asking again produces the same
// malformed response.
for (const get of ["getFileStream", "getThumbnailStream"] as const) {
const { fetch, calls } = scriptedFetch(
new Response(null, { status: 200 }),
new Response(null, { status: 200 }),
);
const client = new ApiClient({ fetch, retry: noWait });
const err: unknown = await client[get](5).catch((e: unknown) => e);
expect(err).toBeInstanceOf(ApiError);
expect((err as ApiError).status).toBe(200);
expect(calls).toHaveLength(1);
}
});
});
describe("ApiClient non-idempotent requests", () => {
/**
* `postJSON` and `putJSON` carry quak's only requests that change server
* state: `/users/srp/create-session`, `/users/two-factor/verify` — which
* consumes one of a small number of 2FA attempts — and `/files/thumbnail`.
*
* They are retried only when the failure proves the request never reached
* the server, which in practice means the connection was never
* established. Everything else is ambiguous: a 5xx proves the server did
* process the request, and a reset or a timeout can arrive after it did.
* Replaying under that ambiguity can burn a 2FA attempt or register a
* thumbnail twice, and neither is worth the round trip it saves.
*/
it("does not replay a POST after a 5xx", async () => {
const { fetch, calls } = scriptedFetch(
textResponse("boom", 500),
jsonResponse({ sessionID: "second" }),
);
const client = new ApiClient({ fetch, retry: noWait });
await expect(
client.postJSON("/users/two-factor/verify", { code: "123456" }),
).rejects.toBeInstanceOf(ApiError);
expect(calls).toHaveLength(1);
});
it("does not replay a POST after a mid-flight connection reset", async () => {
// A reset can happen after the request was fully sent and acted on.
// It is retryable in general — `getJSON` retries it — but it is not
// replay-safe.
const { fetch, calls } = scriptedFetch(
errnoError("ECONNRESET", "socket hang up"),
jsonResponse({ sessionID: "second" }),
);
const client = new ApiClient({ fetch, retry: noWait });
await expect(
client.postJSON("/users/srp/create-session", {}),
).rejects.toThrow(/ECONNRESET|socket hang up/);
expect(calls).toHaveLength(1);
});
it("does not replay a POST after a timeout", async () => {
// A deadline says nothing about whether the server acted.
const { fetch, calls } = scriptedFetch(HANG, jsonResponse({}));
const client = new ApiClient({
fetch,
requestTimeoutMs: 20,
retry: noWait,
});
await expect(client.postJSON("/users/ott", {})).rejects.toThrow();
expect(calls).toHaveLength(1);
}, 5000);
it("replays a POST when the connection was never established", async () => {
// A refused connection or a DNS failure happens before any request
// byte is written, so the server cannot have seen it. This is the one
// case where replaying is provably harmless.
const { fetch, calls } = scriptedFetch(
new TypeError("fetch failed", {
cause: errnoError("ECONNREFUSED", "connect ECONNREFUSED"),
}),
errnoError("EAI_AGAIN", "getaddrinfo EAI_AGAIN api.ente.io"),
jsonResponse({ sessionID: "third" }),
);
const client = new ApiClient({ fetch, retry: noWait });
await expect(
client.postJSON<{ sessionID: string }>(
"/users/srp/create-session",
{},
),
).resolves.toEqual({ sessionID: "third" });
expect(calls).toHaveLength(3);
});
it("applies the same rule to PUT", async () => {
// `/files/thumbnail` is reached through `putJSON`.
const failing = scriptedFetch(
textResponse("boom", 503),
jsonResponse({}),
);
const failingClient = new ApiClient({
fetch: failing.fetch,
retry: noWait,
});
await expect(
failingClient.updateThumbnail(1, "key", "header"),
).rejects.toBeInstanceOf(ApiError);
expect(failing.calls).toHaveLength(1);
const refused = scriptedFetch(
errnoError("ECONNREFUSED", "connect ECONNREFUSED"),
jsonResponse({}),
);
const refusedClient = new ApiClient({
fetch: refused.fetch,
retry: noWait,
});
await refusedClient.updateThumbnail(1, "key", "header");
expect(refused.calls).toHaveLength(2);
});
});

139
test/api/upload.test.ts Normal file
View File

@@ -0,0 +1,139 @@
/**
* Tests for the upload-related ApiClient methods added for thumbnail
* repair: `putJSON`, `putFile`, `getUploadURL`, and `updateThumbnail`.
*
* `putFile` sends a raw PUT to a presigned S3 URL. It must NOT send
* quak's auth headers (X-Auth-Token, X-Client-Package) because the
* presigned URL carries its own S3 auth in the query string. Sending
* extra headers can cause S3 to reject the request.
*
* `putJSON` is like `postJSON` but sends PUT. Used by `updateThumbnail`
* to register the uploaded thumbnail's object key with the Ente API.
*/
import { describe, expect, it } from "vitest";
import { ApiClient } from "../../src/api/client.js";
const jsonResponse = (
body: unknown,
status = 200,
headers: Record<string, string> = {},
): Response =>
new Response(JSON.stringify(body), {
status,
headers: { "content-type": "application/json", ...headers },
});
const recordingFetch = (
...responses: Response[]
): {
fetch: typeof globalThis.fetch;
calls: { url: string; init: RequestInit | undefined }[];
} => {
const calls: { url: string; init: RequestInit | undefined }[] = [];
let i = 0;
const fake = async (
input: RequestInfo | URL,
init?: RequestInit,
): Promise<Response> => {
const url =
typeof input === "string"
? input
: input instanceof URL
? input.href
: input.url;
calls.push({ url, init });
if (i >= responses.length) {
throw new Error(`recordingFetch: no response for call #${i}`);
}
return responses[i++]!;
};
return { fetch: fake as typeof globalThis.fetch, calls };
};
describe("ApiClient.putJSON", () => {
it("sends a PUT request with JSON body and auth headers", async () => {
const { fetch, calls } = recordingFetch(jsonResponse({ ok: true }));
const client = new ApiClient({ fetch, authToken: "tok" });
await client.putJSON("/files/thumbnail", {
fileID: 42,
thumbnail: { objectKey: "k", decryptionHeader: "h" },
});
expect(calls[0]!.init?.method).toBe("PUT");
const headers = new Headers(calls[0]!.init?.headers as HeadersInit);
expect(headers.get("Content-Type")).toBe("application/json");
expect(headers.get("X-Auth-Token")).toBe("tok");
expect(headers.get("X-Client-Package")).toBe("berlin.sneak.quak");
const body = JSON.parse(calls[0]!.init?.body as string);
expect(body.fileID).toBe(42);
expect(body.thumbnail.objectKey).toBe("k");
});
});
describe("ApiClient.putFile", () => {
it("PUTs raw bytes to the presigned URL without auth headers", async () => {
const { fetch, calls } = recordingFetch(
new Response(null, { status: 200 }),
);
const client = new ApiClient({ fetch, authToken: "secret-tok" });
const data = new Uint8Array([1, 2, 3, 4, 5]);
await client.putFile("https://s3.example.com/presigned?sig=abc", data);
expect(calls[0]!.url).toBe("https://s3.example.com/presigned?sig=abc");
expect(calls[0]!.init?.method).toBe("PUT");
const headers = new Headers(calls[0]!.init?.headers as HeadersInit);
expect(headers.get("Content-Type")).toBe("application/octet-stream");
expect(headers.get("Content-Length")).toBe("5");
// Must NOT leak auth headers to S3
expect(headers.has("X-Auth-Token")).toBe(false);
expect(headers.has("X-Client-Package")).toBe(false);
});
it("throws on non-2xx response from presigned URL", async () => {
const { fetch } = recordingFetch(
new Response("Forbidden", { status: 403 }),
);
const client = new ApiClient({ fetch });
await expect(
client.putFile("https://s3.example.com/bad", new Uint8Array(10)),
).rejects.toThrow(/403/);
});
});
describe("ApiClient.getUploadURL", () => {
it("POSTs to /files/upload-url with contentLength and contentMD5", async () => {
const { fetch, calls } = recordingFetch(
jsonResponse({ objectKey: "user/thumb123", url: "https://s3/put" }),
);
const client = new ApiClient({ fetch, authToken: "tok" });
const result = await client.getUploadURL(5000, "abc123==");
expect(calls[0]!.url).toBe("https://api.ente.io/files/upload-url");
const body = JSON.parse(calls[0]!.init?.body as string);
expect(body.contentLength).toBe(5000);
expect(body.contentMD5).toBe("abc123==");
expect(result.objectKey).toBe("user/thumb123");
expect(result.url).toBe("https://s3/put");
});
});
describe("ApiClient.updateThumbnail", () => {
it("PUTs to /files/thumbnail with fileID, objectKey, and decryptionHeader", async () => {
const { fetch, calls } = recordingFetch(jsonResponse({}));
const client = new ApiClient({ fetch, authToken: "tok" });
await client.updateThumbnail(42, "user/obj", "headerBase64==");
expect(calls[0]!.url).toBe("https://api.ente.io/files/thumbnail");
expect(calls[0]!.init?.method).toBe("PUT");
const body = JSON.parse(calls[0]!.init?.body as string);
expect(body.fileID).toBe(42);
expect(body.thumbnail.objectKey).toBe("user/obj");
expect(body.thumbnail.decryptionHeader).toBe("headerBase64==");
});
});

View File

@@ -145,7 +145,7 @@ const buildServerFixture = async (password: string) => {
*/ */
const buildMockFetch = ( const buildMockFetch = (
fixture: Awaited<ReturnType<typeof buildServerFixture>>, fixture: Awaited<ReturnType<typeof buildServerFixture>>,
opts?: { requireTOTP?: boolean }, opts?: { requireTOTP?: boolean; requirePasskeyAndTOTP?: boolean },
) => { ) => {
let srpServer: SrpServer; let srpServer: SrpServer;
const sessionID = "test-session-id"; const sessionID = "test-session-id";
@@ -200,11 +200,47 @@ const buildMockFetch = (
srpServer.checkM1(Buffer.from(body.srpM1, "base64")); srpServer.checkM1(Buffer.from(body.srpM1, "base64"));
const M2 = srpServer.computeM2(); const M2 = srpServer.computeM2();
// IMPORTANT: the museum server's EmailAuthorizationResponse
// (server/ente/user.go) declares passkeySessionID, accountsUrl,
// twoFactorSessionID, and twoFactorSessionIDV2 WITHOUT the
// `omitempty` JSON tag. Go therefore always serializes them,
// sending "" (empty string, NOT null/absent) for any that do
// not apply. These mocks must reproduce that faithfully: a
// client that distinguishes fields with `??` instead of `||`
// passes against an omitting mock but breaks against the real
// server.
if (opts?.requireTOTP) { if (opts?.requireTOTP) {
return new Response( return new Response(
JSON.stringify({ JSON.stringify({
id: 42,
srpM2: M2.toString("base64"), srpM2: M2.toString("base64"),
passkeySessionID: "",
accountsUrl: "",
twoFactorSessionID: "totp-session-999", twoFactorSessionID: "totp-session-999",
twoFactorSessionIDV2: "",
}),
{
status: 200,
headers: { "content-type": "application/json" },
},
);
}
if (opts?.requirePasskeyAndTOTP) {
// When the account has BOTH passkeys and TOTP enabled, the
// server sets passkeySessionID + twoFactorSessionIDV2 (not
// twoFactorSessionID -- that's deliberate, so old clients
// that only know the V1 field keep using the passkey flow).
// The V1 field is still present on the wire as "".
return new Response(
JSON.stringify({
id: 42,
srpM2: M2.toString("base64"),
passkeySessionID: "passkey-session-123",
accountsUrl: "https://accounts.ente.io",
twoFactorSessionID: "",
twoFactorSessionIDV2: "totp-session-v2-456",
}), }),
{ {
status: 200, status: 200,
@@ -219,6 +255,10 @@ const buildMockFetch = (
id: 42, id: 42,
keyAttributes: fixture.keyAttributes, keyAttributes: fixture.keyAttributes,
encryptedToken: fixture.encryptedToken, encryptedToken: fixture.encryptedToken,
passkeySessionID: "",
accountsUrl: "",
twoFactorSessionID: "",
twoFactorSessionIDV2: "",
}), }),
{ {
status: 200, status: 200,
@@ -305,6 +345,26 @@ describe("beginLogin via SRP", () => {
} }
}); });
it("prefers TOTP over passkey when the account has both", async () => {
// Accounts with both passkeys and TOTP get passkeySessionID +
// twoFactorSessionIDV2 in the verify-session response. A CLI
// cannot perform a WebAuthn ceremony, so quak must take the TOTP
// path using the V2 session ID. Returning { kind: "passkey" } here
// would make such accounts unusable from the CLI even though they
// have a perfectly good TOTP secret enrolled.
const fixture = await buildServerFixture(TEST_PASSWORD);
const api = new ApiClient({
fetch: buildMockFetch(fixture, { requirePasskeyAndTOTP: true }),
});
const challenge = await beginLogin(api, TEST_EMAIL, TEST_PASSWORD);
expect(challenge.kind).toBe("totp");
if (challenge.kind === "totp") {
expect(challenge.sessionID).toBe("totp-session-v2-456");
}
});
it("rejects a wrong password (SRP M1 verification fails)", async () => { it("rejects a wrong password (SRP M1 verification fails)", async () => {
const fixture = await buildServerFixture(TEST_PASSWORD); const fixture = await buildServerFixture(TEST_PASSWORD);
const api = new ApiClient({ fetch: buildMockFetch(fixture) }); const api = new ApiClient({ fetch: buildMockFetch(fixture) });

View File

@@ -411,11 +411,22 @@ describe("quak backup", () => {
// File 101 (sunset.jpg) will return HTTP 500. The other two // File 101 (sunset.jpg) will return HTTP 500. The other two
// files must still download. The result must report the failure // files must still download. The result must report the failure
// without throwing. // without throwing.
//
// A 500 is retryable, so this file now costs several requests before
// it is given up on — that is the point of the retry policy, and
// `runBackup`'s own resilience is unchanged by it: the retry lives
// strictly below this loop, and an exhausted file is still logged,
// counted, and stepped over rather than aborting the run. The
// injected `sleep` is what keeps the suite from actually waiting out
// the backoff.
const outDir = join(testDir, "partial-failure"); const outDir = join(testDir, "partial-failure");
const client = await Client.login({ const client = await Client.login({
email: TEST_EMAIL, email: TEST_EMAIL,
password: TEST_PASSWORD, password: TEST_PASSWORD,
apiOptions: { fetch: buildMockFetch(mock, { failFileID: 101 }) }, apiOptions: {
fetch: buildMockFetch(mock, { failFileID: 101 }),
retry: { sleep: () => Promise.resolve(), random: () => 0 },
},
}); });
const result = await runBackup(client, outDir); const result = await runBackup(client, outDir);

View File

@@ -0,0 +1,623 @@
/**
* Tests for `quak backup-metadata <dir>`.
*
* This command dumps all decrypted account metadata into a directory
* tree of plain JSON files, without downloading any file content. It
* is fast (no multi-megabyte downloads) and produces a complete
* plaintext record of every collection name, file title, creation
* date, GPS coordinate, camera model, caption, face label, and any
* other metadata the Ente clients have attached.
*
* Layout:
*
* <dir>/
* account.json { email, userID }
* collections/
* <id>-<sanitized-name>/
* _collection.json { id, name, type, magicMetadata?, ... }
* <fileID>.json { id, metadata, magicMetadata?, pubMagicMetadata? }
*
* The test builds a mock server with two collections, each with files
* that have different combinations of metadata layers, and verifies
* the output tree is correct and complete.
*/
import { gzipSync } from "node:zlib";
import {
existsSync,
mkdtempSync,
readFileSync,
readdirSync,
rmSync,
} from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import sodium from "libsodium-wrappers-sumo";
import { SRP, SrpServer } from "fast-srp-hap";
import { beforeAll, afterAll, describe, expect, it } from "vitest";
import {
init,
toBase64,
deriveKEK,
deriveLoginSubkey,
encryptBlob,
} from "../../src/crypto/index.js";
import * as jpegJs from "jpeg-js";
import { Client } from "../../src/client.js";
import { runMetadataBackup } from "../../src/metadata-backup.js";
import type { KeyAttributes } from "../../src/auth/types.js";
const TEST_EMAIL = "metabackup@example.com";
const TEST_PASSWORD = "metapass";
const TEST_OPS = 2;
const TEST_MEM = 64 * 1024 * 1024;
interface MetaMockState {
verifier: Buffer;
srpAttributes: Record<string, unknown>;
keyAttributes: KeyAttributes;
encryptedToken: string;
collections: Record<string, unknown>[];
filesByCollection: Record<number, Record<string, unknown>[]>;
// For ML data and EXIF tests
encryptedMLData: Record<
number,
{ encryptedData: string; decryptionHeader: string }
>;
fileCiphertexts: Record<number, Uint8Array>;
}
let mock: MetaMockState;
let testDir: string;
const encryptSecretbox = (plaintext: Uint8Array, key: Uint8Array) => {
const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const ciphertext = sodium.crypto_secretbox_easy(plaintext, nonce, key);
return { ciphertext, nonce };
};
const encryptStreamBlob = (plaintext: Uint8Array, key: Uint8Array) => {
const push = sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const ciphertext = sodium.crypto_secretstream_xchacha20poly1305_push(
push.state,
plaintext,
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
return { ciphertext, header: push.header };
};
const buildMetaMock = async (): Promise<MetaMockState> => {
const kekSalt = sodium.randombytes_buf(sodium.crypto_pwhash_SALTBYTES);
const kek = await deriveKEK(TEST_PASSWORD, kekSalt, TEST_OPS, TEST_MEM);
const loginSubKeyBytes = deriveLoginSubkey(kek);
const srpUserID = "meta-srp";
const srpSalt = sodium.randombytes_buf(16);
const verifier = SRP.computeVerifier(
SRP.params["4096"],
Buffer.from(srpSalt),
Buffer.from(srpUserID),
Buffer.from(loginSubKeyBytes),
);
const masterKey = sodium.randombytes_buf(32);
const { ciphertext: encMK, nonce: mkNonce } = encryptSecretbox(
masterKey,
kek,
);
const kp = sodium.crypto_box_keypair();
const { ciphertext: encSK, nonce: skNonce } = encryptSecretbox(
kp.privateKey,
masterKey,
);
const tokenBytes = sodium.randombytes_buf(32);
const encToken = sodium.crypto_box_seal(tokenBytes, kp.publicKey);
const keyAttributes: KeyAttributes = {
kekSalt: toBase64(kekSalt),
encryptedKey: toBase64(encMK),
keyDecryptionNonce: toBase64(mkNonce),
publicKey: toBase64(kp.publicKey),
encryptedSecretKey: toBase64(encSK),
secretKeyDecryptionNonce: toBase64(skNonce),
memLimit: TEST_MEM,
opsLimit: TEST_OPS,
};
// Collection 1: "Vacation" with collection-level pubMagicMetadata
const ck1 = sodium.crypto_secretbox_keygen();
const { ciphertext: encCK1, nonce: ck1N } = encryptSecretbox(
ck1,
masterKey,
);
const { ciphertext: encCN1, nonce: cn1N } = encryptSecretbox(
new TextEncoder().encode("Vacation"),
ck1,
);
const collPubMagic = JSON.stringify({ coverID: 999, sortBy: "date" });
const { ciphertext: encCollPM, header: collPMHeader } = encryptStreamBlob(
new TextEncoder().encode(collPubMagic),
ck1,
);
const rawColl1 = {
id: 10,
owner: { id: 42 },
encryptedKey: toBase64(encCK1),
keyDecryptionNonce: toBase64(ck1N),
encryptedName: toBase64(encCN1),
nameDecryptionNonce: toBase64(cn1N),
type: "album",
updationTime: 1700000000000000,
pubMagicMetadata: {
version: 1,
count: 1,
data: toBase64(encCollPM),
header: toBase64(collPMHeader),
},
};
// Collection 2: "Work" with no magic metadata
const ck2 = sodium.crypto_secretbox_keygen();
const { ciphertext: encCK2, nonce: ck2N } = encryptSecretbox(
ck2,
masterKey,
);
const { ciphertext: encCN2, nonce: cn2N } = encryptSecretbox(
new TextEncoder().encode("Work"),
ck2,
);
const rawColl2 = {
id: 20,
owner: { id: 42 },
encryptedKey: toBase64(encCK2),
keyDecryptionNonce: toBase64(ck2N),
encryptedName: toBase64(encCN2),
nameDecryptionNonce: toBase64(cn2N),
type: "folder",
updationTime: 1700000000000000,
};
// File 100 in coll 10: has metadata + pubMagicMetadata
const fk1 = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { ciphertext: encFK1, nonce: fk1N } = encryptSecretbox(fk1, ck1);
const meta1 = JSON.stringify({
title: "beach.jpg",
fileType: 0,
creationTime: 1700000000000000,
modificationTime: 1700000000000000,
latitude: 35.6762,
longitude: 139.6503,
});
const { ciphertext: encMeta1, header: meta1Header } = encryptStreamBlob(
new TextEncoder().encode(meta1),
fk1,
);
const pubMagic1 = JSON.stringify({
w: 3000,
h: 2000,
cameraMake: "SONY",
cameraModel: "DSC-RX1RM3",
});
const { ciphertext: encPM1, header: pm1Header } = encryptStreamBlob(
new TextEncoder().encode(pubMagic1),
fk1,
);
const rawFile1 = {
id: 100,
collectionID: 10,
ownerID: 42,
encryptedKey: toBase64(encFK1),
keyDecryptionNonce: toBase64(fk1N),
metadata: {
encryptedData: toBase64(encMeta1),
decryptionHeader: toBase64(meta1Header),
},
pubMagicMetadata: {
version: 1,
count: 1,
data: toBase64(encPM1),
header: toBase64(pm1Header),
},
file: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) },
thumbnail: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) },
updationTime: 1700000000000000,
};
// File 200 in coll 20: metadata only, no magic metadata
const fk2 = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { ciphertext: encFK2, nonce: fk2N } = encryptSecretbox(fk2, ck2);
const meta2 = JSON.stringify({
title: "diagram.png",
fileType: 0,
creationTime: 1710000000000000,
modificationTime: 1710000000000000,
});
const { ciphertext: encMeta2, header: meta2Header } = encryptStreamBlob(
new TextEncoder().encode(meta2),
fk2,
);
const rawFile2 = {
id: 200,
collectionID: 20,
ownerID: 42,
encryptedKey: toBase64(encFK2),
keyDecryptionNonce: toBase64(fk2N),
metadata: {
encryptedData: toBase64(encMeta2),
decryptionHeader: toBase64(meta2Header),
},
file: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) },
thumbnail: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) },
updationTime: 1710000000000000,
};
// Encrypt ML data for file 100 (gzipped JSON, encrypted with file key)
const mlPayload = JSON.stringify({
face: {
version: 1,
client: "test",
width: 3000,
height: 2000,
faces: [
{
faceID: "face-abc",
detection: {
box: { x: 0.1, y: 0.2, width: 0.3, height: 0.4 },
landmarks: [
{ x: 0.15, y: 0.25 },
{ x: 0.25, y: 0.25 },
],
},
score: 0.98,
blur: 12.5,
embedding: [0.1, 0.2, 0.3],
},
],
},
clip: {
version: 1,
client: "test",
embedding: [0.5, 0.6, 0.7],
},
});
const gzipped = gzipSync(Buffer.from(mlPayload));
const { header: mlHeader, ciphertext: mlCiphertext } = encryptBlob(
new Uint8Array(gzipped),
fk1,
);
// Generate a real JPEG for EXIF extraction tests
const jw = 100;
const jh = 80;
const jpixels = new Uint8Array(jw * jh * 4);
for (let i = 0; i < jpixels.length; i += 4) {
jpixels[i] = 255;
jpixels[i + 1] = 0;
jpixels[i + 2] = 0;
jpixels[i + 3] = 255;
}
const tinyJpeg = jpegJs.encode(
{ data: jpixels, width: jw, height: jh },
80,
).data;
const filePush1 =
sodium.crypto_secretstream_xchacha20poly1305_init_push(fk1);
const encFileBody1 = sodium.crypto_secretstream_xchacha20poly1305_push(
filePush1.state,
new Uint8Array(tinyJpeg),
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
// Patch rawFile1's file.decryptionHeader to match the push header
rawFile1.file.decryptionHeader = toBase64(filePush1.header);
return {
verifier,
srpAttributes: {
srpUserID,
srpSalt: toBase64(srpSalt),
memLimit: TEST_MEM,
opsLimit: TEST_OPS,
kekSalt: toBase64(kekSalt),
isEmailMFAEnabled: false,
},
keyAttributes,
encryptedToken: toBase64(encToken),
collections: [rawColl1, rawColl2],
filesByCollection: { 10: [rawFile1], 20: [rawFile2] },
encryptedMLData: {
100: {
encryptedData: toBase64(mlCiphertext),
decryptionHeader: toBase64(mlHeader),
},
},
fileCiphertexts: { 100: encFileBody1 },
};
};
const buildMetaFetch = (m: MetaMockState) => {
let srpServer: SrpServer;
return (async (
input: RequestInfo | URL,
init?: RequestInit,
): Promise<Response> => {
const url =
typeof input === "string"
? input
: input instanceof URL
? input.href
: input.url;
const path = new URL(url).pathname;
const json = (body: unknown) =>
new Response(JSON.stringify(body), {
status: 200,
headers: { "content-type": "application/json" },
});
if (path === "/users/srp/attributes")
return json({ attributes: m.srpAttributes });
if (path === "/users/srp/create-session") {
const body = JSON.parse(init?.body as string);
const serverKey = await SRP.genKey();
srpServer = new SrpServer(
SRP.params["4096"],
m.verifier,
serverKey,
);
const B = srpServer.computeB();
srpServer.setA(Buffer.from(body.srpA, "base64"));
return json({ sessionID: "s1", srpB: B.toString("base64") });
}
if (path === "/users/srp/verify-session") {
const body = JSON.parse(init?.body as string);
srpServer.checkM1(Buffer.from(body.srpM1, "base64"));
return json({
srpM2: srpServer.computeM2().toString("base64"),
id: 42,
keyAttributes: m.keyAttributes,
encryptedToken: m.encryptedToken,
});
}
if (path === "/collections/v2")
return json({ collections: m.collections });
if (path === "/collections/v2/diff") {
const collID = Number(
new URL(url).searchParams.get("collectionID"),
);
return json({
diff: m.filesByCollection[collID] ?? [],
hasMore: false,
});
}
if (path === "/files/data/fetch") {
const body = JSON.parse(init?.body as string);
const data = (body.fileIDs as number[])
.filter((id: number) => m.encryptedMLData[id])
.map((id: number) => ({
fileID: id,
...m.encryptedMLData[id],
updatedAt: 1700000000000000,
}));
return json({ data });
}
if (
url.includes("files.ente.io") ||
path.startsWith("/files/download/")
) {
const parsed = new URL(url);
const fileID = Number(
parsed.searchParams.get("fileID") ?? path.split("/").pop(),
);
const ct = m.fileCiphertexts[fileID];
if (ct) return new Response(ct, { status: 200 });
return new Response("not found", { status: 404 });
}
return new Response("not found", { status: 404 });
}) as typeof globalThis.fetch;
};
beforeAll(async () => {
await init();
await sodium.ready;
mock = await buildMetaMock();
testDir = mkdtempSync(join(tmpdir(), "quak-meta-backup-test-"));
});
afterAll(() => {
if (testDir && existsSync(testDir))
rmSync(testDir, { recursive: true, force: true });
});
describe("quak backup-metadata", () => {
it("writes account.json with email and userID", async () => {
const outDir = join(testDir, "full");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
const account = JSON.parse(
readFileSync(join(outDir, "account.json"), "utf-8"),
);
expect(account.email).toBe(TEST_EMAIL);
expect(account.userID).toBe(42);
});
it("creates per-collection directories with _collection.json", async () => {
const outDir = join(testDir, "collections");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
const collDirs = readdirSync(join(outDir, "collections"));
expect(collDirs.length).toBe(2);
// Find the Vacation collection dir (prefixed with ID)
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
expect(vacDir).toBeDefined();
const collMeta = JSON.parse(
readFileSync(
join(outDir, "collections", vacDir, "_collection.json"),
"utf-8",
),
);
expect(collMeta.id).toBe(10);
expect(collMeta.name).toBe("Vacation");
expect(collMeta.type).toBe("album");
});
it("decrypts collection-level pubMagicMetadata", async () => {
const outDir = join(testDir, "coll-magic");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
const collMeta = JSON.parse(
readFileSync(
join(outDir, "collections", vacDir, "_collection.json"),
"utf-8",
),
);
expect(collMeta.pubMagicMetadata).toBeDefined();
expect(collMeta.pubMagicMetadata.coverID).toBe(999);
expect(collMeta.pubMagicMetadata.sortBy).toBe("date");
});
it("writes per-file JSON with all three metadata layers", async () => {
const outDir = join(testDir, "file-meta");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
const fileMeta = JSON.parse(
readFileSync(
join(outDir, "collections", vacDir, "100.json"),
"utf-8",
),
);
expect(fileMeta.id).toBe(100);
expect(fileMeta.metadata.title).toBe("beach.jpg");
expect(fileMeta.metadata.latitude).toBeCloseTo(35.6762);
expect(fileMeta.pubMagicMetadata.cameraMake).toBe("SONY");
expect(fileMeta.pubMagicMetadata.w).toBe(3000);
});
it("handles files with no magic metadata gracefully", async () => {
const outDir = join(testDir, "no-magic");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const workDir = collDirs.find((d) => d.includes("Work"))!;
const fileMeta = JSON.parse(
readFileSync(
join(outDir, "collections", workDir, "200.json"),
"utf-8",
),
);
expect(fileMeta.id).toBe(200);
expect(fileMeta.metadata.title).toBe("diagram.png");
expect(fileMeta.pubMagicMetadata).toBeUndefined();
expect(fileMeta.magicMetadata).toBeUndefined();
});
it("is incremental: second run does not fail", async () => {
const outDir = join(testDir, "incremental");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runMetadataBackup(client, outDir);
const account = JSON.parse(
readFileSync(join(outDir, "account.json"), "utf-8"),
);
expect(account.email).toBe(TEST_EMAIL);
});
it("fetches and decrypts ML data by default", async () => {
const outDir = join(testDir, "ml-data");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
const fileMeta = JSON.parse(
readFileSync(
join(outDir, "collections", vacDir, "100.json"),
"utf-8",
),
);
// ML data should be present and decrypted
expect(fileMeta.mlData).toBeDefined();
expect(fileMeta.mlData.face).toBeDefined();
expect(fileMeta.mlData.face.faces.length).toBe(1);
expect(fileMeta.mlData.face.faces[0].faceID).toBe("face-abc");
expect(fileMeta.mlData.face.faces[0].score).toBeCloseTo(0.98);
expect(fileMeta.mlData.face.faces[0].detection.box.x).toBeCloseTo(0.1);
expect(fileMeta.mlData.clip).toBeDefined();
expect(fileMeta.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
});
it("extracts EXIF from downloaded files when --exif is set", async () => {
const outDir = join(testDir, "exif-data");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir, { exif: true });
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
const fileMeta = JSON.parse(
readFileSync(
join(outDir, "collections", vacDir, "100.json"),
"utf-8",
),
);
// imageMetadata from JPEG parsing should be present
expect(fileMeta.imageMetadata).toBeDefined();
expect(fileMeta.imageMetadata.format).toBe("jpeg");
expect(fileMeta.imageMetadata.width).toBe(100);
expect(fileMeta.imageMetadata.height).toBe(80);
});
});

View File

@@ -91,6 +91,7 @@ interface ServerState {
thumbHeader: Uint8Array; thumbHeader: Uint8Array;
thumbCiphertext: Uint8Array; thumbCiphertext: Uint8Array;
collectionKey: Uint8Array; collectionKey: Uint8Array;
sharedCollectionKey: Uint8Array;
} }
let server: ServerState; let server: ServerState;
@@ -211,6 +212,69 @@ const buildServer = async (): Promise<ServerState> => {
updationTime: 1700000000000000, updationTime: 1700000000000000,
}; };
// A collection another user shared WITH us. The sharer does not have
// our master key, only our public key, so the real server delivers the
// collection key as an anonymous sealed box (crypto_box_seal) to our
// public key and the response carries NO keyDecryptionNonce. Every
// account with an incoming shared album has one of these in its
// /collections/v2 response, so the mock must include one too.
const sharedCollectionKey = sodium.crypto_secretbox_keygen();
const sealedSharedKey = sodium.crypto_box_seal(
sharedCollectionKey,
kp.publicKey,
);
const sharedNameNonce = sodium.randombytes_buf(
sodium.crypto_secretbox_NONCEBYTES,
);
const encSharedName = sodium.crypto_secretbox_easy(
new TextEncoder().encode("Friend's Wedding"),
sharedNameNonce,
sharedCollectionKey,
);
const rawSharedCollection = {
id: 2,
owner: { id: 99 },
encryptedKey: toBase64(sealedSharedKey),
encryptedName: toBase64(encSharedName),
nameDecryptionNonce: toBase64(sharedNameNonce),
type: "album",
updationTime: 1700000000000000,
};
// A DELETED collection. /collections/v2 is a sync API: deleted
// collections stay in the response forever as tombstones with
// isDeleted: true (a long-lived real account accumulates hundreds).
// Their keys still decrypt, but /collections/v2/diff returns
// HTTP 404 for them, so they must never surface from
// listCollections(); a client that naively iterates them dies on
// the first deleted album.
const deletedKey = sodium.crypto_secretbox_keygen();
const dkNonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const encDeletedKey = sodium.crypto_secretbox_easy(
deletedKey,
dkNonce,
masterKey,
);
const deletedNameNonce = sodium.randombytes_buf(
sodium.crypto_secretbox_NONCEBYTES,
);
const encDeletedName = sodium.crypto_secretbox_easy(
new TextEncoder().encode("Old Album"),
deletedNameNonce,
deletedKey,
);
const rawDeletedCollection = {
id: 3,
owner: { id: 42 },
encryptedKey: toBase64(encDeletedKey),
keyDecryptionNonce: toBase64(dkNonce),
encryptedName: toBase64(encDeletedName),
nameDecryptionNonce: toBase64(deletedNameNonce),
type: "album",
updationTime: 1700000000000000,
isDeleted: true,
};
const rawFile = { const rawFile = {
id: 100, id: 100,
collectionID: 1, collectionID: 1,
@@ -230,6 +294,10 @@ const buildServer = async (): Promise<ServerState> => {
// can return them. This is ugly plumbing; in a real program you // can return them. This is ugly plumbing; in a real program you
// never see any of it. // never see any of it.
(globalThis as Record<string, unknown>).__mockRawCollection = rawCollection; (globalThis as Record<string, unknown>).__mockRawCollection = rawCollection;
(globalThis as Record<string, unknown>).__mockRawSharedCollection =
rawSharedCollection;
(globalThis as Record<string, unknown>).__mockRawDeletedCollection =
rawDeletedCollection;
(globalThis as Record<string, unknown>).__mockRawFile = rawFile; (globalThis as Record<string, unknown>).__mockRawFile = rawFile;
return { return {
@@ -253,6 +321,7 @@ const buildServer = async (): Promise<ServerState> => {
thumbHeader: thumbPush.header, thumbHeader: thumbPush.header,
thumbCiphertext, thumbCiphertext,
collectionKey, collectionKey,
sharedCollectionKey,
}; };
}; };
@@ -303,9 +372,14 @@ const buildMockFetch = (s: ServerState) => {
}); });
} }
if (path === "/collections/v2") { if (path === "/collections/v2") {
const raw = (globalThis as Record<string, unknown>) const g = globalThis as Record<string, unknown>;
.__mockRawCollection; return json({
return json({ collections: [raw] }); collections: [
g.__mockRawCollection,
g.__mockRawSharedCollection,
g.__mockRawDeletedCollection,
],
});
} }
if (path === "/collections/v2/diff") { if (path === "/collections/v2/diff") {
const raw = (globalThis as Record<string, unknown>).__mockRawFile; const raw = (globalThis as Record<string, unknown>).__mockRawFile;
@@ -412,6 +486,10 @@ describe("quak Client usage guide", () => {
* encryption key. `listCollections()` fetches them from the server, * encryption key. `listCollections()` fetches them from the server,
* decrypts the keys and names, and returns typed objects. * decrypts the keys and names, and returns typed objects.
* *
* This works transparently for both kinds of collection: ones you
* own (key encrypted with your master key) and ones shared with you
* (key sealed to your public key, flagged with `isShared: true`).
*
* ```ts * ```ts
* const collections = await client.listCollections(); * const collections = await client.listCollections();
* for (const c of collections) { * for (const c of collections) {
@@ -419,7 +497,7 @@ describe("quak Client usage guide", () => {
* } * }
* ``` * ```
*/ */
it("3. list and decrypt collections", async () => { it("3. list and decrypt collections, owned and shared", async () => {
const client = await Client.login({ const client = await Client.login({
email: TEST_EMAIL, email: TEST_EMAIL,
password: TEST_PASSWORD, password: TEST_PASSWORD,
@@ -428,12 +506,25 @@ describe("quak Client usage guide", () => {
const collections = await client.listCollections(); const collections = await client.listCollections();
expect(collections.length).toBe(1); // The mock server also returns a deleted collection (id 3).
expect(collections[0]!.name).toBe("Vacation"); // Deleted collections are tombstones in the sync protocol: their
expect(collections[0]!.type).toBe("album"); // file diff endpoint 404s, so listCollections must drop them.
expect(collections[0]!.id).toBe(1); expect(collections.length).toBe(2);
expect(collections.find((c) => c.id === 3)).toBeUndefined();
const owned = collections.find((c) => c.id === 1)!;
expect(owned.name).toBe("Vacation");
expect(owned.type).toBe("album");
expect(owned.isShared).toBe(false);
// The decrypted collection key is available for advanced use. // The decrypted collection key is available for advanced use.
expect(collections[0]!.key.length).toBe(32); expect(owned.key.length).toBe(32);
const shared = collections.find((c) => c.id === 2)!;
expect(shared.name).toBe("Friend's Wedding");
expect(shared.isShared).toBe(true);
// The key was unsealed with our keypair; the decrypted name above
// already proves it round-trips, but check it exactly too.
expect(shared.key).toEqual(server.sharedCollectionKey);
}); });
/** /**

View File

@@ -0,0 +1,99 @@
/**
* Tests for `crypto.encryptBlob`.
*
* `encryptBlob` is the push-side counterpart to `decryptBlob`. It
* encrypts a small payload as a single secretstream chunk with
* TAG_FINAL and returns the header + ciphertext. Used for encrypting
* thumbnails before upload to the Ente server.
*
* The critical invariant is that `decryptBlob(encryptBlob(...))` is the
* identity function. If either side drifts, uploaded thumbnails become
* unreadable.
*/
import sodium from "libsodium-wrappers-sumo";
import { beforeAll, describe, expect, it } from "vitest";
import {
decryptBlob,
encryptBlob,
init,
STREAM_CHUNK_OVERHEAD,
} from "../../src/crypto/index.js";
describe("crypto.encryptBlob", () => {
beforeAll(async () => {
await init();
await sodium.ready;
});
it("round-trips with decryptBlob for arbitrary data", () => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const plaintext = sodium.randombytes_buf(500);
const { header, ciphertext } = encryptBlob(plaintext, key);
const recovered = decryptBlob(ciphertext, header, key);
expect(recovered).toEqual(plaintext);
});
it("round-trips a zero-length payload", () => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptBlob(new Uint8Array(0), key);
const recovered = decryptBlob(ciphertext, header, key);
expect(recovered.length).toBe(0);
});
it("ciphertext is exactly STREAM_CHUNK_OVERHEAD longer than plaintext", () => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const plaintext = sodium.randombytes_buf(1234);
const { ciphertext } = encryptBlob(plaintext, key);
expect(ciphertext.length).toBe(
plaintext.length + STREAM_CHUNK_OVERHEAD,
);
});
it("header is 24 bytes (secretstream XChaCha20 header size)", () => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header } = encryptBlob(new Uint8Array([1, 2, 3]), key);
expect(header.length).toBe(
sodium.crypto_secretstream_xchacha20poly1305_HEADERBYTES,
);
});
it("produces different ciphertext for different keys", () => {
const k1 = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const k2 = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const plaintext = new Uint8Array([1, 2, 3, 4, 5]);
const enc1 = encryptBlob(plaintext, k1);
const enc2 = encryptBlob(plaintext, k2);
expect(enc1.ciphertext).not.toEqual(enc2.ciphertext);
});
it("produces different ciphertext on each call (random nonce in header)", () => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const plaintext = new Uint8Array([9, 9, 9]);
const a = encryptBlob(plaintext, key);
const b = encryptBlob(plaintext, key);
expect(a.header).not.toEqual(b.header);
expect(a.ciphertext).not.toEqual(b.ciphertext);
});
it("decryptBlob rejects ciphertext encrypted with a different key", () => {
const k1 = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const k2 = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptBlob(
new Uint8Array([1, 2, 3]),
k1,
);
expect(() => decryptBlob(ciphertext, header, k2)).toThrow();
});
it("decryptBlob rejects tampered ciphertext from encryptBlob", () => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptBlob(
sodium.randombytes_buf(100),
key,
);
ciphertext[ciphertext.length - 1] =
ciphertext[ciphertext.length - 1]! ^ 0x01;
expect(() => decryptBlob(ciphertext, header, key)).toThrow();
});
});

View File

@@ -15,7 +15,12 @@
* stream ended on a `TAG_FINAL` chunk and was therefore not truncated. * stream ended on a `TAG_FINAL` chunk and was therefore not truncated.
* *
* These tests pin: * These tests pin:
* - The chunk-size constants match Ente's expectations. * - The chunk-size constants match Ente's expectations, and the
* re-exported `streamTagFinal()` matches the tag a real final chunk
* carries.
* - `streamTagFinal()` reads libsodium's constant at call time rather than
* at import time, which it must, because the constant does not exist yet
* when this library's modules are evaluated.
* - The pull state can decrypt a multi-chunk stream produced by * - The pull state can decrypt a multi-chunk stream produced by
* sodium.crypto_secretstream_xchacha20poly1305_push, in order. * sodium.crypto_secretstream_xchacha20poly1305_push, in order.
* - The tag byte is propagated to the caller. * - The tag byte is propagated to the caller.
@@ -23,13 +28,14 @@
*/ */
import sodium from "libsodium-wrappers-sumo"; import sodium from "libsodium-wrappers-sumo";
import { beforeAll, describe, expect, it } from "vitest"; import { beforeAll, describe, expect, it, vi } from "vitest";
import { import {
init, init,
initStreamPull, initStreamPull,
pullStreamChunk, pullStreamChunk,
STREAM_CHUNK_OVERHEAD, STREAM_CHUNK_OVERHEAD,
STREAM_CHUNK_SIZE, STREAM_CHUNK_SIZE,
streamTagFinal,
} from "../../src/crypto/index.js"; } from "../../src/crypto/index.js";
describe("crypto stream constants", () => { describe("crypto stream constants", () => {
@@ -45,6 +51,85 @@ describe("crypto stream constants", () => {
it("STREAM_CHUNK_OVERHEAD is 17 bytes", () => { it("STREAM_CHUNK_OVERHEAD is 17 bytes", () => {
expect(STREAM_CHUNK_OVERHEAD).toBe(17); expect(STREAM_CHUNK_OVERHEAD).toBe(17);
}); });
/**
* `streamTagFinal()` is re-exported so callers can detect a truncated
* stream (a body that ended on a non-final chunk) without importing
* libsodium themselves.
*
* It is a function, not a constant, and that is load-bearing: libsodium
* attaches its own constants to the module object only after
* `sodium.ready` resolves, which is long after this library's modules are
* evaluated. Reading the value at call time yields libsodium's number;
* reading it at module scope would yield `undefined`, and every
* truncation check downstream would then compare against `undefined` and
* reject good downloads. The eagerness itself is guarded by the next test;
* this one pins the value, comparing it against the tag observed on a real
* final chunk pulled back off the wire format.
*/
it("streamTagFinal() is the tag carried by a real final chunk", async () => {
await init();
await sodium.ready;
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const push =
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const ciphertext = sodium.crypto_secretstream_xchacha20poly1305_push(
push.state,
new TextEncoder().encode("last chunk"),
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
const state = initStreamPull(push.header, key);
expect(pullStreamChunk(state, ciphertext).tag).toBe(streamTagFinal());
});
/**
* The regression guard for the eagerness property described above.
*
* Under vitest, sodium is already initialised in the worker process by
* the time any source module is evaluated, so an eager module-level read
* would happen to pick up a real value and no ordinary test could tell
* the difference. This test recreates the ordering that a plain Node ESM
* consumer sees: a stand-in sodium module whose `TAG_FINAL` property does
* not exist yet when `src/crypto/stream.ts` is evaluated and only appears
* afterwards, exactly as libsodium attaches its constants inside
* `ready.then(...)`.
*
* A call-time read observes the value that appeared after evaluation; a
* module-level read binds `undefined` and this test fails. The stand-in
* uses a sentinel rather than the real tag number so that a read which
* somehow reached the real libsodium would fail too.
*/
it("streamTagFinal() reads the constant at call time, not at import time", async () => {
const SENTINEL = 42;
const late: { tagFinal: number | undefined } = { tagFinal: undefined };
vi.resetModules();
vi.doMock("libsodium-wrappers-sumo", () => ({
default: {
ready: Promise.resolve(),
get crypto_secretstream_xchacha20poly1305_TAG_FINAL() {
return late.tagFinal;
},
},
}));
try {
// Evaluated while the constant is still absent, as it is before
// `sodium.ready` resolves.
const fresh = await import("../../src/crypto/stream.js");
expect(late.tagFinal).toBeUndefined();
// libsodium attaches its constants; a lazy accessor sees them.
late.tagFinal = SENTINEL;
expect(fresh.streamTagFinal()).toBe(SENTINEL);
} finally {
vi.doUnmock("libsodium-wrappers-sumo");
vi.resetModules();
}
});
}); });
describe("crypto.initStreamPull / pullStreamChunk", () => { describe("crypto.initStreamPull / pullStreamChunk", () => {

View File

@@ -9,21 +9,69 @@
* secretstream ciphertext chunks. Each chunk is at most * secretstream ciphertext chunks. Each chunk is at most
* `STREAM_CHUNK_SIZE + STREAM_CHUNK_OVERHEAD` bytes (4 MiB + 17 bytes). * `STREAM_CHUNK_SIZE + STREAM_CHUNK_OVERHEAD` bytes (4 MiB + 17 bytes).
* The download function buffers incoming network data, splits it on the * The download function buffers incoming network data, splits it on the
* chunk boundary, and feeds each piece to `pullStreamChunk`. The last * chunk boundary, and feeds each piece to `pullStreamChunk`.
* chunk carries `TAG_FINAL`; any truncation is detected because the tag *
* will be missing. * Two contracts are load-bearing for anyone using this library as a backup
* tool, and both are documented by the tests below:
*
* 1. **Truncation is an error, never a short file.** Only the final chunk of
* a secretstream carries `TAG_FINAL`. A download cut short by a dropped
* connection still decrypts cleanly up to the last whole chunk, so without
* an explicit `TAG_FINAL` check a truncated body is indistinguishable from
* a complete one. `streamDecrypt` therefore refuses to return unless the
* stream ended on `TAG_FINAL`, and the error says the stream was truncated.
* A transfer that stopped part-way through a chunk is reported the same
* way, since a final chunk that arrived in full always authenticates.
*
* 2. **The destination path is written atomically.** Plaintext goes to a
* temporary sibling file first and is `rename`d into place only after the
* whole stream has decrypted and verified. A caller that sees no exception
* can rely on the destination containing the complete, authenticated file;
* a caller that sees an exception can rely on the destination being
* untouched — whatever was there before is still there, byte for byte, and
* no partial file has appeared. This matters because `runBackup` skips any
* existing non-empty file, so a partial write would be treated as complete
* forever after. The staging file and the rename are observed directly (see
* the `rename` hook below), not inferred from an empty directory.
*
* 3. **A failed transfer is retried as a whole.** A download is a request, a
* stream consumption, and a decryption, and only the first of those three
* happens inside `ApiClient`. A socket reset after the response headers
* have arrived therefore surfaces here, in the download layer — and that is
* the dominant failure mode for multi-megabyte photos over a CDN. So the
* entire sequence is retried as one unit, not just the request. The
* secretstream pull state is not resumable and there is no Range support,
* so a retry starts the file over from byte zero.
* *
* These tests build synthetic encrypted files using sodium's push API, * These tests build synthetic encrypted files using sodium's push API,
* serve them from a mock fetch, and verify the decrypted output on disk. * serve them from a mock fetch, and verify the decrypted output on disk.
*/ */
import { existsSync, readFileSync, rmSync, mkdtempSync } from "node:fs"; import {
import { join } from "node:path"; existsSync,
readdirSync,
readFileSync,
rmSync,
mkdtempSync,
writeFileSync,
} from "node:fs";
import { dirname, join } from "node:path";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { createHash } from "node:crypto";
import sodium from "libsodium-wrappers-sumo"; import sodium from "libsodium-wrappers-sumo";
import { beforeAll, afterAll, describe, expect, it } from "vitest"; import {
import { init, toBase64 } from "../../src/crypto/index.js"; beforeAll,
beforeEach,
afterAll,
describe,
expect,
it,
vi,
} from "vitest";
import { init, toBase64, STREAM_CHUNK_SIZE } from "../../src/crypto/index.js";
import { ApiClient } from "../../src/api/client.js"; import { ApiClient } from "../../src/api/client.js";
import { ApiError, TruncatedStreamError } from "../../src/errors.js";
import type { RetryOptions } from "../../src/retry.js";
import { downloadFile, downloadThumbnail } from "../../src/download/index.js"; import { downloadFile, downloadThumbnail } from "../../src/download/index.js";
import type { EnteFile, FileMetadata } from "../../src/model/types.js"; import type { EnteFile, FileMetadata } from "../../src/model/types.js";
@@ -31,6 +79,52 @@ import type { EnteFile, FileMetadata } from "../../src/model/types.js";
// Test helpers // Test helpers
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/**
* `rename` is intercepted so that the tests can observe — and fail — the
* final step of the atomic write.
*
* Every other failure in this file is injected inside `streamDecrypt`, which
* runs before anything is written to disk. Those tests therefore cannot tell
* an atomic write from a plain `writeFile`: under both, nothing was ever
* created, so an empty directory proves nothing about cleanup. This hook is
* what closes that gap. It records each rename the downloader performs,
* including whether the source existed at that moment (i.e. that the staged
* temp file really was written), and can be told to fail the rename so the
* cleanup path runs with a temp file genuinely on disk.
*
* `vi.hoisted` is required: `vi.mock` factories are hoisted above the imports,
* so a plain module-level `const` would still be in its temporal dead zone by
* the time the factory runs.
*/
const renameHook = vi.hoisted(() => ({
calls: [] as { from: string; to: string; sourceExisted: boolean }[],
failWith: null as Error | null,
}));
vi.mock("node:fs/promises", async (importOriginal) => {
const actual = await importOriginal<typeof import("node:fs/promises")>();
const { existsSync: sourceExists } = await import("node:fs");
return {
...actual,
rename: async (from: string, to: string): Promise<void> => {
renameHook.calls.push({
from,
to,
sourceExisted: sourceExists(from),
});
if (renameHook.failWith !== null) {
throw renameHook.failWith;
}
await actual.rename(from, to);
},
};
});
beforeEach(() => {
renameHook.calls.length = 0;
renameHook.failWith = null;
});
let testDir: string; let testDir: string;
beforeAll(async () => { beforeAll(async () => {
@@ -45,6 +139,35 @@ afterAll(() => {
} }
}); });
/**
* Deterministic stand-in for random test payloads.
*
* Fixture *content* is never load-bearing here — the assertions turn on
* length, framing, and the secretstream tag — but it must not be a constant
* fill either, or a downloader that reordered or repeated chunks would still
* produce the expected bytes. A seeded linear congruential generator gives
* both: byte patterns that differ across every offset and seed, reproducible
* on any machine, which is what the README asks of fixtures.
*
* It is also the difference between a fast suite and a broken one.
* `sodium.randombytes_buf` goes through the wasm wrapper a byte at a time and
* costs roughly 20 seconds for the 4 MiB chunk below — about two hundred
* times what it costs to encrypt the same buffer, and on its own enough to
* push `make test` past the 30-second cap in `script/test`. This loop fills
* 4 MiB in a few milliseconds.
*/
const patternBytes = (length: number, seed: number): Uint8Array => {
const out = new Uint8Array(length);
let x = seed >>> 0;
for (let i = 0; i < length; i++) {
// Numerical Recipes' LCG constants; the high byte is used because
// the low bits of an LCG have short periods.
x = (Math.imul(x, 1664525) + 1013904223) >>> 0;
out[i] = (x >>> 24) & 0xff;
}
return out;
};
/** /**
* Encrypt `plaintext` as a secretstream file body (single chunk with * Encrypt `plaintext` as a secretstream file body (single chunk with
* TAG_FINAL). Returns the key, header, and ciphertext that the mock CDN * TAG_FINAL). Returns the key, header, and ciphertext that the mock CDN
@@ -64,6 +187,79 @@ const encryptFileBody = (
return { header: push.header, ciphertext }; return { header: push.header, ciphertext };
}; };
/**
* Encrypt a body that spans more than one secretstream chunk, the way the
* server does for files larger than the 4 MiB plaintext chunk size.
*
* Framing matters here: the downloader splits the byte stream on fixed
* `STREAM_CHUNK_SIZE + STREAM_CHUNK_OVERHEAD` boundaries, so every chunk
* except the last must carry exactly `STREAM_CHUNK_SIZE` plaintext bytes.
* Only the last chunk is tagged `TAG_FINAL`; the leading ones are
* `TAG_MESSAGE`.
*
* Returns the header, the concatenated body, the plaintext it decrypts to,
* and `finalChunkOffset` — the byte offset at which the `TAG_FINAL` chunk
* begins, so a test can slice it off to simulate a connection that dropped
* before the end of the file.
*/
const encryptMultiChunkBody = (
key: Uint8Array,
leadingChunks: number,
finalChunkPlainSize: number,
): {
header: Uint8Array;
body: Uint8Array;
plaintext: Uint8Array;
finalChunkOffset: number;
} => {
const push = sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const cipherParts: Uint8Array[] = [];
const plainParts: Uint8Array[] = [];
for (let i = 0; i < leadingChunks; i++) {
const plain = patternBytes(STREAM_CHUNK_SIZE, i + 1);
plainParts.push(plain);
cipherParts.push(
sodium.crypto_secretstream_xchacha20poly1305_push(
push.state,
plain,
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_MESSAGE,
),
);
}
const finalPlain = patternBytes(finalChunkPlainSize, leadingChunks + 1);
plainParts.push(finalPlain);
const finalCipher = sodium.crypto_secretstream_xchacha20poly1305_push(
push.state,
finalPlain,
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
const finalChunkOffset = cipherParts.reduce((n, c) => n + c.length, 0);
cipherParts.push(finalCipher);
return {
header: push.header,
body: concat(cipherParts),
plaintext: concat(plainParts),
finalChunkOffset,
};
};
const concat = (parts: Uint8Array[]): Uint8Array => {
const total = parts.reduce((n, p) => n + p.length, 0);
const out = new Uint8Array(total);
let offset = 0;
for (const p of parts) {
out.set(p, offset);
offset += p.length;
}
return out;
};
const buildMockEnteFile = ( const buildMockEnteFile = (
key: Uint8Array, key: Uint8Array,
fileHeader: Uint8Array, fileHeader: Uint8Array,
@@ -90,6 +286,153 @@ const mockFetchForBody = (body: Uint8Array) => {
return fake as typeof globalThis.fetch; return fake as typeof globalThis.fetch;
}; };
/**
* Encrypt a body consisting of one chunk that is *not* tagged TAG_FINAL.
*
* This is the cheap way to present a stream that ended without its final
* chunk: the downloader pulls it, authenticates it, and finds the stream
* over on a TAG_MESSAGE chunk — the same terminal condition as a large file
* whose last chunk was lost, without paying for a 4 MiB fixture. The
* multi-chunk fixture above covers the realistic wire shape; this one is
* used where the test is really about what happens on disk afterwards.
*/
const encryptNonFinalBody = (
plaintext: Uint8Array,
key: Uint8Array,
): { header: Uint8Array; ciphertext: Uint8Array } => {
const push = sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const ciphertext = sodium.crypto_secretstream_xchacha20poly1305_push(
push.state,
plaintext,
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_MESSAGE,
);
return { header: push.header, ciphertext };
};
/**
* Compare file contents by digest rather than with `toEqual`. Vitest's deep
* equality walks multi-megabyte buffers byte by byte, which costs seconds on
* the 4 MiB fixtures; a digest comparison is exact and effectively free.
*/
const expectSameBytes = (actual: Uint8Array, expected: Uint8Array): void => {
expect(actual.length).toBe(expected.length);
expect(createHash("sha256").update(actual).digest("hex")).toBe(
createHash("sha256").update(expected).digest("hex"),
);
};
/**
* A multi-chunk fixture shared by the truncation tests: one full 4 MiB
* `TAG_MESSAGE` chunk followed by a small `TAG_FINAL` chunk. It is built once
* and shared because encrypting 4 MiB costs about 100ms. Its plaintext is
* generated rather than drawn from the CSPRNG, which is what keeps that
* encryption the whole cost of the fixture.
*/
let multiChunk: ReturnType<typeof encryptMultiChunkBody>;
let multiChunkKey: Uint8Array;
beforeAll(() => {
multiChunkKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
multiChunk = encryptMultiChunkBody(multiChunkKey, 1, 1024);
});
/** An error shaped like a Node transport failure: the errno is on `.code`. */
const errnoError = (code: string, message = code): Error =>
Object.assign(new Error(message), { code });
/**
* A retry policy with the waiting removed, used by every fixture in this
* file. Backoff arithmetic belongs to `test/retry/retry.test.ts`; here the
* only interesting quantity is how many requests a download issued, so the
* injected `sleep` returns immediately and nothing in this file waits.
*/
const noWait: RetryOptions = {
sleep: () => Promise.resolve(),
random: () => 0,
};
/**
* One scripted outcome for a single request to the CDN.
*
* - `body` — a complete response body.
* - `status` — an HTTP error response.
* - `reset` — a response whose headers arrive, whose body delivers `bytes`,
* and which then dies with a socket reset. This is the failure that
* motivates retrying the download rather than the request: by the time it
* happens `ApiClient` has already returned successfully.
*/
type BodyStep =
| { kind: "body"; bytes: Uint8Array }
| { kind: "status"; status: number }
| { kind: "reset"; bytes: Uint8Array };
/**
* A fetch that serves one scripted step per call and counts the calls. It
* deliberately refuses to serve more requests than it was given steps for, so
* a retry loop that ran away is a test failure rather than a silent success.
*/
const scriptedCdnFetch = (
...steps: BodyStep[]
): { fetch: typeof globalThis.fetch; requests: () => number } => {
let calls = 0;
const fake = async (): Promise<Response> => {
const step = steps[calls++];
if (step === undefined) {
throw new Error(`scriptedCdnFetch: no step for request #${calls}`);
}
if (step.kind === "status") {
return new Response("error", { status: step.status });
}
if (step.kind === "body") {
return new Response(step.bytes, { status: 200 });
}
const bytes = step.bytes;
return new Response(
new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(bytes);
controller.error(
errnoError("ECONNRESET", "aborted by peer"),
);
},
}),
{ status: 200 },
);
};
return { fetch: fake as typeof globalThis.fetch, requests: () => calls };
};
/**
* Build an EnteFile plus ApiClient whose file *and* thumbnail streams both
* serve `body` under `header`. The download path under test is otherwise
* identical for the two, so every truncation/atomicity case below runs
* against both entry points from a single fixture.
*
* The default policy here is a single attempt. The failure-contract tests are
* about what the caller and the filesystem are left with, not about how many
* times quak asked; pinning attempts to one keeps them saying exactly that,
* and keeps them from re-decrypting a 4 MiB fixture four times over. The
* retry counts have their own tests at the bottom of this file, which set the
* attempt count explicitly.
*/
const fixtureFor = (
key: Uint8Array,
header: Uint8Array,
body: Uint8Array,
retry: RetryOptions = { ...noWait, attempts: 1 },
): { api: ApiClient; file: EnteFile } => ({
api: new ApiClient({ fetch: mockFetchForBody(body), retry }),
file: buildMockEnteFile(key, header, header),
});
// The two entry points share `streamDecrypt` and the atomic-write wrapper,
// so the contract tests are written once and run against both.
const entryPoints = [
{ name: "downloadFile", download: downloadFile },
{ name: "downloadThumbnail", download: downloadThumbnail },
];
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Tests // Tests
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -113,35 +456,45 @@ describe("downloadFile", () => {
const result = await downloadFile(api, file, outPath); const result = await downloadFile(api, file, outPath);
expect(result.path).toBe(outPath); // The whole DownloadResult shape is asserted, not just its fields:
expect(result.bytesWritten).toBe(plaintext.length); // callers depend on `path` being the destination they asked for
// (never the temporary file used along the way) and on
// `bytesWritten` being the plaintext length.
expect(result).toEqual({
path: outPath,
bytesWritten: plaintext.length,
});
expect(readFileSync(outPath)).toEqual(Buffer.from(plaintext)); expect(readFileSync(outPath)).toEqual(Buffer.from(plaintext));
}); });
it("uses metadata.title as filename when outPath is omitted", async () => { it("uses metadata.title as filename when outPath is omitted", async () => {
// With no `outPath`, the destination is `metadata.title`, used
// verbatim as a path. The title here is therefore given inside the
// test's temporary directory: a bare relative name would resolve
// against the process working directory, i.e. the repo root, and
// `make check` must not create files in the repo — a failure between
// the write and any cleanup would leave one behind.
const plaintext = new Uint8Array([1, 2, 3]); const plaintext = new Uint8Array([1, 2, 3]);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen(); const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key); const { header, ciphertext } = encryptFileBody(plaintext, key);
const thumbPush = const thumbPush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(key); sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const file = buildMockEnteFile(key, header, thumbPush.header); const file = buildMockEnteFile(key, header, thumbPush.header);
file.metadata.title = "fallback-name.png"; const titlePath = join(testDir, "fallback-name.png");
file.metadata.title = titlePath;
const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) }); const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) });
const result = await downloadFile(api, file); const result = await downloadFile(api, file);
expect(result.path).toBe("fallback-name.png"); expect(result.path).toBe(titlePath);
// Clean up since it writes to cwd expect(readFileSync(result.path)).toEqual(Buffer.from(plaintext));
if (existsSync(result.path)) rmSync(result.path);
}); });
it("handles a larger single-chunk file (random binary payload)", async () => { it("handles a larger single-chunk file (random binary payload)", async () => {
// Most photos are under 4 MiB and therefore a single secretstream // Most photos are under 4 MiB and therefore a single secretstream
// chunk. This test exercises a non-trivial payload size with // chunk. This test exercises a non-trivial payload size with
// random binary data (not just ASCII) to verify no encoding bugs. // arbitrary binary data (not just ASCII) to verify no encoding bugs.
// Multi-chunk (>4 MiB) decryption is verified by the live const plaintext = patternBytes(100_000, 11);
// integration test against real photos from the dev account.
const plaintext = sodium.randombytes_buf(100_000);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen(); const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key); const { header, ciphertext } = encryptFileBody(plaintext, key);
const thumbPush = const thumbPush =
@@ -153,7 +506,27 @@ describe("downloadFile", () => {
const result = await downloadFile(api, file, outPath); const result = await downloadFile(api, file, outPath);
expect(result.bytesWritten).toBe(100_000); expect(result.bytesWritten).toBe(100_000);
expect(readFileSync(outPath)).toEqual(Buffer.from(plaintext)); expectSameBytes(readFileSync(outPath), plaintext);
});
it("decrypts a body that spans several secretstream chunks", async () => {
// Files over 4 MiB arrive as several ciphertext chunks concatenated
// into one HTTP body. The downloader has to re-split them on the
// exact chunk boundary; getting that wrong corrupts every large
// photo in an account. This is also the positive control for the
// truncation tests below: it proves the multi-chunk fixture itself
// decrypts cleanly when nothing has been removed from it.
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
multiChunk.body,
);
const outPath = join(testDir, "multi-chunk.bin");
const result = await downloadFile(api, file, outPath);
expect(result.bytesWritten).toBe(multiChunk.plaintext.length);
expectSameBytes(readFileSync(outPath), multiChunk.plaintext);
}); });
}); });
@@ -173,7 +546,518 @@ describe("downloadThumbnail", () => {
const result = await downloadThumbnail(api, file, outPath); const result = await downloadThumbnail(api, file, outPath);
expect(result.bytesWritten).toBe(4); expect(result).toEqual({ path: outPath, bytesWritten: 4 });
expect(readFileSync(outPath)).toEqual(Buffer.from(plaintext)); expect(readFileSync(outPath)).toEqual(Buffer.from(plaintext));
}); });
}); });
// ---------------------------------------------------------------------------
// Truncation detection and atomic writes
//
// Everything below is the failure contract. It is deliberately written once
// per entry point via `entryPoints`, because `downloadFile` and
// `downloadThumbnail` must behave identically here: a corrupt thumbnail is
// just as unacceptable as a corrupt original, and `runBackup` trusts both.
// ---------------------------------------------------------------------------
describe.each(entryPoints)(
"$name truncation handling",
({ name, download }) => {
/** A fresh, empty directory so leftover-file assertions are meaningful. */
const freshDir = (): string => {
const dir = mkdtempSync(join(testDir, `${name}-`));
return dir;
};
it("rejects a body whose final TAG_FINAL chunk never arrived", async () => {
// Simulate a connection that dropped after the first 4 MiB chunk.
// Every byte that did arrive decrypts and authenticates perfectly —
// that is precisely the danger. The only signal that the file is
// incomplete is the absence of a chunk tagged TAG_FINAL, so the
// downloader must treat "stream ended on TAG_MESSAGE" as a hard
// error rather than returning a short file.
const truncatedBody = multiChunk.body.slice(
0,
multiChunk.finalChunkOffset,
);
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
truncatedBody,
);
const outPath = join(freshDir(), "truncated.bin");
const err: unknown = await download(api, file, outPath).catch(
(e: unknown) => e,
);
// The type, not the wording, is the contract. The retry policy
// classifies truncation as worth another attempt, and it decides
// that with `instanceof`: matching on message text would make
// rewording a diagnostic silently turn every truncated download
// into a permanent failure.
expect(err).toBeInstanceOf(TruncatedStreamError);
expect((err as Error).message).toMatch(/truncated/i);
});
it("rejects a body whose final chunk arrived only in part", async () => {
// The likelier shape of a dropped connection: the transfer stops
// in the middle of a chunk rather than neatly between two. The
// bytes that arrived are a prefix of a complete chunk, so Poly1305
// rejects them — which is, cryptographically, indistinguishable
// from corruption of a whole chunk.
//
// It is still reported as truncation, because that is what it
// almost always is and because this library's entire reason for
// checking TAG_FINAL is to make a short transfer visible. Calling
// a short transfer "authentication failed" would send a user
// hunting for a corrupt file when their network is at fault. The
// underlying authentication failure is kept as the error's
// `cause`, so the real diagnosis is never lost.
const shortBody = multiChunk.body.slice(
0,
multiChunk.body.length - 8,
);
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
shortBody,
);
const dir = freshDir();
const outPath = join(dir, "partial-final.bin");
const err = await download(api, file, outPath).catch(
(e: unknown) => e,
);
expect(err).toBeInstanceOf(TruncatedStreamError);
expect((err as Error).message).toMatch(/truncated/i);
expect((err as Error).cause).toBeInstanceOf(Error);
expect(((err as Error).cause as Error).message).toMatch(
/authentication failed/i,
);
// And, as with every other failure, the destination is untouched
// and no staged temp file survives.
expect(existsSync(outPath)).toBe(false);
expect(readdirSync(dir)).toEqual([]);
});
it("rejects an empty body instead of writing a zero-byte file", async () => {
// Ente always emits at least one chunk, even for empty content:
// `encryptBlob` shows that a zero-length plaintext still produces a
// TAG_FINAL chunk. A body with no chunks at all therefore means the
// transfer failed, not that the file is empty. Writing a zero-byte
// file here would be the worst outcome, because `runBackup` would
// then see a file it considers present and never retry it.
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
new Uint8Array(0),
);
const outPath = join(freshDir(), "empty.bin");
await expect(download(api, file, outPath)).rejects.toBeInstanceOf(
TruncatedStreamError,
);
});
it("leaves no file at the destination after a truncated download", async () => {
// The caller's contract: if the promise rejects, the destination
// path does not exist. Nothing downstream should ever have to guess
// whether a leftover file is complete.
//
// The body here is a single chunk that was never tagged TAG_FINAL,
// which puts the downloader in exactly the terminal state a lost
// last chunk produces, without the cost of a 4 MiB fixture. What
// this test is really about is the state of the filesystem after
// the rejection.
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptNonFinalBody(
patternBytes(256, 21),
key,
);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "absent.bin");
await expect(download(api, file, outPath)).rejects.toBeInstanceOf(
TruncatedStreamError,
);
expect(existsSync(outPath)).toBe(false);
// And no temporary scratch file is left behind either: the download
// stages plaintext in a sibling temp file, which must be removed on
// the failure path so repeated failures cannot fill the disk.
expect(readdirSync(dir)).toEqual([]);
});
it("reports a corrupt whole chunk as an authentication failure, not truncation", async () => {
// The counterpart to the partial-final-chunk case above, and the
// reason the two are distinguishable at all. A byte is flipped
// inside the first chunk of a multi-chunk body: that chunk arrives
// complete — the stream goes on past it — so its failure cannot be
// a short transfer. It is corruption, and the caller is told so,
// with `pullStreamChunk`'s error propagated unchanged because it is
// the real diagnosis.
//
// The same on-disk guarantee holds for this failure mode as for
// every other: nothing at the destination, nothing left over.
const corrupted = Uint8Array.from(multiChunk.body);
corrupted[10] ^= 0xff;
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
corrupted,
);
const dir = freshDir();
const outPath = join(dir, "corrupt.bin");
const err: unknown = await download(api, file, outPath).catch(
(e: unknown) => e,
);
expect(err).toBeInstanceOf(Error);
expect((err as Error).message).toMatch(/authentication failed/i);
// And explicitly *not* the truncation type, because that type is
// what the retry policy keys on: mislabelling corruption as
// truncation would spend the whole attempt budget re-downloading
// a file that will never decrypt.
expect(err).not.toBeInstanceOf(TruncatedStreamError);
expect(existsSync(outPath)).toBe(false);
expect(readdirSync(dir)).toEqual([]);
});
it("does not clobber an existing file when the download fails", async () => {
// The repair case. A user re-running a backup over a directory that
// already holds good originals must never end up worse off: a failed
// download leaves the previous contents exactly as they were, so the
// old good copy survives until a complete new one is available to
// replace it in a single rename.
const existing = new TextEncoder().encode(
"previously downloaded, known-good contents",
);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptNonFinalBody(
patternBytes(256, 22),
key,
);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "existing.bin");
writeFileSync(outPath, existing);
await expect(download(api, file, outPath)).rejects.toBeInstanceOf(
TruncatedStreamError,
);
expect(readFileSync(outPath)).toEqual(Buffer.from(existing));
expect(readdirSync(dir)).toEqual(["existing.bin"]);
});
it("replaces an existing file when the download succeeds", async () => {
// The mirror image of the previous test: a complete download does
// overwrite whatever was at the destination, atomically, via rename.
const existing = new TextEncoder().encode("stale contents");
const plaintext = patternBytes(512, 23);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "replaced.bin");
writeFileSync(outPath, existing);
const result = await download(api, file, outPath);
expect(result).toEqual({ path: outPath, bytesWritten: 512 });
expect(readFileSync(outPath)).toEqual(Buffer.from(plaintext));
// The temp file is gone once the rename has happened, so a
// successful download leaves exactly one file behind.
expect(readdirSync(dir)).toEqual(["replaced.bin"]);
});
it("stages the plaintext in a sibling temp file and renames it into place", async () => {
// The atomic write, observed directly rather than inferred from an
// empty directory. Every other failure in this file is injected
// inside `streamDecrypt`, which runs before anything is written —
// so under those tests a plain `writeFile` to the destination would
// look identical. This one watches the rename itself.
//
// Two properties are load-bearing. The staging file must exist on
// disk when the rename happens: that is what makes the destination
// appear complete or not at all, instead of filling up as bytes
// land. And it must be a sibling of the destination, because
// `rename` is only atomic within one filesystem — staging in
// `/tmp` and renaming across a mount point would silently become a
// copy, reintroducing the partial-file window this exists to close.
const plaintext = patternBytes(512, 31);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "staged.bin");
await download(api, file, outPath);
expect(renameHook.calls).toHaveLength(1);
const staged = renameHook.calls[0]!;
expect(staged.to).toBe(outPath);
expect(staged.from).not.toBe(outPath);
expect(dirname(staged.from)).toBe(dir);
expect(staged.sourceExisted).toBe(true);
// Afterwards the temp file is gone and only the destination is
// left, holding the complete plaintext.
expect(existsSync(staged.from)).toBe(false);
expect(readFileSync(outPath)).toEqual(Buffer.from(plaintext));
expect(readdirSync(dir)).toEqual(["staged.bin"]);
});
it("removes the staged temp file when the rename itself fails", async () => {
// The cleanup path. It can only run when something fails at or
// after the write, which no amount of bad network data can
// produce: by the time anything is written the whole stream has
// already decrypted and verified. Failing the rename is what
// reaches it — a real possibility on a full disk, a read-only
// mount, or a destination that has become a directory.
//
// This is the only case in which the temp file is on disk at the
// moment of failure, so it is the only one that can show it is
// actually removed rather than merely never created. It also pins
// that the caller sees the original failure: a cleanup that threw
// over the top of it would hide why the download failed.
const existing = new TextEncoder().encode("known-good contents");
const plaintext = patternBytes(512, 32);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "rename-fails.bin");
writeFileSync(outPath, existing);
renameHook.failWith = new Error("simulated rename failure");
await expect(download(api, file, outPath)).rejects.toThrow(
"simulated rename failure",
);
expect(renameHook.calls).toHaveLength(1);
const staged = renameHook.calls[0]!;
expect(staged.sourceExisted).toBe(true);
expect(existsSync(staged.from)).toBe(false);
// The previous contents are still there, untouched, and the
// directory holds nothing else.
expect(readFileSync(outPath)).toEqual(Buffer.from(existing));
expect(readdirSync(dir)).toEqual(["rename-fails.bin"]);
});
},
);
// ---------------------------------------------------------------------------
// Retries
//
// What is retried here is the whole download — request, stream consumption,
// decryption — because only the first of those three happens inside
// `ApiClient`. Every assertion counts requests; none of them measures time.
// ---------------------------------------------------------------------------
describe.each(entryPoints)("$name retries", ({ name, download }) => {
const freshDir = (): string => mkdtempSync(join(testDir, `${name}-retry-`));
/** A cheap single-chunk fixture: no 4 MiB encryption in the retry tests. */
const smallFixture = (
seed: number,
): {
key: Uint8Array;
header: Uint8Array;
ciphertext: Uint8Array;
plaintext: Uint8Array;
} => {
const plaintext = patternBytes(1024, seed);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key);
return { key, header, ciphertext, plaintext };
};
const clientFor = (
fetch: typeof globalThis.fetch,
attempts: number,
): ApiClient => new ApiClient({ fetch, retry: { ...noWait, attempts } });
it("retries a connection reset that happened mid-body", async () => {
// The case `ApiClient` cannot see. Its own request succeeded: headers
// arrived, a `ReadableStream` was handed back, and only then did the
// socket die. Retrying the fetch alone would have caught nothing,
// which is why the retry wraps the whole sequence.
const { key, header, ciphertext, plaintext } = smallFixture(41);
const { fetch, requests } = scriptedCdnFetch(
{ kind: "reset", bytes: ciphertext.slice(0, 16) },
{ kind: "body", bytes: ciphertext },
);
const api = clientFor(fetch, 4);
const file = buildMockEnteFile(key, header, header);
const dir = freshDir();
const outPath = join(dir, "reset-then-ok.bin");
const result = await download(api, file, outPath);
expect(requests()).toBe(2);
expect(result.bytesWritten).toBe(plaintext.length);
expectSameBytes(readFileSync(outPath), plaintext);
});
it("stages one temp file for the attempt that succeeded, not one per attempt", async () => {
// The atomic write stays outside the retry loop. A retried download
// must not leave a trail of half-written scratch files, and the
// destination must be touched exactly once — by the attempt that
// produced a complete, authenticated plaintext.
const { key, header, ciphertext } = smallFixture(42);
const { fetch } = scriptedCdnFetch(
{ kind: "reset", bytes: ciphertext.slice(0, 16) },
{ kind: "reset", bytes: ciphertext.slice(0, 16) },
{ kind: "body", bytes: ciphertext },
);
const api = clientFor(fetch, 4);
const file = buildMockEnteFile(key, header, header);
const dir = freshDir();
const outPath = join(dir, "one-stage.bin");
await download(api, file, outPath);
expect(renameHook.calls).toHaveLength(1);
expect(renameHook.calls[0]!.to).toBe(outPath);
expect(readdirSync(dir)).toEqual(["one-stage.bin"]);
});
it("retries a truncated body and gives up after the configured attempts", async () => {
// Truncation is retryable — the file on the server is intact, the
// transfer was not — but it is not retryable forever. Three attempts
// configured, three requests, then the caller gets the error.
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptNonFinalBody(
patternBytes(256, 43),
key,
);
const { fetch, requests } = scriptedCdnFetch(
{ kind: "body", bytes: ciphertext },
{ kind: "body", bytes: ciphertext },
{ kind: "body", bytes: ciphertext },
{ kind: "body", bytes: ciphertext },
);
const api = clientFor(fetch, 3);
const file = buildMockEnteFile(key, header, header);
const dir = freshDir();
const outPath = join(dir, "always-truncated.bin");
await expect(download(api, file, outPath)).rejects.toBeInstanceOf(
TruncatedStreamError,
);
expect(requests()).toBe(3);
// Every attempt failed before anything was written, so the directory
// is still empty.
expect(readdirSync(dir)).toEqual([]);
});
it("issues exactly one request when the file is gone", async () => {
// A 404 from the CDN is an answer. `runBackup` logs it and moves on;
// spending three more requests and three backoff waits on it would
// slow a large backup down for nothing.
const { key, header } = smallFixture(44);
const { fetch, requests } = scriptedCdnFetch(
{ kind: "status", status: 404 },
{ kind: "status", status: 404 },
{ kind: "status", status: 404 },
{ kind: "status", status: 404 },
);
const api = clientFor(fetch, 4);
const file = buildMockEnteFile(key, header, header);
const outPath = join(freshDir(), "gone.bin");
const err: unknown = await download(api, file, outPath).catch(
(e: unknown) => e,
);
expect(err).toBeInstanceOf(ApiError);
expect((err as ApiError).status).toBe(404);
expect(requests()).toBe(1);
});
it("retries a 503 from the CDN", async () => {
const { key, header, ciphertext, plaintext } = smallFixture(45);
const { fetch, requests } = scriptedCdnFetch(
{ kind: "status", status: 503 },
{ kind: "status", status: 503 },
{ kind: "body", bytes: ciphertext },
);
const api = clientFor(fetch, 4);
const file = buildMockEnteFile(key, header, header);
const outPath = join(freshDir(), "flaky-cdn.bin");
await download(api, file, outPath);
expect(requests()).toBe(3);
expectSameBytes(readFileSync(outPath), plaintext);
});
it("spends one attempt budget, not one per layer", async () => {
// `ApiClient.getFileStream` retries on its own for direct callers.
// The download layer opts out of that and runs its own retry over the
// whole sequence. If it did not, the two budgets would compose: three
// attempts here would become nine requests to the CDN for a single
// file, and the default four would become sixteen.
const { key, header } = smallFixture(46);
const steps: BodyStep[] = Array.from({ length: 12 }, () => ({
kind: "status" as const,
status: 503,
}));
const { fetch, requests } = scriptedCdnFetch(...steps);
const api = clientFor(fetch, 3);
const file = buildMockEnteFile(key, header, header);
const outPath = join(freshDir(), "budget.bin");
await expect(download(api, file, outPath)).rejects.toBeInstanceOf(
ApiError,
);
expect(requests()).toBe(3);
});
});
describe("download retries: corruption is not retried", () => {
it("gives up immediately on a chunk that failed to authenticate", async () => {
// A whole chunk that failed to authenticate while the stream
// continued past it is corruption or a wrong key. Neither is fixed by
// asking again, and a backup run that retried every such file would
// multiply the cost of a genuinely broken file by the attempt count.
//
// This is also the boundary of the single-chunk ambiguity documented
// at the classifier: the split is only achievable because this body
// has more than one chunk.
const corrupted = Uint8Array.from(multiChunk.body);
corrupted[10] ^= 0xff;
const { fetch, requests } = scriptedCdnFetch(
{ kind: "body", bytes: corrupted },
{ kind: "body", bytes: corrupted },
{ kind: "body", bytes: corrupted },
{ kind: "body", bytes: corrupted },
);
const api = new ApiClient({ fetch, retry: { ...noWait, attempts: 4 } });
const file = buildMockEnteFile(
multiChunkKey,
multiChunk.header,
multiChunk.header,
);
const outPath = join(mkdtempSync(join(testDir, "corrupt-")), "c.bin");
await expect(downloadFile(api, file, outPath)).rejects.toThrow(
/authentication failed/i,
);
expect(requests()).toBe(1);
});
});

View File

@@ -25,7 +25,10 @@ const main = async () => {
process.exit(1); process.exit(1);
} }
const { masterKey, token } = await unwrapAuth(challenge.response, PASSWORD); const { masterKey, secretKey, publicKey, token } = await unwrapAuth(
challenge.response,
PASSWORD,
);
api.setAuthToken(token); api.setAuthToken(token);
console.log("Logged in, user ID:", challenge.response.id); console.log("Logged in, user ID:", challenge.response.id);
@@ -37,7 +40,7 @@ const main = async () => {
const userID = challenge.response.id; const userID = challenge.response.id;
const collections = rawCollections.map((raw) => const collections = rawCollections.map((raw) =>
decryptCollection(raw, masterKey, userID), decryptCollection(raw, { masterKey, publicKey, secretKey }, userID),
); );
console.log(`${collections.length} collection(s):`); console.log(`${collections.length} collection(s):`);

View File

@@ -7,12 +7,27 @@
* *
* ## Collection decryption * ## Collection decryption
* *
* The server stores each collection's encryption key sealed under the * How a collection's key is encrypted depends on who owns it:
* owner's master key (secretbox). The collection's name is then sealed
* under that collection key (also secretbox). `decryptCollection`:
* *
* 1. decryptBox(encryptedKey, keyDecryptionNonce, masterKey) -> collectionKey * OWNED collections (owner == current user): the collection key is a
* 2. decryptBox(encryptedName, nameDecryptionNonce, collectionKey) -> name (UTF-8) * secretbox under the owner's master key, and `keyDecryptionNonce`
* carries the nonce.
*
* SHARED collections (owned by someone else, shared with us): the owner
* does not have our master key, so the server instead carries the
* collection key as an anonymous SEALED BOX (crypto_box_seal) to our
* X25519 public key, and `keyDecryptionNonce` is ABSENT from the wire
* (sealed boxes embed an ephemeral public key instead of a nonce).
*
* `decryptCollection` therefore takes the full key material (master key
* plus keypair) and dispatches on the presence of `keyDecryptionNonce`:
*
* 1a. nonce present: decryptBox(encryptedKey, keyDecryptionNonce,
* masterKey) -> collectionKey
* 1b. nonce absent: decryptSealed(encryptedKey, publicKey, secretKey)
* -> collectionKey
* 2. decryptBox(encryptedName, nameDecryptionNonce, collectionKey)
* -> name (UTF-8)
* 3. Maps the string `type` field to a CollectionType union member * 3. Maps the string `type` field to a CollectionType union member
* 4. Returns a Collection with decrypted key, name, and type * 4. Returns a Collection with decrypted key, name, and type
* *
@@ -38,12 +53,28 @@ import sodium from "libsodium-wrappers-sumo";
import { beforeAll, describe, expect, it } from "vitest"; import { beforeAll, describe, expect, it } from "vitest";
import { init, toBase64 } from "../../src/crypto/index.js"; import { init, toBase64 } from "../../src/crypto/index.js";
import { decryptCollection, decryptFile } from "../../src/model/index.js"; import { decryptCollection, decryptFile } from "../../src/model/index.js";
import type { RawCollection, RawEnteFile } from "../../src/model/index.js"; import type {
KeyMaterial,
RawCollection,
RawEnteFile,
} from "../../src/model/index.js";
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Helpers // Helpers
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// The full set of key material a logged-in client holds: the master key
// (decrypts owned collection keys) and the X25519 keypair (unseals shared
// collection keys and the auth token).
const buildKeys = (): KeyMaterial => {
const kp = sodium.crypto_box_keypair();
return {
masterKey: sodium.crypto_secretbox_keygen(),
publicKey: kp.publicKey,
secretKey: kp.privateKey,
};
};
const secretboxEncrypt = ( const secretboxEncrypt = (
plaintext: Uint8Array, plaintext: Uint8Array,
key: Uint8Array, key: Uint8Array,
@@ -80,6 +111,38 @@ const buildRawCollection = (
return { raw, collectionKey }; return { raw, collectionKey };
}; };
// A collection shared with us by another user. The wire format differs
// from owned collections in two ways, both verified against the live
// api.ente.io: `encryptedKey` is an anonymous sealed box to OUR public
// key (80 bytes for a 32-byte key, vs 48 for a secretbox), and
// `keyDecryptionNonce` is entirely ABSENT from the JSON.
const buildSharedRawCollection = (
recipientPublicKey: Uint8Array,
opts?: { name?: string; ownerID?: number },
): { raw: RawCollection; collectionKey: Uint8Array } => {
const collectionKey = sodium.crypto_secretbox_keygen();
const sealedKey = sodium.crypto_box_seal(collectionKey, recipientPublicKey);
const name = opts?.name ?? "Shared Album";
const { ciphertext: encName, nonce: nameNonce } = secretboxEncrypt(
new TextEncoder().encode(name),
collectionKey,
);
const raw: RawCollection = {
id: 101,
owner: { id: opts?.ownerID ?? 99 },
encryptedKey: toBase64(sealedKey),
// NOTE: no keyDecryptionNonce. Do not "fix" this fixture by adding
// one: its absence is exactly what the real server sends, and an
// unfaithful fixture here previously masked a crash on every
// account with an incoming shared album.
encryptedName: toBase64(encName),
nameDecryptionNonce: toBase64(nameNonce),
type: "album",
updationTime: 1700000000000000,
};
return { raw, collectionKey };
};
const buildRawFile = ( const buildRawFile = (
collectionKey: Uint8Array, collectionKey: Uint8Array,
opts?: { title?: string; fileType?: number; creationTime?: number }, opts?: { title?: string; fileType?: number; creationTime?: number },
@@ -137,13 +200,13 @@ describe("model.decryptCollection", () => {
await sodium.ready; await sodium.ready;
}); });
it("decrypts the collection key and name from a raw server response", () => { it("decrypts an owned collection key and name from a raw server response", () => {
const masterKey = sodium.crypto_secretbox_keygen(); const keys = buildKeys();
const { raw, collectionKey } = buildRawCollection(masterKey, { const { raw, collectionKey } = buildRawCollection(keys.masterKey, {
name: "Summer Photos", name: "Summer Photos",
}); });
const col = decryptCollection(raw, masterKey, 1); const col = decryptCollection(raw, keys, 1);
expect(col.id).toBe(100); expect(col.id).toBe(100);
expect(col.key).toEqual(collectionKey); expect(col.key).toEqual(collectionKey);
@@ -154,49 +217,73 @@ describe("model.decryptCollection", () => {
expect(col.isShared).toBe(false); expect(col.isShared).toBe(false);
}); });
it("sets isShared when owner != current user", () => { it("decrypts a shared collection via sealed box when keyDecryptionNonce is absent", () => {
const masterKey = sodium.crypto_secretbox_keygen(); // A collection shared TO us is not encrypted with our master key:
const { raw } = buildRawCollection(masterKey, { ownerID: 99 }); // the sharer only knows our public key, so the collection key
// arrives as crypto_box_seal(collectionKey, ourPublicKey) and the
// response has NO keyDecryptionNonce. decryptCollection must
// recover the key with the keypair, then decrypt the name with it
// as usual. Accounts with any incoming shared album hit this path
// on every listCollections call.
const keys = buildKeys();
const { raw, collectionKey } = buildSharedRawCollection(
keys.publicKey,
{ name: "Friend's Wedding", ownerID: 99 },
);
const col = decryptCollection(raw, masterKey, 1); const col = decryptCollection(raw, keys, 1);
expect(col.key).toEqual(collectionKey);
expect(col.name).toBe("Friend's Wedding");
expect(col.ownerID).toBe(99);
expect(col.isShared).toBe(true); expect(col.isShared).toBe(true);
}); });
it("maps known type strings to CollectionType", () => { it("maps known type strings to CollectionType", () => {
const masterKey = sodium.crypto_secretbox_keygen(); const keys = buildKeys();
for (const type of ["album", "folder", "favorites", "uncategorized"]) { for (const type of ["album", "folder", "favorites", "uncategorized"]) {
const { raw } = buildRawCollection(masterKey, { type }); const { raw } = buildRawCollection(keys.masterKey, { type });
const col = decryptCollection(raw, masterKey, 1); const col = decryptCollection(raw, keys, 1);
expect(col.type).toBe(type); expect(col.type).toBe(type);
} }
}); });
it("maps unrecognised type strings to 'unknown'", () => { it("maps unrecognised type strings to 'unknown'", () => {
const masterKey = sodium.crypto_secretbox_keygen(); const keys = buildKeys();
const { raw } = buildRawCollection(masterKey, { const { raw } = buildRawCollection(keys.masterKey, {
type: "someFutureType", type: "someFutureType",
}); });
const col = decryptCollection(raw, masterKey, 1); const col = decryptCollection(raw, keys, 1);
expect(col.type).toBe("unknown"); expect(col.type).toBe("unknown");
}); });
it("handles a collection with no encrypted name gracefully", () => { it("handles a collection with no encrypted name gracefully", () => {
// Some special collections (e.g. uncategorized) may have no name. // Some special collections (e.g. uncategorized) may have no name.
const masterKey = sodium.crypto_secretbox_keygen(); const keys = buildKeys();
const { raw } = buildRawCollection(masterKey); const { raw } = buildRawCollection(keys.masterKey);
delete raw.encryptedName; delete raw.encryptedName;
delete raw.nameDecryptionNonce; delete raw.nameDecryptionNonce;
const col = decryptCollection(raw, masterKey, 1); const col = decryptCollection(raw, keys, 1);
expect(col.name).toBe(""); expect(col.name).toBe("");
}); });
it("throws when the master key is wrong", () => { it("throws when the master key is wrong for an owned collection", () => {
const masterKey = sodium.crypto_secretbox_keygen(); const keys = buildKeys();
const wrongKey = sodium.crypto_secretbox_keygen(); const { raw } = buildRawCollection(keys.masterKey);
const { raw } = buildRawCollection(masterKey);
expect(() => decryptCollection(raw, wrongKey, 1)).toThrow(); // buildKeys() generates fresh random key material, so this is a
// client holding the wrong master key.
expect(() => decryptCollection(raw, buildKeys(), 1)).toThrow();
});
it("throws when the keypair is wrong for a shared collection", () => {
const keys = buildKeys();
const { raw } = buildSharedRawCollection(keys.publicKey);
// A different keypair must not be able to unseal the key.
const wrongKeys = buildKeys();
expect(() => decryptCollection(raw, wrongKeys, 1)).toThrow();
}); });
}); });

559
test/retry/retry.test.ts Normal file
View File

@@ -0,0 +1,559 @@
/**
* Tests for `src/retry.ts` — the retry policy shared by every network
* operation in quak.
*
* Two things live in that module and they are deliberately separate:
*
* - **`isRetryable(err)`**, a pure classifier. Given an error, is trying
* again capable of producing a different answer? Nothing else about the
* error matters: not how it was logged, not where it came from.
*
* - **`withRetry(fn, opts)`**, the loop. It calls `fn`, and while the error
* is classified retryable and attempts remain, it sleeps and calls `fn`
* again. It never inspects errors itself.
*
* The classifier's default answer is *no*. quak is a backup tool: a wrongly
* retried permanent failure costs a user round trips and delays the rest of
* the run, while a wrongly rejected transient failure costs one file that the
* next run picks up. When in doubt, fail fast.
*
* ## Reading the backoff assertions
*
* `withRetry` takes its `sleep` and its `random` as injected functions. Every
* test here passes a `sleep` that records the delay it was asked for and
* returns immediately, so the suite never waits, and a `random` that returns a
* fixed number, so jitter is exact rather than approximate. **No assertion in
* this file (or anywhere else in the suite) is about elapsed wall-clock time.**
* They are about what `withRetry` asked for, and how many times `fn` ran.
*
* The delay before retry number *n* (1-based) is:
*
* random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))
*
* That is exponential backoff with full jitter: the exponential term is the
* *ceiling*, and the actual wait is drawn uniformly below it. Full jitter,
* rather than a fixed delay plus noise, is what stops a client that lost a
* hundred parallel downloads to one CDN blip from re-sending all hundred at
* the same instant.
*/
import { describe, expect, it } from "vitest";
import {
DEFAULT_RETRY_OPTIONS,
isRetryable,
isSafeToReplay,
resolveRetryOptions,
withRetry,
} from "../../src/retry.js";
import { ApiError, TruncatedStreamError } from "../../src/errors.js";
import { ApiError as ApiErrorFromClient } from "../../src/api/client.js";
// ---------------------------------------------------------------------------
// Test helpers
// ---------------------------------------------------------------------------
/**
* A `sleep` that records what it was asked to wait for and returns
* immediately. This is the whole reason `withRetry` takes an injected sleep:
* the retry policy is exercised in full — every branch, every delay — without
* the suite spending a single millisecond waiting.
*/
const recordingSleep = (): {
sleep: (ms: number) => Promise<void>;
delays: number[];
} => {
const delays: number[] = [];
return {
sleep: (ms: number): Promise<void> => {
delays.push(ms);
return Promise.resolve();
},
delays,
};
};
/** An error shaped like a Node transport failure: the errno is on `.code`. */
const errnoError = (code: string, message = code): Error =>
Object.assign(new Error(message), { code });
// ---------------------------------------------------------------------------
// Classification
// ---------------------------------------------------------------------------
describe("isRetryable: HTTP status codes", () => {
it("does not retry ordinary 4xx responses", () => {
// A 4xx is the server saying the request itself is wrong. Repeating
// it verbatim produces the same answer, so retrying only delays the
// failure the caller has to handle. 404 is the load-bearing case:
// `listMissingThumbnails` depends on a 404 arriving promptly and
// exactly once.
for (const status of [400, 401, 403, 404, 409, 410, 422]) {
expect(isRetryable(new ApiError(`HTTP ${status}`, status))).toBe(
false,
);
}
});
it("retries 408 and 429", () => {
// The two 4xx codes that are statements about timing rather than
// about the request. 408 is the server admitting it gave up waiting;
// 429 is it asking for less traffic — which backoff supplies.
expect(isRetryable(new ApiError("timeout", 408))).toBe(true);
expect(isRetryable(new ApiError("slow down", 429))).toBe(true);
});
it("retries every 5xx response", () => {
// A 5xx is the server failing, not the request being wrong. Ente's
// CDN in particular returns 500 and 503 under load.
for (const status of [500, 502, 503, 504, 599]) {
expect(isRetryable(new ApiError(`HTTP ${status}`, status))).toBe(
true,
);
}
});
it("treats a 2xx or 3xx ApiError as not retryable", () => {
// These exist: `getFileStream` raises an ApiError carrying the
// response status when a 200 arrives with a null body. That is a
// malformed response, not a transport failure, and repeating the
// request will produce the same malformed response.
expect(isRetryable(new ApiError("response body is null", 200))).toBe(
false,
);
expect(isRetryable(new ApiError("redirect", 304))).toBe(false);
});
it("uses the same ApiError class that ApiClient exports", () => {
// `ApiError` lives in `src/errors.ts` and is re-exported from
// `src/api/client.ts`, which is where every existing caller and test
// imports it from. If those ever became two separate classes the
// classifier would silently stop recognising errors raised by the
// client, and every 5xx in the wild would be treated as permanent.
expect(ApiErrorFromClient).toBe(ApiError);
expect(new ApiErrorFromClient("boom", 503)).toBeInstanceOf(ApiError);
expect(isRetryable(new ApiErrorFromClient("boom", 503))).toBe(true);
});
});
describe("isRetryable: transport failures", () => {
it("retries a TypeError, which is how fetch reports a failed request", () => {
// Node's fetch rejects with `TypeError: fetch failed` for everything
// below HTTP: DNS failure, refused connection, TLS error, reset
// socket. The real diagnosis is on `cause`, but there is nothing on
// the object that distinguishes it from a TypeError thrown by a bug,
// so this rule is deliberately literal. The cost of the imprecision
// is bounded by the attempt count; the alternative — demanding a
// recognised `cause` — would classify real network failures as
// permanent and fail backups that should have succeeded.
expect(isRetryable(new TypeError("fetch failed"))).toBe(true);
});
it("retries an errno carried on the error itself", () => {
for (const code of [
"ECONNRESET",
"ETIMEDOUT",
"EPIPE",
"ENOTFOUND",
"EAI_AGAIN",
"ECONNREFUSED",
"EHOSTUNREACH",
"ENETUNREACH",
]) {
expect(isRetryable(errnoError(code))).toBe(true);
}
});
it("retries an errno buried in the cause chain", () => {
// undici does not put the errno on the error it throws; it hangs the
// underlying socket error off `cause`, sometimes more than one level
// down. A classifier that only looked at the top-level error would
// see a bare `Error` and call every dropped connection permanent.
const nested = new Error("request to files.ente.io failed", {
cause: new Error("socket hang up", {
cause: errnoError("ECONNRESET", "read ECONNRESET"),
}),
});
expect(isRetryable(nested)).toBe(true);
});
it("accepts a plain object as a cause", () => {
// Not everything in a cause chain is an Error instance.
expect(
isRetryable(new Error("failed", { cause: { code: "ETIMEDOUT" } })),
).toBe(true);
});
it("retries an aborted request", () => {
// `AbortSignal.timeout()` aborts with a `TimeoutError`; an explicit
// `abort()` produces an `AbortError`. quak only ever aborts a request
// on its own deadline, so both mean "this attempt ran out of time",
// which is exactly the condition a later attempt might not hit.
expect(isRetryable(new DOMException("timed out", "TimeoutError"))).toBe(
true,
);
expect(isRetryable(new DOMException("aborted", "AbortError"))).toBe(
true,
);
});
it("does not confuse an unrelated errno with a transport failure", () => {
// A filesystem error surfaces the same way an errno network error
// does. Retrying a full disk or a missing directory is pointless.
expect(isRetryable(errnoError("ENOSPC", "no space left"))).toBe(false);
expect(isRetryable(errnoError("ENOENT", "no such file"))).toBe(false);
expect(isRetryable(errnoError("EACCES", "permission denied"))).toBe(
false,
);
});
it("terminates on a cause chain that points at itself", () => {
// Defensive: `cause` is an arbitrary user-settable property and
// nothing stops it forming a cycle. Without a bound on the walk this
// classifier would hang the process, which is a worse failure than
// any misclassification.
const looped: Error & { cause?: unknown } = new Error("loop");
looped.cause = looped;
expect(isRetryable(looped)).toBe(false);
});
});
describe("isRetryable: stream truncation versus corruption", () => {
it("retries a truncated stream", () => {
// Truncation is a transfer that stopped early. The bytes that did
// arrive are useless, but the file on the server is fine, so asking
// again is exactly right.
expect(isRetryable(new TruncatedStreamError("stream truncated"))).toBe(
true,
);
});
it("retries a truncated stream whose cause is an authentication failure", () => {
// The single-chunk ambiguity, recorded on issue #2 and inherited from
// the truncation work: when a body ends part-way through a chunk,
// Poly1305 fails and carries no framing signal, so a cut connection
// and genuinely corrupt bytes are indistinguishable. That case is
// reported as truncation with the authentication failure preserved as
// `cause`, and it is therefore retried.
//
// Retrying is the deliberate choice. For a multi-chunk body the split
// is real — a corrupt chunk mid-stream stays an authentication
// failure, see the next test — but for a single-chunk body (most
// thumbnails, every small file) a wrong key and a cut connection look
// identical. The cost of guessing wrong is bounded: a few extra round
// trips before the same failure. The cost of guessing the other way
// is a silently truncated file kept forever.
const err = new TruncatedStreamError("stream truncated", {
cause: new Error("secretstream chunk authentication failed"),
});
expect(isRetryable(err)).toBe(true);
});
it("does not retry an authentication failure that is not truncation", () => {
// A whole chunk that failed to authenticate while the stream carried
// on past it cannot be a short transfer. It is corruption or a wrong
// key, and no number of retries fixes either.
expect(
isRetryable(new Error("secretstream chunk authentication failed")),
).toBe(false);
});
});
describe("isRetryable: everything else", () => {
it("does not retry programming errors or unknown values", () => {
expect(isRetryable(new Error("boom"))).toBe(false);
expect(isRetryable(new RangeError("out of range"))).toBe(false);
expect(isRetryable(new SyntaxError("bad JSON"))).toBe(false);
expect(isRetryable("a string")).toBe(false);
expect(isRetryable(undefined)).toBe(false);
expect(isRetryable(null)).toBe(false);
});
});
// ---------------------------------------------------------------------------
// The replay-safety classifier for non-idempotent requests
// ---------------------------------------------------------------------------
describe("isSafeToReplay", () => {
/**
* `isRetryable` answers "could a retry succeed?". For a POST or a PUT
* that is not the whole question: the other half is "could the first
* attempt already have taken effect on the server?".
*
* quak's non-idempotent calls are `/users/srp/create-session`,
* `/users/two-factor/verify` (which consumes one of a limited number of
* 2FA attempts) and `/files/thumbnail`. A blind replay of any of them can
* do real damage, so they retry only on failures that prove no request
* byte ever reached the server — which means the connection was never
* established.
*/
it("replays only failures where the connection was never established", () => {
for (const code of [
"ENOTFOUND",
"EAI_AGAIN",
"ECONNREFUSED",
"EHOSTUNREACH",
"ENETUNREACH",
]) {
expect(isSafeToReplay(errnoError(code))).toBe(true);
}
// Also when undici has buried it, which is how it actually arrives.
expect(
isSafeToReplay(
new TypeError("fetch failed", {
cause: errnoError("ECONNREFUSED"),
}),
),
).toBe(true);
});
it("does not replay a failure that could have happened after the server acted", () => {
// Every one of these is ambiguous about whether the server processed
// the request. A 5xx proves it did. A reset or a broken pipe can
// arrive after the request was fully sent and handled. A timeout says
// nothing at all about the server's state. A bare `fetch failed` with
// no recognisable cause could be any of them.
expect(isSafeToReplay(new ApiError("HTTP 500", 500))).toBe(false);
expect(isSafeToReplay(new ApiError("HTTP 429", 429))).toBe(false);
expect(isSafeToReplay(errnoError("ECONNRESET"))).toBe(false);
expect(isSafeToReplay(errnoError("EPIPE"))).toBe(false);
expect(isSafeToReplay(errnoError("ETIMEDOUT"))).toBe(false);
expect(
isSafeToReplay(new DOMException("timed out", "TimeoutError")),
).toBe(false);
expect(isSafeToReplay(new TypeError("fetch failed"))).toBe(false);
});
});
// ---------------------------------------------------------------------------
// The retry loop
// ---------------------------------------------------------------------------
describe("withRetry", () => {
it("calls the function once and does not sleep when it succeeds", async () => {
const { sleep, delays } = recordingSleep();
let calls = 0;
const result = await withRetry(
() => {
calls++;
return Promise.resolve("ok");
},
{ sleep },
);
expect(result).toBe("ok");
expect(calls).toBe(1);
expect(delays).toEqual([]);
});
it("stops at the first success and returns its value", async () => {
const { sleep, delays } = recordingSleep();
let calls = 0;
const result = await withRetry(
() => {
calls++;
if (calls < 3) {
return Promise.reject(new ApiError("HTTP 503", 503));
}
return Promise.resolve(calls);
},
{ attempts: 5, sleep },
);
expect(result).toBe(3);
expect(calls).toBe(3);
// Two failures, so two waits — and none after the attempt that
// succeeded.
expect(delays).toHaveLength(2);
});
it("gives up after `attempts` calls and throws the last error", async () => {
// `attempts` counts calls, not retries: `attempts: 3` means the
// function runs three times in total. The error that escapes is the
// one from the final attempt, because that is the current state of
// the world; a caller logging it is logging what is true now.
const { sleep, delays } = recordingSleep();
let calls = 0;
const failure = withRetry(
() => {
calls++;
return Promise.reject(new ApiError(`attempt ${calls}`, 503));
},
{ attempts: 3, sleep },
);
await expect(failure).rejects.toThrow("attempt 3");
expect(calls).toBe(3);
// Three attempts, two gaps between them. Sleeping after the last
// attempt would delay the caller's failure for nothing.
expect(delays).toHaveLength(2);
});
it("does not retry at all when attempts is 1", async () => {
const { sleep, delays } = recordingSleep();
let calls = 0;
await expect(
withRetry(
() => {
calls++;
return Promise.reject(new ApiError("HTTP 500", 500));
},
{ attempts: 1, sleep },
),
).rejects.toThrow("HTTP 500");
expect(calls).toBe(1);
expect(delays).toEqual([]);
});
it("rethrows a non-retryable error immediately", async () => {
const { sleep, delays } = recordingSleep();
let calls = 0;
await expect(
withRetry(
() => {
calls++;
return Promise.reject(new ApiError("HTTP 404", 404));
},
{ attempts: 5, sleep },
),
).rejects.toBeInstanceOf(ApiError);
expect(calls).toBe(1);
expect(delays).toEqual([]);
});
it("preserves the error object, not just its message", async () => {
// Callers classify what escapes: `listMissingThumbnails` needs the
// `ApiError` and its status to tell a genuine 404 from a transient
// failure. Wrapping the error in a "retries exhausted" error would
// break that.
const original = new ApiError("gone", 410, { code: "GONE" });
const err: unknown = await withRetry(() => Promise.reject(original), {
sleep: () => Promise.resolve(),
}).catch((e: unknown) => e);
expect(err).toBe(original);
});
it("honours a caller-supplied classifier", async () => {
// This is how the non-idempotent call sites narrow the policy: same
// loop, same backoff, stricter question.
const { sleep } = recordingSleep();
let calls = 0;
await expect(
withRetry(
() => {
calls++;
// Retryable under the default policy...
return Promise.reject(new ApiError("HTTP 503", 503));
},
{ attempts: 4, sleep, isRetryable: isSafeToReplay },
),
).rejects.toThrow("HTTP 503");
// ...but not under `isSafeToReplay`, so it ran exactly once.
expect(calls).toBe(1);
});
});
describe("withRetry backoff", () => {
it("doubles the ceiling on each retry and caps it", async () => {
// `random: () => 1` pins the jitter to the top of its range, which
// makes the ceiling itself observable. The sequence is
// base, base*2, base*4, ... clamped at maxDelayMs — so a long outage
// settles into a steady poll instead of growing to hours.
const { sleep, delays } = recordingSleep();
await expect(
withRetry(() => Promise.reject(new ApiError("HTTP 500", 500)), {
attempts: 6,
baseDelayMs: 100,
maxDelayMs: 250,
sleep,
random: () => 1,
}),
).rejects.toThrow();
expect(delays).toEqual([100, 200, 250, 250, 250]);
});
it("draws each delay uniformly below its ceiling", async () => {
// Full jitter. The exponential value is the maximum wait, not the
// wait itself, so a fleet of clients that failed together does not
// come back in lockstep.
const { sleep, delays } = recordingSleep();
await expect(
withRetry(() => Promise.reject(new ApiError("HTTP 500", 500)), {
attempts: 4,
baseDelayMs: 100,
maxDelayMs: 10_000,
sleep,
random: () => 0.25,
}),
).rejects.toThrow();
expect(delays).toEqual([25, 50, 100]);
});
it("never asks to sleep longer than the cap or less than zero", async () => {
// Whatever `random` returns from its [0, 1) contract, the delay stays
// inside the configured envelope.
const draws = [0, 0.999_999, 0.5, 0.1, 0.9];
let i = 0;
const { sleep, delays } = recordingSleep();
await expect(
withRetry(() => Promise.reject(new ApiError("HTTP 500", 500)), {
attempts: 6,
baseDelayMs: 1000,
maxDelayMs: 2000,
sleep,
random: () => draws[i++]!,
}),
).rejects.toThrow();
expect(delays).toHaveLength(5);
for (const d of delays) {
expect(d).toBeGreaterThanOrEqual(0);
expect(d).toBeLessThanOrEqual(2000);
}
expect(delays[0]).toBe(0);
});
});
describe("retry defaults", () => {
it("ships a bounded, documented default policy", () => {
// These are the numbers the README documents. They are asserted here
// so the README and the code cannot drift apart silently. Four
// attempts at a 500ms base put the three ceilings at 500, 1000 and
// 2000ms, so a file that is going to fail gives up after at most
// three and a half seconds of waiting — which keeps a
// several-thousand-file backup moving past a bad file rather than
// stalling on it. The 10s cap only comes into play for a caller that
// raises the attempt count.
expect(DEFAULT_RETRY_OPTIONS.attempts).toBe(4);
expect(DEFAULT_RETRY_OPTIONS.baseDelayMs).toBe(500);
expect(DEFAULT_RETRY_OPTIONS.maxDelayMs).toBe(10_000);
});
it("fills in only the fields the caller left out", () => {
const resolved = resolveRetryOptions({ attempts: 2 });
expect(resolved.attempts).toBe(2);
expect(resolved.baseDelayMs).toBe(DEFAULT_RETRY_OPTIONS.baseDelayMs);
expect(resolved.maxDelayMs).toBe(DEFAULT_RETRY_OPTIONS.maxDelayMs);
expect(typeof resolved.sleep).toBe("function");
expect(typeof resolved.random).toBe("function");
});
it("resolves to the defaults when given nothing", () => {
expect(resolveRetryOptions()).toEqual(DEFAULT_RETRY_OPTIONS);
expect(resolveRetryOptions({})).toEqual(DEFAULT_RETRY_OPTIONS);
});
it("defaults random to a real generator in [0, 1)", () => {
const { random } = resolveRetryOptions();
for (let i = 0; i < 100; i++) {
const r = random();
expect(r).toBeGreaterThanOrEqual(0);
expect(r).toBeLessThan(1);
}
});
});

View File

@@ -0,0 +1,616 @@
/**
* Tests for `listMissingThumbnails` and `fixMissingThumbnails`.
*
* These use a mock Ente server that serves encrypted collections, files,
* and thumbnails. The mock server has deliberate gaps: some files have
* working thumbnails, others return 404 or empty bodies. The tests
* verify that the detection and repair logic handles each case correctly.
*
* `fixMissingThumbnails` is the most complex function in quak: it
* downloads the original file, generates a JPEG thumbnail with jpeg-js,
* encrypts it with secretstream push, gets a presigned upload URL,
* uploads to S3, and registers the new thumbnail with the API. The
* test verifies each step actually happened and the uploaded data is
* a valid encrypted blob that decrypts to a JPEG.
*/
import sodium from "libsodium-wrappers-sumo";
import * as jpegJs from "jpeg-js";
import { beforeAll, describe, expect, it } from "vitest";
import {
init,
toBase64,
decryptBlob,
fromBase64,
deriveKEK,
deriveLoginSubkey,
} from "../../src/crypto/index.js";
import { SRP, SrpServer } from "fast-srp-hap";
import { Client } from "../../src/client.js";
import {
listMissingThumbnails,
fixMissingThumbnails,
} from "../../src/thumbnails.js";
import type { KeyAttributes } from "../../src/auth/types.js";
import type { RetryOptions } from "../../src/retry.js";
// ---------------------------------------------------------------------------
// Mock server with controllable thumbnail behavior
// ---------------------------------------------------------------------------
const TEST_EMAIL = "thumb@example.com";
const TEST_PASSWORD = "thumbpass";
const TEST_OPS = 2;
const TEST_MEM = 64 * 1024 * 1024;
interface ThumbMockState {
verifier: Buffer;
srpAttributes: Record<string, unknown>;
keyAttributes: KeyAttributes;
encryptedToken: string;
collections: Record<string, unknown>[];
filesByCollection: Record<number, Record<string, unknown>[]>;
fileCiphertexts: Record<number, Uint8Array>;
fileKeys: Record<number, Uint8Array>;
thumbnailBehavior: Record<number, "ok" | "empty" | "404" | "500">;
// Captures from fix operations
uploadedThumbnails: {
fileID: number;
objectKey: string;
decryptionHeader: string;
ciphertext: Uint8Array;
}[];
}
let mock: ThumbMockState;
const buildThumbMock = async (): Promise<ThumbMockState> => {
const kekSalt = sodium.randombytes_buf(sodium.crypto_pwhash_SALTBYTES);
const kek = await deriveKEK(TEST_PASSWORD, kekSalt, TEST_OPS, TEST_MEM);
const loginSubKeyBytes = deriveLoginSubkey(kek);
const srpUserID = "thumb-srp";
const srpSalt = sodium.randombytes_buf(16);
const verifier = SRP.computeVerifier(
SRP.params["4096"],
Buffer.from(srpSalt),
Buffer.from(srpUserID),
Buffer.from(loginSubKeyBytes),
);
const masterKey = sodium.randombytes_buf(32);
const keyNonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const encryptedKey = sodium.crypto_secretbox_easy(masterKey, keyNonce, kek);
const kp = sodium.crypto_box_keypair();
const skNonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const encSK = sodium.crypto_secretbox_easy(
kp.privateKey,
skNonce,
masterKey,
);
const tokenBytes = sodium.randombytes_buf(32);
const encToken = sodium.crypto_box_seal(tokenBytes, kp.publicKey);
const keyAttributes: KeyAttributes = {
kekSalt: toBase64(kekSalt),
encryptedKey: toBase64(encryptedKey),
keyDecryptionNonce: toBase64(keyNonce),
publicKey: toBase64(kp.publicKey),
encryptedSecretKey: toBase64(encSK),
secretKeyDecryptionNonce: toBase64(skNonce),
memLimit: TEST_MEM,
opsLimit: TEST_OPS,
};
// One collection with 3 files: ok thumbnail, empty thumbnail, 404 thumbnail
const collKey = sodium.crypto_secretbox_keygen();
const ckN = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const encCK = sodium.crypto_secretbox_easy(collKey, ckN, masterKey);
const nameB = new TextEncoder().encode("Photos");
const cnN = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const encCN = sodium.crypto_secretbox_easy(nameB, cnN, collKey);
const rawCollection = {
id: 1,
owner: { id: 42 },
encryptedKey: toBase64(encCK),
keyDecryptionNonce: toBase64(ckN),
encryptedName: toBase64(encCN),
nameDecryptionNonce: toBase64(cnN),
type: "album",
updationTime: 1700000000000000,
};
// Generate a real tiny JPEG via jpeg-js
const w = 100;
const h = 80;
const pixels = new Uint8Array(w * h * 4);
for (let i = 0; i < pixels.length; i += 4) {
pixels[i] = 255; // R
pixels[i + 1] = 0; // G
pixels[i + 2] = 0; // B
pixels[i + 3] = 255; // A
}
const tinyJpeg = jpegJs.encode(
{ data: pixels, width: w, height: h },
80,
).data;
const fileKeys: Record<number, Uint8Array> = {};
const fileCiphertexts: Record<number, Uint8Array> = {};
const rawFiles: Record<string, unknown>[] = [];
for (const fileID of [100, 101, 102]) {
const fk = sodium.crypto_secretstream_xchacha20poly1305_keygen();
fileKeys[fileID] = fk;
const fkN = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const encFK = sodium.crypto_secretbox_easy(fk, fkN, collKey);
const meta = JSON.stringify({
title: `file-${fileID}.jpg`,
fileType: 0,
creationTime: 1700000000000000,
modificationTime: 1700000000000000,
});
const metaPush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(fk);
const encMeta = sodium.crypto_secretstream_xchacha20poly1305_push(
metaPush.state,
new TextEncoder().encode(meta),
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
// Encrypt the tiny JPEG as the file body
const filePush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(fk);
const encFile = sodium.crypto_secretstream_xchacha20poly1305_push(
filePush.state,
new Uint8Array(tinyJpeg),
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
fileCiphertexts[fileID] = encFile;
rawFiles.push({
id: fileID,
collectionID: 1,
ownerID: 42,
encryptedKey: toBase64(encFK),
keyDecryptionNonce: toBase64(fkN),
metadata: {
encryptedData: toBase64(encMeta),
decryptionHeader: toBase64(metaPush.header),
},
file: { decryptionHeader: toBase64(filePush.header) },
thumbnail: {
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
},
updationTime: 1700000000000000,
});
}
return {
verifier,
srpAttributes: {
srpUserID,
srpSalt: toBase64(srpSalt),
memLimit: TEST_MEM,
opsLimit: TEST_OPS,
kekSalt: toBase64(kekSalt),
isEmailMFAEnabled: false,
},
keyAttributes,
encryptedToken: toBase64(encToken),
collections: [rawCollection],
filesByCollection: { 1: rawFiles },
fileCiphertexts,
fileKeys,
thumbnailBehavior: {
100: "ok",
101: "empty",
102: "404",
},
uploadedThumbnails: [],
};
};
const buildThumbFetch = (m: ThumbMockState) => {
let srpServer: SrpServer;
return (async (
input: RequestInfo | URL,
init?: RequestInit,
): Promise<Response> => {
const url =
typeof input === "string"
? input
: input instanceof URL
? input.href
: input.url;
const parsed = new URL(url);
const path = parsed.pathname;
const json = (body: unknown) =>
new Response(JSON.stringify(body), {
status: 200,
headers: { "content-type": "application/json" },
});
// SRP auth flow
if (path === "/users/srp/attributes")
return json({ attributes: m.srpAttributes });
if (path === "/users/srp/create-session") {
const body = JSON.parse(init?.body as string);
const serverKey = await SRP.genKey();
srpServer = new SrpServer(
SRP.params["4096"],
m.verifier,
serverKey,
);
const B = srpServer.computeB();
srpServer.setA(Buffer.from(body.srpA, "base64"));
return json({ sessionID: "s1", srpB: B.toString("base64") });
}
if (path === "/users/srp/verify-session") {
const body = JSON.parse(init?.body as string);
srpServer.checkM1(Buffer.from(body.srpM1, "base64"));
return json({
srpM2: srpServer.computeM2().toString("base64"),
id: 42,
keyAttributes: m.keyAttributes,
encryptedToken: m.encryptedToken,
});
}
// Collections & files
if (path === "/collections/v2")
return json({ collections: m.collections });
if (path === "/collections/v2/diff") {
const collID = Number(parsed.searchParams.get("collectionID"));
return json({
diff: m.filesByCollection[collID] ?? [],
hasMore: false,
});
}
// File download (for fix: download original to generate thumb)
if (
url.includes("files.ente.io") ||
path.startsWith("/files/download/")
) {
const fileID = Number(
parsed.searchParams.get("fileID") ?? path.split("/").pop(),
);
const ct = m.fileCiphertexts[fileID];
if (ct) return new Response(ct, { status: 200 });
return new Response("not found", { status: 404 });
}
// Thumbnail download (for list: check if thumbnail exists)
if (
url.includes("thumbnails.ente.io") ||
path.startsWith("/files/preview/")
) {
const fileID = Number(
parsed.searchParams.get("fileID") ?? path.split("/").pop(),
);
const behavior = m.thumbnailBehavior[fileID];
if (behavior === "ok") {
return new Response(new Uint8Array([0xff, 0xd8, 0xff]), {
status: 200,
});
}
if (behavior === "empty") {
return new Response(new Uint8Array(0), { status: 200 });
}
if (behavior === "500") {
return new Response("Internal Server Error", { status: 500 });
}
return new Response("not found", { status: 404 });
}
// Upload URL minting
if (path === "/files/upload-url") {
return json({
objectKey: `42/thumb-${Date.now()}`,
url: "https://s3.mock.test/presigned-put",
});
}
// Presigned PUT (S3 upload)
if (url.startsWith("https://s3.mock.test/")) {
const body = init?.body;
// Store the uploaded bytes for later inspection
if (body instanceof Uint8Array) {
(m as Record<string, unknown>)._lastUploadedCiphertext = body;
} else if (body instanceof ArrayBuffer) {
(m as Record<string, unknown>)._lastUploadedCiphertext =
new Uint8Array(body);
}
return new Response(null, { status: 200 });
}
// Update thumbnail metadata
if (path === "/files/thumbnail" && init?.method === "PUT") {
const reqBody = JSON.parse(init?.body as string);
m.uploadedThumbnails.push({
fileID: reqBody.fileID,
objectKey: reqBody.thumbnail.objectKey,
decryptionHeader: reqBody.thumbnail.decryptionHeader,
ciphertext:
((m as Record<string, unknown>)
._lastUploadedCiphertext as Uint8Array) ??
new Uint8Array(0),
});
return json({});
}
return new Response("not found", { status: 404 });
}) as typeof globalThis.fetch;
};
/**
* A retry policy with the waiting removed. `listMissingThumbnails` walks every
* file in the account, so a transient failure is retried; without an injected
* `sleep` these tests would spend real seconds waiting out backoff.
*/
const noWait: RetryOptions = {
sleep: () => Promise.resolve(),
random: () => 0,
};
/** Wrap a fetch so the tests can count how often one endpoint was hit. */
const countingFetch = (
inner: typeof globalThis.fetch,
match: (url: string) => boolean,
): { fetch: typeof globalThis.fetch; matched: () => number } => {
let matched = 0;
const fake = async (
input: RequestInfo | URL,
init?: RequestInit,
): Promise<Response> => {
const url =
typeof input === "string"
? input
: input instanceof URL
? input.href
: input.url;
if (match(url)) matched++;
return inner(input, init);
};
return { fetch: fake as typeof globalThis.fetch, matched: () => matched };
};
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
beforeAll(async () => {
await init();
await sodium.ready;
mock = await buildThumbMock();
});
describe("listMissingThumbnails", () => {
it("identifies files with empty and 404 thumbnails, ignores working ones", async () => {
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(mock) },
});
const missing = await listMissingThumbnails(client);
// File 100 has a working thumbnail → not reported
// File 101 has an empty thumbnail → reported
// File 102 has a 404 thumbnail → reported
expect(missing.length).toBe(2);
const ids = missing.map((m) => m.fileID).sort();
expect(ids).toEqual([101, 102]);
const emptyEntry = missing.find((m) => m.fileID === 101)!;
expect(emptyEntry.reason).toContain("empty");
expect(emptyEntry.title).toBe("file-101.jpg");
expect(emptyEntry.collection).toBe("Photos");
const notFoundEntry = missing.find((m) => m.fileID === 102)!;
// A 404 is the server stating the thumbnail is not there. That is the
// only network answer that means "missing", and the reason says so
// rather than the older catch-all "fetch failed" — which used to
// cover a 500 and a dropped connection too.
expect(notFoundEntry.reason).toContain("not found");
});
it("does not report a thumbnail as missing when the server is failing", async () => {
// The distinction that matters for `helper fix-missing-thumbnails`.
// Reporting a file here leads to downloading the original,
// regenerating a thumbnail, and uploading it over a thumbnail that
// was fine all along — because the server was briefly returning 500s.
//
// File 102 serves 500 on every attempt, so the retries are genuinely
// exhausted. It must still not be reported.
const failingMock = await buildThumbMock();
failingMock.thumbnailBehavior[102] = "500";
const counted = countingFetch(
buildThumbFetch(failingMock),
(url) => url.includes("thumbnails.ente.io") && url.includes("102"),
);
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: counted.fetch, retry: { ...noWait } },
});
const missing = await listMissingThumbnails(client);
// Only the genuinely empty thumbnail is reported.
expect(missing.map((m) => m.fileID)).toEqual([101]);
// And the 500 was retried rather than accepted as an answer: four
// attempts is the library default.
expect(counted.matched()).toBe(4);
});
it("does not report a thumbnail as missing when the connection fails", async () => {
// Same rule for a transport failure, which carries no status at all.
const failingMock = await buildThumbMock();
const inner = buildThumbFetch(failingMock);
let thumbRequests = 0;
const fetch = (async (
input: RequestInfo | URL,
init?: RequestInit,
): Promise<Response> => {
const url =
typeof input === "string"
? input
: input instanceof URL
? input.href
: input.url;
if (url.includes("thumbnails.ente.io") && url.includes("102")) {
thumbRequests++;
throw Object.assign(new Error("socket hang up"), {
code: "ECONNRESET",
});
}
return inner(input, init);
}) as typeof globalThis.fetch;
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch, retry: { ...noWait } },
});
const missing = await listMissingThumbnails(client);
expect(missing.map((m) => m.fileID)).toEqual([101]);
expect(thumbRequests).toBe(4);
});
it("deduplicates files seen in multiple collections", async () => {
// Add the same files to a second collection in the mock
const mockWithDupes = await buildThumbMock();
const dupeCollection = {
...mockWithDupes.collections[0]!,
id: 2,
};
mockWithDupes.collections.push(
dupeCollection as Record<string, unknown>,
);
mockWithDupes.filesByCollection[2] =
mockWithDupes.filesByCollection[1]!;
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(mockWithDupes) },
});
const missing = await listMissingThumbnails(client);
// Should still be 2, not 4 (each file checked only once)
expect(missing.length).toBe(2);
});
});
describe("fixMissingThumbnails", () => {
it("downloads original, generates thumbnail, encrypts, uploads, and registers", async () => {
const fixMock = await buildThumbMock();
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(fixMock) },
});
const results = await fixMissingThumbnails(client, [101]);
expect(results.length).toBe(1);
expect(results[0]!.success).toBe(true);
expect(results[0]!.fileID).toBe(101);
expect(results[0]!.title).toBe("file-101.jpg");
expect(results[0]!.collection).toBe("Photos");
// Verify the uploaded thumbnail was registered
expect(fixMock.uploadedThumbnails.length).toBe(1);
const upload = fixMock.uploadedThumbnails[0]!;
expect(upload.fileID).toBe(101);
expect(upload.objectKey).toContain("42/");
expect(upload.decryptionHeader.length).toBeGreaterThan(0);
// Verify the uploaded ciphertext can be decrypted back to a JPEG
const fileKey = fixMock.fileKeys[101]!;
const decrypted = decryptBlob(
upload.ciphertext,
fromBase64(upload.decryptionHeader),
fileKey,
);
// JPEG magic bytes: FF D8 FF
expect(decrypted[0]).toBe(0xff);
expect(decrypted[1]).toBe(0xd8);
expect(decrypted[2]).toBe(0xff);
// Verify jpeg-js produced a reasonably sized thumbnail
expect(decrypted.length).toBeGreaterThan(100);
expect(decrypted.length).toBeLessThan(50000);
});
it("reports failure for nonexistent file IDs without crashing", async () => {
const fixMock = await buildThumbMock();
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(fixMock) },
});
const results = await fixMissingThumbnails(client, [999]);
expect(results.length).toBe(1);
expect(results[0]!.success).toBe(false);
expect(results[0]!.fileID).toBe(999);
expect(results[0]!.error).toContain("not found");
});
it("continues after one file fails and reports mixed results", async () => {
const fixMock = await buildThumbMock();
// Make file 102 fail by removing its ciphertext so download fails
delete fixMock.fileCiphertexts[102];
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(fixMock) },
});
const results = await fixMissingThumbnails(client, [101, 102]);
expect(results.length).toBe(2);
const success = results.find((r) => r.fileID === 101)!;
const failure = results.find((r) => r.fileID === 102)!;
expect(success.success).toBe(true);
expect(failure.success).toBe(false);
});
});
describe("Client.getApiClient", () => {
it("returns the ApiClient when logged in", async () => {
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(mock) },
});
const api = client.getApiClient();
expect(api).toBeDefined();
expect(typeof api.getJSON).toBe("function");
});
it("throws after logout", async () => {
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(mock) },
});
client.logout();
expect(() => client.getApiClient()).toThrow(/logged out/);
});
});

395
yarn.lock
View File

@@ -2,13 +2,6 @@
# yarn lockfile v1 # yarn lockfile v1
"@emnapi/runtime@^1.7.0":
version "1.10.0"
resolved "https://registry.yarnpkg.com/@emnapi/runtime/-/runtime-1.10.0.tgz#4b260c0d3534204e98c6110b8db1a987d26ec87c"
integrity sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==
dependencies:
tslib "^2.4.0"
"@esbuild/aix-ppc64@0.21.5": "@esbuild/aix-ppc64@0.21.5":
version "0.21.5" version "0.21.5"
resolved "https://registry.yarnpkg.com/@esbuild/aix-ppc64/-/aix-ppc64-0.21.5.tgz#c7184a326533fcdf1b8ee0733e21c713b975575f" resolved "https://registry.yarnpkg.com/@esbuild/aix-ppc64/-/aix-ppc64-0.21.5.tgz#c7184a326533fcdf1b8ee0733e21c713b975575f"
@@ -230,152 +223,144 @@
resolved "https://registry.yarnpkg.com/@humanwhocodes/retry/-/retry-0.4.3.tgz#c2b9d2e374ee62c586d3adbea87199b1d7a7a6ba" resolved "https://registry.yarnpkg.com/@humanwhocodes/retry/-/retry-0.4.3.tgz#c2b9d2e374ee62c586d3adbea87199b1d7a7a6ba"
integrity sha512-bV0Tgo9K4hfPCek+aMAn81RppFKv2ySDQeMoSZuvTASywNTnVJCArCZE2FWqpvIatKu7VMRLWlR1EazvVhDyhQ== integrity sha512-bV0Tgo9K4hfPCek+aMAn81RppFKv2ySDQeMoSZuvTASywNTnVJCArCZE2FWqpvIatKu7VMRLWlR1EazvVhDyhQ==
"@img/colour@^1.0.0": "@inquirer/ansi@^2.0.7":
version "1.1.0" version "2.0.7"
resolved "https://registry.yarnpkg.com/@img/colour/-/colour-1.1.0.tgz#b0c2c2fa661adf75effd6b4964497cd80010bb9d" resolved "https://registry.yarnpkg.com/@inquirer/ansi/-/ansi-2.0.7.tgz#86de22810cac3ed406ec10f8d66016815b8226b4"
integrity sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ== integrity sha512-3eTuUO1vH2cZm2ZKHeQxnOqlTi9EfZDGgIe3BL3I4u+rJHocr9Fz86M4fjYABPvFnQG/gGK551HqDiIcETwU6Q==
"@img/sharp-darwin-arm64@0.34.5": "@inquirer/checkbox@^5.2.1":
version "0.34.5" version "5.2.1"
resolved "https://registry.yarnpkg.com/@img/sharp-darwin-arm64/-/sharp-darwin-arm64-0.34.5.tgz#6e0732dcade126b6670af7aa17060b926835ea86" resolved "https://registry.yarnpkg.com/@inquirer/checkbox/-/checkbox-5.2.1.tgz#7f148b3153a776cee202015b10f9a985068d188d"
integrity sha512-imtQ3WMJXbMY4fxb/Ndp6HBTNVtWCUI0WdobyheGf5+ad6xX8VIDO8u2xE4qc/fr08CKG/7dDseFtn6M6g/r3w== integrity sha512-b6xmA/VlTe0ZgDQHDui+Nav470u7u49nRd8/iuhOcQPO9Ch7lGuogydhi2VOmNlZ+zXcM8IcPuNSwQcdJaF/kw==
optionalDependencies:
"@img/sharp-libvips-darwin-arm64" "1.2.4"
"@img/sharp-darwin-x64@0.34.5":
version "0.34.5"
resolved "https://registry.yarnpkg.com/@img/sharp-darwin-x64/-/sharp-darwin-x64-0.34.5.tgz#19bc1dd6eba6d5a96283498b9c9f401180ee9c7b"
integrity sha512-YNEFAF/4KQ/PeW0N+r+aVVsoIY0/qxxikF2SWdp+NRkmMB7y9LBZAVqQ4yhGCm/H3H270OSykqmQMKLBhBJDEw==
optionalDependencies:
"@img/sharp-libvips-darwin-x64" "1.2.4"
"@img/sharp-libvips-darwin-arm64@1.2.4":
version "1.2.4"
resolved "https://registry.yarnpkg.com/@img/sharp-libvips-darwin-arm64/-/sharp-libvips-darwin-arm64-1.2.4.tgz#2894c0cb87d42276c3889942e8e2db517a492c43"
integrity sha512-zqjjo7RatFfFoP0MkQ51jfuFZBnVE2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==
"@img/sharp-libvips-darwin-x64@1.2.4":
version "1.2.4"
resolved "https://registry.yarnpkg.com/@img/sharp-libvips-darwin-x64/-/sharp-libvips-darwin-x64-1.2.4.tgz#e63681f4539a94af9cd17246ed8881734386f8cc"
integrity sha512-1IOd5xfVhlGwX+zXv2N93k0yMONvUlANylbJw1eTah8K/Jtpi15KC+WSiaX/nBmbm2HxRM1gZ0nSdjSsrZbGKg==
"@img/sharp-libvips-linux-arm64@1.2.4":
version "1.2.4"
resolved "https://registry.yarnpkg.com/@img/sharp-libvips-linux-arm64/-/sharp-libvips-linux-arm64-1.2.4.tgz#b1b288b36864b3bce545ad91fa6dadcf1a4ad318"
integrity sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==
"@img/sharp-libvips-linux-arm@1.2.4":
version "1.2.4"
resolved "https://registry.yarnpkg.com/@img/sharp-libvips-linux-arm/-/sharp-libvips-linux-arm-1.2.4.tgz#b9260dd1ebe6f9e3bdbcbdcac9d2ac125f35852d"
integrity sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==
"@img/sharp-libvips-linux-ppc64@1.2.4":
version "1.2.4"
resolved "https://registry.yarnpkg.com/@img/sharp-libvips-linux-ppc64/-/sharp-libvips-linux-ppc64-1.2.4.tgz#4b83ecf2a829057222b38848c7b022e7b4d07aa7"
integrity sha512-FMuvGijLDYG6lW+b/UvyilUWu5Ayu+3r2d1S8notiGCIyYU/76eig1UfMmkZ7vwgOrzKzlQbFSuQfgm7GYUPpA==
"@img/sharp-libvips-linux-riscv64@1.2.4":
version "1.2.4"
resolved "https://registry.yarnpkg.com/@img/sharp-libvips-linux-riscv64/-/sharp-libvips-linux-riscv64-1.2.4.tgz#880b4678009e5a2080af192332b00b0aaf8a48de"
integrity sha512-oVDbcR4zUC0ce82teubSm+x6ETixtKZBh/qbREIOcI3cULzDyb18Sr/Wcyx7NRQeQzOiHTNbZFF1UwPS2scyGA==
"@img/sharp-libvips-linux-s390x@1.2.4":
version "1.2.4"
resolved "https://registry.yarnpkg.com/@img/sharp-libvips-linux-s390x/-/sharp-libvips-linux-s390x-1.2.4.tgz#74f343c8e10fad821b38f75ced30488939dc59ec"
integrity sha512-qmp9VrzgPgMoGZyPvrQHqk02uyjA0/QrTO26Tqk6l4ZV0MPWIW6LTkqOIov+J1yEu7MbFQaDpwdwJKhbJvuRxQ==
"@img/sharp-libvips-linux-x64@1.2.4":
version "1.2.4"
resolved "https://registry.yarnpkg.com/@img/sharp-libvips-linux-x64/-/sharp-libvips-linux-x64-1.2.4.tgz#df4183e8bd8410f7d61b66859a35edeab0a531ce"
integrity sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==
"@img/sharp-libvips-linuxmusl-arm64@1.2.4":
version "1.2.4"
resolved "https://registry.yarnpkg.com/@img/sharp-libvips-linuxmusl-arm64/-/sharp-libvips-linuxmusl-arm64-1.2.4.tgz#c8d6b48211df67137541007ee8d1b7b1f8ca8e06"
integrity sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==
"@img/sharp-libvips-linuxmusl-x64@1.2.4":
version "1.2.4"
resolved "https://registry.yarnpkg.com/@img/sharp-libvips-linuxmusl-x64/-/sharp-libvips-linuxmusl-x64-1.2.4.tgz#be11c75bee5b080cbee31a153a8779448f919f75"
integrity sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==
"@img/sharp-linux-arm64@0.34.5":
version "0.34.5"
resolved "https://registry.yarnpkg.com/@img/sharp-linux-arm64/-/sharp-linux-arm64-0.34.5.tgz#7aa7764ef9c001f15e610546d42fce56911790cc"
integrity sha512-bKQzaJRY/bkPOXyKx5EVup7qkaojECG6NLYswgktOZjaXecSAeCWiZwwiFf3/Y+O1HrauiE3FVsGxFg8c24rZg==
optionalDependencies:
"@img/sharp-libvips-linux-arm64" "1.2.4"
"@img/sharp-linux-arm@0.34.5":
version "0.34.5"
resolved "https://registry.yarnpkg.com/@img/sharp-linux-arm/-/sharp-linux-arm-0.34.5.tgz#5fb0c3695dd12522d39c3ff7a6bc816461780a0d"
integrity sha512-9dLqsvwtg1uuXBGZKsxem9595+ujv0sJ6Vi8wcTANSFpwV/GONat5eCkzQo/1O6zRIkh0m/8+5BjrRr7jDUSZw==
optionalDependencies:
"@img/sharp-libvips-linux-arm" "1.2.4"
"@img/sharp-linux-ppc64@0.34.5":
version "0.34.5"
resolved "https://registry.yarnpkg.com/@img/sharp-linux-ppc64/-/sharp-linux-ppc64-0.34.5.tgz#9c213a81520a20caf66978f3d4c07456ff2e0813"
integrity sha512-7zznwNaqW6YtsfrGGDA6BRkISKAAE1Jo0QdpNYXNMHu2+0dTrPflTLNkpc8l7MUP5M16ZJcUvysVWWrMefZquA==
optionalDependencies:
"@img/sharp-libvips-linux-ppc64" "1.2.4"
"@img/sharp-linux-riscv64@0.34.5":
version "0.34.5"
resolved "https://registry.yarnpkg.com/@img/sharp-linux-riscv64/-/sharp-linux-riscv64-0.34.5.tgz#cdd28182774eadbe04f62675a16aabbccb833f60"
integrity sha512-51gJuLPTKa7piYPaVs8GmByo7/U7/7TZOq+cnXJIHZKavIRHAP77e3N2HEl3dgiqdD/w0yUfiJnII77PuDDFdw==
optionalDependencies:
"@img/sharp-libvips-linux-riscv64" "1.2.4"
"@img/sharp-linux-s390x@0.34.5":
version "0.34.5"
resolved "https://registry.yarnpkg.com/@img/sharp-linux-s390x/-/sharp-linux-s390x-0.34.5.tgz#93eac601b9f329bb27917e0e19098c722d630df7"
integrity sha512-nQtCk0PdKfho3eC5MrbQoigJ2gd1CgddUMkabUj+rBevs8tZ2cULOx46E7oyX+04WGfABgIwmMC0VqieTiR4jg==
optionalDependencies:
"@img/sharp-libvips-linux-s390x" "1.2.4"
"@img/sharp-linux-x64@0.34.5":
version "0.34.5"
resolved "https://registry.yarnpkg.com/@img/sharp-linux-x64/-/sharp-linux-x64-0.34.5.tgz#55abc7cd754ffca5002b6c2b719abdfc846819a8"
integrity sha512-MEzd8HPKxVxVenwAa+JRPwEC7QFjoPWuS5NZnBt6B3pu7EG2Ge0id1oLHZpPJdn3OQK+BQDiw9zStiHBTJQQQQ==
optionalDependencies:
"@img/sharp-libvips-linux-x64" "1.2.4"
"@img/sharp-linuxmusl-arm64@0.34.5":
version "0.34.5"
resolved "https://registry.yarnpkg.com/@img/sharp-linuxmusl-arm64/-/sharp-linuxmusl-arm64-0.34.5.tgz#d6515ee971bb62f73001a4829b9d865a11b77086"
integrity sha512-fprJR6GtRsMt6Kyfq44IsChVZeGN97gTD331weR1ex1c1rypDEABN6Tm2xa1wE6lYb5DdEnk03NZPqA7Id21yg==
optionalDependencies:
"@img/sharp-libvips-linuxmusl-arm64" "1.2.4"
"@img/sharp-linuxmusl-x64@0.34.5":
version "0.34.5"
resolved "https://registry.yarnpkg.com/@img/sharp-linuxmusl-x64/-/sharp-linuxmusl-x64-0.34.5.tgz#d97978aec7c5212f999714f2f5b736457e12ee9f"
integrity sha512-Jg8wNT1MUzIvhBFxViqrEhWDGzqymo3sV7z7ZsaWbZNDLXRJZoRGrjulp60YYtV4wfY8VIKcWidjojlLcWrd8Q==
optionalDependencies:
"@img/sharp-libvips-linuxmusl-x64" "1.2.4"
"@img/sharp-wasm32@0.34.5":
version "0.34.5"
resolved "https://registry.yarnpkg.com/@img/sharp-wasm32/-/sharp-wasm32-0.34.5.tgz#2f15803aa626f8c59dd7c9d0bbc766f1ab52cfa0"
integrity sha512-OdWTEiVkY2PHwqkbBI8frFxQQFekHaSSkUIJkwzclWZe64O1X4UlUjqqqLaPbUpMOQk6FBu/HtlGXNblIs0huw==
dependencies: dependencies:
"@emnapi/runtime" "^1.7.0" "@inquirer/ansi" "^2.0.7"
"@inquirer/core" "^11.2.1"
"@inquirer/figures" "^2.0.7"
"@inquirer/type" "^4.0.7"
"@img/sharp-win32-arm64@0.34.5": "@inquirer/confirm@^6.1.1":
version "0.34.5" version "6.1.1"
resolved "https://registry.yarnpkg.com/@img/sharp-win32-arm64/-/sharp-win32-arm64-0.34.5.tgz#3706e9e3ac35fddfc1c87f94e849f1b75307ce0a" resolved "https://registry.yarnpkg.com/@inquirer/confirm/-/confirm-6.1.1.tgz#9c6a7d79c6132b2af57fdb75747f056204e55356"
integrity sha512-WQ3AgWCWYSb2yt+IG8mnC6Jdk9Whs7O0gxphblsLvdhSpSTtmu69ZG1Gkb6NuvxsNACwiPV6cNSZNzt0KPsw7g== integrity sha512-eb8DBZcz/2qHWQda4rk2JiQk5h9QV/cVHi1yjt0f69WFZMRFn0sJTye3EAP8icut8UDMjQPsaH5KbcOogefrFQ==
dependencies:
"@inquirer/core" "^11.2.1"
"@inquirer/type" "^4.0.7"
"@img/sharp-win32-ia32@0.34.5": "@inquirer/core@^11.2.1":
version "0.34.5" version "11.2.1"
resolved "https://registry.yarnpkg.com/@img/sharp-win32-ia32/-/sharp-win32-ia32-0.34.5.tgz#0b71166599b049e032f085fb9263e02f4e4788de" resolved "https://registry.yarnpkg.com/@inquirer/core/-/core-11.2.1.tgz#54ccd8f7d47852140b6066cbd77d63b2c2b168fd"
integrity sha512-FV9m/7NmeCmSHDD5j4+4pNI8Cp3aW+JvLoXcTUo0IqyjSfAZJ8dIUmijx1qaJsIiU+Hosw6xM5KijAWRJCSgNg== integrity sha512-Qd6GJT1yVyrZZCfN8W2qKF5ApmqryXRhRKCuip8h01x2w/esJQ2XIYc6f9abMIHgKQdBfFTSOdbHRLAhuM09UA==
dependencies:
"@inquirer/ansi" "^2.0.7"
"@inquirer/figures" "^2.0.7"
"@inquirer/type" "^4.0.7"
cli-width "^4.1.0"
fast-wrap-ansi "^0.2.0"
mute-stream "^3.0.0"
signal-exit "^4.1.0"
"@img/sharp-win32-x64@0.34.5": "@inquirer/editor@^5.2.2":
version "0.34.5" version "5.2.2"
resolved "https://registry.yarnpkg.com/@img/sharp-win32-x64/-/sharp-win32-x64-0.34.5.tgz#a81ffb00e69267cd0a1d626eaedb8a8430b2b2f8" resolved "https://registry.yarnpkg.com/@inquirer/editor/-/editor-5.2.2.tgz#7c73e2fc0e7bd4c40cfd38a180ae5bbd24d32b90"
integrity sha512-+29YMsqY2/9eFEiW93eqWnuLcWcufowXewwSNIT6UwZdUUCrM3oFjMWH/Z6/TMmb4hlFenmfAVbpWeup2jryCw== integrity sha512-ZRVd/oD+sYsUd5zVm0NflqEzlqfYCyHNsqkHl2oWXEUHs12tCbcSFi+wVFEvD8+LGRaMUsVrE7qeo6lSG/S1Vg==
dependencies:
"@inquirer/core" "^11.2.1"
"@inquirer/external-editor" "^3.0.3"
"@inquirer/type" "^4.0.7"
"@inquirer/expand@^5.1.1":
version "5.1.1"
resolved "https://registry.yarnpkg.com/@inquirer/expand/-/expand-5.1.1.tgz#e2afeac247d97dd64ee18aa81e902bdd1fe0ea70"
integrity sha512-YmQpenjbFSHAK3sOd44puHh3V1KXXr+JiNpUztoSQ4drLh2rTVzTap/YtlAVu/5xavifIlBfNEzJ/neZJ1a/1g==
dependencies:
"@inquirer/core" "^11.2.1"
"@inquirer/type" "^4.0.7"
"@inquirer/external-editor@^3.0.3":
version "3.0.3"
resolved "https://registry.yarnpkg.com/@inquirer/external-editor/-/external-editor-3.0.3.tgz#d79e772542cf8d340642e9dabd3a1ea7f5a30104"
integrity sha512-6thf5I8q7lZwzGLAxPaaGEREEkZ3nyePPDQ1oyobblxmEE8mqTLguScP7pDjUTAibiyb4hfXl+qjUEJ+di/aNA==
dependencies:
chardet "^2.1.1"
iconv-lite "^0.7.2"
"@inquirer/figures@^2.0.7":
version "2.0.7"
resolved "https://registry.yarnpkg.com/@inquirer/figures/-/figures-2.0.7.tgz#f5cc5843732a81304d06a0db4b53cc7dbda15541"
integrity sha512-aJ8TBPOGB6f/2qziPfElISTCEd5XOYTFckA2SGjhNmiKzfK/u4ot3v0DUzGVdUnKjN10EqnnEPck36BkyfLnJw==
"@inquirer/input@^5.1.2":
version "5.1.2"
resolved "https://registry.yarnpkg.com/@inquirer/input/-/input-5.1.2.tgz#9305cb170dfc3a5323e5eac885a945e7cddd5c4b"
integrity sha512-9K/DDBSQpOyZSkt6sOVP9Vo0TR7atX2kuILsUu0x3wVcVbe97lJwIJKMLdMw25tDYuXl/qp6erT0Xs1rfmcfZg==
dependencies:
"@inquirer/core" "^11.2.1"
"@inquirer/type" "^4.0.7"
"@inquirer/number@^4.1.1":
version "4.1.1"
resolved "https://registry.yarnpkg.com/@inquirer/number/-/number-4.1.1.tgz#b133668d8e0e099b4133abb915221501e0ff75d7"
integrity sha512-XF4IXAbPnGPgw0wsbC/i2tPcyfdZgDpUlhsqU0SfT4IRIGWha6Xm9VRgN5yYxJq+jnyXlfXI/nQ3ulfk0iEICA==
dependencies:
"@inquirer/core" "^11.2.1"
"@inquirer/type" "^4.0.7"
"@inquirer/password@^5.1.1":
version "5.1.1"
resolved "https://registry.yarnpkg.com/@inquirer/password/-/password-5.1.1.tgz#f21efb614da9c905095262f51781fd2a721fceac"
integrity sha512-3XBfF7DAsp5qeDsvN5Rd1HmbNokVvEQoUM0QLrRcybC9nX96w3Pbmu7qUsb3IT3J3jBvs2+mTXaKHOUsgHMLzg==
dependencies:
"@inquirer/ansi" "^2.0.7"
"@inquirer/core" "^11.2.1"
"@inquirer/type" "^4.0.7"
"@inquirer/prompts@8.5.2":
version "8.5.2"
resolved "https://registry.yarnpkg.com/@inquirer/prompts/-/prompts-8.5.2.tgz#09c0132ada2bbba94c91d341115e1e41cb3f1525"
integrity sha512-IYR/3C/paEVVQYQvdDlFZVjRCJVYHHON0XXMH91KO9GSxs0TdKYWlUdvfQl2EfAHDxUaN3IBffkE/BDTh5nJ6g==
dependencies:
"@inquirer/checkbox" "^5.2.1"
"@inquirer/confirm" "^6.1.1"
"@inquirer/editor" "^5.2.2"
"@inquirer/expand" "^5.1.1"
"@inquirer/input" "^5.1.2"
"@inquirer/number" "^4.1.1"
"@inquirer/password" "^5.1.1"
"@inquirer/rawlist" "^5.3.1"
"@inquirer/search" "^4.2.1"
"@inquirer/select" "^5.2.1"
"@inquirer/rawlist@^5.3.1":
version "5.3.1"
resolved "https://registry.yarnpkg.com/@inquirer/rawlist/-/rawlist-5.3.1.tgz#66f6b8e6aa82d47399c433b8262128e7c1a4f9ce"
integrity sha512-QqdTqQddL3qPX/PPrjobpsO25NZ4dWXgTLenrR445L2ptLEYE6Z+PD5c5CNDJNx4ugRgELAIpSIJxZaO2jJ2Og==
dependencies:
"@inquirer/core" "^11.2.1"
"@inquirer/type" "^4.0.7"
"@inquirer/search@^4.2.1":
version "4.2.1"
resolved "https://registry.yarnpkg.com/@inquirer/search/-/search-4.2.1.tgz#c8f4b78ab3f866fdf0503fac0cd08c4a6661c11e"
integrity sha512-xJj8QWKRSrfKoBIITLZK61dD3zwo0Rz11fgDImku30/Oe81zMdIdGgrLY2h6RkJ+KZ/GhNYIRMKnH/62qBTA5g==
dependencies:
"@inquirer/core" "^11.2.1"
"@inquirer/figures" "^2.0.7"
"@inquirer/type" "^4.0.7"
"@inquirer/select@^5.2.1":
version "5.2.1"
resolved "https://registry.yarnpkg.com/@inquirer/select/-/select-5.2.1.tgz#3a05e76e58d9e1bb095e912c3e7093aa04cd4604"
integrity sha512-FlDndEUww8m7BfukO2nJa25vhD+H5jxxCv4oGioKqzyWz3nPHhhw4LKdYRSlXuAx7DsdWia7iyaBPKKS95Evfw==
dependencies:
"@inquirer/ansi" "^2.0.7"
"@inquirer/core" "^11.2.1"
"@inquirer/figures" "^2.0.7"
"@inquirer/type" "^4.0.7"
"@inquirer/type@^4.0.7":
version "4.0.7"
resolved "https://registry.yarnpkg.com/@inquirer/type/-/type-4.0.7.tgz#9c6f0d857fe6ad549a3a932343b64e76acb34b10"
integrity sha512-t28inv14nMQ1PhKpsJPY+kEs/c00qzeCOS2gTNRyTjG5d6qsVA2fItxW4hkvGZ5lvanGLdtCzVIx5dwdRpN1+g==
"@jridgewell/sourcemap-codec@^1.5.5": "@jridgewell/sourcemap-codec@^1.5.5":
version "1.5.5" version "1.5.5"
@@ -557,13 +542,6 @@
dependencies: dependencies:
undici-types "~6.21.0" undici-types "~6.21.0"
"@types/sharp@^0.32.0":
version "0.32.0"
resolved "https://registry.yarnpkg.com/@types/sharp/-/sharp-0.32.0.tgz#fc3ac6df6b456319bae807c3d24efdc6631cdd6f"
integrity sha512-OOi3kL+FZDnPhVzsfD37J88FNeZh6gQsGcLc95NbeURRGvmSjeXiDcyWzF2o3yh/gQAUn2uhh/e+CPCa5nwAxw==
dependencies:
sharp "*"
"@typescript-eslint/eslint-plugin@8.46.2": "@typescript-eslint/eslint-plugin@8.46.2":
version "8.46.2" version "8.46.2"
resolved "https://registry.yarnpkg.com/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.46.2.tgz#dc4ab93ee3d7e6c8e38820a0d6c7c93c7183e2dc" resolved "https://registry.yarnpkg.com/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.46.2.tgz#dc4ab93ee3d7e6c8e38820a0d6c7c93c7183e2dc"
@@ -824,11 +802,21 @@ chalk@^4.0.0:
ansi-styles "^4.1.0" ansi-styles "^4.1.0"
supports-color "^7.1.0" supports-color "^7.1.0"
chardet@^2.1.1:
version "2.1.1"
resolved "https://registry.yarnpkg.com/chardet/-/chardet-2.1.1.tgz#5c75593704a642f71ee53717df234031e65373c8"
integrity sha512-PsezH1rqdV9VvyNhxxOW32/d75r01NY7TQCmOqomRo15ZSOKbpTFVsfjghxo6JloQUCGnH4k1LGu0R4yCLlWQQ==
check-error@^2.1.1: check-error@^2.1.1:
version "2.1.3" version "2.1.3"
resolved "https://registry.yarnpkg.com/check-error/-/check-error-2.1.3.tgz#2427361117b70cca8dc89680ead32b157019caf5" resolved "https://registry.yarnpkg.com/check-error/-/check-error-2.1.3.tgz#2427361117b70cca8dc89680ead32b157019caf5"
integrity sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA== integrity sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==
cli-width@^4.1.0:
version "4.1.0"
resolved "https://registry.yarnpkg.com/cli-width/-/cli-width-4.1.0.tgz#42daac41d3c254ef38ad8ac037672130173691c5"
integrity sha512-ouuZd4/dm2Sw5Gmqy6bGyNNNe1qt9RpmxveLSO7KcgsTnU7RXfsw+/bukWGo1abgBiMAic068rclZsO4IWmmxQ==
color-convert@^2.0.1: color-convert@^2.0.1:
version "2.0.1" version "2.0.1"
resolved "https://registry.yarnpkg.com/color-convert/-/color-convert-2.0.1.tgz#72d3a68d598c9bdb3af2ad1e84f21d896abd4de3" resolved "https://registry.yarnpkg.com/color-convert/-/color-convert-2.0.1.tgz#72d3a68d598c9bdb3af2ad1e84f21d896abd4de3"
@@ -877,11 +865,6 @@ deep-is@^0.1.3:
resolved "https://registry.yarnpkg.com/deep-is/-/deep-is-0.1.4.tgz#a6f2dce612fadd2ef1f519b73551f17e85199831" resolved "https://registry.yarnpkg.com/deep-is/-/deep-is-0.1.4.tgz#a6f2dce612fadd2ef1f519b73551f17e85199831"
integrity sha512-oIPzksmTg4/MriiaYGO+okXDT7ztn/w3Eptv/+gSIdMdKsJo0u4CfYNFJPy+4SKMuCqGw2wxnA+URMg3t8a/bQ== integrity sha512-oIPzksmTg4/MriiaYGO+okXDT7ztn/w3Eptv/+gSIdMdKsJo0u4CfYNFJPy+4SKMuCqGw2wxnA+URMg3t8a/bQ==
detect-libc@^2.1.2:
version "2.1.2"
resolved "https://registry.yarnpkg.com/detect-libc/-/detect-libc-2.1.2.tgz#689c5dcdc1900ef5583a4cb9f6d7b473742074ad"
integrity sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==
env-paths@4.0.0: env-paths@4.0.0:
version "4.0.0" version "4.0.0"
resolved "https://registry.yarnpkg.com/env-paths/-/env-paths-4.0.0.tgz#d0bb1f84a81d2542581bf7b7e8085d0683b39097" resolved "https://registry.yarnpkg.com/env-paths/-/env-paths-4.0.0.tgz#d0bb1f84a81d2542581bf7b7e8085d0683b39097"
@@ -1026,6 +1009,11 @@ esutils@^2.0.2:
resolved "https://registry.yarnpkg.com/esutils/-/esutils-2.0.3.tgz#74d2eb4de0b8da1293711910d50775b9b710ef64" resolved "https://registry.yarnpkg.com/esutils/-/esutils-2.0.3.tgz#74d2eb4de0b8da1293711910d50775b9b710ef64"
integrity sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g== integrity sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g==
exif-reader@2.0.3:
version "2.0.3"
resolved "https://registry.yarnpkg.com/exif-reader/-/exif-reader-2.0.3.tgz#259997735080bc6bb959c37b32c60f004ec4391d"
integrity sha512-zFbQvguwT9JkqyYhR7pjE1Yn8SagwaGLNRU0Oh14xFa1paSf5Gzxn4gxgk0XhnudI0UIqU+HgnBX93+nva592A==
expect-type@^1.1.0: expect-type@^1.1.0:
version "1.3.0" version "1.3.0"
resolved "https://registry.yarnpkg.com/expect-type/-/expect-type-1.3.0.tgz#0d58ed361877a31bbc4dd6cf71bbfef7faf6bd68" resolved "https://registry.yarnpkg.com/expect-type/-/expect-type-1.3.0.tgz#0d58ed361877a31bbc4dd6cf71bbfef7faf6bd68"
@@ -1062,6 +1050,25 @@ fast-srp-hap@2.0.4:
resolved "https://registry.yarnpkg.com/fast-srp-hap/-/fast-srp-hap-2.0.4.tgz#9db296e21a5143951310f99e5a74290106467811" resolved "https://registry.yarnpkg.com/fast-srp-hap/-/fast-srp-hap-2.0.4.tgz#9db296e21a5143951310f99e5a74290106467811"
integrity sha512-lHRYYaaIbMrhZtsdGTwPN82UbqD9Bv8QfOlKs+Dz6YRnByZifOh93EYmf2iEWFtkOEIqR2IK8cFD0UN5wLIWBQ== integrity sha512-lHRYYaaIbMrhZtsdGTwPN82UbqD9Bv8QfOlKs+Dz6YRnByZifOh93EYmf2iEWFtkOEIqR2IK8cFD0UN5wLIWBQ==
fast-string-truncated-width@^3.0.2:
version "3.0.3"
resolved "https://registry.yarnpkg.com/fast-string-truncated-width/-/fast-string-truncated-width-3.0.3.tgz#23afe0da67d752ca0727538f1e6967759728ce49"
integrity sha512-0jjjIEL6+0jag3l2XWWizO64/aZVtpiGE3t0Zgqxv0DPuxiMjvB3M24fCyhZUO4KomJQPj3LTSUnDP3GpdwC0g==
fast-string-width@^3.0.2:
version "3.0.2"
resolved "https://registry.yarnpkg.com/fast-string-width/-/fast-string-width-3.0.2.tgz#16dbabb491ce5585b5ecb675b65c165d71688eeb"
integrity sha512-gX8LrtNEI5hq8DVUfRQMbr5lpaS4nMIWV+7XEbXk2b8kiQIizgnlr12B4dA3ZEx3308ze0O4Q1R+cHts8kyUJg==
dependencies:
fast-string-truncated-width "^3.0.2"
fast-wrap-ansi@^0.2.0:
version "0.2.2"
resolved "https://registry.yarnpkg.com/fast-wrap-ansi/-/fast-wrap-ansi-0.2.2.tgz#95e952a0145bce3f59ad56e179f84c48d4072935"
integrity sha512-7F2Fl+TjRSenLqlU3UjSH0iyqopqoZIu7eZVpEirP2g1GtWa2G/ecEmBdgz31+Mxr+ELclgg6sokpSFIQiZ02Q==
dependencies:
fast-string-width "^3.0.2"
fastq@^1.6.0: fastq@^1.6.0:
version "1.20.1" version "1.20.1"
resolved "https://registry.yarnpkg.com/fastq/-/fastq-1.20.1.tgz#ca750a10dc925bc8b18839fd203e3ef4b3ced675" resolved "https://registry.yarnpkg.com/fastq/-/fastq-1.20.1.tgz#ca750a10dc925bc8b18839fd203e3ef4b3ced675"
@@ -1138,6 +1145,13 @@ has-flag@^4.0.0:
resolved "https://registry.yarnpkg.com/has-flag/-/has-flag-4.0.0.tgz#944771fd9c81c81265c4d6941860da06bb59479b" resolved "https://registry.yarnpkg.com/has-flag/-/has-flag-4.0.0.tgz#944771fd9c81c81265c4d6941860da06bb59479b"
integrity sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ== integrity sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==
iconv-lite@^0.7.2:
version "0.7.2"
resolved "https://registry.yarnpkg.com/iconv-lite/-/iconv-lite-0.7.2.tgz#d0bdeac3f12b4835b7359c2ad89c422a4d1cc72e"
integrity sha512-im9DjEDQ55s9fL4EYzOAv0yMqmMBSZp6G0VvFyTMPKWxiSBHUj9NW/qqLmXUwXrrM7AvqSlTCfvqRb0cM8yYqw==
dependencies:
safer-buffer ">= 2.1.2 < 3.0.0"
ignore@^5.2.0: ignore@^5.2.0:
version "5.3.2" version "5.3.2"
resolved "https://registry.yarnpkg.com/ignore/-/ignore-5.3.2.tgz#3cd40e729f3643fd87cb04e50bf0eb722bc596f5" resolved "https://registry.yarnpkg.com/ignore/-/ignore-5.3.2.tgz#3cd40e729f3643fd87cb04e50bf0eb722bc596f5"
@@ -1188,6 +1202,11 @@ isexe@^2.0.0:
resolved "https://registry.yarnpkg.com/isexe/-/isexe-2.0.0.tgz#e8fbf374dc556ff8947a10dcb0572d633f2cfa10" resolved "https://registry.yarnpkg.com/isexe/-/isexe-2.0.0.tgz#e8fbf374dc556ff8947a10dcb0572d633f2cfa10"
integrity sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw== integrity sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==
jpeg-js@0.4.4:
version "0.4.4"
resolved "https://registry.yarnpkg.com/jpeg-js/-/jpeg-js-0.4.4.tgz#a9f1c6f1f9f0fa80cdb3484ed9635054d28936aa"
integrity sha512-WZzeDOEtTOBK4Mdsar0IqEU5sMr3vSV2RqkAIzUEV2BHnUfKGyswWFPFwK5EeDo93K3FohSHbLAjj0s1Wzd+dg==
js-yaml@^4.1.1: js-yaml@^4.1.1:
version "4.1.1" version "4.1.1"
resolved "https://registry.yarnpkg.com/js-yaml/-/js-yaml-4.1.1.tgz#854c292467705b699476e1a2decc0c8a3458806b" resolved "https://registry.yarnpkg.com/js-yaml/-/js-yaml-4.1.1.tgz#854c292467705b699476e1a2decc0c8a3458806b"
@@ -1293,6 +1312,11 @@ ms@^2.1.3:
resolved "https://registry.yarnpkg.com/ms/-/ms-2.1.3.tgz#574c8138ce1d2b5861f0b44579dbadd60c6615b2" resolved "https://registry.yarnpkg.com/ms/-/ms-2.1.3.tgz#574c8138ce1d2b5861f0b44579dbadd60c6615b2"
integrity sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA== integrity sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==
mute-stream@^3.0.0:
version "3.0.0"
resolved "https://registry.yarnpkg.com/mute-stream/-/mute-stream-3.0.0.tgz#cd8014dd2acb72e1e91bb67c74f0019e620ba2d1"
integrity sha512-dkEJPVvun4FryqBmZ5KhDo0K9iDXAwn08tMLDinNdRBNPcYEDiWYysLcc6k3mjTMlbP9KyylvRpd4wFtwrT9rw==
nanoid@^3.3.11: nanoid@^3.3.11:
version "3.3.12" version "3.3.12"
resolved "https://registry.yarnpkg.com/nanoid/-/nanoid-3.3.12.tgz#ab3d912e217a6d0a514f00a72a16543a28982c05" resolved "https://registry.yarnpkg.com/nanoid/-/nanoid-3.3.12.tgz#ab3d912e217a6d0a514f00a72a16543a28982c05"
@@ -1446,45 +1470,16 @@ run-parallel@^1.1.9:
dependencies: dependencies:
queue-microtask "^1.2.2" queue-microtask "^1.2.2"
semver@^7.6.0, semver@^7.7.3: "safer-buffer@>= 2.1.2 < 3.0.0":
version "2.1.2"
resolved "https://registry.yarnpkg.com/safer-buffer/-/safer-buffer-2.1.2.tgz#44fa161b0187b9549dd84bb91802f9bd8385cd6a"
integrity sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==
semver@^7.6.0:
version "7.8.0" version "7.8.0"
resolved "https://registry.yarnpkg.com/semver/-/semver-7.8.0.tgz#ed0661039fcbcda2ce71f01fa6adbefaa77040df" resolved "https://registry.yarnpkg.com/semver/-/semver-7.8.0.tgz#ed0661039fcbcda2ce71f01fa6adbefaa77040df"
integrity sha512-AcM7dV/5ul4EekoQ29Agm5vri8JNqRyj39o0qpX6vDF2GZrtutZl5RwgD1XnZjiTAfncsJhMI48QQH3sN87YNA== integrity sha512-AcM7dV/5ul4EekoQ29Agm5vri8JNqRyj39o0qpX6vDF2GZrtutZl5RwgD1XnZjiTAfncsJhMI48QQH3sN87YNA==
sharp@*, sharp@^0.34.5:
version "0.34.5"
resolved "https://registry.yarnpkg.com/sharp/-/sharp-0.34.5.tgz#b6f148e4b8c61f1797bde11a9d1cfebbae2c57b0"
integrity sha512-Ou9I5Ft9WNcCbXrU9cMgPBcCK8LiwLqcbywW3t4oDV37n1pzpuNLsYiAV8eODnjbtQlSDwZ2cUEeQz4E54Hltg==
dependencies:
"@img/colour" "^1.0.0"
detect-libc "^2.1.2"
semver "^7.7.3"
optionalDependencies:
"@img/sharp-darwin-arm64" "0.34.5"
"@img/sharp-darwin-x64" "0.34.5"
"@img/sharp-libvips-darwin-arm64" "1.2.4"
"@img/sharp-libvips-darwin-x64" "1.2.4"
"@img/sharp-libvips-linux-arm" "1.2.4"
"@img/sharp-libvips-linux-arm64" "1.2.4"
"@img/sharp-libvips-linux-ppc64" "1.2.4"
"@img/sharp-libvips-linux-riscv64" "1.2.4"
"@img/sharp-libvips-linux-s390x" "1.2.4"
"@img/sharp-libvips-linux-x64" "1.2.4"
"@img/sharp-libvips-linuxmusl-arm64" "1.2.4"
"@img/sharp-libvips-linuxmusl-x64" "1.2.4"
"@img/sharp-linux-arm" "0.34.5"
"@img/sharp-linux-arm64" "0.34.5"
"@img/sharp-linux-ppc64" "0.34.5"
"@img/sharp-linux-riscv64" "0.34.5"
"@img/sharp-linux-s390x" "0.34.5"
"@img/sharp-linux-x64" "0.34.5"
"@img/sharp-linuxmusl-arm64" "0.34.5"
"@img/sharp-linuxmusl-x64" "0.34.5"
"@img/sharp-wasm32" "0.34.5"
"@img/sharp-win32-arm64" "0.34.5"
"@img/sharp-win32-ia32" "0.34.5"
"@img/sharp-win32-x64" "0.34.5"
shebang-command@^2.0.0: shebang-command@^2.0.0:
version "2.0.0" version "2.0.0"
resolved "https://registry.yarnpkg.com/shebang-command/-/shebang-command-2.0.0.tgz#ccd0af4f8835fbdc265b82461aaf0c36663f34ea" resolved "https://registry.yarnpkg.com/shebang-command/-/shebang-command-2.0.0.tgz#ccd0af4f8835fbdc265b82461aaf0c36663f34ea"
@@ -1502,6 +1497,11 @@ siginfo@^2.0.0:
resolved "https://registry.yarnpkg.com/siginfo/-/siginfo-2.0.0.tgz#32e76c70b79724e3bb567cb9d543eb858ccfaf30" resolved "https://registry.yarnpkg.com/siginfo/-/siginfo-2.0.0.tgz#32e76c70b79724e3bb567cb9d543eb858ccfaf30"
integrity sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g== integrity sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==
signal-exit@^4.1.0:
version "4.1.0"
resolved "https://registry.yarnpkg.com/signal-exit/-/signal-exit-4.1.0.tgz#952188c1cbd546070e2dd20d0f41c0ae0530cb04"
integrity sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==
source-map-js@^1.2.1: source-map-js@^1.2.1:
version "1.2.1" version "1.2.1"
resolved "https://registry.yarnpkg.com/source-map-js/-/source-map-js-1.2.1.tgz#1ce5650fddd87abc099eda37dcff024c2667ae46" resolved "https://registry.yarnpkg.com/source-map-js/-/source-map-js-1.2.1.tgz#1ce5650fddd87abc099eda37dcff024c2667ae46"
@@ -1566,11 +1566,6 @@ ts-api-utils@^2.1.0:
resolved "https://registry.yarnpkg.com/ts-api-utils/-/ts-api-utils-2.5.0.tgz#4acd4a155e22734990a5ed1fe9e97f113bcb37c1" resolved "https://registry.yarnpkg.com/ts-api-utils/-/ts-api-utils-2.5.0.tgz#4acd4a155e22734990a5ed1fe9e97f113bcb37c1"
integrity sha512-OJ/ibxhPlqrMM0UiNHJ/0CKQkoKF243/AEmplt3qpRgkW8VG7IfOS41h7V8TjITqdByHzrjcS/2si+y4lIh8NA== integrity sha512-OJ/ibxhPlqrMM0UiNHJ/0CKQkoKF243/AEmplt3qpRgkW8VG7IfOS41h7V8TjITqdByHzrjcS/2si+y4lIh8NA==
tslib@^2.4.0:
version "2.8.1"
resolved "https://registry.yarnpkg.com/tslib/-/tslib-2.8.1.tgz#612efe4ed235d567e8aba5f2a5fab70280ade83f"
integrity sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==
type-check@^0.4.0, type-check@~0.4.0: type-check@^0.4.0, type-check@~0.4.0:
version "0.4.0" version "0.4.0"
resolved "https://registry.yarnpkg.com/type-check/-/type-check-0.4.0.tgz#07b8203bfa7056c0657050e3ccd2c37730bab8f1" resolved "https://registry.yarnpkg.com/type-check/-/type-check-0.4.0.tgz#07b8203bfa7056c0657050e3ccd2c37730bab8f1"