The header of test/packaging/lint-once.test.ts claimed a duplicate prettier
pass is caught wherever it is added. It was not: the walk started at
`make check`, which never reads `Dockerfile`, so appending
`RUN yarn run prettier --check .` to the image that `script/cibuild` builds
left the suite green — two prettier passes on the one path where it matters
most. The walk now also starts at `.gitea/workflows/check.yml` and follows
its `run:` steps into `script/cibuild` and from there into both images, so
the graph under test is the one CI executes rather than the one it was
assumed to execute. Reaching `script/cibuild` and `Dockerfile` is asserted,
and the test and build image is asserted to invoke prettier zero times.
The lockfile assertion was a substring check against the whole of
`script/bootstrap`. That script has two install sites, and the containers
take the second, because the pinned node image ships yarn; changing that
site to a bare `yarn install` kept the suite green while the container's
install stopped being pinned. `install_js_deps` is now resolved out of the
script and split at its `missing yarn` guard, and every `yarn install`
occurrence in each branch is required to carry `--frozen-lockfile`. That the
container runs `script/bootstrap` at all is asserted too, so the lockfile
assertions cannot end up describing a script the image never executes.
Prettier is counted per occurrence instead of per line:
`prettier --check . && prettier --check src` was one invocation by the old
count. The `continue` that followed a counted line also dropped every
script, make, yarn and docker edge sharing that line, so a subtree could be
hidden behind a single `&&`; edges are now extracted from every line.
Undercounting is what would make this file worthless, so every way of
reaching nothing is a thrown error rather than a quiet zero: an unknown
Makefile target, an unknown package.json script, a missing script file, a
node that resolves to no commands, and an unknown node kind. All five are
tested, as is a walk that legitimately counts zero, and the cycle guard.
Every assertion in the file was mutation-tested: changed to assert something
else, run, and confirmed to fail for its own named reason. The two mutations
above were reproduced and both now turn the suite red.
test/packaging/entrypoints.test.ts said `make check` runs test, lint and
fmt-check. Formatting has been part of the lint container since the
duplicate host pass was removed, so the comment now says what it does.
script/check ran script/test, script/lint and script/fmt-check. Since
linting moved into Docker, script/lint is a build of Dockerfile.lint,
which runs `prettier --check .` as a build step — so make check checked
formatting twice over the same tree: once in the container and once on
the host. script/precommit had the same pair.
Drop the script/fmt-check call from both. The container keeps the check,
because a successful Dockerfile.lint build is what CI treats as proof of
a clean tree, and it is the stronger of the two verdicts: its prettier is
digest-pinned and installed under --frozen-lockfile, while the host's is
whatever the working tree happens to have. The pre-commit hook is
unchanged in what it catches — script/lint still fails a badly formatted
tree, and therefore the commit.
script/fmt-check survives as a standalone entrypoint, as REPO_POLICIES.md
requires, for asking the formatting question by itself without docker.
Its verdict cannot drift from the container's: prettier is pinned to an
exact version, installed from yarn.lock in both places, and reads
.gitignore as its default ignore file, which is why .dockerignore keeps
.gitignore in the build context.
The count is asserted rather than promised. test/packaging/lint-once.test.ts
walks the invocation graph from each entrypoint — through the Makefile
shims, the script/ calls, the package.json scripts and the docker build
into Dockerfile.lint's RUN steps — and counts prettier invocations: one
per make check, one per script/precommit, and one each for make lint and
make fmt-check alone, so neither can become a no-op that satisfies the
count trivially. The walk also asserts which nodes it reached, so a
restructure that defeats the resolver fails the test instead of quietly
counting zero.
Observed: 2 prettier invocations per make check before, 1 after.
Linting now happens in one place only: a new root Dockerfile.lint copies
the repo into the digest-pinned node image already used by Dockerfile and
runs eslint and prettier as build steps, so a successful build is a clean
lint. script/lint is reduced to building it, which also works where the
docker daemon is remote and bind mounts are impossible. No host lint path
survives: the "lint" script is gone from package.json, so there is no
second, unpinned way to get a lint verdict.
Caching is waived for lint, because a lint build over an unchanged tree
returns success in well under a second having linted nothing. LINT_EPOCH
is the cache buster and it fails closed exactly as CHECK_EPOCH does: an
unset ARG is the empty string, which is a perfectly stable cache key, so
the guard rejects it and a bare `docker build -f Dockerfile.lint .` errors
out instead of serving a green it did not earn. Both linters sit below the
guard, so a fresh epoch forces them to execute while the bootstrap and
dependency layers above stay cached.
That makes script/lint a docker build, which nothing inside a container
may call. script/check calls script/lint, so the Dockerfile image can no
longer run make check: the lint stage and its COPY --from=lint ordering
hack are deleted, and the remaining stage runs make test and make build
under the existing CHECK_EPOCH guard. script/cibuild is now the composite
gate and builds the lint image first, so a lint failure is reported before
the slower suite runs.
The .dockerignore exclusions are unchanged and still apply to the lint
build, including the .claude/ exclusion (eslint's flat config does not
ignore dot-directories, so a nested worktree in the context would be
linted) and the deliberate exception that keeps .gitignore in the context
for prettier. A new test asserts no per-Dockerfile ignore file shadows the
root one for either image, and test/packaging/lint-docker.test.ts asserts
the whole shape: the docker-only lint path, the digest pin, manifests
copied before sources, the fail-closed guard with both linters below it,
the absence of a lint stage or make check in Dockerfile, and the build
order in script/cibuild.
The image build reported a green it had not earned. `script/cibuild` is a
bare `docker build .`, and with `COPY . .` followed by `RUN make check`,
an unchanged tree served that layer from cache: the suite never ran and
the build still exited 0, while the script's header comment asserted the
opposite.
CHECK_EPOCH, passed by `script/cibuild` and `script/docker`, changes the
cache key of the check and build layers on every invocation. It is
guarded, because an unset ARG is the empty string and therefore a stable
key: without the guard a plain `docker build .` — the command the policy
names, and the one anyone debugging types — would still get the false
green. A missing argument is now a hard failure rather than a silent
degradation to the behaviour the epoch was added to prevent.
The Dockerfile is now two stages: `fmt-check` and `lint` run first, and
the check stage takes a `COPY --from=lint` dependency on them, so a
formatting mistake fails the build in seconds instead of racing the
suite to the finish. Both stages stay pinned to the same digest.
The remaining fixes are one-liners that had made the target unusable:
`script/projectname` still printed the pre-rename name, so `make docker`
tagged its image after a name this project dropped in May;
`script/bootstrap` installed without fetching apt's package lists, which
cannot work on a Debian base; and `.dockerignore` had drifted far enough
from `.gitignore` to ship a ~100 MB compiled binary and any agent
worktree under `.claude/` into the build context. The second of those is
a correctness problem, not a size one — vitest globs a copied worktree's
tests alongside the real ones and runs the suite twice over. `.gitignore`
itself stays in the context, because prettier reads it as a default
ignore file and dropping it would change what `make fmt-check` sees.
script/projectname still prints the pre-rename name, so make docker tags
its image quack, and .dockerignore has drifted from .gitignore: a compiled
bin/quak, the vitest and tsc caches, the CLI's runtime directory and the
local worktree directory all reach the build context. Neither failure is
visible in a build that exits 0.
Red until the fixes land.
rootDir was ./src while include also matched bin/**/*, which is TS6059: tsc
refuses to emit at all when a compiled file sits outside rootDir. rootDir is
now the repository root, which is the smallest change that makes the two
agree and leaves the source layout the README documents alone. Output keeps
the shape of the source tree, so main and types move to dist/src/index.js and
dist/src/index.d.ts while bin.quak stays at dist/bin/quak.js. The alternative,
moving the CLI body into src/ behind a shim in bin/, would hold main at
dist/index.js at the cost of churning the CLI and contradicting the layout
diagram in the README.
Clearing TS6059 exposed two type errors that had never been reached, because
the config error aborts before checking: StateAddress was read as a namespace
member off the default import, and the secretstream pull was called without
the additional-data argument, which libsodium does not make optional. The type
is now taken from the module's named export and the pull passes null for ad,
matching the null already passed on the push side in encryptBlob. Neither
changes what runs.
noEmitOnError stops a failed build from leaving output behind. It emitted
despite the error before, which is how a stale bin/quak.js came to sit next to
bin/quak.ts in a working tree, where eslint then read it and failed make check
on a generated file.
script/build compiles and then checks that the files package.json advertises
are among the ones the compiler wrote, since tsc knows nothing about the
manifest and a green build could still ship a package whose main resolves to
nothing. It also sets the executable bit on the bin entries, which tsc does
not carry over from the source even though it does copy the shebang. The
Makefile target is now a shim over it, as the other targets are, and
package.json's build script points at it so yarn build gets the same checks.
The Dockerfile runs make build after make check, so a branch that does not
compile cannot reach main. What make check itself runs is unchanged.
package.json gains a quak script, so the yarn quak commands the README's
Getting Started block has always listed resolve to the built CLI.
The manifests disagreed and nothing noticed. tsconfig.json set rootDir to
./src while include also matched bin/**/*, which is TS6059, so no build had
succeeded; package.json meanwhile advertised main, types and a bin that a
successful build would have to produce. make check runs test, lint and
fmt-check, so neither half was ever exercised.
These tests read tsconfig.json and package.json and assert the contract
between them without invoking a compiler, which keeps them in the fast unit
suite: every include pattern must root under rootDir, and main, types and
bin.quak must equal the paths tsc will emit for src/index.ts and bin/quak.ts.
They also require a quak script pointing at the built CLI, which the README's
Getting Started block has always told the reader to run.
Three of them fail at this commit.
`CONNECT_CODES` drives `isSafeToReplay`, which is the only thing standing
between a transport failure and a replayed `POST /users/two-factor/verify`.
It included `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` on the stated
grounds that those errnos can only be reported before any request byte was
written. That is not true on Linux: an ICMP destination-unreachable
delivered on an already-established connection sets the socket error and
the next read or write returns `EHOSTUNREACH` or `ENETUNREACH`, and a local
interface going down after the request was fully written surfaces as
`ENETDOWN` the same way. In each case the server may already have received
and acted on the request -- exactly the ambiguity the rule exists to
exclude, on the paths that consume a second-factor attempt or register a
thumbnail.
The three are dropped from `CONNECT_CODES` and stay in `TRANSPORT_CODES`,
so they remain retryable for the idempotent calls; only replay eligibility
narrows. What is left -- `ENOTFOUND`, `EAI_AGAIN`, `ECONNREFUSED` -- means
no TCP connection to the server ever existed, so no request byte can have
been transmitted.
The justification is corrected everywhere it was stated: the comment on
`CONNECT_CODES`, the one on `isSafeToReplay`, the `postJSON` call site, the
README's idempotency section and the `client.test.ts` docblock. All of them
now describe what the narrowed set actually establishes rather than
claiming a proof it did not support.
The narrowing is enforced by the suite rather than asserted in a comment:
the three errnos join `ECONNRESET`/`EPIPE`/`ETIMEDOUT` in the
`isSafeToReplay`-returns-false test, with companion `isRetryable` assertions
so a future edit cannot make them non-retryable by accident. Putting the
three back into `CONNECT_CODES` turns that test red (1 failure, verified).
No retry on 4xx, backoff on 5xx and transport failures, and a deadline on
every request. Before this, one transient 503 or TCP reset failed a file
for good, and a CDN connection that went quiet after accepting the request
blocked `quak backup` forever, because there was no timeout anywhere.
src/retry.ts holds the policy: a classifier that decides whether another
attempt could produce a different answer, and a loop that acts on it with
exponential backoff and full jitter. Retried: 5xx, 408, 429, transport
failures (the errno is read out of the cause chain, which is where Node's
fetch puts it), deadline aborts, and truncated transfers. Not retried:
every other 4xx, and anything unrecognised — a wrongly retried permanent
failure delays every remaining file, while a wrongly abandoned transient
one costs a single file the next run picks up. Attempt count, delays,
sleep and jitter source are all configurable through ApiClientOptions;
sleep being injectable is what lets the suite exercise the policy without
waiting.
Truncation needed a type before it could be classified. streamDecrypt
threw plain Errors whose messages began "download: stream truncated", and
classifying on message text would mean the next reword silently turned
every truncated download into a permanent failure. It now throws
TruncatedStreamError, which lives in src/errors.ts alongside ApiError so
the classifier can recognise both without importing the modules that
import it; api/client.ts re-exports ApiError, so it stays one class and
every existing import path still resolves.
Downloads retry the request, the stream consumption and the decryption
together. Only the first of those happens inside ApiClient: a socket
reset after the headers arrived throws in streamDecrypt, and retrying the
request alone would never see it. The client's own retry is switched off
for those two calls so the budgets do not multiply into sixteen requests
per file, and the atomic write stays outside the loop so a download that
took three attempts still performs one write and one rename.
Non-idempotent requests are not blindly replayed. postJSON and putJSON
reach create-session, two-factor/verify — which burns one of a few
second-factor attempts — and files/thumbnail, so they retry only when the
connection was never established and the server provably never saw the
request. putFile is exempt and retries fully: a presigned PUT stores one
whole object at one key, with no partial state to damage. It now throws
ApiError with the status, as do the two null-body paths, which previously
threw bare Errors that nothing could classify.
Timeouts come from AbortSignal.timeout(), renewed per attempt: 30s for
JSON and upload calls, 10 minutes for file bodies, since a value short
enough to keep a hung API call from stalling a backup would cancel a
legitimate multi-gigabyte download. The download deadline is enforced
over the body rather than only the headers, by racing each read against
the signal, so the guarantee does not depend on the fetch implementation
tearing down a stream it already handed over.
listMissingThumbnails now separates a genuine 404 from an exhausted
retry. Its bare catch reported both as missing, which after this change
would have let a few minutes of 500s talk fix-missing-thumbnails into
regenerating and re-uploading thumbnails that were fine. runBackup and
runMetadataBackup are untouched: the retry sits below them and their
per-file resilience is unchanged.
Tests only; the retry module they import does not exist yet, so the
branch is red at this commit.
New test/retry/retry.test.ts documents the classifier and the backoff:
which errors are worth another attempt, which are not, and how the delay
before each retry is derived. It asserts on the arguments handed to an
injected sleep function rather than on elapsed time, so the suite never
waits and the numbers are exact.
test/api/client.test.ts gains the request-count contract for each of the
six call sites, the deadline behaviour, the ApiError typing that the
presigned PUT and the null-body paths need in order to be classified at
all, and the replay rule for the two non-idempotent methods.
test/download/download.test.ts gains the case that motivates the whole
design: a socket reset after the response headers arrived, which happens
below ApiClient and can only be caught by retrying the request, the
stream consumption and the decryption together. It also pins that a
retried download stages exactly one temp file, and that the two retry
layers do not compose into a multiplied request budget. The existing
truncation tests now assert on the error type rather than its wording,
since that type is what the classifier reads.
test/thumbnails/thumbnails.test.ts separates a genuine 404 from an
exhausted retry, so a failing server can no longer make
fix-missing-thumbnails re-upload thumbnails that already exist.
Rework against review finding B-A on the branch.
The claim was false as written. Commit 8a200be's message, the PR body and
the comments on both sides said the repurposed pinning test in
test/crypto/stream.test.ts fails if streamTagFinal() is ever made eager.
It does not: substituting
const EAGER: number = sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL;
export const streamTagFinal = (): number => EAGER;
leaves the whole suite green. Under vitest, sodium is already initialised
in the worker process by the time a source module is evaluated, so an
eager read picks up a real value there and value equality cannot see the
difference. The danger it claimed to cover is real: a direct probe of the
vendored libsodium-wrappers-sumo gives undefined before await sodium.ready
and 3 after, so an eager read would ship a library that rejects every
valid download as truncated for anyone importing it in plain Node ESM,
while make check stayed green.
Rather than drop the claim, this makes it true. A new test reproduces the
plain-Node ESM ordering that vitest hides: vi.resetModules() plus a
doMock'd stand-in sodium whose TAG_FINAL property is absent while
src/crypto/stream.ts is evaluated and appears only afterwards, exactly as
libsodium attaches its constants inside ready.then(...). A call-time read
sees the value that appeared after evaluation; a module-level read binds
undefined and the test fails. The stand-in hands back a sentinel rather
than 3, so a read that somehow reached the real library, or a return to a
hardcoded literal, fails too.
Demonstrated rather than argued: with the two-line eager variant above in
place, make test reports 1 failed | 140 passed, "expected undefined to be
42", exit 2. Restored with git checkout -- src/crypto/stream.ts, make test
reports 141 passed. The eager variant was applied with the editor and
reverted with git, not by scripted substitution.
The pinning test keeps its original job — value equality against the tag
observed on a real final chunk — and its doc comment now says only that.
The comment on the accessor in src/crypto/stream.ts names the guard test,
and records why the ordinary tests cannot see the bug on their own.
Rework against review findings on the branch.
Test runtime (B1): the suite's cost was never the 4 MiB fixture, it was
filling that fixture from the CSPRNG. sodium.randombytes_buf goes through
the wasm wrapper a byte at a time and takes ~20s for 4 MiB, against ~108ms
to encrypt the same buffer. Fixture content is not load-bearing anywhere in
the file — only length and tag are — so payloads now come from a seeded LCG
instead, which also makes them deterministic and reproducible as the README
asks. A seeded generator rather than a constant fill, so a downloader that
reordered or repeated chunks would still be caught. make test goes from
over the 30s cap in script/test (it was failing outright, then rerunning
verbose) back to 8.96s, against 10.02s on main.
Atomic write coverage (B2): every failure the suite injected originated in
streamDecrypt, which runs before anything is written, so no test observed a
temp file existing or being cleaned up and the catch block in writeAtomic
was dead code. rename is now intercepted in the test file, which adds two
cases per entry point: one asserting the staged file exists at rename time
and is a sibling of the destination, and one failing the rename itself so
the cleanup path runs with a temp file genuinely on disk. Deleting
writeAtomic in favour of a plain writeFile now turns the suite red.
TAG_FINAL (M1): STREAM_TAG_FINAL was a hardcoded 3 plus a test to detect
drift. The premise was right — libsodium attaches its constants inside
ready.then(...), so an eager module-level read binds undefined — but a lazy
read works, and decryptBlob was doing exactly that before. Replaced with
streamTagFinal(), which reads the library's own value at call time. The
drift test is repurposed to pin the accessor against the tag observed on a
real final chunk, which fails if it is ever made eager again.
Partial trailing chunk (M2): a transfer that stopped mid-chunk surfaced as
"authentication failed", which reads as corruption and sends the user after
the wrong problem. A final chunk that arrived in full always authenticates,
so trailing bytes that do not are reported as the truncation they almost
always are, with the authentication failure kept as the error's cause.
Poly1305 cannot separate a partial chunk from a corrupt one, so the message
names both possibilities; a corrupt whole chunk mid-stream is still
reported as an authentication failure, and both are now tested.
The pre-existing test that wrote its output to the process working
directory, i.e. the repo root, now writes into the test's temp directory.
streamDecrypt discarded the secretstream tag, so a download cut short
by a dropped connection decrypted cleanly up to the last whole chunk
and was returned as a success. downloadFile and downloadThumbnail then
wrote straight to the destination, and runBackup skips any existing
non-empty file, so a truncated original was treated as complete on
every subsequent run and never repaired.
streamDecrypt now tracks the tag of each chunk it pulls and throws if
the stream ended on anything other than TAG_FINAL, or if the body
carried no chunks at all — Ente always emits at least one chunk, as
encryptBlob shows by producing a TAG_FINAL chunk even for zero-length
plaintext, so an empty body is a failed transfer rather than an empty
file. Both error messages say the stream was truncated.
Plaintext is now staged in a temporary sibling file (same directory,
so the rename cannot cross a filesystem boundary; random UUID suffix,
so concurrent downloads cannot collide) and renamed into place only
after the whole stream has decrypted and verified. On any error the
temporary file is removed and the original error is rethrown
unchanged, so a cleanup failure never masks the real diagnosis. A
failed download therefore leaves the destination exactly as it was.
Public signatures and the DownloadResult shape are unchanged.
The download layer keeps its no-direct-sodium-import shape: TAG_FINAL
is re-exported from src/crypto as STREAM_TAG_FINAL, which decryptBlob
now uses too. Also moves the pullStreamChunk doc comment off
decryptBlob, where it had been sitting.
Retry and backoff remain out of scope; they stay the Next Step in
TODO.md and are tracked separately.
Covers, for both downloadFile and downloadThumbnail:
- a multi-chunk body whose TAG_FINAL chunk never arrived is rejected
with a truncation error;
- an empty body is rejected as truncation rather than written as a
zero-byte file;
- after a truncation or chunk-authentication failure the destination
path does not exist and no temporary scratch file is left behind;
- an existing file at the destination survives a failed download
byte for byte, and is replaced atomically by a successful one;
- the existing success cases still produce identical bytes and an
identical DownloadResult.
Adds an encryptMultiChunkBody helper that frames leading chunks at
exactly STREAM_CHUNK_SIZE so the downloader's fixed-size re-splitting
lines up, plus a multi-chunk success case as the positive control.
Also pins the new STREAM_TAG_FINAL crypto export against libsodium's
own constant, since it must be declared as a literal: libsodium
attaches its constants only after sodium.ready resolves, well after
this library's modules are evaluated.
These fail until the implementation lands, per the repo's TDD workflow.
/collections/v2 is a sync API: deleted collections remain in the
response forever with isDeleted: true, and their /collections/v2/diff
endpoint returns HTTP 404. A long-lived account accumulates hundreds
of tombstones, so any caller that iterates listCollections() output
(backup, backup-metadata) dies on the first one.
decryptCollection now takes the full key material {masterKey,
publicKey, secretKey} and dispatches on keyDecryptionNonce: present
means an owned collection (secretbox under the master key), absent
means a shared collection (sealed box to our public key). Client
already held the keypair for unsealing the auth token, so it just
passes it through.
Collections shared with the account are not encrypted with the master
key: the sharer only knows the recipient's public key, so the server
delivers encryptedKey as crypto_box_seal to that key and omits
keyDecryptionNonce entirely. decryptCollection assumed the owned-only
wire format and crashed on fromBase64(undefined) for any account with
an incoming shared album, taking down listCollections and every
command built on it (backup, backup-metadata, ...).
The previous "shared" fixture was unfaithful (secretbox + nonce with a
foreign ownerID, a shape the server never sends), which is why the
suite stayed green. These tests model the real wire format and change
decryptCollection to take the full key material {masterKey, publicKey,
secretKey} so it can unseal shared collection keys.
Use || instead of ?? when picking the TOTP session ID. The server
sends "" for unset 2FA fields (no omitempty), and "" ?? v2
short-circuits to "", which made dual-2FA accounts fall through to
the unsupported passkey branch.
The museum server's EmailAuthorizationResponse declares
passkeySessionID, accountsUrl, twoFactorSessionID, and
twoFactorSessionIDV2 without `omitempty`, so Go always sends them,
as "" when unset. The previous mock omitted the unset fields
entirely, which let the ?? -based dispatch pass in tests while the
real server's "" defeated it and dual-2FA logins fell through to
the unsupported passkey branch.
The server signals dual-2FA accounts with passkeySessionID +
twoFactorSessionIDV2. quak now takes the TOTP path in that case
instead of failing with 'Passkey authentication is not supported'.
beginLogin now checks twoFactorSessionID ?? twoFactorSessionIDV2 before
the passkey branch. Accounts with both passkeys and TOTP can now log in
from the CLI using their authenticator app.
When an account has both passkeys and TOTP enrolled, the Ente server
returns passkeySessionID + twoFactorSessionIDV2 (deliberately not the
V1 twoFactorSessionID field, so old clients keep using passkeys).
quak only checked the V1 field, saw the passkey session, and threw
'Passkey authentication is not supported'. A CLI cannot do WebAuthn,
so it must take the TOTP path via the V2 session ID.
Replaces the hand-rolled readline/raw-mode password prompt with
@inquirer/prompts (input + password). The manual approach broke in
bun-compiled binaries because bun doesn't properly re-open stdin
after closing a readline instance. inquirer handles TTY, raw mode,
and password masking correctly across both node and bun runtimes.
Removes the only native dependency (sharp). All image processing
now uses jpeg-js (pure JS JPEG codec) + a bilinear RGBA resize for
thumbnail generation, and raw JPEG APP1/XMP byte parsing + exif-reader
for EXIF extraction. Every dependency is now pure JS or Emscripten.
make build-bin compiles the entire CLI into a 59MB self-contained
binary via bun build --compile (bun from nix-shell). No runtime
dependencies needed to run the binary.
sharp was the only native dependency preventing a single-file binary.
Replaced with:
- jpeg-js (pure JS) for JPEG decode/resize/encode in thumbnail gen
- exif-reader (pure JS) for EXIF tag parsing
- Raw JPEG APP1 marker extraction for EXIF segment discovery
- Raw XMP packet extraction from file bytes
make build-bin produces a ~59MB self-contained Mach-O binary via
bun build --compile (bun installed via nix-shell). Zero runtime
dependencies. Tested: login, whoami, collections, files all work
from the compiled binary.
bin/quak.ts: init() called once at program start before commander
parses, so libsodium is ready for all commands including those that
restore sessions from disk.
118 tests pass.
ML metadata (face detections, CLIP embeddings) is not a separate
category from the rest of the metadata. It is always fetched and
included. The only opt-in is --exif (or --all) which requires
downloading every file for EXIF extraction.
ML data (face detections, CLIP embeddings) is now fetched by default
in backup-metadata. Use --no-ml to skip it. EXIF extraction (which
requires downloading every file) remains opt-in via --exif. --all is
an alias for --exif.
--ml: fetches face detections (bounding boxes, landmarks, embeddings)
and CLIP search embeddings from the /files/data/fetch endpoint. These
are encrypted with the file key and gzipped; quak decrypts and
decompresses them into the per-file JSON output.
--exif: downloads each original file, extracts full image metadata
via sharp (format, dimensions, color space, orientation) and parses
raw EXIF tags via exif-reader (lens, ISO, shutter, aperture, GPS
altitude, software, etc.). Also captures IPTC, XMP, and ICC data.
3 new tests. 119 total, all green.
--ml fetches face detections and CLIP embeddings from the /files/data/fetch
endpoint (type 'mldata'). Each blob is encrypted with the file's key and
gzipped; we decrypt with decryptBlob, gunzip, and include the parsed JSON
as 'mlData' in the per-file output. Fetched in batches of 200 file IDs.
--exif downloads each file, runs sharp().metadata() to extract image
properties (format, dimensions, color space, orientation), then parses
the raw EXIF buffer with exif-reader for structured tags (lens, ISO,
shutter, aperture, GPS altitude, etc.). Also captures raw IPTC, XMP,
and ICC profile data. Included as 'imageMetadata' in the per-file output.
Without either flag, behavior is unchanged (fast metadata-only dump).
Adds exif-reader 2.0.3 as a runtime dependency.
3 new tests (ML data decrypted, ML data absent when flag not set, EXIF
extraction). 119 total tests, all green.
Dumps all decrypted account metadata to a directory tree of plain JSON
files. No file content downloads. Includes collection-level magic
metadata decryption (visibility, sort order, cover photo) which was
previously missing. 6 new tests, 116 total.
New command: quak backup-metadata <dir>
Dumps every piece of decrypted account metadata into a directory tree
of plain JSON files without downloading any file content. Layout:
<dir>/
account.json { email, userID }
collections/
<id>-<name>/
_collection.json { id, name, type, pubMagicMetadata?, ... }
<fileID>.json { id, metadata, magicMetadata?, pubMagicMetadata? }
Also adds collection-level magic metadata decryption (magicMetadata,
pubMagicMetadata, sharedMagicMetadata) to decryptCollection, which was
previously only done for files. The server sends these for visibility
settings, sort order, cover photo selection, etc.
6 new tests covering: account.json, per-collection dirs with
_collection.json, collection pubMagicMetadata decryption, per-file
JSON with all three metadata layers, graceful handling of files with
no magic metadata, and incremental re-run safety. 116 total.
Fixes accumulated drift from the original spec-first README:
- Intro: added paragraph describing backup, metadata decryption, and
thumbnail repair capabilities
- Rationale: removed the 'deliberately scoped to read operations'
claim (no longer true since thumbnail upload exists); called out
the Go CLI's crash-on-failure bug as explicit motivation
- Getting Started: fixed library example to use actual Client.login()
API, removed nonexistent fromSavedSession/getFile methods, added
backup CLI example
- Layout: fixed to match actual directory structure (download/,
backup.ts, thumbnails.ts; removed nonexistent session/)
- Session handling: replaced the fictional encrypted-keychain
SessionStore description with the actual implementation (plain JSON
via env-paths, consumer-managed persistence via toJSON/fromJSON)
- CLI surface: added backup, helper list-missing-thumbnails, and
helper fix-missing-thumbnails commands
- Backup layout: new section documenting the originals/ + collections/
symlink structure
- API reference: replaced the stale type declarations with pointers
to the actual source files and a note that test/client/usage.test.ts
is the authoritative API tutorial
- TODO: collapsed completed phases, kept only open items
- For LLMs: new section summarizing repo policies, TDD workflow,
required checks, formatting rules, and pointers to REPO_POLICIES.md
and LLM_PROSE_TELLS.md
quak helper list-missing-thumbnails: scans all files, fetches each
thumbnail, reports missing/empty ones with deduplication.
quak helper fix-missing-thumbnails: downloads originals, generates
720px JPEG via sharp, encrypts with secretstream push (encryptBlob),
uploads via presigned URL, registers via PUT /files/thumbnail.
New crypto: encryptBlob (secretstream push, single chunk TAG_FINAL).
New ApiClient: putJSON, putFile (no auth headers for S3), getUploadURL,
updateThumbnail. New Client: getApiClient() accessor.
20 new tests covering: encryptBlob round-trips and edge cases (8),
upload API methods including putFile auth-header leakage check (4),
listMissingThumbnails detection and dedup (2), fixMissingThumbnails
full pipeline with JPEG magic byte verification on decrypted upload (3),
getApiClient logged-in/logged-out behavior (2), error resilience (1).
110 total tests, all green.
test/crypto/encrypt-blob.test.ts (8 tests):
Round-trip with decryptBlob, zero-length payload, ciphertext
overhead check, header size check, different keys produce different
output, same key produces different output each call (random nonce),
wrong-key rejection, tamper detection.
test/api/upload.test.ts (4 tests):
putJSON sends PUT with auth headers and JSON body. putFile sends
PUT to the exact presigned URL with Content-Type octet-stream and
does NOT send X-Auth-Token or X-Client-Package (S3 would reject
them). getUploadURL POSTs with contentLength and contentMD5.
updateThumbnail PUTs to /files/thumbnail with correct body shape.
test/thumbnails/thumbnails.test.ts (8 tests):
listMissingThumbnails identifies empty (0 byte) and 404 thumbnails
while ignoring working ones; deduplicates across collections.
fixMissingThumbnails verifies the full pipeline: download original,
generate JPEG via sharp, encrypt with encryptBlob, upload via
presigned URL, register via PUT /files/thumbnail. The test
decrypts the uploaded ciphertext and verifies it starts with JPEG
magic bytes (FF D8 FF). Also tests: nonexistent file ID reports
failure without crashing; mixed success/failure across multiple
files; Client.getApiClient() works when logged in, throws after
logout.
list-missing-thumbnails: iterates all files across all collections,
fetches each thumbnail from the CDN, reports any that are missing or
empty. Deduplicates by file ID across collections.
fix-missing-thumbnails: for each missing thumbnail, downloads the
original file, generates a 720px JPEG thumbnail via sharp, encrypts
it with secretstream push (encryptBlob), uploads to a presigned URL,
and registers the new thumbnail via PUT /files/thumbnail.
New crypto: encryptBlob (secretstream push, single chunk TAG_FINAL).
New ApiClient methods: getUploadURL, putFile, putJSON, updateThumbnail.
New Client method: getApiClient() for modules that need raw API access.
Deps: sharp 0.34.5 (image processing), @types/sharp 0.32.0.
Complete CLI surface:
quak login interactive or QUAK_EMAIL/QUAK_PASSWORD
quak whoami print logged-in account
quak logout delete session
quak collections list all albums (--json)
quak files list files in a collection (--json)
quak get <id> download+decrypt a file (--out, --collection)
quak get-thumb <id> download+decrypt a thumbnail
quak backup <dir> full incremental backup
get/get-thumb search all collections for the file ID when --collection
is not specified. All listing commands support --json.
Live-tested: collections list, file list, single file download (472 KB
JPEG from the dev account, verified as valid JPEG with EXIF intact).
All three metadata layers (basic, magicMetadata, pubMagicMetadata) are
now decrypted from secretstream blobs and exposed on EnteFile. Backup
writes originals/<fileID>.json with the full decrypted metadata
including camera make/model, dimensions, datetime, and any face/keyword
data the Ente clients have added.
