`fetchMLDataBatch` dropped an entry that did not decrypt, gunzip or parse,
so the file looked like one with no ML data and the run exited 0. It now
returns the reason beside the payloads, also for ML data that is not a
JSON object. The library gives the reasons from its last ML data fetch
that succeeded in `status().mlDataErrors` and the `fetchMLData` "done"
event; a fetch that fails leaves them as they were. `quak backup` writes
the reason as `mlDataError` and records the file in `failures.json`;
`backup-metadata` writes `mlDataError` and exits 1.
Judgement call: a file whose ML data is cached keeps it in its JSON when a
newer copy does not decode, as when the fetch fails.
Model: opus-5-5
test/cli/backup-end-to-end.test.ts runs backupCommand through run, as
bin/quak.ts does, with the real loadSession reading a session saved after
logging in, and every request answered by a fake of the Ente HTTP API serving
data encrypted with libsodium. It covers the backup tree of an account with
two albums of its own and one shared with it, a second run that downloads
nothing, a download that succeeds after two 503s, a 404 recorded in
failures.json with exit 1, a refresh answered 401 with exit 3, and --verify
putting back an original changed on disk.
The fake server and the account it logs in moved from the backup-metadata
test into test/ente-server.ts, unchanged, so both tests use them.
Model: opus-5-5
Each ML data request of up to 200 files is now tried on its own. A request
that still fails after its retries is logged, its files are written with the
reason in `mlDataError`, and the command exits 1 once the dump is complete.
`fetchMLData`, used only here, is removed in favour of a per-batch loop over
`fetchMLDataBatch`.
Model: opus-5-5
The segment scan behind `backup-metadata --exif` now checks every
segment length against the bytes that remain and stops on lengths
under 2, so a truncated or corrupt original can neither throw nor loop.
A malformed or unparseable EXIF segment is recorded as
`imageMetadata.exifError`, and a failure to read the original as
`imageMetadataError` in the per-file JSON, instead of the field being
silently left out. Tests use short hand-built byte arrays.
Model: opus-5-5
A file title or album name decrypted from server data could name a path
outside the chosen directory (`../../.ssh/authorized_keys`). One module,
src/filename.ts, now makes such names safe for `quak get`/`get-thumb`
without `--out`, downloadFile/downloadThumbnail without outPath, and the
backup and metadata backup trees. Originals-cache extensions are limited to
letters and digits. A user-supplied path is still used as is. decryptFile
reads a missing or non-string title as "" and rejects metadata that is not
a JSON object.
Model: opus-5-5
Ports the CLI to the library API. collections/files/get/get-thumb use the fresh read variants (Library.fresh(), current server state); backup runs lib.backup; backup-metadata and the missing-thumbnail helpers enumerate via the library. files/get output is byte-identical to the pre-port CLI — raw metadata.title, microsecond creationTime, pre-port row order — exit codes unchanged. Adds --cache-dir; fixes the helper JPEG-only assumption (closes#17).
Model: opus-4-8
sharp was the only native dependency preventing a single-file binary.
Replaced with:
- jpeg-js (pure JS) for JPEG decode/resize/encode in thumbnail gen
- exif-reader (pure JS) for EXIF tag parsing
- Raw JPEG APP1 marker extraction for EXIF segment discovery
- Raw XMP packet extraction from file bytes
make build-bin produces a ~59MB self-contained Mach-O binary via
bun build --compile (bun installed via nix-shell). Zero runtime
dependencies. Tested: login, whoami, collections, files all work
from the compiled binary.
bin/quak.ts: init() called once at program start before commander
parses, so libsodium is ready for all commands including those that
restore sessions from disk.
118 tests pass.
ML metadata (face detections, CLIP embeddings) is not a separate
category from the rest of the metadata. It is always fetched and
included. The only opt-in is --exif (or --all) which requires
downloading every file for EXIF extraction.
ML data (face detections, CLIP embeddings) is now fetched by default
in backup-metadata. Use --no-ml to skip it. EXIF extraction (which
requires downloading every file) remains opt-in via --exif. --all is
an alias for --exif.
--ml fetches face detections and CLIP embeddings from the /files/data/fetch
endpoint (type 'mldata'). Each blob is encrypted with the file's key and
gzipped; we decrypt with decryptBlob, gunzip, and include the parsed JSON
as 'mlData' in the per-file output. Fetched in batches of 200 file IDs.
--exif downloads each file, runs sharp().metadata() to extract image
properties (format, dimensions, color space, orientation), then parses
the raw EXIF buffer with exif-reader for structured tags (lens, ISO,
shutter, aperture, GPS altitude, etc.). Also captures raw IPTC, XMP,
and ICC profile data. Included as 'imageMetadata' in the per-file output.
Without either flag, behavior is unchanged (fast metadata-only dump).
Adds exif-reader 2.0.3 as a runtime dependency.
3 new tests (ML data decrypted, ML data absent when flag not set, EXIF
extraction). 119 total tests, all green.
New command: quak backup-metadata <dir>
Dumps every piece of decrypted account metadata into a directory tree
of plain JSON files without downloading any file content. Layout:
<dir>/
account.json { email, userID }
collections/
<id>-<name>/
_collection.json { id, name, type, pubMagicMetadata?, ... }
<fileID>.json { id, metadata, magicMetadata?, pubMagicMetadata? }
Also adds collection-level magic metadata decryption (magicMetadata,
pubMagicMetadata, sharedMagicMetadata) to decryptCollection, which was
previously only done for files. The server sends these for visibility
settings, sort order, cover photo selection, etc.
6 new tests covering: account.json, per-collection dirs with
_collection.json, collection pubMagicMetadata decryption, per-file
JSON with all three metadata layers, graceful handling of files with
no magic metadata, and incremental re-run safety. 116 total.
clawbot
sneak