Two background fills start inside open() with no caller input, through the
shared pools (#45) at background priority, so on-demand work always preempts
them. Thumbnails: every file newest first until all are on disk, sharing the
thumbnail pool with thumbnails.ensure. Originals: the pinned set — the
favorites album, then the latest-week window ending at the newest file —
through the content pool. The pinned set is the eviction predicate (#47), so a
pinned original is never evicted and a file that leaves the set (favorite
removed or window moved on a later refresh) becomes an ordinary, evictable
original. A cached file costs one map lookup and no fetch; each refresh re-kicks
the fills to pick up new files and retry failures. Progress via open()
onProgress and status() (thumbnailsCached/Total, originalsCached/Pinned).
Model: opus-4-8
Rewrites backup on the library API. backup() refreshes, fetches pending originals (and optionally thumbnails) through the pools reusing the content cache, then materialises the unchanged collections/ symlink views + per-collection JSON + sidecars from the model. A symlink failure no longer aborts the run (closes#8). failures.json reconciles against each run's attempted set — since-deleted/out-of-scope/resolved entries clear, still-failing retained, one attempt per file per run — and the exit-code contract is preserved.
Model: opus-4-8
Tests only; the retry module they import does not exist yet, so the
branch is red at this commit.
New test/retry/retry.test.ts documents the classifier and the backoff:
which errors are worth another attempt, which are not, and how the delay
before each retry is derived. It asserts on the arguments handed to an
injected sleep function rather than on elapsed time, so the suite never
waits and the numbers are exact.
test/api/client.test.ts gains the request-count contract for each of the
six call sites, the deadline behaviour, the ApiError typing that the
presigned PUT and the null-body paths need in order to be classified at
all, and the replay rule for the two non-idempotent methods.
test/download/download.test.ts gains the case that motivates the whole
design: a socket reset after the response headers arrived, which happens
below ApiClient and can only be caught by retrying the request, the
stream consumption and the decryption together. It also pins that a
retried download stages exactly one temp file, and that the two retry
layers do not compose into a multiplied request budget. The existing
truncation tests now assert on the error type rather than its wording,
since that type is what the classifier reads.
test/thumbnails/thumbnails.test.ts separates a genuine 404 from an
exhausted retry, so a failing server can no longer make
fix-missing-thumbnails re-upload thumbnails that already exist.
sharp was the only native dependency preventing a single-file binary.
Replaced with:
- jpeg-js (pure JS) for JPEG decode/resize/encode in thumbnail gen
- exif-reader (pure JS) for EXIF tag parsing
- Raw JPEG APP1 marker extraction for EXIF segment discovery
- Raw XMP packet extraction from file bytes
make build-bin produces a ~59MB self-contained Mach-O binary via
bun build --compile (bun installed via nix-shell). Zero runtime
dependencies. Tested: login, whoami, collections, files all work
from the compiled binary.
bin/quak.ts: init() called once at program start before commander
parses, so libsodium is ready for all commands including those that
restore sessions from disk.
118 tests pass.
ML metadata (face detections, CLIP embeddings) is not a separate
category from the rest of the metadata. It is always fetched and
included. The only opt-in is --exif (or --all) which requires
downloading every file for EXIF extraction.
ML data (face detections, CLIP embeddings) is now fetched by default
in backup-metadata. Use --no-ml to skip it. EXIF extraction (which
requires downloading every file) remains opt-in via --exif. --all is
an alias for --exif.
--ml fetches face detections and CLIP embeddings from the /files/data/fetch
endpoint (type 'mldata'). Each blob is encrypted with the file's key and
gzipped; we decrypt with decryptBlob, gunzip, and include the parsed JSON
as 'mlData' in the per-file output. Fetched in batches of 200 file IDs.
--exif downloads each file, runs sharp().metadata() to extract image
properties (format, dimensions, color space, orientation), then parses
the raw EXIF buffer with exif-reader for structured tags (lens, ISO,
shutter, aperture, GPS altitude, etc.). Also captures raw IPTC, XMP,
and ICC profile data. Included as 'imageMetadata' in the per-file output.
Without either flag, behavior is unchanged (fast metadata-only dump).
Adds exif-reader 2.0.3 as a runtime dependency.
3 new tests (ML data decrypted, ML data absent when flag not set, EXIF
extraction). 119 total tests, all green.
New command: quak backup-metadata <dir>
Dumps every piece of decrypted account metadata into a directory tree
of plain JSON files without downloading any file content. Layout:
<dir>/
account.json { email, userID }
collections/
<id>-<name>/
_collection.json { id, name, type, pubMagicMetadata?, ... }
<fileID>.json { id, metadata, magicMetadata?, pubMagicMetadata? }
Also adds collection-level magic metadata decryption (magicMetadata,
pubMagicMetadata, sharedMagicMetadata) to decryptCollection, which was
previously only done for files. The server sends these for visibility
settings, sort order, cover photo selection, etc.
6 new tests covering: account.json, per-collection dirs with
_collection.json, collection pubMagicMetadata decryption, per-file
JSON with all three metadata layers, graceful handling of files with
no magic metadata, and incremental re-run safety. 116 total.
bin/quak.ts: commander-based CLI with login (interactive + QUAK_EMAIL/
QUAK_PASSWORD env vars), whoami, logout, backup commands. Session
stored at env-paths('quak').data/session.json (~/Library/Application
Support/quak/ on macOS, XDG on Linux).
src/backup.ts: runBackup downloads all files into originals/<id>.<ext>,
symlinks into collections/<name>/<title>, writes per-collection JSON
metadata at collections/<name>.json. Deduplicates across collections
(each file downloaded once). Skips existing originals on incremental
runs. Never crashes on single-file failure.
4 backup tests + live-tested against real Ente account.
4 tests for runBackup: full download into collection-named dirs,
incremental skip of existing files, resilient continuation after
single-file HTTP 500, and metadata.json output.
Adds commander 14.0.3 and env-paths 4.0.0 as runtime deps.
sneak
clawbot